Skip to contentSkip to conversation

Turn rdmsr, rdtsc into inline functions, clarify names

PATCH Andi Kleen 9 participants 25 messages Aug 6, 2006 – Aug 15, 2006 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andi Kleen

author
#1 lore raw
Quoted 5 lines · Rusty Russell

[Andi, sorry, x86_64 part untested, so sending straight to you]

rdmsr and rdtsc are macros, altering their arguments directly. An inline function would offer decent typechecking, but needs to take pointer args. The comment notes that gcc produces better code with

I think I prefer the macro variant actually. Sorry. It just looks better without the &s.

We don't care very much about the code quality here because rdmsr/wrmsr are always very slow in microcode anyways and tend to synchronize the CPUs.

If you feel a need to clean up I would suggest you convert more users over to the ll variants which take a single 64bit value instead of two 32bit ones.

-Andi

#2 lore raw
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Rusty Russell

[Andi, sorry, x86_64 part untested, so sending straight to you]

rdmsr and rdtsc are macros, altering their arguments directly. An inline function would offer decent typechecking, but needs to take pointer args. The comment notes that gcc produces better code with

I think I prefer the macro variant actually. Sorry. It just looks better without the &s.

Hi Andi,

Please reconsider. This isn't about being pretty, it's about not
having hidden side-effects, and having typechecking.

Quoted 3 lines

We don't care very much about the code quality here because rdmsr/wrmsr are always very slow in microcode anyways and tend to synchronize the CPUs.

Agreed, but comment about it above the macros made me wary, so I checked it. No significant code difference with gcc >= 4.0, at least.

Quoted 3 lines

If you feel a need to clean up I would suggest you convert more users over to the ll variants which take a single 64bit value instead of two 32bit ones.

You mean the l and ll variants? The 64 bit variants are rdmsrl and rdtscll, not to be confused with rdtscl, which returns the lower 32 bits. This confusion caused the x86_64 bug in gameport.c which the patch comment mentioned (at least, seems to be a bug to me).

See why I want to fix these names?

So if you would prefer u64 rdtsc64(), u32 rdtsc_low(), u64 rdmsr64(int msr), u32 rdmsr_low(int msr), I can convert everyone to that, although it's a more invasive change...

Thanks,
Rusty.

Signature · 1 line
Help! Save Australia from the worst of the DMCA: http://linux.org.au/law
#3 lore raw
Quoted 11 lines · Rusty Russell

You mean the l and ll variants? The 64 bit variants are rdmsrl and rdtscll, not to be confused with rdtscl, which returns the lower 32 bits. This confusion caused the x86_64 bug in gameport.c which the patch comment mentioned (at least, seems to be a bug to me).

See why I want to fix these names?

So if you would prefer u64 rdtsc64(), u32 rdtsc_low(), u64 rdmsr64(int msr), u32 rdmsr_low(int msr), I can convert everyone to that, although it's a more invasive change...

rdmsrl is really misnamed. It should have been rdmsrq to be consistent, and have rdmsrl return the low 32 bits.

-hpa

#4 lore raw
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Rusty Russell

You mean the l and ll variants? The 64 bit variants are rdmsrl and rdtscll, not to be confused with rdtscl, which returns the lower 32 bits. This confusion caused the x86_64 bug in gameport.c which the patch comment mentioned (at least, seems to be a bug to me).

Quoted 8 lines
Quoted 5 lines

See why I want to fix these names?

So if you would prefer u64 rdtsc64(), u32 rdtsc_low(), u64 rdmsr64(int msr), u32 rdmsr_low(int msr), I can convert everyone to that, although it's a more invasive change...

rdmsrl is really misnamed. It should have been rdmsrq to be consistent, and have rdmsrl return the low 32 bits.

I prefer the more explicit linux-style naming of rdmsr_low32/rdmsr64, myself, even though this is x86-specific code. Noone has an excuse for misunderstanding then...

Thanks!
Rusty.

Signature · 1 line
Help! Save Australia from the worst of the DMCA: http://linux.org.au/law
#5 lore raw
Quoted 10 lines · Rusty Russell
Quoted 5 lines
Quoted 4 lines

So if you would prefer u64 rdtsc64(), u32 rdtsc_low(), u64 rdmsr64(int msr), u32 rdmsr_low(int msr), I can convert everyone to that, although it's a more invasive change...

rdmsrl is really misnamed. It should have been rdmsrq to be consistent, and have rdmsrl return the low 32 bits.

I prefer the more explicit linux-style naming of rdmsr_low32/rdmsr64, myself, even though this is x86-specific code. Noone has an excuse for misunderstanding then...

Well, we *do* have readb/readw/readl/readq etc.

-hpa

#6 lore raw
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines · Rusty Russell
Quoted 4 lines
Quoted 4 lines

So if you would prefer u64 rdtsc64(), u32 rdtsc_low(), u64 rdmsr64(int msr), u32 rdmsr_low(int msr), I can convert everyone to that, although it's a more invasive change...

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

rdmsrl is really misnamed. It should have been rdmsrq to be consistent, and have rdmsrl return the low 32 bits.

I prefer the more explicit linux-style naming of rdmsr_low32/rdmsr64, myself, even though this is x86-specific code. Noone has an excuse for misunderstanding then...

Well, we *do* have readb/readw/readl/readq etc.

Exactly. And I modelled the name rdtsc64 on the "new-style" ioread8, ioread16 and ioread32 from asm-generic/iomap.h 8)

Rusty.

Signature · 1 line
Help! Save Australia from the worst of the DMCA: http://linux.org.au/law
author
#7 lore raw
In reply to Rusty Russell
Quoted 2 lines

Please reconsider. This isn't about being pretty, it's about not
having hidden side-effects,

I wouldn't call it hidden, it's well defined in the architecture.

Quoted 1 line

and having typechecking.

The existing code will already reject any non integer and I don't see a particular need to be more strict than that.

Quoted 7 lines
Quoted 3 lines

If you feel a need to clean up I would suggest you convert more users over to the ll variants which take a single 64bit value instead of two 32bit ones.

You mean the l and ll variants? The 64 bit variants are rdmsrl and rdtscll, not to be confused with rdtscl, which returns the lower 32 bits. This confusion caused the x86_64 bug in gameport.c

Why does gameport access MSRs or TSCs? That sounds like a bug in itself.

<looking at the code>

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Dmitry, I would suggest to convert it over to do_gettimeofday and remove all the architecture ifdefs.

Or maybe just remove it completely. Who cares about the speed of a gameport anyways? And why can't they measure it in user space?

Quoted 1 line

See why I want to fix these names?

No.

Quoted 3 lines

So if you would prefer u64 rdtsc64(), u32 rdtsc_low(), u64 rdmsr64(int msr), u32 rdmsr_low(int msr), I can convert everyone to that, although it's a more invasive change...

I think things are most fine right now, except that I think most users of high low should be converted to work directly on 64bit quantities.

-Andi

#8 lore raw
Quoted 4 lines · Andi Kleen
Quoted 2 lines

Please reconsider. This isn't about being pretty, it's about not
having hidden side-effects,

I wouldn't call it hidden, it's well defined in the architecture.

Sorry Andi, I'm not talking about the asm, which is fine. I'm talking about the function-style macro which alters its arguments directly. It's very bad form.

	int a, b;
	rdtsc(a, b);
Quoted 4 lines
Quoted 1 line

and having typechecking.

The existing code will already reject any non integer and I don't see a particular need to be more strict than that.

Um?

	u64 c;
	int a,b;

	rdtsc(&a, &b);
	rdtscl(c);

These macros are badly named and don't check for bad usage. Sure, less than 1% of uses is wrong at the moment, but I'm volunteering to fix them because I think it sets a bad example and sets us up for more bugs.

I feel fairly strongly about this, but I'll drop the x86_64 changes.

Rusty.

Signature · 1 line
Help! Save Australia from the worst of the DMCA: http://linux.org.au/law
author
#9 lore raw
Quoted 1 line

I feel fairly strongly about this, but I'll drop the x86_64 changes.

I would prefer if you leave them alone in i386 too.

-andi

#10 lore raw
In reply to Andi Kleen
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Quoted 6 lines

Dmitry, I would suggest to convert it over to do_gettimeofday and remove all the architecture ifdefs.

Or maybe just remove it completely.  Who cares about the speed of a gameport anyways? And why can't they measure it in user space?

Analog driver uses it to adjust timing. Vojtech should have more background on that..

Signature · 1 line
Dmitry
author
#11 lore raw
Quoted 6 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

-Andi

#12 lore raw
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast. The joystick drivers will not work if the timing calls take significant time compared to an inb() from ISA/LPC space.

That's why they don't fallback to PIT timing when TSC is not available - io counting is a better option.

Signature · 2 lines
Vojtech Pavlik
Director SuSE Labs
author
#13 lore raw
Quoted 7 lines · Vojtech Pavlik
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Would not preempt_disable fix that?

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

Please change that code.

-Andi

#14 lore raw
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Vojtech Pavlik
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

I assume you wanted to say "When gettimeofday() is slow, it means TSC is not reliable", which I agree with.

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

I assume we could make it work with the monotonic timer instead.

Quoted 1 line

Please change that code.

I'm not arguing that the code is correct. ;)

Signature · 2 lines
Vojtech Pavlik
Director SuSE Labs
author
#15 lore raw
Quoted 20 lines · Vojtech Pavlik
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Vojtech Pavlik
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

Quoted 6 lines

I assume you wanted to say "When gettimeofday() is slow, it means TSC is not reliable", which I agree with.

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

Hmm if that is the only obstacle I can export a "slow gettimeofday" flag.

However it would be some work to implement it for all architectures.

Quoted 2 lines

I assume we could make it work with the monotonic timer instead.

The monotonic timer is the right thing to use to make you independent of ntpd, but it's normally not faster or slower than gettimeofday.

-Andi

#16 lore raw
Quoted 21 lines · Andi Kleen
Quoted 20 lines · Vojtech Pavlik
Quoted 18 lines · Andi Kleen
Quoted 7 lines · Vojtech Pavlik
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Would not preempt_disable fix that?

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

Quoted 8 lines
Quoted 6 lines

I assume you wanted to say "When gettimeofday() is slow, it means TSC is not reliable", which I agree with.

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

Hmm if that is the only obstacle I can export a "slow gettimeofday" flag.

That would help.

Quoted 6 lines

However it would be some work to implement it for all architectures.

Quoted 1 line

I assume we could make it work with the monotonic timer instead.

The monotonic timer is the right thing to use to make you independent of ntpd, but it's normally not faster or slower than gettimeofday.

Yup.

Signature · 2 lines
Vojtech Pavlik
Director SuSE Labs
#17 lore raw
In reply to Andi Kleen
Quoted 24 lines · Andi Kleen
Quoted 23 lines · Vojtech Pavlik
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Vojtech Pavlik
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Dmitry Torokhov
Quoted 3 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

I assume you wanted to say "When gettimeofday() is slow, it means TSC is not reliable", which I agree with.

Quoted 8 lines
Quoted 3 lines

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

Hmm if that is the only obstacle I can export a "slow gettimeofday" flag.

However it would be some work to implement it for all architectures.

Hmm, would it be easier to export "fast gettimeofday" and assume that we have slow gettimeofday by default (so gameport will fall back on io counting)?

Btw, could anyone point me to the origin of the thread - I can't find it in any of the archives of LKML list (including my personal). Thanks!

Signature · 1 line
Dmitry
author
#18 lore raw
Quoted 30 lines · Dmitry Torokhov
Quoted 29 lines · Andi Kleen
Quoted 6 lines · Vojtech Pavlik
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Vojtech Pavlik
Quoted 3 lines · Andi Kleen
Quoted 2 lines · Dmitry Torokhov
Quoted 1 line · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when

the

Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

code can switch CPUs

Would not preempt_disable fix that?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

I assume you wanted to say "When gettimeofday() is slow, it means TSC is not reliable", which I agree with.

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

Quoted 8 lines
Quoted 4 lines

Hmm if that is the only obstacle I can export a "slow gettimeofday" flag.

However it would be some work to implement it for all architectures.

Hmm, would it be easier to export "fast gettimeofday" and assume that we have slow gettimeofday by default (so gameport will fall back on io counting)?

I would expect fast gettimeofday to be more common than slow.

-Andi

#19 lore raw
In reply to Andi Kleen
Quoted 6 lines · Andi Kleen
Quoted 3 lines · Vojtech Pavlik

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

Hmm if that is the only obstacle I can export a "slow gettimeofday" flag.

Wouldn't it be much more useful to normalize this versus (e.g.) CPU cycles for much more information than a plain "this is fast/this is slow" flag, to be measured on bootup?
That way a driver could use

	if (gtod_cpu_cycles_needed <= 500)
		gettimeofday();
	else
		funky_fast_workaround();

OK, in total we have at least four ways of doing this:

a) gtod_is_slow flag
b) number of CPU cycles needed
c) number of nanoseconds needed (but this is less useful since it doesn't properly take into account the fast vs. slow CPUs behaviour, I think)
d) providing all three items above together, for optimal flexibility??

This is somewhat related to an idea of mine which would be to benchmark all clock sources on bootup and print a timing summary, optionally warning users if grave performance issues have been found with a specific source (and especially if that one is active!).
Additionally, print timing summary of gettimeofday() itself on bootup?

Andreas Mohr

author
#20 lore raw
Quoted 6 lines

That way a driver could use

	if (gtod_cpu_cycles_needed <= 500)
		gettimeofday();
	else
		funky_fast_workaround();

Sounds like overengineering to me. I prefer something simple.

Quoted 1 line

OK, in total we have at least four ways of doing this:

Please don't get carried away with this. I'm really not interested in any complex solutions here.

-Andi

#21 lore raw
Quoted 8 lines · Andi Kleen
Quoted 6 lines

That way a driver could use

	if (gtod_cpu_cycles_needed <= 500)
		gettimeofday();
	else
		funky_fast_workaround();

Sounds like overengineering to me. I prefer something simple.

I could as well benchmark gettimeofday() in the gameport init.

Quoted 4 lines
Quoted 1 line

OK, in total we have at least four ways of doing this:

Please don't get carried away with this. I'm really not interested in any complex solutions here.

Signature · 2 lines
Vojtech Pavlik
Director SuSE Labs
#22 lore raw
Quoted 11 lines · Vojtech Pavlik
Quoted 8 lines · Andi Kleen
Quoted 6 lines

That way a driver could use

	if (gtod_cpu_cycles_needed <= 500)
		gettimeofday();
	else
		funky_fast_workaround();

Sounds like overengineering to me. I prefer something simple.

I could as well benchmark gettimeofday() in the gameport init.

Except you can't because the only way to benchmark it would be to use gettimeofday already and then you don't know how much is overhead and what is the real time.

-Andi

#23 lore raw
Quoted 10 lines · Andi Kleen
Quoted 8 lines · Vojtech Pavlik
Quoted 7 lines · Andi Kleen
Quoted 6 lines

That way a driver could use

	if (gtod_cpu_cycles_needed <= 500)
		gettimeofday();
	else
		funky_fast_workaround();
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Sounds like overengineering to me. I prefer something simple.

I could as well benchmark gettimeofday() in the gameport init.

Except you can't because the only way to benchmark it would be to use gettimeofday already and then you don't know how much is overhead and what is the real time.

I run it for 1/10th of a second in a loop and see how many iterations I've done. That's enough to check if its fast enough to use in the gameport routine. Less than 50k iterations and I can't use it.

Signature · 2 lines
Vojtech Pavlik
Director SuSE Labs
#24 lore raw
In reply to Vojtech Pavlik

Hi!

Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Would not preempt_disable fix that?

Partially, but you still have other problems. Please just get rid of it. Why do we have timer code in the kernel if you then chose not to use it?

The problem is that gettimeofday() is not always fast.

When it is not fast that means it is not reliable and then you're also not well off using it anyways.

Quoted 6 lines

I assume you wanted to say "When gettimeofday() is slow, it means TSC is not reliable", which I agree with.

But I need, in the driver, in the no-TSC case use i/o counting, not a slow but reliable method. And I can't say, from outside the timing subsystem, whether gettimeofday() is fast or slow.

do gettimeofday(); gettimeofday(); gettimeofday();, then compare the result with gettimeofday(); inb(); gettimeofday(); ?

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#25 lore raw
In reply to Dmitry Torokhov
Quoted 6 lines · Dmitry Torokhov
Quoted 2 lines · Andi Kleen

This whole thing is broken, e.g. on a preemptive kernel when the code can switch CPUs

Would not preempt_disable fix that?

Quoted 8 lines
Quoted 5 lines

Dmitry, I would suggest to convert it over to do_gettimeofday and remove all the architecture ifdefs.

Or maybe just remove it completely.  Who cares about the speed of a gameport anyways? And why can't they measure it in user space?

Analog driver uses it to adjust timing. Vojtech should have more background on that..

The gameport speed is used by digital-over-gameport joystick drivers to keep track of time based on the number of inb()s done to the gameport.

We can't use anything slower than rdtsc there, and since rdtsc isn't always available, the count of inb()s is the best approximation.

The speed can't easily be measured in userspace - you need to make sure IRQs, DMA and other stuff doesn't interfere with the measurement, which needs to be rather exact. AND, you need it for the kernel to work, so it doesn't make sense to move it to an userspace tool that the kernel would have to call.

The analog joystick driver does even more timing magic to get precise readings from the port, where the value read depends on the time for a bit on the port to change from 0 to 1.

I agree that gameports are mostly dead today, but people are still using them.

Signature · 2 lines
Vojtech Pavlik
Director SuSE Labs