Skip to contentSkip to conversation

provide /proc/sal/itc_drift through AUX?

Linux-ia64 Jes Sorensen 3 participants 8 messages Mar 19, 2003 – Mar 21, 2003

lore
Share & tools
lore
#0 lore raw

Hi

I was wondering what people think about providing the information of /proc/sal/itc_drift as an AUX vector?

The problem is that on some NUMA boxen (such as the SGI boxes), the ITC isn't synchronized across nodes and we can't rely on ar.itc in userland for implementing the high-precision timing. I believe the IBM NUMA-Q team has a similar problem that could be solved in a similar way on ia32?

Instead one can switch to using gettimeofday() for the timing, which with the new fast syscalls should be quite pleasant.

I have a patch which implements this for glibc-2.2 (will do 2.3 later), however what I don't like about it is that one ends up opening and reading /proc/sal/itc_drift in every single binary executed. To avoid the overhead of this it seems a good idea to me to provide this information via an AUX vector.

If anybody is interested in the glibc patch, feel free to grab it from
http://www.wildopensource.com/~jes/glibc/itc-drift-patch.diff
For now it's a test patch, though it does seem to behave as expected.

Thoughts?

Cheers,
Jes

#1 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jes Sorensen

Jes> Hi I was wondering what people think about providing the Jes> information of /proc/sal/itc_drift as an AUX vector?

Jes> The problem is that on some NUMA boxen (such as the SGI boxes), Jes> the ITC isn't synchronized across nodes and we can't rely on Jes> ar.itc in userland for implementing the high-precision Jes> timing. I believe the IBM NUMA-Q team has a similar problem Jes> that could be solved in a similar way on ia32?

Jes> Instead one can switch to using gettimeofday() for the timing, Jes> which with the new fast syscalls should be quite pleasant.

Jes> I have a patch which implements this for glibc-2.2 (will do 2.3 Jes> later), however what I don't like about it is that one ends up Jes> opening and reading /proc/sal/itc_drift in every single binary Jes> executed. To avoid the overhead of this it seems a good idea to Jes> me to provide this information via an AUX vector.

Jes> If anybody is interested in the glibc patch, feel free to grab Jes> it from
Jes> http://www.wildopensource.com/~jes/glibc/itc-drift-patch.diff
Jes> For now it's a test patch, though it does seem to behave as Jes> expected.

Jes> Thoughts?

Why do you need to export it if you just use a light-weight gettimeofday()? The light-weight syscall can readily access the ITC-drift info via kernel memory.

BTW: I think someone should explore using an NTP-like approach to keep ITC drift small enough that it's still usable for ITC-based interpolation. Granted, this assumes that hw drifts are reasonably small and you'd still need to worry about different clock-frequencies, but I suspect that in practice, this would work extremely well. It would be nice because it would be much faster than an HPET-based approach, would work on all ia64 machines, and would provide better resolution. Furthermore, we already have all the logic in the time-interpolation to ensure that there is never an observable time discontinuity.

	--david
#2 lore raw
In reply to Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"David" = David Mosberger <[email protected]> writes:

David> Why do you need to export it if you just use a light-weight David> gettimeofday()? The light-weight syscall can readily access David> the ITC-drift info via kernel memory.

Let me get sure I understand you right here, you are suggesting I add a new syscall to provide the info currently found in /proc/sal/itc_drift? Right now it's just returning true/false (1/0) as to whether the ITC drifts or not.

The reason I was thinking about using the AUX was that it's still faster than making a system call and the dynamic linker will always be calling it once. It doesn't make a big difference of course whether we do it one way or another.

David> BTW: I think someone should explore using an NTP-like approach David> to keep ITC drift small enough that it's still usable for David> ITC-based interpolation. Granted, this assumes that hw drifts David> are reasonably small and you'd still need to worry about David> different clock-frequencies, but I suspect that in practice, David> this would work extremely well. It would be nice because it David> would be much faster than an HPET-based approach, would work on David> all ia64 machines, and would provide better resolution. David> Furthermore, we already have all the logic in the David> time-interpolation to ensure that there is never an observable David> time discontinuity.

I think this would work if it wasn't for the different clock frequency problem. What real life ITC drifts are like I must admit I have no idea.

Pardon my ignorance, what are you referring to by HPET? I am sure it's something obvious and I know I have heard it before, but it's just not ringing a bell here right now.

Cheers,
Jes

#3 lore raw
In reply to Jes Sorensen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Jes Sorensen

"David" = David Mosberger <[email protected]> writes:

David> Why do you need to export it if you just use a light-weight David> gettimeofday()? The light-weight syscall can readily access David> the ITC-drift info via kernel memory.

Jes> Let me get sure I understand you right here, you are suggesting Jes> I add a new syscall to provide the info currently found in Jes> /proc/sal/itc_drift? Right now it's just returning true/false Jes> (1/0) as to whether the ITC drifts or not.

No, not at all. I was simply suggesting to always use gettimeofday(). The light-weight implementation of gettimeofday() runs in ~130 cycles, which isn't much slower than what you could do in user-level.

Jes> what are you referring to by HPET?

HPET = High Precision Event Timer. See:

http://www.intel.com/labs/platcomp/hpet/hpetspec.htm

	--david
#4 lore raw
In reply to Jes Sorensen

Jes, regarding your query on real life ITC drifts, I have this input. Typical crystals used for the CPU clock today have an accuracy of 250ppm. That is 250 parts per million. Which means for each million ticks, it may actually be 250 realtime units too fast or too slow. So in one second's worth of 1GHz ticks, it may be wrong versus realtime by 250 microseconds.
(not to belabor the point, but then a 1 millisecond error accumulates each 4 seconds).

In case you ask "how can this be, my digital watch is better than that". Your watch is better designed, plus it has more temperature stability on your arm. Plus nobody is willing to buy expensive crystals for the computer...

FYI, Rich
Rich Altmaier
SGI

Jes Sorensen wrote:
...

Quoted 5 lines

I think this would work if it wasn't for the different clock frequency problem. What real life ITC drifts are like I must admit I have no idea.

#5 lore raw
In reply to Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Rich Altmaier

Rich> Jes, regarding your query on real life ITC drifts, I have this Rich> input. Typical crystals used for the CPU clock today have an Rich> accuracy of 250ppm. That is 250 parts per million. Which Rich> means for each million ticks, it may actually be 250 realtime Rich> units too fast or too slow. So in one second's worth of 1GHz Rich> ticks, it may be wrong versus realtime by 250 microseconds. Rich> (not to belabor the point, but then a 1 millisecond error Rich> accumulates each 4 seconds).

Good point.

Rich> In case you ask "how can this be, my digital watch is better Rich> than that". Your watch is better designed, plus it has more Rich> temperature stability on your arm. Plus nobody is willing to Rich> buy expensive crystals for the computer...

Well, most watches don't run at 1GHz either! ;-)

	--david
#6 lore raw
In reply to Jes Sorensen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

"David" = David Mosberger <[email protected]> writes:

On 19 Mar 2003 21:06:20 -0500, Jes Sorensen <[email protected]> said:

Jes> Let me get sure I understand you right here, you are suggesting I Jes> add a new syscall to provide the info currently found in Jes> /proc/sal/itc_drift? Right now it's just returning true/false Jes> (1/0) as to whether the ITC drifts or not.

David> No, not at all. I was simply suggesting to always use David> gettimeofday(). The light-weight implementation of David> gettimeofday() runs in ~130 cycles, which isn't much slower David> than what you could do in user-level.

Ok, that makes it even simpler, I will do that then.

Cheers,
Jes

#7 lore raw
In reply to Jes Sorensen
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

"Jes" = Jes Sorensen <[email protected]> writes:

"David" = David Mosberger <[email protected]> writes:

David> No, not at all. I was simply suggesting to always use David> gettimeofday(). The light-weight implementation of David> gettimeofday() runs in ~130 cycles, which isn't much slower David> than what you could do in user-level.

Jes> Ok, that makes it even simpler, I will do that then.

Thinking about this, I still think we need the patch for glibc-2.2 and maybe glibc-2.3 for compiling with kernels < 2.5.x? Though I'd probably just say we should take the hit for people using 2.4 kernels with glibc-2.3. The fast syscall support isn't going into glibc-2.2.

I assume some of the BuzzwordCompliantServer packages from various distributions are going to stick to glibc-2.2 and kernel-2.4.x for a while.

Back to mangling glibc-2.3.

Cheers,
Jes