Running an Ivy Bridge cpu at fixed frequency
Is there any way to persuade the intel_pstate driver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've set no_turbo to 1).
An alternative would be a variable frequency TSC - might give more consistent values.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 10 lines · David Laight
Is there any way to persuade the
intel_pstatedriver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've setno_turboto 1).
I don't remember. I'm sure I could figure out what MSR to write, but that's not the answer you're looking for. Someone else will know :)
Quoted 1 line
An alternative would be a variable frequency TSC - might give more consistent values.
You can quite easily use perf to count cycles. I never really finished it, but this is a tiny little library that should do exactly what you need. It's a bit messy.
Quoted 2 lines · David Laight
Is there any way to persuade the
intel_pstatedriver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
You can, use performance governor and put scaling_{min,max}_freq at the base_frequency (I _think_, I never quite remember if that is the max non turbo P state).
If that doesn't work, simply put it at cpuinfo_min_freq. It's slow, but it's guaranteed stable.
Quoted 2 lines
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
As Andy already wrote, perf is really good for this.
Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.
Quoted 2 lines
I thought I'd managed by setting the '
scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
You also have to set the min I think, and select the performance governor, otherwise it's too tempted to be 'smart' about stuff.
Quoted 4 lines
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've set
no_turboto 1).An alternative would be a variable frequency TSC - might give more consistent values.
perf :-)
From: Peter Zijlstra
Quoted 5 lines
Sent: 05 December 2019 09:46
As Andy already wrote, perf is really good for this.Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.
You are in a maze of incomplete documentation all disjoint.
The x86 instruction set doc (eg 325462.pdf) defines the rdpmc instruction, tells you how many counters each cpu type has, but doesn't even contain a reference to how they are incremented.
I guess there are some processor-specific MSR for that.
perf_event_open(2) tells you a few things, but doesn't actually what anything is. It contains all but the last 'if' clause of this function, without really saying what any of it does - or why you might do it this way.
static inline u64 mmap_read_self(void *addr)
{
struct perf_event_mmap_page *pc = addr;
u32 seq, idx, time_mult = 0, time_shift = 0, width = 0;
u64 count, cyc = 0, time_offset = 0, enabled, running, delta;
s64 pmc = 0;
do {
seq = pc->lock;
barrier();
enabled = pc->time_enabled;
running = pc->time_running;
if (pc->cap_user_time && enabled != running) {
cyc = rdtsc();
time_mult = pc->time_mult;
time_shift = pc->time_shift;
time_offset = pc->time_offset;
}
idx = pc->index;
count = pc->offset;
if (pc->cap_user_rdpmc && idx) {
width = pc->pmc_width;
pmc = rdpmc(idx - 1);
}
barrier();
} while (pc->lock != seq);
if (idx) {
pmc <<= 64 - width;
pmc >>= 64 - width; /* shift right signed */
count += pmc;
}
if (enabled != running) {
u64 quot, rem;
quot = (cyc >> time_shift);
rem = cyc & ((1 << time_shift) - 1);
delta = time_offset + quot * time_mult +
((rem * time_mult) >> time_shift);
enabled += delta;
if (idx)
running += delta;
quot = count / running;
rem = count % running;
count = quot * enabled + (rem * enabled) / running;
}
return count;
}AFAICT:
1) The last clause is scaling the count up to allow for time when the hardware counter couldn't be allocated.
I'm not convinced that is useful, better to ignore the entire measurement. Half this got deleted from the man page, leaving strange 'set but unused' variables.
2) The hardware counters are disabled while the process is asleep. On wake a different pmc counter might be used (maybe on a different cpu). The new cpu might not even have a counter available.
3) If you don't want to scale up for missing periods it is probably enough to do:
do {
seq = pc->offset;
barrier();
idx = pc->index;
if (!index)
return -1;
count = pc->offset + rdpmc(idx - 1);
} while (seq != pc->seq);
return (unsigned int)count;Not tried it yet :-)
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 11 lines · David Laight
From: Peter Zijlstra
Quoted 4 lines
Sent: 05 December 2019 09:46
As Andy already wrote, perf is really good for this.
Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.You are in a maze of incomplete documentation all disjoint.
I don’t see any documentation. Maybe you shouldn’t have turned your flashlight on.
Quoted 85 lines
The x86 instruction set doc (eg 325462.pdf) defines the rdpmc instruction, tells you how many counters each cpu type has, but doesn't even contain a reference to how they are incremented.
I guess there are some processor-specific MSR for that.
perf_event_open(2)tells you a few things, but doesn't actually what anything is. It contains all but the last 'if' clause of this function, without really saying what any of it does - or why you might do it this way.static inline u64
mmap_read_self(void *addr){ struct perf_event_mmap_page *pc = addr; u32 seq, idx, time_mult = 0, time_shift = 0, width = 0; u64 count, cyc = 0, time_offset = 0, enabled, running, delta; s64 pmc = 0; do { seq = pc->lock; barrier(); enabled = pc->time_enabled; running = pc->time_running; if (pc->cap_user_time && enabled != running) { cyc = rdtsc(); time_mult = pc->time_mult; time_shift = pc->time_shift; time_offset = pc->time_offset; } idx = pc->index; count = pc->offset; if (pc->cap_user_rdpmc && idx) { width = pc->pmc_width; pmc = rdpmc(idx - 1); } barrier(); } while (pc->lock != seq); if (idx) { pmc <<= 64 - width; pmc >>= 64 - width; /* shift right signed */ count += pmc; } if (enabled != running) { u64 quot, rem; quot = (cyc >> time_shift); rem = cyc & ((1 << time_shift) - 1); delta = time_offset + quot * time_mult + ((rem * time_mult) >> time_shift); enabled += delta; if (idx) running += delta; quot = count / running; rem = count % running; count = quot * enabled + (rem * enabled) / running; } return count; }AFAICT:
1) The last clause is scaling the count up to allow for time when the hardware counter couldn't be allocated.
I'm not convinced that is useful, better to ignore the entire measurement. Half this got deleted from the man page, leaving strange 'set but unused' variables.2) The hardware counters are disabled while the process is asleep. On wake a different pmc counter might be used (maybe on a different cpu). The new cpu might not even have a counter available.
3) If you don't want to scale up for missing periods it is probably enough to do:
do { seq = pc->offset; barrier(); idx = pc->index; if (!index) return -1; count = pc->offset + rdpmc(idx - 1);
Quoted 4 lines
} while (seq !=
pc->seq);
return (unsigned int)count;Not tried it yet :-)
Use my version :). I just throw out the sample if we were preempted or if it was otherwise suspicious.
—Andy
Quoted 6 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 8 lines · David Laight
From: Peter Zijlstra
Quoted 5 lines
Sent: 05 December 2019 09:46
As Andy already wrote, perf is really good for this.Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.
You are in a maze of incomplete documentation all disjoint.
I'm sure..
Quoted 3 lines
The x86 instruction set doc (eg 325462.pdf) defines the rdpmc instruction, tells you how many counters each cpu type has, but doesn't even contain a reference to how they are incremented.
There's book 3, chapter 18, performance monitoring overview, that should explain how the counters work, and chapter 19 that lists many of the available events.
TL;DR, they're (48bit) signed counters that increment and raise an interrupt when the sign flips. This means we set them to '-period' and then upon read (either early or on interrupt) compute the delta and accumulate elsewhere.
Quoted 3 lines
perf_event_open(2)tells you a few things, but doesn't actually what anything is. It contains all but the last 'if' clause of this function, without really saying what any of it does - or why you might do it this way.
I don't actually know what's in that manpage. But it really shouldn't be too hard to understand.
It's a seqcount protected set of value, there's the RDPMC counter index, and the counter offset. If the idx!=0 it means the counter is actually programmed and we must RDPMC, the result of which we must add to the offset.
The whole counter scaling crud is just that, crud you can mostly forget about if you want to quickly hack something together. See mmap_read_pinned() for the simplified (and much faster version) that ignores all that.
Quoted 5 lines
AFAICT:
1) The last clause is scaling the count up to allow for time when the hardware counter couldn't be allocated.
I'm not convinced that is useful, better to ignore the entire measurement. Half this got deleted from the man page, leaving strange 'set but unused' variables.
Depending on the usecase, sure. I don't mave use for it either. I know other people find it useful.
Quoted 3 lines
2) The hardware counters are disabled while the process is asleep. On wake a different pmc counter might be used (maybe on a different cpu). The new cpu might not even have a counter available.
Right, but if this is all you're running that is unlikely to happen.
Quoted 10 lines
3) If you don't want to scale up for missing periods it is probably enough to do:
do { seq = pc->offset; barrier(); idx = pc->index; if (!index) return -1; count = pc->offset + rdpmc(idx - 1); } while (seq != pc->seq); return (unsigned int)count;
You still need to do the rdpmc sign extent crud, but see mmap_read_pinned() that does just about that.
As the name suggests it relies on using perf_event_attr::pinned = 1.
From: Peter Zijlstra
Quoted 2 lines
Sent: 06 December 2019 10:16
To: David Laight <[email protected]>
...
Quoted 4 lines
The whole counter scaling crud is just that, crud you can mostly forget about if you want to quickly hack something together. See
mmap_read_pinned()for the simplified (and much faster version) that ignores all that.
I noticed that version later :-(
The 'seqcount' is interesting, since it only protects against updates that happen while the process itself is in kernel space. It doesn't allow arbitrary kernel updates of the memory area.
...
Quoted 2 lines
You still need to do the rdpmc sign extent crud, but see
mmap_read_pinned()that does just about that.
Actually for what I'm doing i can truncate the counter to 32 bits and not worry about when it wraps.
Anyway I've not got some histograms of the elapsed cycle counts for recvfrom() and recvmsg() with, and without, some of the HARDENED_USERCOPY costs.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 11 lines · David Laight
Is there any way to persuade the
intel_pstatedriver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've setno_turboto 1).An alternative would be a variable frequency TSC - might give more consistent values.
Have you tried intel_pstate=passive parameter in cmdline? You'll be able to fix the frequency using governors or sysfs. Not sure that this is what you're looking for. I personally also don't know that 'passive' mode will work on Ivy Bridge.
Best regards,
Alexey