Skip to contentSkip to conversation

Running an Ivy Bridge cpu at fixed frequency

DISCUSSION David Laight 5 participants 8 messages Dec 4, 2019 – Dec 6, 2019

lore
Share & tools
lore
#0 lore raw

Is there any way to persuade the intel_pstate driver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've set no_turbo to 1).

An alternative would be a variable frequency TSC - might give more consistent values.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#1 lore raw
Quoted 10 lines · David Laight

Is there any way to persuade the intel_pstate driver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've set no_turbo to 1).

I don't remember. I'm sure I could figure out what MSR to write, but that's not the answer you're looking for. Someone else will know :)

Quoted 1 line

An alternative would be a variable frequency TSC - might give more consistent values.

You can quite easily use perf to count cycles. I never really finished it, but this is a tiny little library that should do exactly what you need. It's a bit messy.

https://git.kernel.org/pub/scm/linux/kernel/git/luto/misc-tests.git/tree/tight_loop/perf_self_monitor.c

#2 lore raw
In reply to David Laight
Quoted 2 lines · David Laight

Is there any way to persuade the intel_pstate driver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?

You can, use performance governor and put scaling_{min,max}_freq at the base_frequency (I _think_, I never quite remember if that is the max non turbo P state).

If that doesn't work, simply put it at cpuinfo_min_freq. It's slow, but it's guaranteed stable.

Quoted 2 lines

It is really difficult to compare code execution times when the cpu clock speed keeps changing.

As Andy already wrote, perf is really good for this.

Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.

Quoted 2 lines

I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.

You also have to set the min I think, and select the performance governor, otherwise it's too tempted to be 'smart' about stuff.

Quoted 4 lines

It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've set no_turbo to 1).

An alternative would be a variable frequency TSC - might give more consistent values.

perf :-)

#3 lore raw

From: Peter Zijlstra

Quoted 5 lines

Sent: 05 December 2019 09:46
As Andy already wrote, perf is really good for this.

Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.

You are in a maze of incomplete documentation all disjoint.

The x86 instruction set doc (eg 325462.pdf) defines the rdpmc instruction, tells you how many counters each cpu type has, but doesn't even contain a reference to how they are incremented.
I guess there are some processor-specific MSR for that.

perf_event_open(2) tells you a few things, but doesn't actually what anything is. It contains all but the last 'if' clause of this function, without really saying what any of it does - or why you might do it this way.

static inline u64 mmap_read_self(void *addr)

{
        struct perf_event_mmap_page *pc = addr;
        u32 seq, idx, time_mult = 0, time_shift = 0, width = 0;
        u64 count, cyc = 0, time_offset = 0, enabled, running, delta;
        s64 pmc = 0;

        do {
                seq = pc->lock;
                barrier();

                enabled = pc->time_enabled;
                running = pc->time_running;

                if (pc->cap_user_time && enabled != running) {
                        cyc = rdtsc();
                        time_mult = pc->time_mult;
                        time_shift = pc->time_shift;
                        time_offset = pc->time_offset;
                }

                idx = pc->index;
                count = pc->offset;
                if (pc->cap_user_rdpmc && idx) {
                        width = pc->pmc_width;
                        pmc = rdpmc(idx - 1);
                }

                barrier();
        } while (pc->lock != seq);

        if (idx) {
                pmc <<= 64 - width;
                pmc >>= 64 - width; /* shift right signed */
                count += pmc;
        }

        if (enabled != running) {
                u64 quot, rem;

                quot = (cyc >> time_shift);
                rem = cyc & ((1 << time_shift) - 1);
                delta = time_offset + quot * time_mult +
                        ((rem * time_mult) >> time_shift);

                enabled += delta;
                if (idx)
                        running += delta;

                quot = count / running;
                rem = count % running;
                count = quot * enabled + (rem * enabled) / running;
        }

        return count;
}

AFAICT:
1) The last clause is scaling the count up to allow for time when the hardware counter couldn't be allocated.
I'm not convinced that is useful, better to ignore the entire measurement. Half this got deleted from the man page, leaving strange 'set but unused' variables.

2) The hardware counters are disabled while the process is asleep. On wake a different pmc counter might be used (maybe on a different cpu). The new cpu might not even have a counter available.

3) If you don't want to scale up for missing periods it is probably enough to do:

	do {
		seq = pc->offset;
		barrier();
		idx = pc->index;
		if (!index)
			return -1;
		count = pc->offset + rdpmc(idx - 1);
	} while (seq != pc->seq);
	return (unsigned int)count;

Not tried it yet :-)

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#4 lore raw
Quoted 11 lines · David Laight

From: Peter Zijlstra

Quoted 4 lines

Sent: 05 December 2019 09:46
As Andy already wrote, perf is really good for this.
Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.

You are in a maze of incomplete documentation all disjoint.

I don’t see any documentation. Maybe you shouldn’t have turned your flashlight on.

Quoted 85 lines

The x86 instruction set doc (eg 325462.pdf) defines the rdpmc instruction, tells you how many counters each cpu type has, but doesn't even contain a reference to how they are incremented.
I guess there are some processor-specific MSR for that.

perf_event_open(2) tells you a few things, but doesn't actually what anything is. It contains all but the last 'if' clause of this function, without really saying what any of it does - or why you might do it this way.

static inline u64 mmap_read_self(void *addr)

{
      struct perf_event_mmap_page *pc = addr;
      u32 seq, idx, time_mult = 0, time_shift = 0, width = 0;
      u64 count, cyc = 0, time_offset = 0, enabled, running, delta;
      s64 pmc = 0;

      do {
              seq = pc->lock;
              barrier();

              enabled = pc->time_enabled;
              running = pc->time_running;

              if (pc->cap_user_time && enabled != running) {
                      cyc = rdtsc();
                      time_mult = pc->time_mult;
                      time_shift = pc->time_shift;
                      time_offset = pc->time_offset;
              }

              idx = pc->index;
              count = pc->offset;
              if (pc->cap_user_rdpmc && idx) {
                      width = pc->pmc_width;
                      pmc = rdpmc(idx - 1);
              }

              barrier();
      } while (pc->lock != seq);

      if (idx) {
              pmc <<= 64 - width;
              pmc >>= 64 - width; /* shift right signed */
              count += pmc;
      }

      if (enabled != running) {
              u64 quot, rem;

              quot = (cyc >> time_shift);
              rem = cyc & ((1 << time_shift) - 1);
              delta = time_offset + quot * time_mult +
                      ((rem * time_mult) >> time_shift);

              enabled += delta;
              if (idx)
                      running += delta;

              quot = count / running;
              rem = count % running;
              count = quot * enabled + (rem * enabled) / running;
      }

      return count;
}

AFAICT:
1) The last clause is scaling the count up to allow for time when the hardware counter couldn't be allocated.
I'm not convinced that is useful, better to ignore the entire measurement. Half this got deleted from the man page, leaving strange 'set but unused' variables.

2) The hardware counters are disabled while the process is asleep. On wake a different pmc counter might be used (maybe on a different cpu). The new cpu might not even have a counter available.

3) If you don't want to scale up for missing periods it is probably enough to do:

  do {
      seq = pc->offset;
      barrier();
      idx = pc->index;
      if (!index)
          return -1;
      count = pc->offset + rdpmc(idx - 1);
Quoted 4 lines

} while (seq != pc->seq);
return (unsigned int)count;

Not tried it yet :-)

Use my version :). I just throw out the sample if we were preempted or if it was otherwise suspicious.

—Andy

Quoted 6 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#5 lore raw
In reply to David Laight
Quoted 8 lines · David Laight

From: Peter Zijlstra

Quoted 5 lines

Sent: 05 December 2019 09:46
As Andy already wrote, perf is really good for this.

Find attached, it probably is less shiny than what Andy handed you, but contains all the bits required to frob something.

You are in a maze of incomplete documentation all disjoint.

I'm sure..

Quoted 3 lines

The x86 instruction set doc (eg 325462.pdf) defines the rdpmc instruction, tells you how many counters each cpu type has, but doesn't even contain a reference to how they are incremented.

There's book 3, chapter 18, performance monitoring overview, that should explain how the counters work, and chapter 19 that lists many of the available events.

TL;DR, they're (48bit) signed counters that increment and raise an interrupt when the sign flips. This means we set them to '-period' and then upon read (either early or on interrupt) compute the delta and accumulate elsewhere.

Quoted 3 lines

perf_event_open(2) tells you a few things, but doesn't actually what anything is. It contains all but the last 'if' clause of this function, without really saying what any of it does - or why you might do it this way.

I don't actually know what's in that manpage. But it really shouldn't be too hard to understand.

It's a seqcount protected set of value, there's the RDPMC counter index, and the counter offset. If the idx!=0 it means the counter is actually programmed and we must RDPMC, the result of which we must add to the offset.

The whole counter scaling crud is just that, crud you can mostly forget about if you want to quickly hack something together. See mmap_read_pinned() for the simplified (and much faster version) that ignores all that.

Quoted 5 lines

AFAICT:
1) The last clause is scaling the count up to allow for time when the hardware counter couldn't be allocated.
I'm not convinced that is useful, better to ignore the entire measurement. Half this got deleted from the man page, leaving strange 'set but unused' variables.

Depending on the usecase, sure. I don't mave use for it either. I know other people find it useful.

Quoted 3 lines

2) The hardware counters are disabled while the process is asleep. On wake a different pmc counter might be used (maybe on a different cpu). The new cpu might not even have a counter available.

Right, but if this is all you're running that is unlikely to happen.

Quoted 10 lines

3) If you don't want to scale up for missing periods it is probably enough to do:

	do {
		seq = pc->offset;
		barrier();
		idx = pc->index;
		if (!index)
			return -1;
		count = pc->offset + rdpmc(idx - 1);
	} while (seq != pc->seq);
	return (unsigned int)count;

You still need to do the rdpmc sign extent crud, but see mmap_read_pinned() that does just about that.

As the name suggests it relies on using perf_event_attr::pinned = 1.

#6 lore raw

From: Peter Zijlstra

Quoted 2 lines

Sent: 06 December 2019 10:16
To: David Laight <[email protected]>

...

Quoted 4 lines

The whole counter scaling crud is just that, crud you can mostly forget about if you want to quickly hack something together. See mmap_read_pinned() for the simplified (and much faster version) that ignores all that.

I noticed that version later :-(
The 'seqcount' is interesting, since it only protects against updates that happen while the process itself is in kernel space. It doesn't allow arbitrary kernel updates of the memory area.

...

Quoted 2 lines

You still need to do the rdpmc sign extent crud, but see mmap_read_pinned() that does just about that.

Actually for what I'm doing i can truncate the counter to 32 bits and not worry about when it wraps.

Anyway I've not got some histograms of the elapsed cycle counts for recvfrom() and recvmsg() with, and without, some of the HARDENED_USERCOPY costs.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#7 lore raw
In reply to David Laight
Quoted 11 lines · David Laight

Is there any way to persuade the intel_pstate driver to make an Ivy bridge (i7-3770) cpu run at a fixed frequency?
It is really difficult to compare code execution times when the cpu clock speed keeps changing.
I thought I'd managed by setting the 'scaling_max_freq' to 1.7GHz, but even that doesn't seem to be working now.
It would also be nice to run a little faster than that - but without it 'randomly' going to 'turbo' frequencies (which it is doing even after I've set no_turbo to 1).

An alternative would be a variable frequency TSC - might give more consistent values.

Have you tried intel_pstate=passive parameter in cmdline? You'll be able to fix the frequency using governors or sysfs. Not sure that this is what you're looking for. I personally also don't know that 'passive' mode will work on Ivy Bridge.

Best regards,
Alexey