perf_events: fix remapped count support
Subsystems (3)
PATCH 1/1 perf_events: fix remapped count support
This patch fixes the remapped counter support such that it now works on X86 processors.
There were several issues:
- needed a way to allow (permission) user level read (via RDPMC on x86)
- there was no way of scaling the count correctly
- values returned by RDPMC and prev_count did not have the same "sign" extensions
- did work with Intel X86 fixed counters
This patch adds a new sysctl control:
/proc/sys/kernel/perf_event_paranoid_user_read
It is set by default. You must clear it to allow direct user read. I chose not to include this in perf_event_paranoid because the priority there did not make sense w.r.t to this particular control point. It would have required an ABI change, i.e, changing the meaning of the existing values. The fact that we may want to restrict user read is independent of trace points, or system-wide monitoring.
The patch adds a new field to the perf_mmap_page buffer header (delta_base). It is necessary to correctly scale the count. This new value represents the counter value when the event was last scheduled in. It cannot be substracted by the kernel from the total count because of scaling. The correct formula is:
count = offset * time_enabled/time_running + (raw_count - prev_count)
We expose prev_count. The rest was already there. We also correct prev_count to allow the substraction to work. On X86 (and possibly others), the user read instruction (RDPMC) returns the N-bit count where N is the counter width, the upper bits are zeroed. But prev_count is always a 64-bit negative number. To avoid having to expose the raw counter width, the kernel now adjusts the prev_count value exposed in perf_mmap_page to have only N-bit worth of data.
User read support on X86 is implemented for AMD processors, Intel X86 processors except Pentium 4. I know it is also supported there, but I don't have a machine to test this on. A specific patch will be submitted later on.
For other architectures, e.g., PPC, SPARC, assuming they do offer the ability to read counts directly from user space, all that is needed is a couple of new arch specific functions:
- hw_perf_event_counter_width(): actual counter width
- hw_perf_event_index(): register index to pass to the user instruction
- hw_perf_update_user_read_perm(): toggle permission to read from userspace
--
arch/x86/include/asm/perf_event.h | 2 - arch/x86/kernel/cpu/perf_event.c | 39 ++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_amd.c | 2 + arch/x86/kernel/cpu/perf_event_intel.c | 2 + arch/x86/kernel/cpu/perf_event_p6.c | 7 +++ include/linux/perf_event.h | 15 ++++++- kernel/perf_event.c | 63 +++++++++++++++++++++++++++++---- kernel/sysctl.c | 7 +++ 8 files changed, 126 insertions(+), 11 deletions(-)
arch/x86/include/asm/perf_event.h +1 -1
@@ -134,7 +134,7 @@ union cpuid10_edx {extern void init_hw_perf_events (void );extern void perf_events_lapic_init (void ); -#define PERF_EVENT_INDEX_OFFSET 0 +#define X86_RDPMC_FIXED_BASE 0x40000000/* * Abuse bit 3 of the cpu eflags register to indicate proper PEBS IP fixups.
arch/x86/kernel/cpu/perf_event.c +39 -0
@@ -253,6 +253,12 @@ static int x86_perf_event_set_period(struct perf_event *event);#define C (x) PERF_COUNT_HW_CACHE_##x +int hw_perf_event_counter_width (struct perf_event *event) +{ +/* XXX: not all counters may have the same width */ +return x86_pmu.event_bits; +} +static u64 __read_mostly hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX ] [PERF_COUNT_HW_CACHE_OP_MAX ]
@@ -779,6 +785,39 @@ static inline int match_prev_assignment(struct hw_perf_event *hwc,static int x86_pmu_start (struct perf_event *event);static void x86_pmu_stop (struct perf_event *event); +int hw_perf_event_index (struct perf_event *event) +{ +struct hw_perf_event *hwc = &event->hw; + +if (hwc->idx ==X86_PMC_IDX_FIXED_BTS ) +return -1 ; + +if (hwc->idx >=X86_PMC_IDX_FIXED ) +return X86_RDPMC_FIXED_BASE + hwc->idx -X86_PMC_IDX_FIXED ; + +return hwc->idx; +} + +static void __perf_update_user_read_perm (void *data) +{ +/* +* enable/disable secure monitoring on +* current CPU +*/ +if (perf_paranoid_user_read ()) +set_in_cr4 (X86_CR4_PCE ); +else +clear_in_cr4 (X86_CR4_PCE ); +} + +int +hw_event_update_user_read_perm (void ) +{ +__perf_update_user_read_perm (NULL ); +smp_call_function (__perf_update_user_read_perm,NULL ,1 ); +return 0 ; +} +void hw_perf_enable (void ) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);
arch/x86/kernel/cpu/perf_event_amd.c +2 -0
@@ -299,6 +299,8 @@ static void amd_pmu_cpu_online(int cpu)struct amd_nb *nb =NULL ;int i, nb_id; +__perf_update_user_read_perm (NULL ); +if (boot_cpu_data.x86_max_cores <2 )return ;
arch/x86/kernel/cpu/perf_event_intel.c +2 -0
@@ -774,6 +774,8 @@ static void intel_pmu_cpu_starting(int cpu)* Deal with CPUs that don't clear their LBRs on power-up. */ intel_pmu_lbr_reset (); + +__perf_update_user_read_perm (NULL ); }static void intel_pmu_cpu_dying (int cpu)
arch/x86/kernel/cpu/perf_event_p6.c +7 -0
@@ -102,6 +102,11 @@ static void p6_pmu_enable_event(struct perf_event *event) (void )checking_wrmsrl (hwc->config_base + hwc->idx, val); } +static void p6_pmu_cpu_starting (int cpu) +{ +__perf_update_user_read_perm (NULL ); +} +static __initconst struct x86_pmu p6_pmu = { .name ="p6" , .handle_irq = x86_pmu_handle_irq,
@@ -131,6 +136,8 @@ static __initconst struct x86_pmu p6_pmu = { .event_mask = (1ULL << 32) - 1, .get_event_constraints = x86_get_event_constraints, .event_constraints = p6_event_constraints, + + .cpu_starting = p6_pmu_cpu_starting, };static __init int p6_pmu_init (void )
include/linux/perf_event.h +12 -3
@@ -250,8 +250,8 @@ struct perf_event_mmap_page {* * barrier() * if (pc->index) { -* count = pmc_read(pc->index - 1); -* count += pc->offset; +* count = pmc_read(pc->index - 1) - pc->delta_base; +* count += pc->offset * pc->time_enabled/pc->time_running; * } else * goto regular_read; *
@@ -266,12 +266,13 @@ struct perf_event_mmap_page {__s64 offset;/* add to hardware event value */ __u64 time_enabled;/* time event active */ __u64 time_running;/* time event on cpu */ +__u64 delta_base;/* raw counter base value */ /* * Hole for extension of the self monitor capabilities */ -__u64 __reserved[123 ];/* align to 1k */ +__u64 __reserved[122 ];/* align to 1k */ /* * Control data for the mmap() data buffer.
@@ -919,6 +920,7 @@ extern void perf_event_fork(struct task_struct *tsk);extern struct perf_callchain_entry *perf_callchain (struct pt_regs *regs);extern int sysctl_perf_event_paranoid; +extern int sysctl_perf_event_paranoid_user_read;extern int sysctl_perf_event_mlock;extern int sysctl_perf_event_sample_rate;
@@ -937,6 +939,11 @@ static inline bool perf_paranoid_kernel(void)return sysctl_perf_event_paranoid >1 ; } +static inline bool perf_paranoid_user_read (void ) +{ +return !!sysctl_perf_event_paranoid_user_read; +} +extern void perf_event_init (void );extern void perf_tp_event (int event_id,u64 addr,u64 count,void *record,int entry_size,struct pt_regs *regs);
@@ -958,6 +965,8 @@ extern int perf_swevent_get_recursion_context(void);extern void perf_swevent_put_recursion_context (int rctx);extern void perf_event_enable (struct perf_event *event);extern void perf_event_disable (struct perf_event *event); +extern int proc_perf_event_paranoid_user_read (struct ctl_table *table,int write, +void __user *buffer,size_t *length,loff_t *ppos);#else static inline void perf_event_task_sched_in (struct task_struct *task) { }
kernel/perf_event.c +56 -7
100 lines not on this page · raw message
kernel/sysctl.c +7 -0
@@ -929,6 +929,13 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, + { + .procname = "perf_event_paranoid_user_read", + .data = &sysctl_perf_event_paranoid_user_read, + .maxlen = sizeof(sysctl_perf_event_sample_rate), + .mode = 0644, + .proc_handler = proc_perf_event_paranoid_user_read, + },#endif #ifdef CONFIG_KMEMCHECK {
Peter Zijlstra
maintainerQuoted 2 lines · Stephane Eranian
This patch fixes the remapped counter support such that it now works on X86 processors.
(could you please not add all this whitespace in front? and make sure
it's no wider than 70 chars)
Also, I wouldn't say it fixes it, it's currently not broken, its plain not implemented. So this patch adds support for rdpmc.
Quoted 3 lines
For other architectures, e.g., PPC, SPARC, assuming they do offer the ability to read counts directly from user space, all that is needed is a couple of new arch specific functions:
Power already supports this, so if you just broke this Paul is going to be unhappy.
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Stephane Eranian
This patch fixes the remapped counter support such that it now works on X86 processors.
(could you please not add all this whitespace in front? and make sure it's no wider than 70 chars)
I will fix that.
Quoted 3 lines
Also, I wouldn't say it fixes it, it's currently not broken, its plain not implemented. So this patch adds support for rdpmc.
There you go, then.
Quoted 7 lines
Quoted 3 lines
For other architectures, e.g., PPC, SPARC, assuming they do offer the ability to read counts directly from user space, all that is needed is a couple of new arch specific functions:
Power already supports this, so if you just broke this Paul is going to be unhappy.
I certainly changed the content of hdr->offset. But I don't see how PPC could have this working otherwise in the presence of multiplex and thus when you need the timing information to scale.
I'll wait for Paul's comments, then.
Quoted 6 lines · Stephane Eranian
The patch adds a new field to the
perf_mmap_pagebuffer header (delta_base). It is necessary to correctly scale the count. This new value represents the counter value when the event was last scheduled in. It cannot be substracted by the kernel from the total count because of scaling. The correct formula is:count = offset * time_enabled/time_running + (
raw_count-prev_count)
Interesting. On powerpc I took the view that what was needed from a user-space direct_counter_read() function was to return the same value as a read() on the event counter fd would give, that is, the unscaled count. The stuff that's in the mmapped page is sufficient to compute the unscaled count, and I already have userspace code on powerpc that does that.
We don't currently have a way to work out up-to-the-nanosecond values of time_enabled and time_running in userspace, though it could be done. Userspace would have to read the timebase or TSC (or equivalent), subtract a base value, multiply by a scale factor, and add that on to the time_enabled/time_running values stored in the mmapped page. Although this is all quite doable on powerpc where the timebase counts at constant frequency and is synchronized across cpus, it seemed like it could get complicated on x86 where the TSC might change speed and might not be synchronized (though I suppose we only need to supply the values relevant to the cpu where the monitored thread is running).
Your equation above contains the insight that if one wants to scale readings by time_enabled/time_running, the scaling only needs to be applied to the count as of the last time the event got scheduled, and then we can add on the delta count since then, unscaled. (That isn't mathematically exactly the same if the event rate is changing, but it should be close.)
That's probably a useful thing to do, but I think there are other uses of time_enabled and time_running, for example for estimating error bounds on the scaled count, or for knowing whether the event has been on the PMU at all in a given time interval. That's the main reason that the read() interface gives the unscaled count plus (optionally) time_enabled and time_running, rather than giving only the scaled count.
Quoted 6 lines
For other architectures, e.g., PPC, SPARC, assuming they do offer the ability to read counts directly from user space, all that is needed is a couple of new arch specific functions:
-hw_perf_event_counter_width(): actual counter width
-hw_perf_event_index(): register index to pass to the user instruction
-hw_perf_update_user_read_perm(): toggle permission to read from userspace
I'm puzzled why powerpc now needs to supply more stuff when we already have userspace reading of (unscaled) counts working just fine.
We don't have any way to stop userspace reading the counters, so the sysctl_perf_event_paranoid_user_read is useless on powerpc and quite possibly confusing to users.
Quoted 5 lines
include/linux/perf_event.h +0
-0
include/linux/perf_event.h struct perf_event_mmap_page { · line 256
Quoted 9 lines
include/linux/perf_event.h +2
-2@@ -250,8 +250,8 @@ struct perf_event_mmap_page {* * barrier() * if (pc->index) { -* count = pmc_read(pc->index - 1); -* count += pc->offset; +* count = pmc_read(pc->index - 1) - pc->delta_base; +* count += pc->offset * pc->time_enabled/pc->time_running; * } else * goto regular_read;
This changes what's returned by this code snippet from an unscaled count to a scaled count, and thus makes it not what read() returns, which I think is a bad idea. It would be OK to add another alternative code snippet in the comment that returns a scaled count, and make it clear that one returns the unscaled count and the other the scaled count.
include/linux/perf_event.h void perf_event_update_userpage(struct perf_event *event) · line 2287
Quoted 11 lines
include/linux/perf_event.h +5
-2@@ -2249,8 +2280,11 @@ void perf_event_update_userpage(struct perf_event *event)barrier (); userpg->index =perf_event_index (event); userpg->offset =atomic64_read (&event->count); -if (event->state ==PERF_EVENT_STATE_ACTIVE ) - userpg->offset -=atomic64_read (&event->hw.prev_count); +if (event->state ==PERF_EVENT_STATE_ACTIVE + && !perf_paranoid_user_read ()) { +update_event_times (event); + userpg->delta_base =perf_adjust_event_prev_count (event); + }
And this changes the meaning of the offset field and thus breaks the ABI, and makes existing userspace code on powerpc give the wrong answer.
I don't mind if you add fields to make it possible to compute a scaled count more easily, but please don't change the meaning of the existing fields.
Regards,
Paul.
Quoted 7 lines · Paul Mackerras
Quoted 5 lines · Stephane Eranian
The patch adds a new field to the
perf_mmap_pagebuffer header (delta_base). It is necessary to correctly scale the count. This new value represents the counter value when the event was last scheduled in. It cannot be substracted by the kernel from the total count because of scaling. The correct formula is:
Quoted 9 lines
Quoted 1 line
count = offset * time_enabled/time_running + (
raw_count-prev_count)Interesting. On powerpc I took the view that what was needed from a user-space
direct_counter_read()function was to return the same value as aread()on the event counter fd would give, that is, the unscaled count. The stuff that's in the mmapped page is sufficient to compute the unscaled count, and I already have userspace code on powerpc that does that.
We agree on that. When you combined mmapped data + direct_counter_read() you should get the unscaled count.
As for the scaling, you are right, my formula is not quite correct. You also need to scale the delta you obtain for the direct_read_counter(). But for that, you would need the timebase relative to when prev_count was last touched. Then, you would need to collect time at the user level (TSC) and compute the time delta t. In that case, I think the correct formula would become:
(offset + (raw_count - prev_count)) * (time_enabled + t)
count = --------------------------------------------------------------------------------
time_running + t
Quoted 12 lines
We don't currently have a way to work out up-to-the-nanosecond values of
time_enabledandtime_runningin userspace, though it could be done. Userspace would have to read the timebase or TSC (or equivalent), subtract a base value, multiply by a scale factor, and add that on to the time_enabled/time_running values stored in the mmapped page. Although this is all quite doable on powerpc where the timebase counts at constant frequency and is synchronized across cpus, it seemed like it could get complicated on x86 where the TSC might change speed and might not be synchronized (though I suppose we only need to supply the values relevant to the cpu where the monitored thread is running).
Recent X86 processors do have constant TSC, i.e., not subject to frequency scaling.
You don't need to have TSC synchronized as long as you guarantee that prev_count and the kernel TSC snapshot is updated each time the event is scheduled.
Quoted 11 lines
Quoted 7 lines
For other architectures, e.g., PPC, SPARC, assuming they do offer the ability to read counts directly from user space, all that is needed is a couple of new arch specific functions:
-hw_perf_event_counter_width(): actual counter width
-hw_perf_event_index(): register index to pass to the user instruction
-hw_perf_update_user_read_perm(): toggle permission to read from userspaceI'm puzzled why powerpc now needs to supply more stuff when we already have userspace reading of (unscaled) counts working just fine.
Going back to the formula above, and if I revert the change I made to userpage->offset. Then the code would look as follows:
userpg->offset -= atomic64_read(&event->hw.prev_count);
Prev_count is a 64-bit negative value on X86. Whereas direct_read_counter() on X86, returns only the N-bits of the counter, where N is the width of the counter.
For instance, with a 40-bit counter:
direct_read_counter() = 0xffce91a62aprev_count = 0xffffffffce91a600
Thus, offset + (raw_count - prev_count)) would not be correct. That is why in my patch, there is this small adjust to mask off bits 63-N. You may not have to do this on PPC. The alternative would be expose the width to user and fix the direct_read_counter() count.
Quoted 4 lines
I don't mind if you add fields to make it possible to compute a scaled count more easily, but please don't change the meaning of the existing fields.
I can revert the change. I just did not know this was already working on PPC. I guess I was not part of that discussion.