perf: use hrtimer for event multiplexing
Subsystems (2)
cover perf: use hrtimer for event multiplexing
The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.
This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.
In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.
In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().
The second patch provide a sysctl control to adjust the multiplexing interval. Unit is milliseconds.
Here is a simple before/after example with two event groups which do require multiplexing. This is done in system-wide mode on an idle system. What matters here is the scaling factor in [] in not the total counts.
Before:
# perf stat -a -e ref-cycles,ref-cycles sleep 10 Performance counter stats for 'sleep 10': 34,319,545 ref-cycles [56.51%] 31,917,229 ref-cycles [43.50%]
10.000827569 seconds time elapsed
After:
# perf stat -a -e ref-cycles,ref-cycles sleep 10 Performance counter stats for 'sleep 10': 11,144,822,193 ref-cycles [50.00%] 11,103,760,513 ref-cycles [50.00%]
10.000672946 seconds time elapsed
---
Stephane Eranian (3):
perf: use hrtimer for event multiplexing
perf: add sysctl control to adjust multiplexing interval
perf: remove jiffies_interval
include/linux/perf_event.h | 6 ++- init/main.c | 2 +- kernel/events/core.c | 155 +++++++++++++++++++++++++++++++++++++++++--- kernel/sysctl.c | 8 ++ 4 files changed, 160 insertions(+), 11 deletions(-)
Signature · 1 line
1.7.5.4
PATCH 1/3 perf: use hrtimer for event multiplexing
The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.
This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.
In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.
In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().
The default hrtimer interval in milliseconds is set to a timer tick just like with the old code. We will provide a sysctl to tune this in another patch.
--- init/main.c | 2 +- kernel/events/core.c | 125 ++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 118 insertions(+), 9 deletions(-)
init/main.c +1 -1
@@ -541,7 +541,6 @@ asmlinkage void __init start_kernel(void)local_irq_disable (); }idr_init_cache (); -perf_event_init ();rcu_init ();radix_tree_init ();/* init some links before init_ISA_irqs() */
@@ -553,6 +552,7 @@ asmlinkage void __init start_kernel(void)softirq_init ();timekeeping_init ();time_init (); +perf_event_init ();profile_init ();call_function_init ();if (!irqs_disabled ())
kernel/events/core.c +117 -8
192 lines not on this page · raw message
Signature · 1 line
1.7.5.4
Peter Zijlstra
maintainerkernel/events/core.c static LIST_HEAD(pmus); · line 158
Quoted 12 lines · Stephane Eranian
kernel/events/core.c +8
-0@@ -148,6 +148,15 @@ static LIST_HEAD(pmus);static DEFINE_MUTEX (pmus_lock);static struct srcu_struct pmus_srcu; +struct perf_cpu_hrtimer { +struct hrtimer hrtimer; +int active; +}; + +static DEFINE_PER_CPU(struct list_head, rotation_list); + +static DEFINE_PER_CPU(struct perf_cpu_hrtimer, perf_cpu_hrtimer);
How about sticking the hrtimer in perf_cpu_context so you can have a different rotation interval per PMU ?
Sorta like e9d2b064149ff7ef4acbc65a1b9374ac8b218d3e removed. Stopping the timer when the PMU isn't over committed should solve the NOHZ problem I think.
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
kernel/events/core.c +0
-0@@ -148,6 +148,15 @@ static LIST_HEAD(pmus);static DEFINE_MUTEX (pmus_lock);static struct srcu_struct pmus_srcu;
Quoted 13 lines
kernel/events/core.c static LIST_HEAD(pmus); · line 158
Quoted 8 lines
kernel/events/core.c +8
-0+struct perf_cpu_hrtimer { +struct hrtimer hrtimer; +int active; +}; + +static DEFINE_PER_CPU(struct list_head, rotation_list); + +static DEFINE_PER_CPU(struct perf_cpu_hrtimer, perf_cpu_hrtimer);How about sticking the hrtimer in
perf_cpu_contextso you can have a different rotation interval per PMU ?
I think having different intervals would be a good thing, especially for uncore. But now, I am wondering how this could work without too much overhead. Looks like you're suggesting arming multiple hrtimers if multiple PMU are overcommitted. Is that right? As opposed to having a PMU multiplier off of a single per-cpu hrtimer.
Quoted 3 lines
Sorta like
e9d2b064149ff7ef4acbc65a1b9374ac8b218d3eremoved. Stopping the timer when the PMU isn't over committed should solve the NOHZ problem I think.
Yeah, my patch does solve this. That's what I show in my example in the intro msg.
Peter Zijlstra
maintainerQuoted 4 lines · Stephane Eranian
I think having different intervals would be a good thing, especially for uncore. But now, I am wondering how this could work without too much overhead. Looks like you're suggesting arming multiple hrtimers if multiple PMU are overcommitted. Is that right?
Right, we shoulnd't have too many PMUs anyway, let alone over committed ones, so a timer per cpu per pmu should be fine.
Quoted 2 lines
As opposed to having a PMU multiplier off of a
single per-cpu hrtimer.
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
I think having different intervals would be a good thing, especially for uncore. But now, I am wondering how this could work without too much overhead. Looks like you're suggesting arming multiple hrtimers if multiple PMU are overcommitted. Is that right?
Right, we shoulnd't have too many PMUs anyway, let alone over committed ones, so a timer per cpu per pmu should be fine.
Quoted 2 lines
As opposed to having a PMU multiplier off of a
single per-cpu hrtimer.
That's true. I started modifying my code to implement your suggestion. We'll see how it goes. Then we would have to export that mux interval via sysfs for each PMU.
Peter Zijlstra
maintainerQuoted 4 lines · Stephane Eranian
That's true. I started modifying my code to implement your suggestion. We'll see how it goes. Then we would have to export that mux interval via sysfs for each PMU.
Indeed. Thanks!
Peter Zijlstra
maintainerOn Fri, 2012-09-07 at 16:29 +0200, Stephane Eranian wrote:
Style nit:
kernel/events/core.c perf_cgroup_mark_enabled(struct perf_event *event, · line 725
Quoted 8 lines
kernel/events/core.c +6
-0+if (h->active) +list_for_each_entry_safe (cpuctx, tmp, head, rotation_list) + rotations +=perf_rotate_context (cpuctx); +if (!hrtimer_callback_running (hr)) +__hrtimer_start_range_ns (hr,ns_to_ktime (PERF_CPU_HRTIMER ), +0 ,HRTIMER_MODE_REL_PINNED ,0 );
All multi-line statements should be a stmt block, even if not strictly required.
PATCH 2/3 perf: add sysctl control to adjust multiplexing interval
In reply to Stephane Eranianadd /proc/sys/kernel/perf_event_mux_interval_ms
entry to adjust the multiplexing period.
Unit is milliseconds.
--- include/linux/perf_event.h | 5 +++++ kernel/events/core.c | 43 ++++++++++++++++++++++++++++++++++++------- kernel/sysctl.c | 8 ++++++++ 3 files changed, 49 insertions(+), 7 deletions(-)
include/linux/perf_event.h +5 -0
@@ -1297,11 +1297,16 @@ static inline void perf_callchain_store(struct perf_callchain_entry *entry, u64extern int sysctl_perf_event_paranoid;extern int sysctl_perf_event_mlock;extern int sysctl_perf_event_sample_rate; +extern int sysctl_perf_event_mux_interval_ms;extern int perf_proc_update_handler (struct ctl_table *table,int write,void __user *buffer,size_t *lenp,loff_t *ppos); +extern int perf_proc_mux_interval_ms_handler (struct ctl_table *table,int write, +void __user *buffer,size_t *lenp, +loff_t *ppos); +static inline bool perf_paranoid_tracepoint_raw (void ) {return sysctl_perf_event_paranoid >-1 ;
kernel/events/core.c +36 -7
85 lines not on this page · raw message
kernel/sysctl.c +8 -0
@@ -981,6 +981,14 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = perf_proc_update_handler, }, + { + .procname = "perf_event_mux_interval_ms", + .data = &sysctl_perf_event_mux_interval_ms, + .maxlen = sizeof(sysctl_perf_event_mux_interval_ms), + .mode = 0644, + .proc_handler = perf_proc_mux_interval_ms_handler, + }, +#endif #ifdef CONFIG_KMEMCHECK {
Signature · 1 line
1.7.5.4
PATCH 3/3 perf: remove jiffies_interval
In reply to Stephane EranianObsolete because superseded by hrtimer based
multiplexing.
--- include/linux/perf_event.h | 1 - kernel/events/core.c | 1 - 2 files changed, 0 insertions(+), 2 deletions(-)
include/linux/perf_event.h +0 -1
@@ -1107,7 +1107,6 @@ struct perf_cpu_context {int active_oncpu;int exclusive;struct list_head rotation_list; -int jiffies_interval;struct pmu *active_pmu;struct perf_cgroup *cgrp; };
kernel/events/core.c +0 -1
@@ -6126,7 +6126,6 @@ int perf_pmu_register(struct pmu *pmu, char *name, int type)lockdep_set_class (&cpuctx->ctx.lock, &cpuctx_lock); cpuctx->ctx.type = cpu_context; cpuctx->ctx.pmu = pmu; - cpuctx->jiffies_interval =1 ;INIT_LIST_HEAD (&cpuctx->rotation_list); cpuctx->active_pmu = pmu; }
Signature · 1 line
1.7.5.4
Peter Zijlstra
maintainerRe: [PATCH 3/3] perf: remove jiffies_interval
Quoted 2 lines · Stephane Eranian
Obsolete because superseded by hrtimer based
multiplexing.
Not entirely, the jiffies_interval allows different PMUs to have different rotation speeds. Your code doesn't allow this.