perf: use hrtimer for event multiplexing
v3 cover perf: use hrtimer for event multiplexing
The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.
This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.
In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.
In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().
The second patch provide a sysctl control to adjust the multiplexing interval. Unit is milliseconds.
Here is a simple before/after example with two event groups which do require multiplexing. This is done in system-wide mode on an idle system. What matters here is the scaling factor in [] in not the total counts.
Before:
# perf stat -a -e ref-cycles,ref-cycles sleep 10 Performance counter stats for 'sleep 10': 34,319,545 ref-cycles [56.51%] 31,917,229 ref-cycles [43.50%]
10.000827569 seconds time elapsed
After:
# perf stat -a -e ref-cycles,ref-cycles sleep 10 Performance counter stats for 'sleep 10': 11,144,822,193 ref-cycles [50.00%] 11,103,760,513 ref-cycles [50.00%]
10.000672946 seconds time elapsed
In this second version of the patchset, we now have the hrtimer_interval per PMU instance. The tunable is in /sys/devices/XXX/mux_interval_ms, where XXX is the name of the PMU instance. Due to initialization changes of each hrtimer, we had to introduce hrtimer_init_cpu() to initialize a hrtimer from another CPU.
In the 3rd version, we simplify the code a bit by using hrtimer_active(). We stopped using the rotation_list for perf_cpu_hrtimer_cancel(). We also fix an intialization problem.
---
Stephane Eranian (3):
hrtimer: add hrtimer_init_cpu()
perf: use hrtimer for event multiplexing
perf: add sysfs entry to adjust multiplexing interval per PMU
include/linux/hrtimer.h | 2 + include/linux/perf_event.h | 5 +- init/main.c | 2 +- kernel/events/core.c | 166 +++++++++++++++++++++++++++++++++++++++++--- kernel/hrtimer.c | 17 +++-- 5 files changed, 176 insertions(+), 16 deletions(-)
Signature · 1 line
1.7.5.4
PATCH v3 1/2 perf: use hrtimer for event multiplexing
The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.
This patch changes the timer source for multiplexing. It introduces a per-PMU per-cpu hrtimer. The advantage is that even when a core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.
The per-PMU implementation (suggested by PeterZ) enables adjusting the multiplexing interval per PMU. The preferred interval is stashed into the struct pmu. If not set, it will be forced to the default interval value.
In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.
In order for this to work properly, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().
The default interval in milliseconds is set to a timer tick just like with the old code. We will provide a sysctl to tune this in another patch.
--- include/linux/perf_event.h | 3 +- init/main.c | 2 +- kernel/events/core.c | 117 +++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 113 insertions(+), 9 deletions(-)
include/linux/perf_event.h +2 -1
@@ -1106,8 +1106,9 @@ struct perf_cpu_context {struct perf_event_context *task_ctx;int active_oncpu;int exclusive; +struct hrtimer hrtimer; +ktime_t hrtimer_interval;struct list_head rotation_list; -int jiffies_interval;struct pmu *active_pmu;struct perf_cgroup *cgrp; };
init/main.c +1 -1
@@ -541,7 +541,6 @@ asmlinkage void __init start_kernel(void)local_irq_disable (); }idr_init_cache (); -perf_event_init ();rcu_init ();radix_tree_init ();/* init some links before init_ISA_irqs() */
@@ -553,6 +552,7 @@ asmlinkage void __init start_kernel(void)softirq_init ();timekeeping_init ();time_init (); +perf_event_init ();profile_init ();call_function_init ();if (!irqs_disabled ())
kernel/events/core.c +110 -7
178 lines not on this page · raw message
Signature · 1 line
1.7.5.4
PATCH v3 2/2 perf: add sysfs entry to adjust multiplexing interval per PMU
In reply to Stephane EranianThis patch adds /sys/device/xxx/mux_interval_ms to ajust the multiplexing interval per PMU. The unit is milliseconds. Value has to be >= 1.
--- include/linux/perf_event.h | 1 + kernel/events/core.c | 54 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 51 insertions(+), 4 deletions(-)
include/linux/perf_event.h +1 -0
@@ -784,6 +784,7 @@ struct pmu {int *__percpu pmu_disable_count;struct perf_cpu_context *__percpu pmu_cpu_context;int task_ctx_nr; +int hrtimer_interval_ms;/* * Fully disable/enable this PMU, can be used to protect from the PMI
kernel/events/core.c +50 -4
72 lines not on this page · raw message
Signature · 1 line
1.7.5.4
Any comment on this patch set?
This is an important improvement for any system-wide measurement.
Quoted 84 lines · Stephane Eranian
The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.
This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.
In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.
In order for this to work properly with
HOTPLUG_CPU, we had to change the order of initialization instart_kernel()such thathrtimer_init()is run beforeperf_event_init().The second patch provide a sysctl control to adjust the multiplexing interval. Unit is milliseconds.
Here is a simple before/after example with two event groups which do require multiplexing. This is done in system-wide mode on an idle system. What matters here is the scaling factor in [] in not the total counts.
Before:
# perf stat -a -e ref-cycles,ref-cycles sleep 10 Performance counter stats for 'sleep 10': 34,319,545 ref-cycles [56.51%] 31,917,229 ref-cycles [43.50%]10.000827569 seconds time elapsed
After:
# perf stat -a -e ref-cycles,ref-cycles sleep 10 Performance counter stats for 'sleep 10': 11,144,822,193 ref-cycles [50.00%] 11,103,760,513 ref-cycles [50.00%]10.000672946 seconds time elapsed
In this second version of the patchset, we now have the
hrtimer_intervalper PMU instance. The tunable is in/sys/devices/XXX/mux_interval_ms, where XXX is the name of the PMU instance. Due to initialization changes of each hrtimer, we had to introducehrtimer_init_cpu()to initialize a hrtimer from another CPU.In the 3rd version, we simplify the code a bit by using
hrtimer_active(). We stopped using therotation_listforperf_cpu_hrtimer_cancel(). We also fix an intialization problem.Signed-off-by Stephane Eranian <[email protected]>---
Stephane Eranian (3):
hrtimer: addhrtimer_init_cpu()
perf: use hrtimer for event multiplexing
perf: add sysfs entry to adjust multiplexing interval per PMUinclude/linux/hrtimer.h | 2 + include/linux/perf_event.h | 5 +- init/main.c | 2 +- kernel/events/core.c | 166 +++++++++++++++++++++++++++++++++++++++++--- kernel/hrtimer.c | 17 +++-- 5 files changed, 176 insertions(+), 16 deletions(-)--
1.7.5.4