Skip to contentSkip to conversation

perf: use hrtimer for event multiplexing

PATCH Superseded Stephane Eranian 4 participants 12 messages Mar 22, 2013 – Mar 25, 2013 perf events

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover perf: use hrtimer for event multiplexing

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.

In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommitted, the hrtimer is activated. It is stopped when the PMU is not overcommitted.

In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().

The second patch provide a sysctl control to adjust the multiplexing interval. Unit is milliseconds.

Here is a simple before/after example with two event groups which do require multiplexing. This is done in system-wide mode on an idle system. What matters here is the scaling factor in [] in not the total counts.

Before:

# perf stat -a -e ref-cycles,ref-cycles sleep 10
 Performance counter stats for 'sleep 10':
 34,319,545 ref-cycles  [56.51%]
 31,917,229 ref-cycles  [43.50%]

10.000827569 seconds time elapsed

After:

# perf stat -a -e ref-cycles,ref-cycles sleep 10
 Performance counter stats for 'sleep 10':
 11,144,822,193 ref-cycles [50.00%]
 11,103,760,513 ref-cycles [50.00%]

10.000672946 seconds time elapsed

What matters here is the 50% not the actual count. Ref-cycles runs only on one fixed counter. With two instances, each should get 50% of the PMU which is now true. This helps mitigate the error introduced by the scaling.

In this second version of the patchset, we now have the hrtimer_interval per PMU instance. The tunable is in /sys/devices/XXX/mux_interval_ms, where XXX is the name of the PMU instance. Due to initialization changes of each hrtimer, we had to introduce hrtimer_init_cpu() to initialize a hrtimer from another CPU.

In the 3rd version, we simplify the code a bit by using hrtimer_active(). We stopped using the rotation_list for perf_cpu_hrtimer_cancel(). We also fix an intialization problem.

In the 4th version, we rebase to 3.8.0-rc7 and we kept SW event on the rotation list which is now used only for unthrottling. We also renamed the sysfs tunable to perf_event_mux_interval_ms to be more consistent with the existing sysctl entries.

In the 5th version, we modified the code such that a new hrtimer interval is applied immediately to any active hrtimer as suggested by Jiri Olsa. Also got rid of the CPU notifier for hrtimer, it was useless and unreliable. The code is rebased to 3.9.0-rc3.

---

Stephane Eranian (2):
perf: use hrtimer for event multiplexing
perf: add sysfs entry to adjust multiplexing interval per PMU

 include/linux/perf_event.h |    4 +-
 init/main.c                |    2 +-
 kernel/events/core.c       |  173 +++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 167 insertions(+), 12 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH v5 1/2 perf: use hrtimer for event multiplexing

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

This patch changes the timer source for multiplexing. It introduces a per-PMU per-cpu hrtimer. The advantage is that even when a core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.

The per-PMU implementation (suggested by PeterZ) enables adjusting the multiplexing interval per PMU. The preferred interval is stashed into the struct pmu. If not set, it will be forced to the default interval value.

In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.

In order for this to work properly, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().

The default interval in milliseconds is set to a timer tick just like with the old code. We will provide a sysctl to tune this in another patch.

---
 include/linux/perf_event.h |    3 +-
 init/main.c                |    2 +-
 kernel/events/core.c       |  114 ++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 109 insertions(+), 10 deletions(-)
include/linux/perf_event.h +2 -1
index 12a1aa2..e2e6c7e 100644
@@ -500,8 +500,9 @@ struct perf_cpu_context {
 	struct perf_event_context	*task_ctx;
 	int				active_oncpu;
 	int				exclusive;
+	struct hrtimer			hrtimer;
+	ktime_t				hrtimer_interval;
 	struct list_head		rotation_list;
-	int				jiffies_interval;
 	struct pmu			*unique_pmu;
 	struct perf_cgroup		*cgrp;
 };
init/main.c +1 -1
index b3e0614..1542b38 100644
@@ -544,7 +544,6 @@ asmlinkage void __init start_kernel(void)
 		local_irq_disable();
 	}
 	idr_init_cache();
-	perf_event_init();
 	rcu_init();
 	radix_tree_init();
 	/* init some links before init_ISA_irqs() */
@@ -556,6 +555,7 @@ asmlinkage void __init start_kernel(void)
 	softirq_init();
 	timekeeping_init();
 	time_init();
+	perf_event_init();
 	profile_init();
 	call_function_init();
 	if (!irqs_disabled())
kernel/events/core.c +106 -8

175 lines not on this page · raw message

Signature · 1 line
1.7.9.5
maintainer
#2 lore raw
Quoted 6 lines · Stephane Eranian
(quoted diff) +2 -0
+#define PERF_CPU_HRTIMER (1000 / HZ)
+static enum hrtimer_restart perf_cpu_hrtimer_handler(struct hrtimer

*hr)

kernel/events/core.c +3 -0
+{
+       struct perf_cpu_context *cpuctx;
+       enum hrtimer_restart ret = HRTIMER_NORESTART;

kernel/events/core.c perf_cgroup_mark_enabled(struct perf_event *event, · line 665

Quoted 10 lines
kernel/events/core.c +10 -0
+       unsigned long flags;
+       int rotations = 0;
+
+       cpuctx = container_of(hr, struct perf_cpu_context, hrtimer);
+
+       local_irq_save(flags);
+
+       rotations = perf_rotate_context(cpuctx);
+
+       local_irq_restore(flags);

IIRC hrtimer callbacks are always ran with interrupts disabled.. not a big deal though. I'll continue reading.

#3 lore raw
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
(quoted diff) +2 -0
+#define PERF_CPU_HRTIMER (1000 / HZ)
+static enum hrtimer_restart perf_cpu_hrtimer_handler(struct hrtimer

*hr)

kernel/events/core.c +8 -0
+{
+       struct perf_cpu_context *cpuctx;
+       enum hrtimer_restart ret = HRTIMER_NORESTART;
+       unsigned long flags;
+       int rotations = 0;
+
+       cpuctx = container_of(hr, struct perf_cpu_context, hrtimer);
+
Quoted 8 lines

kernel/events/core.c perf_cgroup_mark_enabled(struct perf_event *event, · line 665

Quoted 5 lines
kernel/events/core.c +5 -0
+       local_irq_save(flags);
+
+       rotations = perf_rotate_context(cpuctx);
+
+       local_irq_restore(flags);

IIRC hrtimer callbacks are always ran with interrupts disabled.. not a big deal though. I'll continue reading.

I can create a V6 to remove this if necessary.

#4 lore raw

PATCH v5 2/2 perf: add sysfs entry to adjust multiplexing interval per PMU

In reply to Stephane Eranian

This patch adds /sys/device/xxx/perf_event_mux_interval_ms to ajust the multiplexing interval per PMU. The unit is milliseconds. Value has to be >= 1.

In the 4th version, we renamed the sysfs file to be more consistent with the other /proc/sys/kernel entries for perf_events.

In the 5th version, we handle the reprogramming of the hrtimer using hrtimer_forward_now(). That way, we sync up to new timer value quickly (suggested by Jiri Olsa).

---
 include/linux/perf_event.h |    1 +
 kernel/events/core.c       |   63 +++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 60 insertions(+), 4 deletions(-)
include/linux/perf_event.h +1 -0
index e2e6c7e..e057b06 100644
@@ -193,6 +193,7 @@ struct pmu {
 	int * __percpu			pmu_disable_count;
 	struct perf_cpu_context * __percpu pmu_cpu_context;
 	int				task_ctx_nr;
+	int				hrtimer_interval_ms;
 
 	/*
 	 * Fully disable/enable this PMU, can be used to protect from the PMI
kernel/events/core.c +59 -4

81 lines not on this page · raw message

Signature · 1 line
1.7.9.5
maintainer
#5 lore raw

Re: [PATCH v5 2/2] perf: add sysfs entry to adjust multiplexing interval per PMU

kernel/events/core.c type_show(struct device *dev, struct device_attribute *attr, char *page) · line 6076

Quoted 11 lines · Stephane Eranian

In the 4th version, we renamed the sysfs file to be more consistent with the other /proc/sys/kernel entries for perf_events.

static struct device_attribute pmu_dev_attrs[] = {

kernel/events/core.c +3 -2
-       __ATTR_RO(type),
-       __ATTR_NULL,
+       __ATTR_RO(type),
+       __ATTR_RW(perf_event_mux_interval_ms),
+       __ATTR_NULL,
 };

but but but, the file doesn't live in /proc/sys/kernel/, its in /sys/bus/event_source/devices/*/$foo and none of the files there are perf_event_* prefixed.

Other than that, the patches look fine.

#6 lore raw

Re: [PATCH v5 2/2] perf: add sysfs entry to adjust multiplexing interval per PMU

Quoted 14 lines · Peter Zijlstra

kernel/events/core.c type_show(struct device *dev, struct device_attribute *attr, char *page) · line 6076

Quoted 11 lines · Stephane Eranian

In the 4th version, we renamed the sysfs file to be more consistent with the other /proc/sys/kernel entries for perf_events.

static struct device_attribute pmu_dev_attrs[] = {

kernel/events/core.c +3 -2
-       __ATTR_RO(type),
-       __ATTR_NULL,
+       __ATTR_RO(type),
+       __ATTR_RW(perf_event_mux_interval_ms),
+       __ATTR_NULL,
 };
Quoted 4 lines

but but but, the file doesn't live in /proc/sys/kernel/, its in /sys/bus/event_source/devices/*/$foo and none of the files there are perf_event_* prefixed.

True. Some leftover from the earlier patch. The sysfs entry is now per pmu instance. So yes, it is in /sys/devices/*/

Quoted 3 lines

Other than that, the patches look fine.

Thanks.

#7 lore raw
In reply to Stephane Eranian

2013/3/22 Stephane Eranian <[email protected]>:

Quoted 79 lines

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.

In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommitted, the hrtimer is activated. It is stopped when the PMU is not overcommitted.

In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().

The second patch provide a sysctl control to adjust the multiplexing interval. Unit is milliseconds.

Here is a simple before/after example with two event groups which do require multiplexing. This is done in system-wide mode on an idle system. What matters here is the scaling factor in [] in not the total counts.

Before:

# perf stat -a -e ref-cycles,ref-cycles sleep 10
 Performance counter stats for 'sleep 10':
 34,319,545 ref-cycles  [56.51%]
 31,917,229 ref-cycles  [43.50%]

10.000827569 seconds time elapsed

After:

# perf stat -a -e ref-cycles,ref-cycles sleep 10
 Performance counter stats for 'sleep 10':
 11,144,822,193 ref-cycles [50.00%]
 11,103,760,513 ref-cycles [50.00%]

10.000672946 seconds time elapsed

What matters here is the 50% not the actual count. Ref-cycles runs only on one fixed counter. With two instances, each should get 50% of the PMU which is now true. This helps mitigate the error introduced by the scaling.

In this second version of the patchset, we now have the hrtimer_interval per PMU instance. The tunable is in /sys/devices/XXX/mux_interval_ms, where XXX is the name of the PMU instance. Due to initialization changes of each hrtimer, we had to introduce hrtimer_init_cpu() to initialize a hrtimer from another CPU.

In the 3rd version, we simplify the code a bit by using hrtimer_active(). We stopped using the rotation_list for perf_cpu_hrtimer_cancel(). We also fix an intialization problem.

In the 4th version, we rebase to 3.8.0-rc7 and we kept SW event on the rotation list which is now used only for unthrottling. We also renamed the sysfs tunable to perf_event_mux_interval_ms to be more consistent with the existing sysctl entries.

Quoted 8 lines

In the 5th version, we modified the code such that a new hrtimer interval is applied immediately to any active hrtimer as suggested by Jiri Olsa. Also got rid of the CPU notifier for hrtimer, it was useless and unreliable. The code is rebased to 3.9.0-rc3.

And I have to say this patch is going to be very useful for the full dynticks tree. We are happy to get rid of that tick hook.

maintainer
#8 lore raw
Quoted 2 lines · Frederic Weisbecker

And I have to say this patch is going to be very useful for the full dynticks tree. We are happy to get rid of that tick hook.

I'm sorry to have to disappoint, but the tick hook is still there ;-), its just no longer used for counter rotation. Of course we have multiple uses for the tick, what did you think :-)

#9 lore raw

2013/3/25 Peter Zijlstra <[email protected]>:

Quoted 8 lines · Frederic Weisbecker
Quoted 2 lines

And I have to say this patch is going to be very useful for the full dynticks tree. We are happy to get rid of that tick hook.

I'm sorry to have to disappoint, but the tick hook is still there ;-), its just no longer used for counter rotation. Of course we have multiple uses for the tick, what did you think :-)

Haha!
Bah, at least some significant part of the work is pulled outside the hook :)

#10 lore raw
In reply to Stephane Eranian
Quoted 9 lines · Stephane Eranian

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

Would it be possible to only do this when uncore events are active? Otherwise it may have a large power cost and actually change results, as the core counters will tick more.

-Andi

#11 lore raw
Quoted 12 lines · Andi Kleen
Quoted 9 lines · Stephane Eranian

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

Quoted 5 lines

Would it be possible to only do this when uncore events are active? Otherwise it may have a large power cost and actually change results, as the core counters will tick more.

the hrtimer is activated only when multiplexing is needed and that's for any PMU.