Skip to contentSkip to conversation

perf: use hrtimer for event multiplexing

PATCH Superseded Stephane Eranian 2 participants 11 messages Sep 7, 2012

Subsystems (2)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

cover perf: use hrtimer for event multiplexing

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.

In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.

In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().

The second patch provide a sysctl control to adjust the multiplexing interval. Unit is milliseconds.

Here is a simple before/after example with two event groups which do require multiplexing. This is done in system-wide mode on an idle system. What matters here is the scaling factor in [] in not the total counts.

Before:

# perf stat -a -e ref-cycles,ref-cycles sleep 10
 Performance counter stats for 'sleep 10':
 34,319,545 ref-cycles  [56.51%]
 31,917,229 ref-cycles  [43.50%]

10.000827569 seconds time elapsed

After:

# perf stat -a -e ref-cycles,ref-cycles sleep 10
 Performance counter stats for 'sleep 10':
 11,144,822,193 ref-cycles [50.00%]
 11,103,760,513 ref-cycles [50.00%]

10.000672946 seconds time elapsed

---

Stephane Eranian (3):
perf: use hrtimer for event multiplexing
perf: add sysctl control to adjust multiplexing interval
perf: remove jiffies_interval

 include/linux/perf_event.h |    6 ++-
 init/main.c                |    2 +-
 kernel/events/core.c       |  155 +++++++++++++++++++++++++++++++++++++++++---
 kernel/sysctl.c            |    8 ++
 4 files changed, 160 insertions(+), 11 deletions(-)
Signature · 1 line
1.7.5.4
#1 lore raw

PATCH 1/3 perf: use hrtimer for event multiplexing

The current scheme of using the timer tick was fine for per-thread events. However, it was causing bias issues in system-wide mode (including for uncore PMUs). Event groups would not get their fair share of runtime on the PMU. With tickless kernels, if a core is idle there is no timer tick, and thus no event rotation (multiplexing). However, there are events (especially uncore events) which do count even though cores are asleep.

This patch changes the timer source for multiplexing. It introduces a per-cpu hrtimer. The advantage is that even when the core goes idle, it will come back to service the hrtimer, thus multiplexing on system-wide events works much better.

In order to minimize the impact of the hrtimer, it is turned on and off on demand. When the PMU on a CPU is overcommited, the hrtimer is activated. It is stopped when the PMU is not overcommitted.

In order for this to work properly with HOTPLUG_CPU, we had to change the order of initialization in start_kernel() such that hrtimer_init() is run before perf_event_init().

The default hrtimer interval in milliseconds is set to a timer tick just like with the old code. We will provide a sysctl to tune this in another patch.

---
 init/main.c          |    2 +-
 kernel/events/core.c |  125 ++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 118 insertions(+), 9 deletions(-)
init/main.c +1 -1
index b286730..8ffe441 100644
@@ -541,7 +541,6 @@ asmlinkage void __init start_kernel(void)
 		local_irq_disable();
 	}
 	idr_init_cache();
-	perf_event_init();
 	rcu_init();
 	radix_tree_init();
 	/* init some links before init_ISA_irqs() */
@@ -553,6 +552,7 @@ asmlinkage void __init start_kernel(void)
 	softirq_init();
 	timekeeping_init();
 	time_init();
+	perf_event_init();
 	profile_init();
 	call_function_init();
 	if (!irqs_disabled())
kernel/events/core.c +117 -8

192 lines not on this page · raw message

Signature · 1 line
1.7.5.4
maintainer
#2 lore raw

kernel/events/core.c static LIST_HEAD(pmus); · line 158

Quoted 12 lines · Stephane Eranian
kernel/events/core.c +8 -0
@@ -148,6 +148,15 @@ static LIST_HEAD(pmus);
 static DEFINE_MUTEX(pmus_lock);
 static struct srcu_struct pmus_srcu;
 
+struct perf_cpu_hrtimer {
+       struct hrtimer hrtimer;
+       int active;
+};
+
+static DEFINE_PER_CPU(struct list_head, rotation_list);
+
+static DEFINE_PER_CPU(struct perf_cpu_hrtimer, perf_cpu_hrtimer);

How about sticking the hrtimer in perf_cpu_context so you can have a different rotation interval per PMU ?

Sorta like e9d2b064149ff7ef4acbc65a1b9374ac8b218d3e removed. Stopping the timer when the PMU isn't over committed should solve the NOHZ problem I think.

#3 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
kernel/events/core.c +0 -0
@@ -148,6 +148,15 @@ static LIST_HEAD(pmus);
 static DEFINE_MUTEX(pmus_lock);
 static struct srcu_struct pmus_srcu;
 
Quoted 13 lines

kernel/events/core.c static LIST_HEAD(pmus); · line 158

Quoted 8 lines
kernel/events/core.c +8 -0
+struct perf_cpu_hrtimer {
+       struct hrtimer hrtimer;
+       int active;
+};
+
+static DEFINE_PER_CPU(struct list_head, rotation_list);
+
+static DEFINE_PER_CPU(struct perf_cpu_hrtimer, perf_cpu_hrtimer);

How about sticking the hrtimer in perf_cpu_context so you can have a different rotation interval per PMU ?

I think having different intervals would be a good thing, especially for uncore. But now, I am wondering how this could work without too much overhead. Looks like you're suggesting arming multiple hrtimers if multiple PMU are overcommitted. Is that right? As opposed to having a PMU multiplier off of a single per-cpu hrtimer.

Quoted 3 lines

Sorta like e9d2b064149ff7ef4acbc65a1b9374ac8b218d3e removed. Stopping the timer when the PMU isn't over committed should solve the NOHZ problem I think.

Yeah, my patch does solve this. That's what I show in my example in the intro msg.

maintainer
#4 lore raw
Quoted 4 lines · Stephane Eranian

I think having different intervals would be a good thing, especially for uncore. But now, I am wondering how this could work without too much overhead. Looks like you're suggesting arming multiple hrtimers if multiple PMU are overcommitted. Is that right?

Right, we shoulnd't have too many PMUs anyway, let alone over committed ones, so a timer per cpu per pmu should be fine.

Quoted 2 lines

As opposed to having a PMU multiplier off of a
single per-cpu hrtimer.

#5 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

I think having different intervals would be a good thing, especially for uncore. But now, I am wondering how this could work without too much overhead. Looks like you're suggesting arming multiple hrtimers if multiple PMU are overcommitted. Is that right?

Right, we shoulnd't have too many PMUs anyway, let alone over committed ones, so a timer per cpu per pmu should be fine.

Quoted 2 lines

As opposed to having a PMU multiplier off of a
single per-cpu hrtimer.

That's true. I started modifying my code to implement your suggestion. We'll see how it goes. Then we would have to export that mux interval via sysfs for each PMU.

maintainer
#6 lore raw
Quoted 4 lines · Stephane Eranian

That's true. I started modifying my code to implement your suggestion. We'll see how it goes. Then we would have to export that mux interval via sysfs for each PMU.

Indeed. Thanks!

maintainer
#7 lore raw
In reply to Stephane Eranian

On Fri, 2012-09-07 at 16:29 +0200, Stephane Eranian wrote:

Style nit:

kernel/events/core.c perf_cgroup_mark_enabled(struct perf_event *event, · line 725

Quoted 8 lines
kernel/events/core.c +6 -0
+	if (h->active)
+		list_for_each_entry_safe(cpuctx, tmp, head, rotation_list)
+			rotations += perf_rotate_context(cpuctx);
 
 
+	if (!hrtimer_callback_running(hr))
+		__hrtimer_start_range_ns(hr, ns_to_ktime(PERF_CPU_HRTIMER),
+					 0, HRTIMER_MODE_REL_PINNED, 0);

All multi-line statements should be a stmt block, even if not strictly required.

#8 lore raw

PATCH 2/3 perf: add sysctl control to adjust multiplexing interval

In reply to Stephane Eranian

add /proc/sys/kernel/perf_event_mux_interval_ms
entry to adjust the multiplexing period.

Unit is milliseconds.

---
 include/linux/perf_event.h |    5 +++++
 kernel/events/core.c       |   43 ++++++++++++++++++++++++++++++++++++-------
 kernel/sysctl.c            |    8 ++++++++
 3 files changed, 49 insertions(+), 7 deletions(-)
include/linux/perf_event.h +5 -0
index cc5e2cd..21c3bb0 100644
@@ -1297,11 +1297,16 @@ static inline void perf_callchain_store(struct perf_callchain_entry *entry, u64
 extern int sysctl_perf_event_paranoid;
 extern int sysctl_perf_event_mlock;
 extern int sysctl_perf_event_sample_rate;
+extern int sysctl_perf_event_mux_interval_ms;
 
 extern int perf_proc_update_handler(struct ctl_table *table, int write,
 		void __user *buffer, size_t *lenp,
 		loff_t *ppos);
 
+extern int perf_proc_mux_interval_ms_handler(struct ctl_table *table, int write,
+		void __user *buffer, size_t *lenp,
+		loff_t *ppos);
+
 static inline bool perf_paranoid_tracepoint_raw(void)
 {
 	return sysctl_perf_event_paranoid > -1;
kernel/events/core.c +36 -7

85 lines not on this page · raw message

kernel/sysctl.c +8 -0
index 81c7b1a..0d7b2c8 100644
@@ -981,6 +981,14 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= perf_proc_update_handler,
 	},
+	{
+		.procname	= "perf_event_mux_interval_ms",
+		.data		= &sysctl_perf_event_mux_interval_ms,
+		.maxlen		= sizeof(sysctl_perf_event_mux_interval_ms),
+		.mode		= 0644,
+		.proc_handler	= perf_proc_mux_interval_ms_handler,
+	},
+
 #endif
 #ifdef CONFIG_KMEMCHECK
 	{
Signature · 1 line
1.7.5.4
#9 lore raw

PATCH 3/3 perf: remove jiffies_interval

In reply to Stephane Eranian

Obsolete because superseded by hrtimer based
multiplexing.

---
 include/linux/perf_event.h |    1 -
 kernel/events/core.c       |    1 -
 2 files changed, 0 insertions(+), 2 deletions(-)
include/linux/perf_event.h +0 -1
index 21c3bb0..b0cb209 100644
@@ -1107,7 +1107,6 @@ struct perf_cpu_context {
 	int				active_oncpu;
 	int				exclusive;
 	struct list_head		rotation_list;
-	int				jiffies_interval;
 	struct pmu			*active_pmu;
 	struct perf_cgroup		*cgrp;
 };
kernel/events/core.c +0 -1
index fd53509..e0db140 100644
@@ -6126,7 +6126,6 @@ int perf_pmu_register(struct pmu *pmu, char *name, int type)
 		lockdep_set_class(&cpuctx->ctx.lock, &cpuctx_lock);
 		cpuctx->ctx.type = cpu_context;
 		cpuctx->ctx.pmu = pmu;
-		cpuctx->jiffies_interval = 1;
 		INIT_LIST_HEAD(&cpuctx->rotation_list);
 		cpuctx->active_pmu = pmu;
 	}
Signature · 1 line
1.7.5.4
maintainer
#10 lore raw

Re: [PATCH 3/3] perf: remove jiffies_interval

Quoted 2 lines · Stephane Eranian

Obsolete because superseded by hrtimer based
multiplexing.

Not entirely, the jiffies_interval allows different PMUs to have different rotation speeds. Your code doesn't allow this.