Skip to contentSkip to conversation

perf_counter: mmap output of overflow data

RFC PATCH In review Peter Zijlstra 3 participants 8 messages Mar 20, 2009 – Mar 21, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover perf_counter: mmap output of overflow data

Only compiled for now, but wanted to get it out for discussion.

#1 lore raw

RFC PATCH 1/3 perf_counter: add an mmap method to allow userspace to read hardware counters

From: Paul Mackerras <[email protected]>

Impact: new feature giving performance improvement

This adds the ability for userspace to do an mmap on a hardware counter fd and get access to a read-only page that contains the information needed to translate a hardware counter value to the full 64-bit counter value that would be returned by a read on the fd. This is useful on architectures that allow user programs to read the hardware counters, such as PowerPC.

The mmap will only succeed if the counter is a hardware counter monitoring the current process.

On my quad 2.5GHz PowerPC 970MP machine, userspace can read a counter and translate it to the full 64-bit value in about 30ns using the mmapped page, compared to about 830ns for the read syscall on the counter, so this does give a significant performance improvement.

---
 arch/powerpc/kernel/perf_counter.c |    6 ++
 include/linux/perf_counter.h       |   15 +++++++
 kernel/perf_counter.c              |   76 +++++++++++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+)
arch/powerpc/kernel/perf_counter.c +6 -0
Index: linux-2.6/arch/powerpc/kernel/perf_counter.c
===================================================================
@@ -417,6 +417,8 @@ void hw_perf_restore(u64 disable)
 		atomic64_set(&counter->hw.prev_count, val);
 		counter->hw.idx = hwc_index[i] + 1;
 		write_pmc(counter->hw.idx, val);
+		if (counter->user_page)
+			perf_counter_update_userpage(counter);
 	}
 	mb();
 	cpuhw->mmcr[0] |= MMCR0_PMXE | MMCR0_FCECE;
@@ -572,6 +574,8 @@ static void power_perf_disable(struct pe
 			ppmu->disable_pmc(counter->hw.idx - 1, cpuhw->mmcr);
 			write_pmc(counter->hw.idx, 0);
 			counter->hw.idx = 0;
+			if (counter->user_page)
+				perf_counter_update_userpage(counter);
 			break;
 		}
 	}
@@ -697,6 +701,8 @@ static void record_and_restart(struct pe
 	write_pmc(counter->hw.idx, val);
 	atomic64_set(&counter->hw.prev_count, val);
 	atomic64_set(&counter->hw.period_left, left);
+	if (counter->user_page)
+		perf_counter_update_userpage(counter);
 
 	/*
 	 * Finally record data if requested.
include/linux/perf_counter.h +15 -0
Index: linux-2.6/include/linux/perf_counter.h
===================================================================
@@ -126,6 +126,17 @@ struct perf_counter_hw_event {
 #define PERF_COUNTER_IOC_ENABLE		_IO('$', 0)
 #define PERF_COUNTER_IOC_DISABLE	_IO('$', 1)
 
+/*
+ * Structure of the page that can be mapped via mmap
+ */
+struct perf_counter_mmap_page {
+	__u32	version;		/* version number of this structure */
+	__u32	compat_version;		/* lowest version this is compat with */
+	__u32	lock;			/* seqlock for synchronization */
+	__u32	index;			/* hardware counter identifier */
+	__s64	offset;			/* add to hardware counter value */
+};
+
 #ifdef __KERNEL__
 /*
  * Kernel-internal data types and definitions:
@@ -240,6 +251,9 @@ struct perf_counter {
 	int				oncpu;
 	int				cpu;
 
+	/* pointer to page shared with userspace via mmap */
+	unsigned long			user_page;
+
 	/* read() / irq related data */
 	wait_queue_head_t		waitq;
 	/* optional: for NMIs */
@@ -316,6 +330,7 @@ extern int perf_counter_task_enable(void
 extern int hw_perf_group_sched_in(struct perf_counter *group_leader,
 	       struct perf_cpu_context *cpuctx,
 	       struct perf_counter_context *ctx, int cpu);
+extern void perf_counter_update_userpage(struct perf_counter *counter);
 
 extern void perf_counter_output(struct perf_counter *counter,
 				int nmi, struct pt_regs *regs);
kernel/perf_counter.c +76 -0

94 lines not on this page · raw message

#3 lore raw

RFC PATCH 3/3 perf_counter: new output ABI - part 1

In reply to Peter Zijlstra

Rework the output ABI

use sys_read() only for instant data and provide mmap() output for all async overflow data.

The first mmap() determines the size of the output buffer. The mmap() size must be a PAGE_SIZE multiple of 1+pages, where pages must be a power of 2 or 0. Further mmap()s of the same fd must have the same size. Once all maps are gone, you can again mmap() with a new size.

In case of 0 extra pages there is no data output and the first page only contains meta data.

When there are data pages, a poll() event will be generated for each full page of data. Furthermore, the output is circular. This means that although 1 page is a valid configuration, its useless, since we'll start overwriting it the instant we report a full page.

Future work will focus on the output format (currently maintained) where we'll likey want each entry denoted by a header which includes a type and length.

Further future work will allow to splice() the fd, also containing the async overflow data -- splice() would be mutually exclusive with mmap() of the data.

---
 include/linux/perf_counter.h |   38 +--
 kernel/perf_counter.c        |  439 +++++++++++++++++++++----------------------
 2 files changed, 241 insertions(+), 236 deletions(-)
include/linux/perf_counter.h +17 -21
Index: linux-2.6/include/linux/perf_counter.h
===================================================================
@@ -135,6 +135,10 @@ struct perf_counter_mmap_page {
 	__u32	lock;			/* seqlock for synchronization */
 	__u32	index;			/* hardware counter identifier */
 	__s64	offset;			/* add to hardware counter value */
+
+	__u32   data_head;		/* head in the data section */
+
+	char	data[0] __attribute__((aligned(PAGE_SIZE)));
 };
 
 #ifdef __KERNEL__
@@ -180,21 +184,6 @@ struct hw_perf_counter {
 #endif
 };
 
-/*
- * Hardcoded buffer length limit for now, for IRQ-fed events:
- */
-#define PERF_DATA_BUFLEN		2048
-
-/**
- * struct perf_data - performance counter IRQ data sampling ...
- */
-struct perf_data {
-	int				len;
-	int				rd_idx;
-	int				overrun;
-	u8				data[PERF_DATA_BUFLEN];
-};
-
 struct perf_counter;
 
 /**
@@ -218,6 +207,14 @@ enum perf_counter_active_state {
 
 struct file;
 
+struct perf_mmap_data {
+	struct rcu_head			rcu_head;
+	int				nr_pages;
+	atomic_t			head;
+	struct perf_counter_mmap_page   *user_page;
+	u8 				*data_pages[0];
+};
+
 /**
  * struct perf_counter - performance counter kernel representation:
  */
@@ -251,16 +248,15 @@ struct perf_counter {
 	int				oncpu;
 	int				cpu;
 
-	/* pointer to page shared with userspace via mmap */
-	unsigned long			user_page;
+	/* mmap bits */
+	struct mutex			mmap_mutex;
+	atomic_t			mmap_count;
+	struct perf_mmap_data		*data;
 
-	/* read() / irq related data */
+	/* poll related */
 	wait_queue_head_t		waitq;
 	/* optional: for NMIs */
 	int				wakeup_pending;
-	struct perf_data		*irqdata;
-	struct perf_data		*usrdata;
-	struct perf_data		data[2];
 
 	void (*destroy)(struct perf_counter *);
 	struct rcu_head			rcu_head;
kernel/perf_counter.c +224 -215

566 lines not on this page · raw message

#2 lore raw

RFC PATCH 2/3 mutex: add atomic_dec_and_mutex_lock]

In reply to Peter Zijlstra

From: Eric Paris <[email protected]>

Much like the atomic_dec_and_lock() function in which we take an hold a spin_lock if we drop the atomic to 0 this function takes and holds the mutex if we dec the atomic to 0.

---
 include/linux/mutex.h |   23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
include/linux/mutex.h +23 -0
Index: linux-2.6/include/linux/mutex.h
===================================================================
@@ -151,4 +151,27 @@ extern int __must_check mutex_lock_killa
 extern int mutex_trylock(struct mutex *lock);
 extern void mutex_unlock(struct mutex *lock);
 
+/**
+ * atomic_dec_and_mutex_lock - return holding mutex if we dec to 0
+ * @cnt: the atomic which we are to dec
+ * @lock: the mutex to return holding if we dec to 0
+ *
+ * return true and hold lock if we dec to 0, return false otherwise
+ */
+static inline int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock)
+{
+	/* dec if we can't possibly hit 0 */
+	if (atomic_add_unless(cnt, -1, 1))
+		return 0;
+	/* we might hit 0, so take the lock */
+	mutex_lock(lock);
+	if (!atomic_dec_and_test(cnt)) {
+		/* when we actually did the dec, we didn't hit 0 */
+		mutex_unlock(lock);
+		return 0;
+	}
+	/* we hit 0, and we hold the lock */
+	return 1;
+}
+
 #endif
#4 lore raw

Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1

In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra

Rework the output ABI

use sys_read() only for instant data and provide mmap() output for all async overflow data.

i like this.

Quoted 6 lines

The first mmap() determines the size of the output buffer. The mmap() size must be a PAGE_SIZE multiple of 1+pages, where pages must be a power of 2 or 0. Further mmap()s of the same fd must have the same size. Once all maps are gone, you can again mmap() with a new size.

Quoted 8 lines

In case of 0 extra pages there is no data output and the first page only contains meta data.

When there are data pages, a poll() event will be generated for each full page of data. Furthermore, the output is circular. This means that although 1 page is a valid configuration, its useless, since we'll start overwriting it the instant we report a full page.

i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult. mmap() seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)

per-CPU-ness will be handled naturally via per-cpu counters.

Paul, can you see any hole/quirkiness in this scheme?

Ingo

#5 lore raw

Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1

Quoted 9 lines · Ingo Molnar

i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult. mmap() seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)

per-CPU-ness will be handled naturally via per-cpu counters.

Paul, can you see any hole/quirkiness in this scheme?

The one thing I can see that we would lose is the ability to have a signal delivered on every event. The PAPI developers want to be able to get a signal generated every time the counter overflows, and previously we could do that using the O_ASYNC flag, but now we'll only get a signal every page's worth of events.

So I think we want userspace to be able to say how often we should generate a poll event, i.e. provide a way for userspace to say "please generate a poll event every N counter events". That would also solve the problem of 1 page not being a valid configuration - you could set the poll interval to the number of events that fit in half a page, for instance.

Paul.

#6 lore raw

Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1

Quoted 10 lines · Paul Mackerras
Quoted 8 lines · Ingo Molnar

i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult. mmap() seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)

per-CPU-ness will be handled naturally via per-cpu counters.

Quoted 7 lines
Quoted 1 line

Paul, can you see any hole/quirkiness in this scheme?

The one thing I can see that we would lose is the ability to have a signal delivered on every event. The PAPI developers want to be able to get a signal generated every time the counter overflows, and previously we could do that using the O_ASYNC flag, but now we'll only get a signal every page's worth of events.

Ah, nice, didn't know about O_ASYNC and was thinking we should perhaps provide some signal too, seems that's already taken care of, sweet :-)

Quoted 6 lines

So I think we want userspace to be able to say how often we should generate a poll event, i.e. provide a way for userspace to say "please generate a poll event every N counter events". That would also solve the problem of 1 page not being a valid configuration - you could set the poll interval to the number of events that fit in half a page, for instance.

Sure, can do, sounds like s sensible extension -- except it will be hard to guess the event size for some future events like callchains and mmap data.

#7 lore raw

Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1

Quoted 29 lines · Peter Zijlstra
Quoted 10 lines · Paul Mackerras
Quoted 8 lines · Ingo Molnar

i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult. mmap() seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)

per-CPU-ness will be handled naturally via per-cpu counters.

Quoted 7 lines
Quoted 1 line

Paul, can you see any hole/quirkiness in this scheme?

The one thing I can see that we would lose is the ability to have a signal delivered on every event. The PAPI developers want to be able to get a signal generated every time the counter overflows, and previously we could do that using the O_ASYNC flag, but now we'll only get a signal every page's worth of events.

Ah, nice, didn't know about O_ASYNC and was thinking we should perhaps provide some signal too, seems that's already taken care of, sweet :-)

Quoted 6 lines

So I think we want userspace to be able to say how often we should generate a poll event, i.e. provide a way for userspace to say "please generate a poll event every N counter events". That would also solve the problem of 1 page not being a valid configuration - you could set the poll interval to the number of events that fit in half a page, for instance.

Quoted 3 lines

Sure, can do, sounds like s sensible extension -- except it will be hard to guess the event size for some future events like callchains and mmap data.

Agreed. Looks like a perfect fit for yet another ioctl/fcntl method?

Ingo