perf_counter: mmap output of overflow data
Subsystems (2)
cover perf_counter: mmap output of overflow data
Only compiled for now, but wanted to get it out for discussion.
RFC PATCH 1/3 perf_counter: add an mmap method to allow userspace to read hardware counters
From: Paul Mackerras <[email protected]>
Impact: new feature giving performance improvement
This adds the ability for userspace to do an mmap on a hardware counter fd and get access to a read-only page that contains the information needed to translate a hardware counter value to the full 64-bit counter value that would be returned by a read on the fd. This is useful on architectures that allow user programs to read the hardware counters, such as PowerPC.
The mmap will only succeed if the counter is a hardware counter monitoring the current process.
On my quad 2.5GHz PowerPC 970MP machine, userspace can read a counter and translate it to the full 64-bit value in about 30ns using the mmapped page, compared to about 830ns for the read syscall on the counter, so this does give a significant performance improvement.
--- arch/powerpc/kernel/perf_counter.c | 6 ++ include/linux/perf_counter.h | 15 +++++++ kernel/perf_counter.c | 76 +++++++++++++++++++++++++++++++++++++ 3 files changed, 97 insertions(+)
arch/powerpc/kernel/perf_counter.c +6 -0
@@ -417,6 +417,8 @@ void hw_perf_restore(u64 disable)atomic64_set (&counter->hw.prev_count, val); counter->hw.idx = hwc_index[i] +1 ;write_pmc (counter->hw.idx, val); +if (counter->user_page) +perf_counter_update_userpage (counter); }mb (); cpuhw->mmcr[0 ] |=MMCR0_PMXE |MMCR0_FCECE ;
@@ -572,6 +574,8 @@ static void power_perf_disable(struct pe ppmu->disable_pmc (counter->hw.idx -1 , cpuhw->mmcr);write_pmc (counter->hw.idx,0 ); counter->hw.idx =0 ; +if (counter->user_page) +perf_counter_update_userpage (counter);break ; } }
@@ -697,6 +701,8 @@ static void record_and_restart(struct pewrite_pmc (counter->hw.idx, val);atomic64_set (&counter->hw.prev_count, val);atomic64_set (&counter->hw.period_left, left); +if (counter->user_page) +perf_counter_update_userpage (counter);/* * Finally record data if requested.
include/linux/perf_counter.h +15 -0
@@ -126,6 +126,17 @@ struct perf_counter_hw_event {#define PERF_COUNTER_IOC_ENABLE _IO('$', 0)#define PERF_COUNTER_IOC_DISABLE _IO('$', 1) +/* +* Structure of the page that can be mapped via mmap +*/ +struct perf_counter_mmap_page { +__u32 version;/* version number of this structure */ +__u32 compat_version;/* lowest version this is compat with */ +__u32 lock;/* seqlock for synchronization */ +__u32 index;/* hardware counter identifier */ +__s64 offset;/* add to hardware counter value */ +}; +#ifdef __KERNEL__/* * Kernel-internal data types and definitions:
@@ -240,6 +251,9 @@ struct perf_counter {int oncpu;int cpu; +/* pointer to page shared with userspace via mmap */ +unsigned long user_page; +/* read() / irq related data */ wait_queue_head_t waitq;/* optional: for NMIs */
@@ -316,6 +330,7 @@ extern int perf_counter_task_enable(voidextern int hw_perf_group_sched_in (struct perf_counter *group_leader,struct perf_cpu_context *cpuctx,struct perf_counter_context *ctx,int cpu); +extern void perf_counter_update_userpage (struct perf_counter *counter);extern void perf_counter_output (struct perf_counter *counter,int nmi,struct pt_regs *regs);
kernel/perf_counter.c +76 -0
94 lines not on this page · raw message
RFC PATCH 2/3 mutex: add atomic_dec_and_mutex_lock]
In reply to Peter ZijlstraFrom: Eric Paris <[email protected]>
Much like the atomic_dec_and_lock() function in which we take an hold a spin_lock if we drop the atomic to 0 this function takes and holds the mutex if we dec the atomic to 0.
--- include/linux/mutex.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
include/linux/mutex.h +23 -0
@@ -151,4 +151,27 @@ extern int __must_check mutex_lock_killaextern int mutex_trylock (struct mutex *lock);extern void mutex_unlock (struct mutex *lock); +/** +* atomic_dec_and_mutex_lock - return holding mutex if we dec to 0 +* @cnt: the atomic which we are to dec +* @lock: the mutex to return holding if we dec to 0 +* +* return true and hold lock if we dec to 0, return false otherwise +*/ +static inline int atomic_dec_and_mutex_lock (atomic_t *cnt,struct mutex *lock) +{ +/* dec if we can't possibly hit 0 */ +if (atomic_add_unless (cnt,-1 ,1 )) +return 0 ; +/* we might hit 0, so take the lock */ +mutex_lock (lock); +if (!atomic_dec_and_test (cnt)) { +/* when we actually did the dec, we didn't hit 0 */ +mutex_unlock (lock); +return 0 ; + } +/* we hit 0, and we hold the lock */ +return 1 ; +} +#endif
RFC PATCH 3/3 perf_counter: new output ABI - part 1
In reply to Peter ZijlstraRework the output ABI
use sys_read() only for instant data and provide mmap() output for all async overflow data.
The first mmap() determines the size of the output buffer. The mmap() size must be a PAGE_SIZE multiple of 1+pages, where pages must be a power of 2 or 0. Further mmap()s of the same fd must have the same size. Once all maps are gone, you can again mmap() with a new size.
In case of 0 extra pages there is no data output and the first page only contains meta data.
When there are data pages, a poll() event will be generated for each full page of data. Furthermore, the output is circular. This means that although 1 page is a valid configuration, its useless, since we'll start overwriting it the instant we report a full page.
Future work will focus on the output format (currently maintained) where we'll likey want each entry denoted by a header which includes a type and length.
Further future work will allow to splice() the fd, also containing the async overflow data -- splice() would be mutually exclusive with mmap() of the data.
--- include/linux/perf_counter.h | 38 +-- kernel/perf_counter.c | 439 +++++++++++++++++++++---------------------- 2 files changed, 241 insertions(+), 236 deletions(-)
include/linux/perf_counter.h +17 -21
@@ -135,6 +135,10 @@ struct perf_counter_mmap_page {__u32 lock;/* seqlock for synchronization */ __u32 index;/* hardware counter identifier */ __s64 offset;/* add to hardware counter value */ + +__u32 data_head;/* head in the data section */ + + char data[0] __attribute__((aligned(PAGE_SIZE))); };#ifdef __KERNEL__
@@ -180,21 +184,6 @@ struct hw_perf_counter {#endif }; -/* -* Hardcoded buffer length limit for now, for IRQ-fed events: -*/ -#define PERF_DATA_BUFLEN 2048 - -/** -* struct perf_data - performance counter IRQ data sampling ... -*/ -struct perf_data { -int len; -int rd_idx; -int overrun; -u8 data[PERF_DATA_BUFLEN ]; -}; -struct perf_counter ;/**
@@ -218,6 +207,14 @@ enum perf_counter_active_state {struct file ; +struct perf_mmap_data { +struct rcu_head rcu_head; +int nr_pages; +atomic_t head; +struct perf_counter_mmap_page *user_page; +u8 *data_pages[0 ]; +}; +/** * struct perf_counter - performance counter kernel representation: */
@@ -251,16 +248,15 @@ struct perf_counter {int oncpu;int cpu; -/* pointer to page shared with userspace via mmap */ -unsigned long user_page; +/* mmap bits */ +struct mutex mmap_mutex; +atomic_t mmap_count; +struct perf_mmap_data *data; -/* read() / irq related data */ +/* poll related */ wait_queue_head_t waitq;/* optional: for NMIs */ int wakeup_pending; -struct perf_data *irqdata; -struct perf_data *usrdata; -struct perf_data data[2 ];void (*destroy)(struct perf_counter *);struct rcu_head rcu_head;
kernel/perf_counter.c +224 -215
566 lines not on this page · raw message
Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1
Quoted 4 lines · Peter Zijlstra
Rework the output ABI
use
sys_read()only for instant data and providemmap()output for all async overflow data.
i like this.
Quoted 6 lines
The first
mmap()determines the size of the output buffer. Themmap()size must be aPAGE_SIZEmultiple of 1+pages, where pages must be a power of 2 or 0. Further mmap()s of the same fd must have the same size. Once all maps are gone, you can againmmap()with a new size.
Quoted 8 lines
In case of 0 extra pages there is no data output and the first page only contains meta data.
When there are data pages, a
poll()event will be generated for each full page of data. Furthermore, the output is circular. This means that although 1 page is a valid configuration, its useless, since we'll start overwriting it the instant we report a full page.
i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult. mmap() seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)
per-CPU-ness will be handled naturally via per-cpu counters.
Paul, can you see any hole/quirkiness in this scheme?
Ingo
Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1
Quoted 9 lines · Ingo Molnar
i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult.
mmap()seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)per-CPU-ness will be handled naturally via per-cpu counters.
Paul, can you see any hole/quirkiness in this scheme?
The one thing I can see that we would lose is the ability to have a signal delivered on every event. The PAPI developers want to be able to get a signal generated every time the counter overflows, and previously we could do that using the O_ASYNC flag, but now we'll only get a signal every page's worth of events.
So I think we want userspace to be able to say how often we should generate a poll event, i.e. provide a way for userspace to say "please generate a poll event every N counter events". That would also solve the problem of 1 page not being a valid configuration - you could set the poll interval to the number of events that fit in half a page, for instance.
Paul.
Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1
Quoted 10 lines · Paul Mackerras
Quoted 8 lines · Ingo Molnar
i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult.
mmap()seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)per-CPU-ness will be handled naturally via per-cpu counters.
Quoted 7 lines
Quoted 1 line
Paul, can you see any hole/quirkiness in this scheme?
The one thing I can see that we would lose is the ability to have a signal delivered on every event. The PAPI developers want to be able to get a signal generated every time the counter overflows, and previously we could do that using the
O_ASYNCflag, but now we'll only get a signal every page's worth of events.
Ah, nice, didn't know about O_ASYNC and was thinking we should perhaps provide some signal too, seems that's already taken care of, sweet :-)
Quoted 6 lines
So I think we want userspace to be able to say how often we should generate a poll event, i.e. provide a way for userspace to say "please generate a poll event every N counter events". That would also solve the problem of 1 page not being a valid configuration - you could set the poll interval to the number of events that fit in half a page, for instance.
Sure, can do, sounds like s sensible extension -- except it will be hard to guess the event size for some future events like callchains and mmap data.
Re: [RFC][PATCH 3/3] perf_counter: new output ABI - part 1
Quoted 29 lines · Peter Zijlstra
Quoted 10 lines · Paul Mackerras
Quoted 8 lines · Ingo Molnar
i think it would still be nice to allow plain old-fashioned poll()+read() loops ... but the logistics of that seem difficult.
mmap()seems to fit this better - and it's probably faster as well. (as we have to construct the kernel-space pages anyway, so mapping them isnt that big of an issue)per-CPU-ness will be handled naturally via per-cpu counters.
Quoted 7 lines
Quoted 1 line
Paul, can you see any hole/quirkiness in this scheme?
The one thing I can see that we would lose is the ability to have a signal delivered on every event. The PAPI developers want to be able to get a signal generated every time the counter overflows, and previously we could do that using the
O_ASYNCflag, but now we'll only get a signal every page's worth of events.Ah, nice, didn't know about
O_ASYNCand was thinking we should perhaps provide some signal too, seems that's already taken care of, sweet :-)Quoted 6 lines
So I think we want userspace to be able to say how often we should generate a poll event, i.e. provide a way for userspace to say "please generate a poll event every N counter events". That would also solve the problem of 1 page not being a valid configuration - you could set the poll interval to the number of events that fit in half a page, for instance.
Quoted 3 lines
Sure, can do, sounds like s sensible extension -- except it will be hard to guess the event size for some future events like callchains and mmap data.
Agreed. Looks like a perfect fit for yet another ioctl/fcntl method?
Ingo