perf_event: fix loss of notification with multi-event
Stephane Eranian
Reviewed-byHere is a fixed version of Peter's patch. I fixed the compilation errors + added the missing linked list + spinlock initializations.
I also added the mmap_mutex in perf_poll() to avoid the race with perf_set_output(). It's not pretty but it solves the problem of a process getting stuck on a sampling buffer which is never going to get any samples. This was an unlikely scenario but we should take care of it anyway.
This patch solves the issue with loss of buffer notifications when sampling on multiple events at the same time. In that case fewer than expected samples would be captured.
---
include/linux/perf_event.h +1 -0
@@ -822,6 +822,7 @@ struct perf_event {int mmap_locked;struct user_struct *mmap_user;struct ring_buffer *rb; +struct list_head rb_entry;/* poll related */ wait_queue_head_t waitq;
kernel/events/core.c +84 -2
142 lines not on this page · raw message
kernel/events/internal.h +3 -0
@@ -26,6 +26,9 @@ struct ring_buffer {local_t lost;/* nr records lost */ long watermark;/* wakeup watermark */ +/* poll crap */ +spinlock_t event_lock; +struct list_head event_list;struct perf_event_mmap_page *user_page;void *data_pages[0 ];
kernel/events/ring_buffer.c +3 -0
@@ -209,6 +209,9 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) rb->writable =1 ;atomic_set (&rb->refcount,1 ); + +INIT_LIST_HEAD (&rb->event_list); +spin_lock_init (&rb->event_lock); }#ifndef CONFIG_PERF_USE_VMALLOC
Peter Zijlstra
maintainerQuoted 10 lines · Stephane Eranian
Here is a fixed version of Peter's patch. I fixed the compilation errors + added the missing linked list + spinlock initializations.
I also added the
mmap_mutexinperf_poll()to avoid the race withperf_set_output(). It's not pretty but it solves the problem of a process getting stuck on a sampling buffer which is never going to get any samples. This was an unlikely scenario but we should take care of it anyway.
Quoted 3 lines
This patch solves the issue with loss of buffer notifications when sampling on multiple events at the same time. In that case fewer than expected samples would be captured.
Hehe, so it actually works eh.. amazing stuff ;-)
OK, I'll stick it in perf/urgent.. Thanks Stephane!
If anybody happens upon a prettier solution, don't be shy and please do tell us ;-)
Peter Zijlstra
maintainerQuoted 1 line · Peter Zijlstra
OK, I'll stick it in perf/urgent.. Thanks Stephane!
I'll use you as test-subject for my quilt2git script ( http://programming.kicks-ass.net/misc/quilt2git.sh )
My exploded quilt tree lives at:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git
With this particular commit:
in the perf/urgent branch.
Peter Zijlstra
maintainerQuoted 9 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
OK, I'll stick it in perf/urgent.. Thanks Stephane!
I'll use you as test-subject for my quilt2git script ( http://programming.kicks-ass.net/misc/quilt2git.sh )
My exploded quilt tree lives at:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git
With a note that this tree is completely regenerated every time I run this script, so its a complete rebase. Its only purpose in life is to expose my quilt tree in git topic branch form for whoemever thinks this is more convenient than grabbing my tarball and fiddling with quilt.
Also note that the master branch of the tree includes all topic master branches, which in turn include all topic subject branches. IOW the master branch contains my complete quilt tree of the moment (which might or might not actually compile, eat your pet or cook you dinner).
[tip:perf/urgent] perf: Fix loss of notification with multi-event
In reply to Stephane EranianCommit-ID: 10c6db110d0eb4466b59812c49088ab56218fc2e Gitweb: http://git.kernel.org/tip/10c6db110d0eb4466b59812c49088ab56218fc2e Author: Peter Zijlstra <[email protected]> AuthorDate: Sat, 26 Nov 2011 02:47:31 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 5 Dec 2011 09:33:03 +0100
perf: Fix loss of notification with multi-event
When you do:
$ perf record -e cycles,cycles,cycles noploop 10
You expect about 10,000 samples for each event, i.e., 10s at 1000samples/sec. However, this is not what's happening. You get much fewer samples, maybe 3700 samples/event:
$ perf report -D | tail -15
Aggregated stats:
TOTAL events: 10998
MMAP events: 66
COMM events: 2
SAMPLE events: 10930
cycles stats:
TOTAL events: 3644
SAMPLE events: 3644
cycles stats:
TOTAL events: 3642
SAMPLE events: 3642
cycles stats:
TOTAL events: 3644
SAMPLE events: 3644On a Intel Nehalem or even AMD64, there are 4 counters capable of measuring cycles, so there is plenty of space to measure those events without multiplexing (even with the NMI watchdog active). And even with multiplexing, we'd expect roughly the same number of samples per event.
The root of the problem was that when the event that caused the buffer to become full was not the first event passed on the cmdline, the user notification would get lost. The notification was sent to the file descriptor of the overflowed event but the perf tool was not polling on it. The perf tool aggregates all samples into a single buffer, i.e., the buffer of the first event. Consequently, it assumes notifications for any event will come via that descriptor.
The seemingly straight forward solution of moving the waitq into the ringbuffer object doesn't work because of life-time issues. One could perf_event_set_output() on a fd that you're also blocking on and cause the old rb object to be freed while its waitq would still be referenced by the blocked thread -> FAIL.
Therefore link all events to the ringbuffer and broadcast the wakeup from the ringbuffer object to all possible events that could be waited upon. This is rather ugly, and we're open to better solutions but it works for now.
--- include/linux/perf_event.h | 1 + kernel/events/core.c | 86 ++++++++++++++++++++++++++++++++++++++++++- kernel/events/internal.h | 3 + kernel/events/ring_buffer.c | 3 + 4 files changed, 91 insertions(+), 2 deletions(-)
include/linux/perf_event.h +1 -0
@@ -822,6 +822,7 @@ struct perf_event {int mmap_locked;struct user_struct *mmap_user;struct ring_buffer *rb; +struct list_head rb_entry;/* poll related */ wait_queue_head_t waitq;
kernel/events/core.c +84 -2
142 lines not on this page · raw message
kernel/events/internal.h +3 -0
@@ -22,6 +22,9 @@ struct ring_buffer {local_t lost;/* nr records lost */ long watermark;/* wakeup watermark */ +/* poll crap */ +spinlock_t event_lock; +struct list_head event_list;struct perf_event_mmap_page *user_page;void *data_pages[0 ];
kernel/events/ring_buffer.c +3 -0
@@ -209,6 +209,9 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) rb->writable =1 ;atomic_set (&rb->refcount,1 ); + +INIT_LIST_HEAD (&rb->event_list); +spin_lock_init (&rb->event_lock); }#ifndef CONFIG_PERF_USE_VMALLOC