perf: Add support for Intel Processor Trace
Subsystems (6)
Alexander Shishkin
reviewercover perf: Add support for Intel Processor Trace
Hi Peter and all,
Here's the second attempt at Intel PT support patchset, this time I only include the kernel part, since it requires more scrutiny. The whole patchset including the userspace currently can be found in my github repo [1]. Major changes since the previous version are:
* magic mmap() offset got replaced with a separate file descriptor, which is a 2nd ring buffer attached to the same event; this way, the first ring buffer (perf stream) receives trace buffer related events such as the one that signals trace data being lost (ITRACE_LOST), in addition to the normal sideband data,
* added a driver for BTS per Ingo's request, now BTS can be used via the same interface as Intel PT, thus illustrating the capabilities of "itrace" framework to those who are interested,
* massive patches got split into more digestible ones for the benefit of the reviewer,
* added support for multiple itrace pmus (since we have to accomodate both PT and BTS now),
* various small changes.
This patchset adds support for Intel Processor Trace (PT) extension [2] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel and userspace infrastructure. We provide an abstraction for it called "itrace" for "instruction trace" ([3]).
The single most notable thing is that while PT outputs trace data in a compressed binary format, it will still generate hundreds of megabytes of trace data per second per core. Decoding this binary stream takes 2-3 orders of magnitude the cpu time that it takes to generate it. These considerations make it impossible to carry out decoding in kernel space. Therefore, the trace data is exported to userspace as a zero-copy mapping that userspace can collect and store for later decoding. To that end, perf is extended to support an additional ring buffer per event, which will export the trace data. This ring buffer is mapped from a file descriptor, which is derived from the event's file descriptor. This ring buffer has its own user page with data_head and data_tail (in case the buffer is mapped writable) pointers used as read/write pointers in the buffer.
This way we get a normal perf data stream that provides sideband information that is required to decode the trace data, such as MMAPs, COMMs etc, plus the actual trace in a separate buffer.
If the trace buffer is mapped writable, the driver will stop tracing when it fills up (data_head approaches data_tail), till data is read, data_tail pointer is moved forward and an ioctl() is issued to re-enable tracing. If the trace buffer is mapped read only, the tracing will continue, overwriting older data, so that the buffer always contains the most recent data. Tracing can be stopped with an ioctl() and restarted once the data is collected.
Another use case is annotating samples of other perf events: if you set PERF_SAMPLE_ITRACE, attr.itrace_sample_size bytes of trace will be included in each event's sample.
Also, itrace data can be included in process core dumps, which can be enabled with a new rlimit -- RLIMIT_ITRACE.
[1] https://github.com/virtuoso/linux-perf/tree/intel_pt
[2] http://download-software.intel.com/sites/default/files/managed/50/1a/319433-018.pdf
[3] http://events.linuxfoundation.org/sites/events/files/slides/lcna13_kleen.pdf
Alexander Shishkin (11):
x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
perf: Abstract ring_buffer backing store operations
perf: Allow for multiple ring buffers per event
itrace: Infrastructure for instruction flow tracing units
itrace: Add functionality to include traces in perf event samples
itrace: Add functionality to include traces in process core dumps
x86: perf: intel_pt: Intel PT PMU driver
x86: perf: intel_pt: Add sampling functionality
x86: perf: intel_pt: Add core dump functionality
x86: perf: intel_bts: Add BTS PMU driver
x86: perf: intel_bts: Add core dump related functionality
arch/x86/include/asm/cpufeature.h | 1 + arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 129 +++ arch/x86/kernel/cpu/perf_event.c | 4 + arch/x86/kernel/cpu/perf_event.h | 6 + arch/x86/kernel/cpu/perf_event_intel.c | 16 +- arch/x86/kernel/cpu/perf_event_intel_bts.c | 500 ++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 3 +- arch/x86/kernel/cpu/perf_event_intel_pt.c | 1180 ++++++++++++++++++++++++++++ arch/x86/kernel/cpu/scattered.c | 1 + fs/binfmt_elf.c | 6 + fs/proc/base.c | 1 + include/asm-generic/resource.h | 1 + include/linux/itrace.h | 162 ++++ include/linux/perf_event.h | 34 +- include/uapi/asm-generic/resource.h | 3 +- include/uapi/linux/elf.h | 1 + include/uapi/linux/perf_event.h | 22 +- kernel/events/Makefile | 2 +- kernel/events/core.c | 341 +++++--- kernel/events/internal.h | 39 +- kernel/events/itrace.c | 705 +++++++++++++++++ kernel/events/ring_buffer.c | 178 +++-- kernel/exit.c | 3 + kernel/sys.c | 5 + 26 files changed, 3189 insertions(+), 173 deletions(-) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c create mode 100644 include/linux/itrace.h create mode 100644 kernel/events/itrace.c
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 1/11 x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
Intel Processor Trace is an architecture extension that allows for program flow tracing.
--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
@@ -186,6 +186,7 @@#define X86_FEATURE_DTHERM (7*32+ 7)/* Digital Thermal Sensor */ #define X86_FEATURE_HW_PSTATE (7*32+ 8)/* AMD HW-PState */ #define X86_FEATURE_PROC_FEEDBACK (7*32+ 9)/* AMD ProcFeedbackInterface */ +#define X86_FEATURE_INTEL_PT (7*32+10)/* Intel Processor Trace */ /* Virtualization flags: Linux defined, word 8 */ #define X86_FEATURE_TPR_SHADOW (8*32+ 0)/* Intel TPR Shadow */
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -36,6 +36,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c) { X86_FEATURE_ARAT, CR_EAX, 2, 0x00000006, 0 }, { X86_FEATURE_PLN, CR_EAX, 4, 0x00000006, 0 }, { X86_FEATURE_PTS, CR_EAX, 6, 0x00000006, 0 }, + { X86_FEATURE_INTEL_PT, CR_EBX,25, 0x00000007, 0 }, { X86_FEATURE_APERFMPERF, CR_ECX, 0, 0x00000006, 0 }, { X86_FEATURE_EPB, CR_ECX, 3, 0x00000006, 0 }, { X86_FEATURE_XSAVEOPT, CR_EAX, 0, 0x0000000d, 1 },
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 2/11 perf: Abstract ring_buffer backing store operations
In reply to Alexander ShishkinThis patch extends perf's ring_buffer code so that buffers with different backing can be allocated simultaneously with rb_alloc(). This allows the reuse of ring_buffer code for exporting hardware-written trace buffers (such as those of Intel PT) to userspace.
--- kernel/events/core.c | 4 +- kernel/events/internal.h | 32 +++++++- kernel/events/ring_buffer.c | 176 +++++++++++++++++++++++++++----------------- 3 files changed, 143 insertions(+), 69 deletions(-)
kernel/events/core.c +2 -2
@@ -4105,9 +4105,9 @@ again:if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; - rb =rb_alloc (nr_pages, + rb =rb_alloc (event, nr_pages, event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); + event->cpu, flags,NULL );if (!rb) { ret = -ENOMEM ;
kernel/events/internal.h +31 -1
@@ -6,6 +6,33 @@/* Buffer handling */ +struct ring_buffer ; + +struct ring_buffer_ops { +/* +* How much memory should be allocated for struct ring_buffer, taking into +* account data_pages[] array. +*/ +unsigned long (*get_size)(int ); +/* +* Allocate user_page for this buffer, can be NULL, in which case it is +* allocated by alloc_data_page(). +*/ +int (*alloc_user_page)(struct ring_buffer *,int ,int ); +/* +* Allocate data_pages for this buffer. +*/ +int (*alloc_data_page)(struct ring_buffer *,int ,int ,int ); +/* +* Free the buffer. +*/ +void (*free_buffer)(struct ring_buffer *); +/* +* Get a struct page for a given page index in the buffer. +*/ +struct page *(*mmap_to_page)(struct ring_buffer *,unsigned long ); +}; +#define RING_BUFFER_WRITABLE 0x01struct ring_buffer {
@@ -15,6 +42,8 @@ struct ring_buffer {struct work_struct work;int page_order;/* allocation order */ #endif +struct ring_buffer_ops *ops; +struct perf_event *event;int nr_pages;/* nr of data pages */ int overwrite;/* can overwrite itself */
@@ -41,7 +70,8 @@ struct ring_buffer {extern void rb_free (struct ring_buffer *rb);extern struct ring_buffer * -rb_alloc (int nr_pages,long watermark,int cpu,int flags); +rb_alloc (struct perf_event *event,int nr_pages,long watermark,int cpu, +int flags,struct ring_buffer_ops *rb_ops);extern void perf_event_wakeup (struct perf_event *event); extern void
kernel/events/ring_buffer.c +110 -66
262 lines not on this page · raw message
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 3/11 perf: Allow for multiple ring buffers per event
In reply to Alexander ShishkinCurrently, a perf event can have one ring buffer associated with it, that is used for perf record stream. However, some pmus, such as instruction tracing units, will generate binary streams of their own, for which it is convenient to reuse the ring buffer code to export such streams to the userspace. So, this patch extends the perf code to support more than one ring buffer per event. All the existing functionality will default to using the main ring buffer for everything and only the main buffer is exported to userspace at this point.
--- include/linux/perf_event.h | 11 ++- kernel/events/core.c | 186 ++++++++++++++++++++++++-------------------- kernel/events/internal.h | 7 ++ kernel/events/ring_buffer.c | 2 +- 4 files changed, 118 insertions(+), 88 deletions(-)
include/linux/perf_event.h +8 -3
@@ -289,6 +289,11 @@ struct swevent_hlist {struct perf_cgroup ;struct ring_buffer ; +enum perf_event_rb { +PERF_RB_MAIN =0 , +PERF_NR_RB , +}; +/** * struct perf_event - performance event kernel representation: */
@@ -398,10 +403,10 @@ struct perf_event {/* mmap bits */ struct mutex mmap_mutex; -atomic_t mmap_count; +atomic_t mmap_count[PERF_NR_RB ]; -struct ring_buffer *rb; -struct list_head rb_entry; +struct ring_buffer *rb[PERF_NR_RB ]; +struct list_head rb_entry[PERF_NR_RB ];/* poll related */ wait_queue_head_t waitq;
kernel/events/core.c +102 -84
389 lines not on this page · raw message
kernel/events/internal.h +7 -0
@@ -63,6 +63,7 @@ struct ring_buffer {atomic_t mmap_count;unsigned long mmap_locked;struct user_struct *mmap_user; +void *priv;struct perf_event_mmap_page *user_page;void *data_pages[0 ];
@@ -73,6 +74,12 @@ extern struct ring_buffer * rb_alloc(struct perf_event *event, int nr_pages, long watermark, int cpu, int flags, struct ring_buffer_ops *rb_ops);extern void perf_event_wakeup (struct perf_event *event); +extern struct ring_buffer *ring_buffer_get (struct perf_event *event,int rbx); +extern void ring_buffer_put (struct ring_buffer *rb); +extern void ring_buffer_attach (struct perf_event *event, +struct ring_buffer *rb); +extern void ring_buffer_detach (struct perf_event *event, +struct ring_buffer *rb); extern void perf_event_header__init_id(struct perf_event_header *header,
kernel/events/ring_buffer.c +1 -1
@@ -120,7 +120,7 @@ int perf_output_begin(struct perf_output_handle *handle,if (event->parent) event = event->parent; - rb =rcu_dereference (event->rb); + rb =rcu_dereference (event->rb[PERF_RB_MAIN ]);if (unlikely (!rb)) goto out;
Signature · 1 line
1.8.5.2
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 6 lines · Alexander Shishkin
Currently, a perf event can have one ring buffer associated with it, that is used for perf record stream. However, some pmus, such as instruction tracing units, will generate binary streams of their own, for which it is convenient to reuse the ring buffer code to export such streams to the userspace. So, this patch extends the perf code to support more than one ring buffer per event.
No-no-no-no... like I said last time around, 'splice' whatever results you get into a perf buffer and make it look like perf events.
I'm not convinced it needs to be a PERF_RECORD_SAMPLE; but some PERF_RECORD_* type for sure. Also it must allow interleaving with other events.
I understand your use-case wants sideband events in another buffer due to generation speed and not particularly caring about itrace data that's lost but wanting a coherent side-band stream.
And that's fine, use two events for this; but that doesn't mean it shouldn't be possible to mix them.
So for example:
/*
* struct {
* struct perf_event_header header;
* u64 extended_size;
* u64 data_offset;
* u64 data_size;
* struct sample_id sample_id;
* }PERF_RECORD_DATA
Now; suppose your itrace data is 1mb, allocate an event of 1mb+sizeof(PERF_RECORD_DATA)+PAGE_SIZE-1.
Then write the PERF_RECORD_DATA structure into the normal ring-buffer location; set data_offset to point to the first page boundary, data_size to 1mb.
Then frob things such that perf_mmap_to_page() for the next 1mb of pages points to your buffer pages and wipe the page-table entries.
Then we need to somehow shoot down TLBs, and that's tricky, because up to this point we're in interrupt context (ideally the whole itrace nonsense gets dropped out of the PMI through an irq_work ASAP, no point in doing it in NMI context anyhow).
So for TLB shootdown we can do a number of vile-ish things; but I think the prettiest is relying (and thus mandating) that the consumer wait in poll()/select()/etc. And either adding something like poll_work() which gets ran on poll-wakeup on the right task, or doing something ugly with task-work.
The point being that the consumer only needs to flush the TLBs before trying to access the buffer and that its clearly not doing so when its poll()-ing.
Another vile option is shooting down page-table entries and TLBs for the entire buffer when writing into the control page to update the tail -- that has some other 'fun' issues, but should be possible as well.
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 2 lines
I'm not convinced it needs to be a
PERF_RECORD_SAMPLE; but some PERF_RECORD_* type for sure.
Adding a header shouldn't be a problem, it's merely wasting 4K.
Quoted 1 line
Also it must allow interleaving with other > events.
But can you describe a concrete use case where interleaving is better?
I'm not aware of any. Anything that could be usefully interleaved can just be in the side band stream, and if you want a unified uncompressed stream you just run perf inject. The standard tools don't care for it as they have to reorder everything anyways to deal with multi CPU reordering.
Your scheme is very complex and adds a lot of use restrictions over the current code, so there should be a good reason for it at least.
Especially the TLB hac^wproposal sounds horrible to me, compared to the straight forward zero copy ring buffer used today.
-Andi
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 8 lines · Andi Kleen
Quoted 2 lines
I'm not convinced it needs to be a
PERF_RECORD_SAMPLE; but some PERF_RECORD_* type for sure.Adding a header shouldn't be a problem, it's merely wasting 4K.
Quoted 1 line
Also it must allow interleaving with other > events.
But can you describe a concrete use case where interleaving is better?
I really don't want the multi-buffer nonsense proposed. An event gets _1_ buffer, that's it.
That also means that if someone redirect another event into this buffer, it needs to just work.
And because its a perf buffer, people expect it to look like one. So we've got 'wrap' it.
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 4 lines
I really don't want the multi-buffer nonsense proposed.
An event gets
_1_ buffer, that's it.
But we already have multi buffer. Just profile multiple CPUs Then you have one buffer per CPU that need to be combined.
This just has two buffers per CPU.
Quoted 2 lines
That also means that if someone redirect another event into this buffer, it needs to just work.
All the tools already handle multiple buffers (for multi CPUs). So they don't need it.
Quoted 2 lines
And because its a perf buffer, people expect it to look like one. So we've got 'wrap' it.
Flushing TLBs from NMIs or irq work or any interrupt context is just a non starter.
The start/stop hardware and create gigantic gaps was also pretty bad and would completely change the perf format too
It seem to me you're trying to solve a non-problem.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 9 lines · Andi Kleen
Quoted 4 lines
I really don't want the multi-buffer nonsense proposed.
An event gets
_1_ buffer, that's it.But we already have multi buffer. Just profile multiple CPUs Then you have one buffer per CPU that need to be combined.
This just has two buffers per CPU.
Well, an event still gets *one* *perf* buffer in our implementation, which is consistent with how things are done now, plus one trace buffer. We could also export the trace buffer as a device node or something, so that no software would expect to see perf headers in that buffer.
Quoted 9 lines
Quoted 2 lines
That also means that if someone redirect another event into this buffer, it needs to just work.
All the tools already handle multiple buffers (for multi CPUs). So they don't need it.
Quoted 2 lines
And because its a perf buffer, people expect it to look like one. So we've got 'wrap' it.
Quoted 7 lines
Flushing TLBs from NMIs or irq work or any interrupt context is just a non starter.
The start/stop hardware and create gigantic gaps was also pretty bad and would completely change the perf format too
It seem to me you're trying to solve a non-problem.
Look at it this way: if the only way for it to be part of perf is by wrapping trace data in perf headers, perf framework is simply not suitable for instruction tracing. Therefore, it seems logical to have a standalone driver or, considering ETM/PTM and others, a standalone framework for exporting instruction trace data to userspace as plain mmap interface with out the overhead of overwriting userspace ptes, flushing tlbs, having inconsistent mappings in threads and that would still work for hardware that doesn't support sg lists.
What do you think?
Regards,
--
Alex
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
Then write the
PERF_RECORD_DATAstructure into the normal ring-buffer location; setdata_offsetto point to the first page boundary,data_sizeto 1mb.Then frob things such that
perf_mmap_to_page()for the next 1mb of pages points to your buffer pages and wipe the page-table entries.
Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt? mm->page_table_lock is a plain non-irq spinlock.
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 11 lines · Dave Hansen
Quoted 6 lines · Peter Zijlstra
Then write the
PERF_RECORD_DATAstructure into the normal ring-buffer location; setdata_offsetto point to the first page boundary,data_sizeto 1mb.Then frob things such that
perf_mmap_to_page()for the next 1mb of pages points to your buffer pages and wipe the page-table entries.Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt?
mm->page_table_lockis a plain non-irq spinlock.
Yes, this does look more than just tricky even if we move the bulk of interrupt code to an irq_work. Peter, are you quite sure this is what we want to do just for exporting trace buffers to userspace?
Regards,
--
Alex
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 7 lines
Quoted 3 lines
Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt?
mm->page_table_lockis a plain non-irq spinlock.Yes, this does look more than just tricky even if we move the bulk of interrupt code to an
irq_work. Peter, are you quite sure this is what we want to do just for exporting trace buffers to userspace?
The other big problem is scalability. Even if it was somehow possible to make this scheme work the IPIs for flushing would kill performance on any multi threaded client. Given perf is not multi-threaded today, but it doesn't seem a good idea to design the interface assuming no client ever will be.
-Andi
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 8 lines · Andi Kleen
Quoted 7 lines
Quoted 3 lines
Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt?
mm->page_table_lockis a plain non-irq spinlock.Yes, this does look more than just tricky even if we move the bulk of interrupt code to an
irq_work. Peter, are you quite sure this is what we want to do just for exporting trace buffers to userspace?
Quoted 5 lines
The other big problem is scalability. Even if it was somehow possible to make this scheme work the IPIs for flushing would kill performance on any multi threaded client. Given perf is not multi-threaded today, but it doesn't seem a good idea to design the interface assuming no client ever will be.
Well any mmap()ed interface that wants to swap buffers will have this same problem.
You can restrict the TLB flushing to the threads that poll() on the relevant events. This just means other threads will see old/partial data, but that shouldn't be a problem as they shouldn't be looking in the first place.
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 8 lines
Quoted 5 lines
The other big problem is scalability. Even if it was somehow possible to make this scheme work the IPIs for flushing would kill performance on any multi threaded client. Given perf is not multi-threaded today, but it doesn't seem a good idea to design the interface assuming no client ever will be.
Well any mmap()ed interface that wants to swap buffers will have this same problem.
There's no need to swap buffers in a sane design. The perf ring buffer or the ftrace buffer don't need this. There's no need for a PT buffer to do so either.
Quoted 5 lines
You can restrict the TLB flushing to the threads that
poll()on the relevant events. This just means other threads will see old/partial data, but that shouldn't be a problem as they shouldn't be looking in the first place.
Then we get incoherent processes. You're not serious about that are you?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
In reply to Dave HansenQuoted 11 lines · Dave Hansen
Quoted 6 lines · Peter Zijlstra
Then write the
PERF_RECORD_DATAstructure into the normal ring-buffer location; setdata_offsetto point to the first page boundary,data_sizeto 1mb.Then frob things such that
perf_mmap_to_page()for the next 1mb of pages points to your buffer pages and wipe the page-table entries.Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt?
mm->page_table_lockis a plain non-irq spinlock.
One could modify existing page tables the same way we do the lockless lookup for GUP. But instead of doing the get_page() we do a pte modification.
But I suppose we can push all that to task context by having the polling task do it before it gets to userspace again.
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
In reply to Alexander ShishkinHow about something like this for the itrace thing?
You would mmap() the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a second mmap() with the .pgoff matching the aux_offset you gave and .length matching the aux_size you gave.
This way the mmap() content still looks like a single linear file (could be sparse if you leave a hole, although we could require the aux_offset to match the end of the data section).
And there is still the single event->rb, not more.
Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.
With this there should be no difficult page table tricks or anything.
The patch is way incomplete but should sketch enough of the idea..
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
--- include/uapi/linux/perf_event.h | 19 +++++++++++++++ kernel/events/core.c | 51 +++++++++++++++++++++++++++++++++++++---- kernel/events/internal.h | 6 +++++ kernel/events/ring_buffer.c | 8 +------ 4 files changed, 72 insertions(+), 12 deletions(-)
include/uapi/linux/perf_event.h +19 -0
@@ -491,6 +491,13 @@ struct perf_event_mmap_page {*/ __u64 data_head;/* head in the data section */ __u64 data_tail;/* user-space written tail */ +__u64 data_offset; +__u64 data_size; + +__u64 aux_head; +__u64 aux_tail; +__u64 aux_offset; +__u64 aux_size; };#define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)
@@ -705,6 +712,18 @@ enum perf_event_type {*/ PERF_RECORD_MMAP2 =10 , +/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* }; +*/ +PERF_RECORD_AUX =11 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +46 -5
75 lines not on this page · raw message
kernel/events/internal.h +1 -0
@@ -36,6 +36,7 @@ struct ring_buffer {struct user_struct *mmap_user;struct perf_event_mmap_page *user_page; +struct radix_tree_root page_tree;void *data_pages[0 ]; };
kernel/events/ring_buffer.c +1 -7
@@ -251,13 +251,7 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)struct page *perf_mmap_to_page (struct ring_buffer *rb,unsigned long pgoff) { -if (pgoff > rb->nr_pages) -return NULL ; - -if (pgoff ==0 ) -return virt_to_page (rb->user_page); - -return virt_to_page (rb->data_pages[pgoff -1 ]); +return radix_tree_lookup (&rb->page_tree, pgoff); }static void *perf_mmap_alloc_page (int cpu)
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 17 lines · Peter Zijlstra
How about something like this for the itrace thing?
You would
mmap()the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a secondmmap()with the .pgoff matching theaux_offsetyou gave and .length matching theaux_sizeyou gave.This way the
mmap()content still looks like a single linear file (could be sparse if you leave a hole, although we could require theaux_offsetto match the end of the data section).And there is still the single
event->rb, not more.Then, when data inside that aux data store changes they should inject an
PERF_RECORD_AUXto indicate this did happen, which ties it back into the normal event flow.
Quoted 7 lines
With this there should be no difficult page table tricks or anything.
The patch is way incomplete but should sketch enough of the idea..
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
This looks like a pretty good concept to me, to support the buffering quirks/constraints that itrace CPUs apparently have.
Thanks,
Ingo
Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
In reply to Peter ZijlstraQuoted 3 lines
Then, when data inside that aux data store changes they should inject an
PERF_RECORD_AUXto indicate this did happen, which ties it back into the normal event flow.
What happens when the aux buffer wraps? How would the client know if the data belongs to this _AUX entry or some later one?
May need some extra sequence numbers in the mmap header and the aux entry to handle this.
-Andi
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 6 lines · Andi Kleen
Quoted 3 lines
Then, when data inside that aux data store changes they should inject an
PERF_RECORD_AUXto indicate this did happen, which ties it back into the normal event flow.What happens when the aux buffer wraps? How would the client know if the data belongs to this
_AUXentry or some later one?
It belongs to the last one. Rewind them from 'now' until you hit collisions in AUX space, then you're done.
Quoted 2 lines
May need some extra sequence numbers in the mmap header and the aux entry to handle this.
You're thinking of overwrite mode, right? We should update the tail in that case, I've not thought about how to do that for the AUX buffer.
There have been some patches for the normal buffer, but they stalled;
https://lkml.org/lkml/2013/7/8/154
I'm all for merging that patch (or a fixed on, since it has fail in) if we can show the current !overwrite case doesn't regress.
Also, would anybody want different mode for the data and aux parts? In that case we do need to add some extra state to the control page to indicate such.
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Andi Kleen
Quoted 3 lines
Then, when data inside that aux data store changes they should inject an
PERF_RECORD_AUXto indicate this did happen, which ties it back into the normal event flow.What happens when the aux buffer wraps? How would the client know if the data belongs to this
_AUXentry or some later one?It belongs to the last one. Rewind them from 'now' until you hit collisions in AUX space, then you're done.
I guess the point here is that if we don't want to lose any data in aux space, we need to stop the perf_event when it fills up. Also there's a question if we need a separate wake up watermark for the AUX buffer or do we simply wake up the poller every time there's new data.
Quoted 5 lines
Quoted 2 lines
May need some extra sequence numbers in the mmap header and the aux entry to handle this.
You're thinking of overwrite mode, right? We should update the tail in that case, I've not thought about how to do that for the AUX buffer.
In the overwrite mode we don't have to write out AUX records at all before we stop the trace, we don't care how many times data in the AUX space wraps.
Quoted 10 lines
There have been some patches for the normal buffer, but they stalled;
https://lkml.org/lkml/2013/7/8/154
I'm all for merging that patch (or a fixed on, since it has fail in) if we can show the current !overwrite case doesn't regress.
Also, would anybody want different mode for the data and aux parts? In that case we do need to add some extra state to the control page to indicate such.
For the decoder to make sense of the trace, it needs all the data in the normal buffer (MMAPs, sched_switches), not just the latest bits, so it's a good idea to have it.
Regards,
--
Alex
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
In reply to Peter ZijlstraQuoted 1 line · Peter Zijlstra
How about something like this for the itrace thing?
It's much nicer than the page swizzling draft I was about to send you.
Quoted 4 lines
You would
mmap()the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a secondmmap()with the .pgoff matching theaux_offsetyou gave and .length matching theaux_sizeyou gave.
Why do we need aux_{offset,size} at all, then? Userspace should know how they mmap()ed it.
Quoted 5 lines
This way the
mmap()content still looks like a single linear file (could be sparse if you leave a hole, although we could require theaux_offsetto match the end of the data section).And there is still the single
event->rb, not more.
Fair enough.
Quoted 5 lines
Then, when data inside that aux data store changes they should inject an
PERF_RECORD_AUXto indicate this did happen, which ties it back into the normal event flow.With this there should be no difficult page table tricks or anything.
True.
Quoted 1 line
The patch is way incomplete but should sketch enough of the idea..
Can I take it over?
Quoted 2 lines
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
With PERF_RECORD_AUX carrying offset and size, we shouldn't need aux_{head,tail} either, don't you think?
Quoted 80 lines
--- include/uapi/linux/perf_event.h | 19 +++++++++++++++ kernel/events/core.c | 51 +++++++++++++++++++++++++++++++++++++---- kernel/events/internal.h | 6 +++++ kernel/events/ring_buffer.c | 8 +------ 4 files changed, 72 insertions(+), 12 deletions(-)include/uapi/linux/perf_event.h +19
-0@@ -491,6 +491,13 @@ struct perf_event_mmap_page {*/ __u64 data_head;/* head in the data section */ __u64 data_tail;/* user-space written tail */ +__u64 data_offset; +__u64 data_size; + +__u64 aux_head; +__u64 aux_tail; +__u64 aux_offset; +__u64 aux_size; };#define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)@@ -705,6 +712,18 @@ enum perf_event_type {*/ PERF_RECORD_MMAP2 =10 , +/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* }; +*/ +PERF_RECORD_AUX =11 , + PERF_RECORD_MAX, /* non-ABI */ };kernel/events/core.c +11
-2@@ -4016,7 +4016,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)static const struct vm_operations_struct perf_mmap_vmops = { .open = perf_mmap_open, - .close = perf_mmap_close, + .close = perf_mmap_close,/* non mergable */ .fault = perf_mmap_fault, .page_mkwrite = perf_mmap_fault, };@@ -4030,6 +4030,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)struct ring_buffer *rb;unsigned long vma_size;unsigned long nr_pages; +unsigned long pgoff;long user_extra, extra;int ret =0 , flags =0 ;@@ -4045,7 +4046,50 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)return -EINVAL ; vma_size = vma->vm_end - vma->vm_start; - nr_pages = (vma_size /PAGE_SIZE ) -1 ; + +if (vma->vm_pgoff ==0 ) { + nr_pages = (vma_size /PAGE_SIZE ) -1 ; + }else { +if (!event->rb) +return -EINVAL ; + + nr_pages = vma_size /PAGE_SIZE ; +
kernel/events/core.c static int perf_mmap(struct file *file, struct vm_area_struct *vma) · line 4067
Quoted 10 lines
kernel/events/core.c +10
-0+mutex_lock (&event->mmap_mutex); + ret = -EINVAL ; +if (!event->rb) + goto err_aux_unlock; + +if (!atomic_inc_not_zero (&event->rb->mmap_count)) + goto err_aux_unlock; + +if (userpg->aux_offset < userpg->data_offset + userpg->data_size) + goto err_aux_unlock;
The data_{offset,size} seem to be only set by userspace too, maybe we can do away with these altogether unless we want to allow for it to be a sparse file?
kernel/events/core.c static int perf_mmap(struct file *file, struct vm_area_struct *vma) · line 4069
Quoted 1 line
kernel/events/core.c +1
-0+ pgoff = userpg->aux_offset;
..and simply do a
pgoff = event->rb->nr_pages + 1;
?
Quoted 54 lines
kernel/events/core.c +23
-3+if (pgoff & ~PAGE_MASK ) + goto err_aux_unlock; + + pgoff >>=PAGE_SHIFT ; +if (pgoff != vma->vm_pgoff) + goto err_aux_unlock; + +/* XXX do we want to allow !power_of_2 sizes, for AUX? */ +if (nr_pages ==0 || !is_power_of_2 (nr_pages)) + goto err_aux_unlock; + +if (vma_size !=PAGE_SIZE * nr_pages) + goto err_aux_unlock; + +if (userpg->aux_size != vma_size) + goto err_aux_unlock; + + ret =rb_alloc_aux (event->rb, userpg->aux_offset >>PAGE_SHIFT , nr_pages); + +err_aux_unlock: +mutex_unlock (&event->mmap_mutex); +return ret; + }/* * If we have rb pages ensure they're a power-of-two number, so we @@ -4057,9 +4101,6 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (vma_size !=PAGE_SIZE * (1 + nr_pages))return -EINVAL ; -if (vma->vm_pgoff !=0 ) -return -EINVAL ; -WARN_ON_ONCE (event->ctx->parent_ctx); again:mutex_lock (&event->mmap_mutex);kernel/events/internal.h +1
-0@@ -36,6 +36,7 @@ struct ring_buffer {struct user_struct *mmap_user;struct perf_event_mmap_page *user_page; +struct radix_tree_root page_tree;void *data_pages[0 ]; };kernel/events/ring_buffer.c +0
-0@@ -251,13 +251,7 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) struct page *
kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 254
Quoted 10 lines
kernel/events/ring_buffer.c +1
-7perf_mmap_to_page(struct ring_buffer *rb, unsigned long pgoff) { -if (pgoff > rb->nr_pages) -return NULL ; - -if (pgoff ==0 ) -return virt_to_page (rb->user_page); - -return virt_to_page (rb->data_pages[pgoff -1 ]); +return radix_tree_lookup (&rb->page_tree, pgoff);
This can instead call into the underlying driver, which will likely maintain an array similar to data_pages[] anyway.
Regards,
--
Alex
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 32 lines · Alexander Shishkin
Quoted 1 line · Peter Zijlstra
How about something like this for the itrace thing?
It's much nicer than the page swizzling draft I was about to send you.
Quoted 4 lines
You would
mmap()the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a secondmmap()with the .pgoff matching theaux_offsetyou gave and .length matching theaux_sizeyou gave.Why do we need aux_{offset,size} at all, then? Userspace should know how they mmap()ed it.
Quoted 5 lines
This way the
mmap()content still looks like a single linear file (could be sparse if you leave a hole, although we could require theaux_offsetto match the end of the data section).And there is still the single
event->rb, not more.Fair enough.
Quoted 5 lines
Then, when data inside that aux data store changes they should inject an
PERF_RECORD_AUXto indicate this did happen, which ties it back into the normal event flow.With this there should be no difficult page table tricks or anything.
True.
Quoted 1 line
The patch is way incomplete but should sketch enough of the idea..
Quoted 7 lines
Can I take it over?
Quoted 2 lines
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
With
PERF_RECORD_AUXcarrying offset and size, we shouldn't need aux_{head,tail} either, don't you think?
I take this one back, as perf record doesn't actually parse records from the buffer, it would still need the pointers.
Regards,
--
Alex
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
In reply to Peter ZijlstraQuoted 2 lines · Peter Zijlstra
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 9 lines · Alexander Shishkin
Quoted 2 lines · Peter Zijlstra
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.
Yeah, this might make it easier. So then the rule is *_offset + *_{head,tail} is the actual file offset into the buffer.
Alexander Shishkin
reviewerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines · Peter Zijlstra
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
Quoted 7 lines
Quoted 4 lines
Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.
Yeah, this might make it easier. So then the rule is
*_offset+ *_{head,tail} is the actual file offset into the buffer.
You mean *_offset + (*_{head,tail} % *_size) ?
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event
Quoted 13 lines · Alexander Shishkin
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
Quoted 2 lines · Peter Zijlstra
So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.
Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.
Quoted 4 lines
Quoted 2 lines
Yeah, this might make it easier. So then the rule is
*_offset+ *_{head,tail} is the actual file offset into the buffer.You mean
*_offset+ (*_{head,tail} %*_size) ?
Indeed I did..
Alexander Shishkin
reviewerPATCH 4/11 itrace: Infrastructure for instruction flow tracing units
In reply to Alexander ShishkinInstruction tracing PMUs are capable of recording a log of instruction execution flow on a cpu core, which can be useful for profiling and crash analysis. This patch adds itrace infrastructure for perf events and the rest of the kernel to use.
Since such PMUs can produce copious amounts of trace data at the rate for hundreds of megabytes per second per core, it may be impractical to process it inside the kernel in real time, but instead export raw trace streams to userspace for subsequent analysis. Thus, itrace PMUs may export their trace buffers, which can be mmap()ed to userspace from a special file descriptor, which can be obtained from the perf_event_open() syscall by using a PERF_FLAG_FD_ITRACE flag and the original perf event descriptor.
This infrastructure should also be useful for ARM ETM/PTM and other program flow tracing units that can potentially generate a lot of trace data very fast.
--- include/linux/itrace.h | 89 +++++++++++++ include/linux/perf_event.h | 5 + include/uapi/linux/perf_event.h | 17 +++ kernel/events/Makefile | 2 +- kernel/events/core.c | 130 ++++++++++++++++--- kernel/events/itrace.c | 271 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 495 insertions(+), 19 deletions(-) create mode 100644 include/linux/itrace.h create mode 100644 kernel/events/itrace.c
include/linux/itrace.h new file +89 -0
@@ -0,0 +1,89 @@ +/* +* Instruction flow trace unit infrastructure +* Copyright (c) 2013, Intel Corporation. +* +* This program is free software; you can redistribute it and/or modify it +* under the terms and conditions of the GNU General Public License, +* version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +* +* You should have received a copy of the GNU General Public License along with +* this program; if not, write to the Free Software Foundation, Inc., +* 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. +* +*/ + +#ifndef _LINUX_ITRACE_H +#define _LINUX_ITRACE_H + +#include <linux/perf_event.h> +#include <linux/file.h> + +extern struct ring_buffer_ops itrace_rb_ops; + +static inline bool is_itrace_vma (struct vm_area_struct *vma) +{ +if (vma->vm_file) { +struct perf_event *event = vma->vm_file->private_data; +if (event->hw.itrace_file == vma->vm_file) +return true; + } + +return false; +} + +void *itrace_priv (struct perf_event *event); + +void *itrace_event_get_priv (struct perf_event *event); +void itrace_event_put (struct perf_event *event); + +struct itrace_pmu { +struct pmu pmu; +struct list_head entry; +/* +* Allocate/free ring_buffer backing store +*/ +void *(*alloc_buffer)(int cpu,int nr_pages,bool overwrite, +void **pages, +struct perf_event_mmap_page **user_page); +void (*free_buffer)(void *buffer); + +int (*event_init)(struct perf_event *event); + +char *name; +}; + +#define to_itrace_pmu (x) container_of((x), struct itrace_pmu, pmu) + +#ifdef CONFIG_PERF_EVENTS +extern int itrace_inherit_event (struct perf_event *event, +struct task_struct *task); +extern void itrace_lost_data (struct perf_event *event,u64 offset); +extern int itrace_pmu_register (struct itrace_pmu *ipmu); + +extern int itrace_event_installable (struct perf_event *event, +struct perf_event_context *ctx); + +extern void itrace_wake_up (struct perf_event *event); + +extern bool is_itrace_event (struct perf_event *event); + +#else +static int itrace_inherit_event (struct perf_event *event, +struct task_struct *task) {return 0 ; } +static inline void +itrace_lost_data (struct perf_event *event,u64 offset) {} +static inline int itrace_pmu_register (struct itrace_pmu *ipmu) {return -EINVAL ; } + +static inline int +itrace_event_installable (struct perf_event *event, +struct perf_event_context *ctx) {return -EINVAL ; } +static inline void itrace_wake_up (struct perf_event *event) {} +static inline bool is_itrace_event (struct perf_event *event) {return false; } +#endif + +#endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +5 -0
@@ -126,6 +126,10 @@ struct hw_perf_event {/* for tp_event->class */ struct list_head tp_list; }; +struct {/* itrace */ +struct file *itrace_file; +struct task_struct *itrace_target; + }; #ifdef CONFIG_HAVE_HW_BREAKPOINT struct { /* breakpoint */ /*
@@ -291,6 +295,7 @@ struct ring_buffer;enum perf_event_rb {PERF_RB_MAIN =0 , +PERF_RB_ITRACE ,PERF_NR_RB , };
include/uapi/linux/perf_event.h +17 -0
@@ -237,6 +237,10 @@ enum perf_event_read_format {#define PERF_ATTR_SIZE_VER2 80/* add: branch_sample_type */ #define PERF_ATTR_SIZE_VER3 96/* add: sample_regs_user */ /* add: sample_stack_user */ +#define PERF_ATTR_SIZE_VER4 120/* add: itrace_config */ +/* add: itrace_watermark */ +/* add: itrace_sample_type */ +/* add: itrace_sample_size */ /* * Hardware event_id to monitor via a performance monitoring event:
@@ -333,6 +337,11 @@ struct perf_event_attr {/* Align to u64. */ __u32 __reserved_2; + +__u64 itrace_config; +__u32 itrace_watermark;/* wakeup every n pages */ +__u32 itrace_sample_type;/* pmu->type of the itrace PMU */ +__u64 itrace_sample_size; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
@@ -705,6 +714,13 @@ enum perf_event_type {*/ PERF_RECORD_MMAP2 =10 , +/* +* struct { +* u64 offset; +* } +*/ +PERF_RECORD_ITRACE_LOST =11 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -726,6 +742,7 @@ enum perf_callchain_context {#define PERF_FLAG_FD_OUTPUT (1U << 1)#define PERF_FLAG_PID_CGROUP (1U << 2)/* pid=cgroup id, per-cpu mode only */ #define PERF_FLAG_FD_CLOEXEC (1U << 3)/* O_CLOEXEC */ +#define PERF_FLAG_FD_ITRACE (1U << 4)/* request itrace fd */ union perf_mem_data_src {__u64 val;
kernel/events/Makefile +1 -1
@@ -2,7 +2,7 @@ ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_core.o = -pg endif -obj-y := core.o ring_buffer.o callchain.o +obj-y := core.o ring_buffer.o callchain.o itrace.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o obj-$(CONFIG_UPROBES) += uprobes.o
kernel/events/core.c +112 -18
263 lines not on this page · raw message
kernel/events/itrace.c new file +271 -0
271 lines not on this page · raw message
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 5/11 itrace: Add functionality to include traces in perf event samples
In reply to Alexander ShishkinTrace data from itrace PMUs can be used to annotate other perf events by including it in sample records when PERF_SAMPLE_ITRACE flag is set. In this case, a PT kernel counter is created for each such event and trace data is retrieved from it and stored in the perf data stream.
--- include/linux/itrace.h | 37 +++++++++ include/linux/perf_event.h | 15 ++++ include/uapi/linux/perf_event.h | 5 +- kernel/events/core.c | 35 +++++++++ kernel/events/itrace.c | 169 ++++++++++++++++++++++++++++++++++++++-- 5 files changed, 252 insertions(+), 9 deletions(-)
include/linux/itrace.h +37 -0
@@ -54,12 +54,27 @@ struct itrace_pmu {int (*event_init)(struct perf_event *event); +/* +* Calculate the size of a sample to be written out +*/ +unsigned long (*sample_trace)(struct perf_event *event, +struct perf_sample_data *data); + +/* +* Write out a trace sample to the given output handle +*/ +void (*sample_output)(struct perf_event *event, +struct perf_output_handle *handle, +struct perf_sample_data *data);char *name; };#define to_itrace_pmu (x) container_of((x), struct itrace_pmu, pmu)#ifdef CONFIG_PERF_EVENTS + +extern int itrace_kernel_event (struct perf_event *event, +struct task_struct *task);extern int itrace_inherit_event (struct perf_event *event,struct task_struct *task);extern void itrace_lost_data (struct perf_event *event,u64 offset);
@@ -72,7 +87,17 @@ extern void itrace_wake_up(struct perf_event *event);extern bool is_itrace_event (struct perf_event *event); +extern int itrace_sampler_init (struct perf_event *event, +struct task_struct *task); +extern void itrace_sampler_fini (struct perf_event *event); +extern unsigned long itrace_sampler_trace (struct perf_event *event, +struct perf_sample_data *data); +extern void itrace_sampler_output (struct perf_event *event, +struct perf_output_handle *handle, +struct perf_sample_data *data);#else +static int itrace_kernel_event (struct perf_event *event, +struct task_struct *task) {return 0 ; }static int itrace_inherit_event (struct perf_event *event,struct task_struct *task) {return 0 ; } static inline void
@@ -84,6 +109,18 @@ itrace_event_installable(struct perf_event *event, struct perf_event_context *ctx) { return -EINVAL; }static inline void itrace_wake_up (struct perf_event *event) {}static inline bool is_itrace_event (struct perf_event *event) {return false; } + +static inline int itrace_sampler_init (struct perf_event *event, +struct task_struct *task) {} +static inline void +itrace_sampler_fini (struct perf_event *event) {} +static inline unsigned long +itrace_sampler_trace (struct perf_event *event, +struct perf_sample_data *data) {return 0 ; } +static inline void +itrace_sampler_output (struct perf_event *event, +struct perf_output_handle *handle, +struct perf_sample_data *data) {}#endif #endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +15 -0
@@ -83,6 +83,12 @@ struct perf_regs_user {struct pt_regs *regs; }; +struct perf_trace_record { +u64 size; +unsigned long from; +unsigned long to; +}; +struct task_struct ;/*
@@ -97,6 +103,11 @@ struct hw_perf_event_extra {struct event_constraint ; +enum perf_itrace_counter_type { +PERF_ITRACE_USER =BIT (1 ), +PERF_ITRACE_SAMPLING =BIT (2 ), +}; +/** * struct hw_perf_event - performance event hardware details: */
@@ -129,6 +140,7 @@ struct hw_perf_event {struct {/* itrace */ struct file *itrace_file;struct task_struct *itrace_target; +unsigned int counter_type; }; #ifdef CONFIG_HAVE_HW_BREAKPOINT struct { /* breakpoint */
@@ -434,6 +446,7 @@ struct perf_event {perf_overflow_handler_t overflow_handler;void *overflow_handler_context; +struct perf_event *trace_event;#ifdef CONFIG_EVENT_TRACING struct ftrace_event_call *tp_event;struct event_filter *filter;
@@ -591,6 +604,7 @@ struct perf_sample_data {union perf_mem_data_src data_src;struct perf_callchain_entry *callchain;struct perf_raw_record *raw; +struct perf_trace_record trace;struct perf_branch_stack *br_stack;struct perf_regs_user regs_user;u64 stack_user_size;
@@ -611,6 +625,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data, data->period = period; data->regs_user.abi =PERF_SAMPLE_REGS_ABI_NONE ; data->regs_user.regs =NULL ; + data->trace.from = data->trace.to = data->trace.size =0 ; data->stack_user_size =0 ; data->weight =0 ; data->data_src.val =0 ;
include/uapi/linux/perf_event.h +4 -1
@@ -137,8 +137,9 @@ enum perf_event_sample_format {PERF_SAMPLE_DATA_SRC =1U <<15 ,PERF_SAMPLE_IDENTIFIER =1U <<16 ,PERF_SAMPLE_TRANSACTION =1U <<17 , +PERF_SAMPLE_ITRACE =1U <<18 , - PERF_SAMPLE_MAX = 1U << 18, /* non-ABI */ + PERF_SAMPLE_MAX = 1U << 19, /* non-ABI */ };/*
@@ -689,6 +690,8 @@ enum perf_event_type {* { u64 weight; } && PERF_SAMPLE_WEIGHT * { u64 data_src; } && PERF_SAMPLE_DATA_SRC * { u64 transaction; } && PERF_SAMPLE_TRANSACTION +* { u64 size; +* char data[size]; } && PERF_SAMPLE_ITRACE * }; */ PERF_RECORD_SAMPLE = 9,
kernel/events/core.c +35 -0
@@ -1576,6 +1576,9 @@ void perf_event_disable(struct perf_event *event)struct perf_event_context *ctx = event->ctx;struct task_struct *task = ctx->task; +if (event->trace_event) +perf_event_disable (event->trace_event); +if (!task) {/* * Disable the event on the cpu that it's on
@@ -2070,6 +2073,8 @@ void perf_event_enable(struct perf_event *event)struct perf_event_context *ctx = event->ctx;struct task_struct *task = ctx->task; +if (event->trace_event) +perf_event_enable (event->trace_event);if (!task) {/* * Enable the event on the cpu that it's on
@@ -3209,6 +3214,8 @@ static void unaccount_event(struct perf_event *event)static_key_slow_dec_deferred (&perf_sched_events);if (has_branch_stack (event))static_key_slow_dec_deferred (&perf_sched_events); +if ((event->attr.sample_type &PERF_SAMPLE_ITRACE ) && event->trace_event) +itrace_sampler_fini (event);unaccount_event_cpu (event, event->cpu); }
@@ -4664,6 +4671,13 @@ void perf_output_sample(struct perf_output_handle *handle,if (sample_type &PERF_SAMPLE_TRANSACTION )perf_output_put (handle, data->txn); +if (sample_type &PERF_SAMPLE_ITRACE ) { +perf_output_put (handle, data->trace.size); + +if (data->trace.size) +itrace_sampler_output (event, handle, data); + } +if (!event->attr.watermark) {int wakeup_events = event->attr.wakeup_events;
@@ -4771,6 +4785,14 @@ void perf_prepare_sample(struct perf_event_header *header, data->stack_user_size = stack_size; header->size += size; } + +if (sample_type &PERF_SAMPLE_ITRACE ) { +u64 size =sizeof (u64); + + size +=itrace_sampler_trace (event, data); + + header->size += size; + } } static void perf_event_output(struct perf_event *event,
@@ -6795,6 +6817,15 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,if (err) goto err_pmu; } + +if (event->attr.sample_type &PERF_SAMPLE_ITRACE ) { + err =itrace_sampler_init (event, task); +if (err) { +/* XXX: either clean up callchain buffers too +or forbid them to go together */ + goto err_pmu; + } + } }return event;
@@ -7369,6 +7400,10 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,account_event (event); + err =itrace_kernel_event (event, task); +if (err) + goto err_free; + ctx =find_get_context (event->pmu, task, cpu);if (IS_ERR (ctx)) { err =PTR_ERR (ctx);
kernel/events/itrace.c +161 -8
201 lines not on this page · raw message
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 6/11 itrace: Add functionality to include traces in process core dumps
In reply to Alexander ShishkinPer thread trace data that is provided by itrace PMUs can be included in process core dumps, which is controlled via a new rlimit parameter RLIMIT_ITRACE. This is done by a per-thread kernel counter that is created when this RLIMIT_ITRACE is set.
The value of RLIMIT_ITRACE indicates the size of the per-thread elf note in a core dump and the buffer size used to collect corresponding trace.
--- fs/binfmt_elf.c | 6 + fs/proc/base.c | 1 + include/asm-generic/resource.h | 1 + include/linux/itrace.h | 36 +++++ include/linux/perf_event.h | 3 + include/uapi/asm-generic/resource.h | 3 +- include/uapi/linux/elf.h | 1 + kernel/events/itrace.c | 289 +++++++++++++++++++++++++++++++++++- kernel/exit.c | 3 + kernel/sys.c | 5 + 10 files changed, 343 insertions(+), 5 deletions(-)
fs/binfmt_elf.c +6 -0
@@ -34,6 +34,7 @@#include <linux/utsname.h> #include <linux/coredump.h> #include <linux/sched.h> +#include <linux/itrace.h> #include <asm/uaccess.h> #include <asm/param.h> #include <asm/page.h>
@@ -1576,6 +1577,8 @@ static int fill_thread_core_info(struct elf_thread_core_info *t, } } + *total +=itrace_elf_note_size (t->task); +return 1 ; }
@@ -1608,6 +1611,7 @@ static int fill_note_info(struct elfhdr *elf, int phdrs,for (i =0 ; i < view->n; ++i)if (view->regsets[i].core_note_type !=0 ) ++info->thread_notes; + info->thread_notes++;/* ITRACE */ /* * Sanity check. We rely on regset 0 being in NT_PRSTATUS,
@@ -1710,6 +1714,8 @@ static int write_note_info(struct elf_note_info *info, !writenote(&t->notes[i], cprm))return 0 ; +itrace_elf_note_write (cprm, t->task); + first =0 ; t = t->next; }while (t);
fs/proc/base.c +1 -0
@@ -471,6 +471,7 @@ static const struct limit_names lnames[RLIM_NLIMITS] = { [RLIMIT_NICE] = {"Max nice priority", NULL}, [RLIMIT_RTPRIO] = {"Max realtime priority", NULL}, [RLIMIT_RTTIME] = {"Max realtime timeout", "us"}, + [RLIMIT_ITRACE] = {"Max ITRACE buffer size", "bytes"}, };/* Display limits for a process */
include/asm-generic/resource.h +1 -0
@@ -25,6 +25,7 @@ [RLIMIT_NICE] = { 0, 0 }, \ [RLIMIT_RTPRIO] = { 0, 0 }, \ [RLIMIT_RTTIME] = { RLIM_INFINITY, RLIM_INFINITY }, \ + [RLIMIT_ITRACE] = { 0, RLIM_INFINITY }, \ } #endif
include/linux/itrace.h +36 -0
@@ -22,6 +22,7 @@#include <linux/perf_event.h> #include <linux/file.h> +#include <linux/coredump.h> extern struct ring_buffer_ops itrace_rb_ops;
@@ -66,6 +67,19 @@ struct itrace_pmu {void (*sample_output)(struct perf_event *event,struct perf_output_handle *handle,struct perf_sample_data *data); + +/* +* Get the PMU-specific part of a core dump note +*/ +size_t (*core_size)(struct perf_event *event); + +/* +* Write out the core dump note +*/ +void (*core_output)(struct coredump_params *cprm, +struct perf_event *event, +unsigned long len); +u64 coredump_config;char *name; };
@@ -95,6 +109,17 @@ extern unsigned long itrace_sampler_trace(struct perf_event *event,extern void itrace_sampler_output (struct perf_event *event,struct perf_output_handle *handle,struct perf_sample_data *data); + +extern int update_itrace_rlimit (struct task_struct *,unsigned long ); +extern void exit_itrace (struct task_struct *); + +struct itrace_note { +u64 itrace_config; +}; + +extern size_t itrace_elf_note_size (struct task_struct *tsk); +extern void itrace_elf_note_write (struct coredump_params *cprm, +struct task_struct *task);#else static int itrace_kernel_event (struct perf_event *event,struct task_struct *task) {return 0 ; }
@@ -121,6 +146,17 @@ static inline void itrace_sampler_output(struct perf_event *event, struct perf_output_handle *handle, struct perf_sample_data *data) {} + +static inline int +update_itrace_rlimit(struct task_struct *, unsigned long) { return -EINVAL; } +static inline void exit_itrace (struct task_struct *) {} + +static inline size_t +itrace_elf_note_size (struct task_struct *tsk) {return 0 ; } +static inline void +itrace_elf_note_write (struct coredump_params *cprm, +struct task_struct *task) {} +#endif #endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +3 -0
@@ -106,6 +106,9 @@ struct event_constraint;enum perf_itrace_counter_type {PERF_ITRACE_USER =BIT (1 ),PERF_ITRACE_SAMPLING =BIT (2 ), +PERF_ITRACE_COREDUMP =BIT (3 ), +PERF_ITRACE_KERNEL = (PERF_ITRACE_SAMPLING |PERF_ITRACE_COREDUMP ), +PERF_ITRACE_ANY = (PERF_ITRACE_KERNEL |PERF_ITRACE_USER ), };/**
include/uapi/asm-generic/resource.h +2 -1
@@ -45,7 +45,8 @@0-39 for nice level 19 .. -20 */ #define RLIMIT_RTPRIO 14/* maximum realtime priority */ #define RLIMIT_RTTIME 15/* timeout for RT tasks in us */ -#define RLIM_NLIMITS 16 +#define RLIMIT_ITRACE 16/* max itrace size */ +#define RLIM_NLIMITS 17/* * SuS says limits have to be unsigned.
include/uapi/linux/elf.h +1 -0
@@ -369,6 +369,7 @@ typedef struct elf64_shdr {#define NT_PRPSINFO 3#define NT_TASKSTRUCT 4#define NT_AUXV 6 +#define NT_ITRACE 7/* * Note to userspace developers: size of NT_SIGINFO note may increase * in the future to accomodate more fields, don't assume it is fixed!
kernel/events/itrace.c +285 -4
345 lines not on this page · raw message
kernel/exit.c +3 -0
@@ -48,6 +48,7 @@#include <linux/fs_struct.h> #include <linux/init_task.h> #include <linux/perf_event.h> +#include <linux/itrace.h> #include <trace/events/sched.h> #include <linux/hw_breakpoint.h> #include <linux/oom.h>
@@ -788,6 +789,8 @@ void do_exit(long code)check_stack_usage ();exit_thread (); +exit_itrace (tsk); +/* * Flush inherited counters to the parent - before the parent * gets woken up by child-exit notifications.
kernel/sys.c +5 -0
@@ -14,6 +14,7 @@#include <linux/fs.h> #include <linux/kmod.h> #include <linux/perf_event.h> +#include <linux/itrace.h> #include <linux/resource.h> #include <linux/kernel.h> #include <linux/workqueue.h>
@@ -1402,6 +1403,10 @@ int do_prlimit(struct task_struct *tsk, unsigned int resource,update_rlimit_cpu (tsk, new_rlim->rlim_cur); out:read_unlock (&tasklist_lock); + +if (!retval && new_rlim && resource ==RLIMIT_ITRACE ) + retval =update_itrace_rlimit (tsk, new_rlim->rlim_cur); +return retval; }
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 7/11 x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinAdd support for Intel Processor Trace (PT) to kernel's perf/itrace events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel. Thus, buffers containing this binary stream are zero-copy mapped to the debug tools in userspace for subsequent decoding and analysis.
--- arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 127 ++++ arch/x86/kernel/cpu/perf_event.c | 4 + arch/x86/kernel/cpu/perf_event_intel.c | 10 + arch/x86/kernel/cpu/perf_event_intel_pt.c | 991 ++++++++++++++++++++++++++++++ 6 files changed, 1151 insertions(+) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
arch/x86/include/uapi/asm/msr-index.h +18 -0
@@ -74,6 +74,24 @@#define MSR_IA32_PERF_CAPABILITIES 0x00000345#define MSR_PEBS_LD_LAT_THRESHOLD 0x000003f6 +#define MSR_IA32_RTIT_CTL 0x00000570 +#define RTIT_CTL_TRACEEN BIT(0) +#define RTIT_CTL_OS BIT(2) +#define RTIT_CTL_USR BIT(3) +#define RTIT_CTL_CR3EN BIT(7) +#define RTIT_CTL_TOPA BIT(8) +#define RTIT_CTL_TSC_EN BIT(10) +#define RTIT_CTL_DISRETC BIT(11) +#define RTIT_CTL_BRANCH_EN BIT(13) +#define MSR_IA32_RTIT_STATUS 0x00000571 +#define RTIT_STATUS_CONTEXTEN BIT(1) +#define RTIT_STATUS_TRIGGEREN BIT(2) +#define RTIT_STATUS_ERROR BIT(4) +#define RTIT_STATUS_STOPPED BIT(5) +#define MSR_IA32_RTIT_CR3_MATCH 0x00000572 +#define MSR_IA32_RTIT_OUTPUT_BASE 0x00000560 +#define MSR_IA32_RTIT_OUTPUT_MASK 0x00000561 +#define MSR_MTRRfix64K_00000 0x00000250#define MSR_MTRRfix16K_80000 0x00000258#define MSR_MTRRfix16K_A0000 0x00000259
arch/x86/kernel/cpu/Makefile +1 -0
@@ -37,6 +37,7 @@ endif obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_p6.o perf_event_knc.o perf_event_p4.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore.o perf_event_intel_rapl.o +obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o endif
arch/x86/kernel/cpu/intel_pt.h new file +127 -0
127 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event.c +4 -0
@@ -385,6 +385,10 @@ static inline int precise_br_compat(struct perf_event *event)int x86_pmu_hw_config (struct perf_event *event) { +if (event->attr.sample_type &PERF_SAMPLE_ITRACE && + event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ) +return -EINVAL ; +if (event->attr.precise_ip) {int precise =0 ;
arch/x86/kernel/cpu/perf_event_intel.c +10 -0
@@ -1312,6 +1312,8 @@ int intel_pmu_save_and_restart(struct perf_event *event)return x86_perf_event_set_period (event); } +void intel_pt_interrupt (void ); +static void intel_pmu_reset (void ) {struct debug_store *ds =__this_cpu_read (cpu_hw_events.ds);
@@ -1393,6 +1395,14 @@ again: }/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } + +/* * Checkpointed counters can lead to 'spurious' PMIs because the * rollback caused by the PMI will have cleared the overflow status * bit. Therefore always force probe these counters.
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +991 -0
991 lines not on this page · raw message
Signature · 1 line
1.8.5.2
Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver
arch/x86/kernel/cpu/perf_event_intel.c int intel_pmu_save_and_restart(struct perf_event *event) · line 1315
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel.c +1
-0@@ -1312,6 +1312,8 @@ int intel_pmu_save_and_restart(struct perf_event *event)return x86_perf_event_set_period (event); } +void intel_pt_interrupt (void );
Should be in $(pwd)/perf_event.h
Quoted 12 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +6
-0@@ -0,0 +1,991 @@ +/* +* Intel(R) Processor Trace PMU driver for perf +* Copyright (c) 2013-2014, Intel Corporation. +* +* This program is free software; you can redistribute it and/or modify it +* under the terms and conditions of the GNU General Public License,
arch/x86/kernel/cpu/perf_event_intel_pt.c line 16
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+* version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +* +* You should have received a copy of the GNU General Public License along with +* this program; if not, write to the Free Software Foundation, Inc., +* 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
Remove the address, and add a pointer to the specification
Similar in the other files.
arch/x86/kernel/cpu/perf_event_intel_pt.c line 54
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+/* +* Capabilities of Intel PT hardware, such as number of address bits or +* supported output schemes, are cached and exported to userspace as "caps" +* attribute group of pt pmu device +* (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store +* relevant bits together with intel_pt traces. +* +* Currently, for debugging purposes, these attributes are also writable; this +* should be removed in the final version.
Already remove that code?
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +4
-0+{ +u64 reg; + + reg =RTIT_CTL_TOPA |RTIT_CTL_BRANCH_EN ;
arch/x86/kernel/cpu/perf_event_intel_pt.c line 224
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+ +if (!event->attr.exclude_kernel) + reg |=RTIT_CTL_OS ; +if (!event->attr.exclude_user) + reg |=RTIT_CTL_USR ; + + reg |= (event->attr.itrace_config &PT_CONFIG_MASK ); + +if (wrmsr_safe (MSR_IA32_RTIT_CTL , reg,0 ) <0 ) { +pr_warn ("Failed to enable PT on cpu %d\n" , event->cpu);
Should rate limit this warning
arch/x86/kernel/cpu/perf_event_intel_pt.c line 235
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+return -EINVAL ; + } +return 0 ; +} + +static void pt_config_start (bool start) +{ +u64 ctl; + +rdmsrl (MSR_IA32_RTIT_CTL , ctl);
Should bail out here if someone else already started (e.g. hardware debugger) The read needs to be moved to before we overwrite other MSRs
Quoted 12 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+if (start) + ctl |=RTIT_CTL_TRACEEN ; +else + ctl &= ~RTIT_CTL_TRACEEN ; +wrmsrl (MSR_IA32_RTIT_CTL , ctl); + +/** +* pt_handle_status - take care of possible status conditions +* @event: currently active PT event +*/
arch/x86/kernel/cpu/perf_event_intel_pt.c
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+static void pt_handle_status (struct perf_event *event) +{ +struct pt_buffer *buf =itrace_priv (event); +int advance =0 ; +u64 status; + +rdmsrl (MSR_IA32_RTIT_STATUS , status); + +if (status &RTIT_STATUS_ERROR ) { +pr_err ("ToPA ERROR encountered, trying to recover\n" );
Add perf: prefix here (or better redefine pr_fmt at the beginning) Should be rate limited
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+static struct pt_buffer *pt_buffer_alloc (int cpu,size_t size, +unsigned long watermark, +bool snapshot,gfp_t gfp, +void **pages) +{ +struct pt_buffer *buf; +int node; +int ret; +
arch/x86/kernel/cpu/perf_event_intel_pt.c line 678
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+if (!size || watermark <<PAGE_SHIFT > size) +return NULL ; + +if (cpu ==-1 ) + cpu =raw_smp_processor_id (); + node =cpu_to_node (cpu); + + buf =kzalloc (sizeof (struct pt_buffer ), gfp); +if (!buf) +return NULL ;
Should be kzalloc_node()
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Peter Zijlstra
maintainerRe: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 42 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +41
-0+static int topa_insert_pages (struct pt_buffer *buf,gfp_t gfp, +enum topa_sz sz) +{ +struct topa *topa = buf->last; +int node =cpu_to_node (buf->cpu); +int order =get_order (sizes (sz)); +struct page *p; +unsigned long pn; + + p =alloc_pages_node (node, gfp |GFP_USER | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY, order); +if (!p) +return -ENOMEM ; + +return 0 ; +} + +static int pt_buffer_init_topa (struct pt_buffer *buf,size_t size,gfp_t gfp) +{ +struct topa *topa; +int err, region_size; + + topa =topa_alloc (buf->cpu, gfp); +if (!topa) +return -ENOMEM ; + +topa_insert_table (buf, topa); + + region_size =pt_get_topa_region_size (buf->snapshot, size); +if (region_size <0 ) { +pt_buffer_fini_topa (buf); +return region_size; + } + +while (region_size &&get_order (sizes (region_size)) >MAX_ORDER ) + region_size--; + +/* fixup watermark in case of higher order allocations */ +if (buf->watermark < (sizes (region_size) >>PAGE_SHIFT )) + buf->watermark =sizes (region_size) >>PAGE_SHIFT ; + +while (buf->size < size) {
arch/x86/kernel/cpu/perf_event_intel_pt.c line 479
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+ err =topa_insert_pages (buf, gfp, region_size); +if (err) { +if (region_size) { + region_size--; +continue ; + } +pt_buffer_fini_topa (buf); +return -ENOMEM ; + } + }
So MAX_ORDER is 11, which gives us (11+12) 23bit or 8M max allocations, right?
Given this TOPA stuff is completely fucked in the first release, that's about all we can get it, right?
Now given you said 100s MB/s data rate for this itrace stuff, we're at ~0.1s traces. And that's in the very best case where we can actually get 8M.
Is that a workable amount?
Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver
Quoted 10 lines
So
MAX_ORDERis 11, which gives us (11+12) 23bit or 8M max allocations, right?Given this TOPA stuff is completely fucked in the first release, that's about all we can get it, right?
Now given you said 100s MB/s data rate for this itrace stuff, we're at ~0.1s traces. And that's in the very best case where we can actually get 8M.
Is that a workable amount
The PMI handler switches to a new buffer to allow longer traces in memory. It just costs one PMI.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Peter Zijlstra
maintainerRe: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 10 lines · Alexander Shishkin
Add support for Intel Processor Trace (PT) to kernel's perf/itrace events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel. Thus, buffers containing this binary stream are zero-copy mapped to the debug tools in userspace for subsequent decoding and analysis.
Signed-off-by Alexander Shishkin <[email protected]>
Quoted 10 lines
--- arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 127 ++++ arch/x86/kernel/cpu/perf_event.c | 4 + arch/x86/kernel/cpu/perf_event_intel.c | 10 + arch/x86/kernel/cpu/perf_event_intel_pt.c | 991 ++++++++++++++++++++++++++++++ 6 files changed, 1151 insertions(+) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
Andi said that when itrace is enabled the LBR is wrecked; this patch seems to fail to deal with that.
Alexander Shishkin
reviewerRe: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver
Quoted 22 lines · Peter Zijlstra
Quoted 10 lines · Alexander Shishkin
Add support for Intel Processor Trace (PT) to kernel's perf/itrace events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel. Thus, buffers containing this binary stream are zero-copy mapped to the debug tools in userspace for subsequent decoding and analysis.
Signed-off-by Alexander Shishkin <[email protected]>Quoted 10 lines
--- arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 127 ++++ arch/x86/kernel/cpu/perf_event.c | 4 + arch/x86/kernel/cpu/perf_event_intel.c | 10 + arch/x86/kernel/cpu/perf_event_intel_pt.c | 991 ++++++++++++++++++++++++++++++ 6 files changed, 1151 insertions(+) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
Quoted 2 lines
Andi said that when itrace is enabled the LBR is wrecked; this patch seems to fail to deal with that.
True, there needs to be a _safe() msr access before any configuration is done, I have a fix for that, but let's first deal with the buffer management.
Regards,
--
Alex
Alexander Shishkin
reviewerPATCH 8/11 x86: perf: intel_pt: Add sampling functionality
In reply to Alexander ShishkinIntel Processor Trace (PT) data can be used in perf event samples to annotate other perf events. This patch implements itrace sampling related hooks that configure and output sampling data to perf stream. Users will need to include PERF_SAMPLE_ITRACE flag in the attr.sample_type mask and specify PT's PMU type in attr.itrace_sample_type field.
--- arch/x86/kernel/cpu/perf_event_intel_pt.c | 115 ++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_pt.c +115 -0
@@ -946,17 +946,130 @@ static void pt_event_read(struct perf_event *event) { } +typedef unsigned int (*pt_copyfn )(void *data,const void *src, +unsigned int len); + +/** +* pt_buffer_output - copy part of pt buffer to perf stream +* @buf: buffer to copy from +* @from: initial offset +* @to: final offset +* @copyfn: function that copies data out (like perf_output_copy()) +* @data: data to be passed on to the copy function (like perf_output_handle) +*/ +static int pt_buffer_output (struct pt_buffer *buf,unsigned long from, +unsigned long to,pt_copyfn copyfn,void *data) +{ +unsigned long tocopy; +unsigned int len =0 , remainder; +void *page; + +do { + tocopy =PAGE_SIZE -offset_in_page (from); +if (to > from) + tocopy =min (tocopy, to - from); +if (!tocopy) +break ; + + page =pt_buffer_get_page (buf, from >>PAGE_SHIFT ); +if (WARN_ONCE (!page,"no data page for %lx offset\n" , from)) +break ; + + page +=offset_in_page (from); + + remainder =copyfn (data, page, tocopy); +if (remainder) +return -EFAULT ; + + len += tocopy; + from += tocopy; +if (from == buf->size) + from =0 ; + }while (to != from); +return len; +} +static int pt_event_init (struct perf_event *event) {if (event->attr.type != pt_pmu.itrace.pmu.type)return -ENOENT ; +/* can't be both */ +if (event->attr.sample_type &PERF_SAMPLE_ITRACE ) +return -ENOENT ; +if (!pt_event_valid (event))return -EINVAL ;return 0 ; } +static unsigned long pt_trace_sampler_trace (struct perf_event *event, +struct perf_sample_data *data) +{ +struct pt_buffer *buf; + +pt_event_stop (event,0 ); + + buf =itrace_event_get_priv (event); +if (!buf) { + data->trace.size =0 ; + goto out; + } + +pt_read_offset (buf); +pt_update_head (buf); + + data->trace.to =local64_read (&buf->head); + +if (data->trace.to < event->attr.itrace_sample_size) + data->trace.from = buf->size + data->trace.to - + event->attr.itrace_sample_size; +else + data->trace.from = data->trace.to - + event->attr.itrace_sample_size; + data->trace.size =ALIGN (event->attr.itrace_sample_size,sizeof (u64)); + +itrace_event_put (event); + +out: +if (!data->trace.size) +pt_event_start (event,0 ); + +return data->trace.size; +} + +static void pt_trace_sampler_output (struct perf_event *event, +struct perf_output_handle *handle, +struct perf_sample_data *data) +{ +unsigned long padding; +struct pt_buffer *buf; +int ret; + + buf =itrace_event_get_priv (event); +if (!buf) +return ; + + ret =pt_buffer_output (buf, data->trace.from, data->trace.to, + (pt_copyfn )perf_output_copy, handle); +itrace_event_put (event); +if (ret <0 ) { +pr_warn ("%s: failed to copy trace data\n" , __func__); + goto out; + } + + padding = data->trace.size - ret; +if (padding) { +u64 u =0 ; + +perf_output_copy (handle, &u, padding); + } + +out: +pt_event_start (event,0 ); +} +static __init int pt_init (void ) {int ret, cpu;
@@ -982,6 +1095,8 @@ static __init int pt_init(void) pt_pmu.itrace.pmu.read = pt_event_read; pt_pmu.itrace.alloc_buffer = pt_buffer_itrace_alloc; pt_pmu.itrace.free_buffer = pt_buffer_itrace_free; + pt_pmu.itrace.sample_trace = pt_trace_sampler_trace; + pt_pmu.itrace.sample_output = pt_trace_sampler_output; pt_pmu.itrace.name ="intel_pt" ; ret =itrace_pmu_register (&pt_pmu.itrace);
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 9/11 x86: perf: intel_pt: Add core dump functionality
In reply to Alexander ShishkinIntel Processor Trace (PT) data can be used in process core dumps. This is done by implementing itrace code dump related hooks that configure and output trace data to a core file. The driver will also include the list of PT capabilities in itrace core dump notes so that the decoder can make assumptions about the binary stream.
--- arch/x86/kernel/cpu/intel_pt.h | 2 + arch/x86/kernel/cpu/perf_event_intel_pt.c | 74 +++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+)
arch/x86/kernel/cpu/intel_pt.h +2 -0
@@ -84,6 +84,8 @@ enum pt_capabilities {struct pt_pmu {struct itrace_pmu itrace;u32 caps[4 *PT_CPUID_LEAVES ]; +char *capstr; +unsigned int caplen; };/**
arch/x86/kernel/cpu/perf_event_intel_pt.c +74 -0
110 lines not on this page · raw message
Signature · 1 line
1.8.5.2
Re: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality
arch/x86/kernel/cpu/perf_event_intel_pt.c static ssize_t pt_cap_store(struct device *cdev, · line 146
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c +1
-0char *buf)@@ -114,6 +143,7 @@ static ssize_t pt_cap_store(struct device *cdev,return -EINVAL ;pt_cap_set (cap, new); +pt_cap_string ();
Don't we need some lock here? Otherwise it may leak memory with racing writes and become inconsistent.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Alexander Shishkin
reviewerRe: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality
Quoted 9 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_pt.c static ssize_t pt_cap_store(struct device *cdev, · line 146
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c +1
-0char *buf)@@ -114,6 +143,7 @@ static ssize_t pt_cap_store(struct device *cdev,return -EINVAL ;pt_cap_set (cap, new); +pt_cap_string ();Don't we need some lock here? Otherwise it may leak memory with racing writes and become inconsistent.
Good point.
Regards,
--
Alex
Re: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality
In reply to Alexander Shishkinarch/x86/kernel/cpu/perf_event_intel_pt.c out: · line 1131
Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c +13
-0+ +static void pt_trace_core_output (struct coredump_params *cprm, +struct perf_event *event, +unsigned long len) +{ +struct pt_buffer *buf; +u64 from, to; +int ret; + + buf =itrace_priv (event); + +if (!dump_emit (cprm, pt_pmu.capstr, pt_pmu.caplen)) +return ;
It would be nicer if this was a separate note, instead of just being concatenated with the rest of the data.
Would make simpler parsing and be cleaner.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Alexander Shishkin
reviewerRe: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality
Quoted 18 lines · Andi Kleen
arch/x86/kernel/cpu/perf_event_intel_pt.c out: · line 1131
Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c +13
-0+ +static void pt_trace_core_output (struct coredump_params *cprm, +struct perf_event *event, +unsigned long len) +{ +struct pt_buffer *buf; +u64 from, to; +int ret; + + buf =itrace_priv (event); + +if (!dump_emit (cprm, pt_pmu.capstr, pt_pmu.caplen)) +return ;It would be nicer if this was a separate note, instead of just being concatenated with the rest of the data.
Quoted 1 line
Would make simpler parsing and be cleaner.
So long as we won't have to include traces from two different pmus in the same core file, then matching these sections may provide another challenge. Doesn't seem like a sensible scenario, though.
Regards,
--
Alex
Alexander Shishkin
reviewerPATCH 10/11 x86: perf: intel_bts: Add BTS PMU driver
In reply to Alexander ShishkinAdd support for Branch Trace Store (BTS) via kernel perf/itrace event infrastructure. The difference with the existing implementation of BTS support is that this one is a separate PMU that exports events' trace buffers to userspace the same way as Intel PT PMU does. The immediate benefit is that the buffer size can be much bigger, resulting in fewer interrupts and no kernel side copying is involved. Also, of the kernel code is possible. Additionally, it is now possible to include BTS traces into process core dumps.
The old way of collecting BTS traces still works.
--- arch/x86/kernel/cpu/Makefile | 2 +- arch/x86/kernel/cpu/perf_event.h | 6 + arch/x86/kernel/cpu/perf_event_intel.c | 6 +- arch/x86/kernel/cpu/perf_event_intel_bts.c | 478 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 3 +- 5 files changed, 492 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
arch/x86/kernel/cpu/Makefile +1 -1
@@ -37,7 +37,7 @@ endif obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_p6.o perf_event_knc.o perf_event_p4.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore.o perf_event_intel_rapl.o -obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o +obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o perf_event_intel_bts.o endif
arch/x86/kernel/cpu/perf_event.h +6 -0
@@ -702,6 +702,12 @@ void intel_pmu_lbr_init_snb(void);int intel_pmu_setup_lbr_filter (struct perf_event *event); +int intel_bts_interrupt (void ); + +void intel_bts_enable_local (void ); + +void intel_bts_disable_local (void ); +int p4_pmu_init (void );int p6_pmu_init (void );
arch/x86/kernel/cpu/perf_event_intel.c +5 -1
@@ -1052,6 +1052,8 @@ static void intel_pmu_disable_all(void)if (test_bit (INTEL_PMC_IDX_FIXED_BTS , cpuc->active_mask))intel_pmu_disable_bts (); +else +intel_bts_disable_local ();intel_pmu_pebs_disable_all ();intel_pmu_lbr_disable_all ();
@@ -1074,7 +1076,8 @@ static void intel_pmu_enable_all(int added)return ;intel_pmu_enable_bts (event->hw.config); - } + }else +intel_bts_enable_local (); }/*
@@ -1362,6 +1365,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)apic_write (APIC_LVTPC ,APIC_DM_NMI );intel_pmu_disable_all (); handled =intel_pmu_drain_bts_buffer (); + handled +=intel_bts_interrupt (); status =intel_pmu_get_status ();if (!status) {intel_pmu_enable_all (0 );
arch/x86/kernel/cpu/perf_event_intel_bts.c new file +478 -0
478 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event_intel_ds.c +2 -1
@@ -444,7 +444,8 @@ void intel_pmu_enable_bts(u64 config) debugctlmsr |=DEBUGCTLMSR_TR ; debugctlmsr |=DEBUGCTLMSR_BTS ; - debugctlmsr |=DEBUGCTLMSR_BTINT ; +if (config &ARCH_PERFMON_EVENTSEL_INT ) + debugctlmsr |=DEBUGCTLMSR_BTINT ;if (!(config &ARCH_PERFMON_EVENTSEL_OS )) debugctlmsr |=DEBUGCTLMSR_BTS_OFF_OS ;
Signature · 1 line
1.8.5.2
Alexander Shishkin
reviewerPATCH 11/11 x86: perf: intel_bts: Add core dump related functionality
In reply to Alexander ShishkinBTS data can be used in process core dumps. This patch implements itrace core dump related hooks that will configure and output BTS traces into a core file.
--- arch/x86/kernel/cpu/perf_event_intel_bts.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_bts.c +22 -0
@@ -436,6 +436,26 @@ static void bts_event_read(struct perf_event *event) { } +static size_t bts_trace_core_size (struct perf_event *event) +{ +return 0 ; +} + +static void bts_trace_core_output (struct coredump_params *cprm, +struct perf_event *event, +unsigned long len) +{ +struct bts_buffer *buf =itrace_event_get_priv (event); +u64 head =local64_read (&buf->head); + +if (head < len) { +dump_emit (cprm, buf->buf + head, buf->real_size - head); +dump_emit (cprm, buf->buf, len - buf->real_size + head); + }else +dump_emit (cprm, buf->buf + head - len, len); +itrace_event_put (event); +} +static __init int bts_init (void ) {int ret, cpu;
@@ -459,6 +479,8 @@ static __init int bts_init(void) bts_pmu.pmu.read = bts_event_read; bts_pmu.alloc_buffer = bts_buffer_itrace_alloc; bts_pmu.free_buffer = bts_buffer_itrace_free; + bts_pmu.core_size = bts_trace_core_size; + bts_pmu.core_output = bts_trace_core_output; bts_pmu.name ="intel_bts" ; ret =itrace_pmu_register (&bts_pmu);
Signature · 1 line
1.8.5.2
Re: [PATCH v1 11/11] x86: perf: intel_bts: Add core dump related functionality
Quoted 3 lines · Alexander Shishkin
BTS data can be used in process core dumps. This patch implements itrace core dump related hooks that will configure and output BTS traces into a core file.
Don't we need a different note number here?
Otherwise how should the debugger know if it's BTS or PT.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Alexander Shishkin
reviewerRe: [PATCH v1 11/11] x86: perf: intel_bts: Add core dump related functionality
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Alexander Shishkin
BTS data can be used in process core dumps. This patch implements itrace core dump related hooks that will configure and output BTS traces into a core file.
Don't we need a different note number here?
Otherwise how should the debugger know if it's BTS or PT.
The pmu name is part of the note as well as its itrace_config.
Regards,
--
Alex