Skip to contentSkip to conversation

perf: Add support for Intel Processor Trace

PATCH In review Alexander Shishkin 6 participants 45 messages Feb 6, 2014 – May 8, 2014

Subsystems (6)
lore
Share & tools
lore

Alexander Shishkin

author reviewer
#0 lore raw

cover perf: Add support for Intel Processor Trace

Hi Peter and all,

Here's the second attempt at Intel PT support patchset, this time I only include the kernel part, since it requires more scrutiny. The whole patchset including the userspace currently can be found in my github repo [1]. Major changes since the previous version are:

* magic mmap() offset got replaced with a separate file descriptor, which is a 2nd ring buffer attached to the same event; this way, the first ring buffer (perf stream) receives trace buffer related events such as the one that signals trace data being lost (ITRACE_LOST), in addition to the normal sideband data,
* added a driver for BTS per Ingo's request, now BTS can be used via the same interface as Intel PT, thus illustrating the capabilities of "itrace" framework to those who are interested,
* massive patches got split into more digestible ones for the benefit of the reviewer,
* added support for multiple itrace pmus (since we have to accomodate both PT and BTS now),
* various small changes.

This patchset adds support for Intel Processor Trace (PT) extension [2] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel and userspace infrastructure. We provide an abstraction for it called "itrace" for "instruction trace" ([3]).

The single most notable thing is that while PT outputs trace data in a compressed binary format, it will still generate hundreds of megabytes of trace data per second per core. Decoding this binary stream takes 2-3 orders of magnitude the cpu time that it takes to generate it. These considerations make it impossible to carry out decoding in kernel space. Therefore, the trace data is exported to userspace as a zero-copy mapping that userspace can collect and store for later decoding. To that end, perf is extended to support an additional ring buffer per event, which will export the trace data. This ring buffer is mapped from a file descriptor, which is derived from the event's file descriptor. This ring buffer has its own user page with data_head and data_tail (in case the buffer is mapped writable) pointers used as read/write pointers in the buffer.

This way we get a normal perf data stream that provides sideband information that is required to decode the trace data, such as MMAPs, COMMs etc, plus the actual trace in a separate buffer.

If the trace buffer is mapped writable, the driver will stop tracing when it fills up (data_head approaches data_tail), till data is read, data_tail pointer is moved forward and an ioctl() is issued to re-enable tracing. If the trace buffer is mapped read only, the tracing will continue, overwriting older data, so that the buffer always contains the most recent data. Tracing can be stopped with an ioctl() and restarted once the data is collected.

Another use case is annotating samples of other perf events: if you set PERF_SAMPLE_ITRACE, attr.itrace_sample_size bytes of trace will be included in each event's sample.

Also, itrace data can be included in process core dumps, which can be enabled with a new rlimit -- RLIMIT_ITRACE.

[1] https://github.com/virtuoso/linux-perf/tree/intel_pt
[2] http://download-software.intel.com/sites/default/files/managed/50/1a/319433-018.pdf
[3] http://events.linuxfoundation.org/sites/events/files/slides/lcna13_kleen.pdf

Alexander Shishkin (11):
x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
perf: Abstract ring_buffer backing store operations
perf: Allow for multiple ring buffers per event
itrace: Infrastructure for instruction flow tracing units
itrace: Add functionality to include traces in perf event samples
itrace: Add functionality to include traces in process core dumps
x86: perf: intel_pt: Intel PT PMU driver
x86: perf: intel_pt: Add sampling functionality
x86: perf: intel_pt: Add core dump functionality
x86: perf: intel_bts: Add BTS PMU driver
x86: perf: intel_bts: Add core dump related functionality

 arch/x86/include/asm/cpufeature.h          |    1 +
 arch/x86/include/uapi/asm/msr-index.h      |   18 +
 arch/x86/kernel/cpu/Makefile               |    1 +
 arch/x86/kernel/cpu/intel_pt.h             |  129 +++
 arch/x86/kernel/cpu/perf_event.c           |    4 +
 arch/x86/kernel/cpu/perf_event.h           |    6 +
 arch/x86/kernel/cpu/perf_event_intel.c     |   16 +-
 arch/x86/kernel/cpu/perf_event_intel_bts.c |  500 ++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |    3 +-
 arch/x86/kernel/cpu/perf_event_intel_pt.c  | 1180 ++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/scattered.c            |    1 +
 fs/binfmt_elf.c                            |    6 +
 fs/proc/base.c                             |    1 +
 include/asm-generic/resource.h             |    1 +
 include/linux/itrace.h                     |  162 ++++
 include/linux/perf_event.h                 |   34 +-
 include/uapi/asm-generic/resource.h        |    3 +-
 include/uapi/linux/elf.h                   |    1 +
 include/uapi/linux/perf_event.h            |   22 +-
 kernel/events/Makefile                     |    2 +-
 kernel/events/core.c                       |  341 +++++---
 kernel/events/internal.h                   |   39 +-
 kernel/events/itrace.c                     |  705 +++++++++++++++++
 kernel/events/ring_buffer.c                |  178 +++--
 kernel/exit.c                              |    3 +
 kernel/sys.c                               |    5 +
 26 files changed, 3189 insertions(+), 173 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
 create mode 100644 include/linux/itrace.h
 create mode 100644 kernel/events/itrace.c
Signature · 1 line
1.8.5.2
author reviewer
#1 lore raw

PATCH 1/11 x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection

Intel Processor Trace is an architecture extension that allows for program flow tracing.

---
 arch/x86/include/asm/cpufeature.h | 1 +
 arch/x86/kernel/cpu/scattered.c   | 1 +
 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index 89270b4..cb9864f 100644
@@ -186,6 +186,7 @@
 #define X86_FEATURE_DTHERM	(7*32+ 7) /* Digital Thermal Sensor */
 #define X86_FEATURE_HW_PSTATE	(7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK (7*32+ 9) /* AMD ProcFeedbackInterface */
+#define X86_FEATURE_INTEL_PT	(7*32+10) /* Intel Processor Trace */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW  (8*32+ 0) /* Intel TPR Shadow */
arch/x86/kernel/cpu/scattered.c +1 -0
index b6f794a..726e6a3 100644
@@ -36,6 +36,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c)
 		{ X86_FEATURE_ARAT,		CR_EAX, 2, 0x00000006, 0 },
 		{ X86_FEATURE_PLN,		CR_EAX, 4, 0x00000006, 0 },
 		{ X86_FEATURE_PTS,		CR_EAX, 6, 0x00000006, 0 },
+		{ X86_FEATURE_INTEL_PT,		CR_EBX,25, 0x00000007, 0 },
 		{ X86_FEATURE_APERFMPERF,	CR_ECX, 0, 0x00000006, 0 },
 		{ X86_FEATURE_EPB,		CR_ECX, 3, 0x00000006, 0 },
 		{ X86_FEATURE_XSAVEOPT,		CR_EAX,	0, 0x0000000d, 1 },
Signature · 1 line
1.8.5.2
author reviewer
#2 lore raw

PATCH 2/11 perf: Abstract ring_buffer backing store operations

In reply to Alexander Shishkin

This patch extends perf's ring_buffer code so that buffers with different backing can be allocated simultaneously with rb_alloc(). This allows the reuse of ring_buffer code for exporting hardware-written trace buffers (such as those of Intel PT) to userspace.

---
 kernel/events/core.c        |   4 +-
 kernel/events/internal.h    |  32 +++++++-
 kernel/events/ring_buffer.c | 176 +++++++++++++++++++++++++++-----------------
 3 files changed, 143 insertions(+), 69 deletions(-)
kernel/events/core.c +2 -2
index 56003c6..6899741 100644
@@ -4105,9 +4105,9 @@ again:
 	if (vma->vm_flags & VM_WRITE)
 		flags |= RING_BUFFER_WRITABLE;
 
-	rb = rb_alloc(nr_pages, 
+	rb = rb_alloc(event, nr_pages,
 		event->attr.watermark ? event->attr.wakeup_watermark : 0,
-		event->cpu, flags);
+		event->cpu, flags, NULL);
 
 	if (!rb) {
 		ret = -ENOMEM;
kernel/events/internal.h +31 -1
index 569b2187..6cb208f 100644
@@ -6,6 +6,33 @@
 
 /* Buffer handling */
 
+struct ring_buffer;
+
+struct ring_buffer_ops {
+	/*
+	 * How much memory should be allocated for struct ring_buffer, taking into
+	 * account data_pages[] array.
+	 */
+	unsigned long	(*get_size)(int);
+	/*
+	 * Allocate user_page for this buffer, can be NULL, in which case it is
+	 * allocated by alloc_data_page().
+	 */
+	int		(*alloc_user_page)(struct ring_buffer *, int, int);
+	/*
+	 * Allocate data_pages for this buffer.
+	 */
+	int		(*alloc_data_page)(struct ring_buffer *, int, int, int);
+	/*
+	 * Free the buffer.
+	 */
+	void		(*free_buffer)(struct ring_buffer *);
+	/*
+	 * Get a struct page for a given page index in the buffer.
+	 */
+	struct page	*(*mmap_to_page)(struct ring_buffer *, unsigned long);
+};
+
 #define RING_BUFFER_WRITABLE		0x01
 
 struct ring_buffer {
@@ -15,6 +42,8 @@ struct ring_buffer {
 	struct work_struct		work;
 	int				page_order;	/* allocation order  */
 #endif
+	struct ring_buffer_ops		*ops;
+	struct perf_event		*event;
 	int				nr_pages;	/* nr of data pages  */
 	int				overwrite;	/* can overwrite itself */
 
@@ -41,7 +70,8 @@ struct ring_buffer {
 
 extern void rb_free(struct ring_buffer *rb);
 extern struct ring_buffer *
-rb_alloc(int nr_pages, long watermark, int cpu, int flags);
+rb_alloc(struct perf_event *event, int nr_pages, long watermark, int cpu,
+	 int flags, struct ring_buffer_ops *rb_ops);
 extern void perf_event_wakeup(struct perf_event *event);
 
 extern void
kernel/events/ring_buffer.c +110 -66

262 lines not on this page · raw message

Signature · 1 line
1.8.5.2
author reviewer
#3 lore raw

PATCH 3/11 perf: Allow for multiple ring buffers per event

In reply to Alexander Shishkin

Currently, a perf event can have one ring buffer associated with it, that is used for perf record stream. However, some pmus, such as instruction tracing units, will generate binary streams of their own, for which it is convenient to reuse the ring buffer code to export such streams to the userspace. So, this patch extends the perf code to support more than one ring buffer per event. All the existing functionality will default to using the main ring buffer for everything and only the main buffer is exported to userspace at this point.

---
 include/linux/perf_event.h  |  11 ++-
 kernel/events/core.c        | 186 ++++++++++++++++++++++++--------------------
 kernel/events/internal.h    |   7 ++
 kernel/events/ring_buffer.c |   2 +-
 4 files changed, 118 insertions(+), 88 deletions(-)
include/linux/perf_event.h +8 -3
index e56b07f..93cefb6 100644
@@ -289,6 +289,11 @@ struct swevent_hlist {
 struct perf_cgroup;
 struct ring_buffer;
 
+enum perf_event_rb {
+	PERF_RB_MAIN = 0,
+	PERF_NR_RB,
+};
+
 /**
  * struct perf_event - performance event kernel representation:
  */
@@ -398,10 +403,10 @@ struct perf_event {
 
 	/* mmap bits */
 	struct mutex			mmap_mutex;
-	atomic_t			mmap_count;
+	atomic_t			mmap_count[PERF_NR_RB];
 
-	struct ring_buffer		*rb;
-	struct list_head		rb_entry;
+	struct ring_buffer		*rb[PERF_NR_RB];
+	struct list_head		rb_entry[PERF_NR_RB];
 
 	/* poll related */
 	wait_queue_head_t		waitq;
kernel/events/core.c +102 -84

389 lines not on this page · raw message

kernel/events/internal.h +7 -0
index 6cb208f..841f7c4 100644
@@ -63,6 +63,7 @@ struct ring_buffer {
 	atomic_t			mmap_count;
 	unsigned long			mmap_locked;
 	struct user_struct		*mmap_user;
+	void				*priv;
 
 	struct perf_event_mmap_page	*user_page;
 	void				*data_pages[0];
@@ -73,6 +74,12 @@ extern struct ring_buffer *
 rb_alloc(struct perf_event *event, int nr_pages, long watermark, int cpu,
 	 int flags, struct ring_buffer_ops *rb_ops);
 extern void perf_event_wakeup(struct perf_event *event);
+extern struct ring_buffer *ring_buffer_get(struct perf_event *event, int rbx);
+extern void ring_buffer_put(struct ring_buffer *rb);
+extern void ring_buffer_attach(struct perf_event *event,
+			       struct ring_buffer *rb);
+extern void ring_buffer_detach(struct perf_event *event,
+			       struct ring_buffer *rb);
 
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
kernel/events/ring_buffer.c +1 -1
index 161a676..232d7de 100644
@@ -120,7 +120,7 @@ int perf_output_begin(struct perf_output_handle *handle,
 	if (event->parent)
 		event = event->parent;
 
-	rb = rcu_dereference(event->rb);
+	rb = rcu_dereference(event->rb[PERF_RB_MAIN]);
 	if (unlikely(!rb))
 		goto out;
 
Signature · 1 line
1.8.5.2
maintainer
#4 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 6 lines · Alexander Shishkin

Currently, a perf event can have one ring buffer associated with it, that is used for perf record stream. However, some pmus, such as instruction tracing units, will generate binary streams of their own, for which it is convenient to reuse the ring buffer code to export such streams to the userspace. So, this patch extends the perf code to support more than one ring buffer per event.

No-no-no-no... like I said last time around, 'splice' whatever results you get into a perf buffer and make it look like perf events.

I'm not convinced it needs to be a PERF_RECORD_SAMPLE; but some PERF_RECORD_* type for sure. Also it must allow interleaving with other events.

I understand your use-case wants sideband events in another buffer due to generation speed and not particularly caring about itrace data that's lost but wanting a coherent side-band stream.

And that's fine, use two events for this; but that doesn't mean it shouldn't be possible to mix them.

So for example:

/*
* struct {
* struct perf_event_header header;
* u64 extended_size;
* u64 data_offset;
* u64 data_size;
* struct sample_id sample_id;
* }
PERF_RECORD_DATA

Now; suppose your itrace data is 1mb, allocate an event of 1mb+sizeof(PERF_RECORD_DATA)+PAGE_SIZE-1.

Then write the PERF_RECORD_DATA structure into the normal ring-buffer location; set data_offset to point to the first page boundary, data_size to 1mb.

Then frob things such that perf_mmap_to_page() for the next 1mb of pages points to your buffer pages and wipe the page-table entries.

Then we need to somehow shoot down TLBs, and that's tricky, because up to this point we're in interrupt context (ideally the whole itrace nonsense gets dropped out of the PMI through an irq_work ASAP, no point in doing it in NMI context anyhow).

So for TLB shootdown we can do a number of vile-ish things; but I think the prettiest is relying (and thus mandating) that the consumer wait in poll()/select()/etc. And either adding something like poll_work() which gets ran on poll-wakeup on the right task, or doing something ugly with task-work.

The point being that the consumer only needs to flush the TLBs before trying to access the buffer and that its clearly not doing so when its poll()-ing.

Another vile option is shooting down page-table entries and TLBs for the entire buffer when writing into the control page to update the tail -- that has some other 'fun' issues, but should be possible as well.

#5 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 2 lines

I'm not convinced it needs to be a PERF_RECORD_SAMPLE; but some PERF_RECORD_* type for sure.

Adding a header shouldn't be a problem, it's merely wasting 4K.

Quoted 1 line

Also it must allow interleaving with other > events.

But can you describe a concrete use case where interleaving is better?

I'm not aware of any. Anything that could be usefully interleaved can just be in the side band stream, and if you want a unified uncompressed stream you just run perf inject. The standard tools don't care for it as they have to reorder everything anyways to deal with multi CPU reordering.

Your scheme is very complex and adds a lot of use restrictions over the current code, so there should be a good reason for it at least.

Especially the TLB hac^wproposal sounds horrible to me, compared to the straight forward zero copy ring buffer used today.

-Andi

maintainer
#6 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 8 lines · Andi Kleen
Quoted 2 lines

I'm not convinced it needs to be a PERF_RECORD_SAMPLE; but some PERF_RECORD_* type for sure.

Adding a header shouldn't be a problem, it's merely wasting 4K.

Quoted 1 line

Also it must allow interleaving with other > events.

But can you describe a concrete use case where interleaving is better?

I really don't want the multi-buffer nonsense proposed. An event gets _1_ buffer, that's it.

That also means that if someone redirect another event into this buffer, it needs to just work.

And because its a perf buffer, people expect it to look like one. So we've got 'wrap' it.

#7 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 4 lines

I really don't want the multi-buffer nonsense proposed.

An event gets
_1_ buffer, that's it.

But we already have multi buffer. Just profile multiple CPUs Then you have one buffer per CPU that need to be combined.

This just has two buffers per CPU.

Quoted 2 lines

That also means that if someone redirect another event into this buffer, it needs to just work.

All the tools already handle multiple buffers (for multi CPUs). So they don't need it.

Quoted 2 lines

And because its a perf buffer, people expect it to look like one. So we've got 'wrap' it.

Flushing TLBs from NMIs or irq work or any interrupt context is just a non starter.

The start/stop hardware and create gigantic gaps was also pretty bad and would completely change the perf format too

It seem to me you're trying to solve a non-problem.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
author reviewer
#8 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 9 lines · Andi Kleen
Quoted 4 lines

I really don't want the multi-buffer nonsense proposed.

An event gets
_1_ buffer, that's it.

But we already have multi buffer. Just profile multiple CPUs Then you have one buffer per CPU that need to be combined.

This just has two buffers per CPU.

Well, an event still gets *one* *perf* buffer in our implementation, which is consistent with how things are done now, plus one trace buffer. We could also export the trace buffer as a device node or something, so that no software would expect to see perf headers in that buffer.

Quoted 9 lines
Quoted 2 lines

That also means that if someone redirect another event into this buffer, it needs to just work.

All the tools already handle multiple buffers (for multi CPUs). So they don't need it.

Quoted 2 lines

And because its a perf buffer, people expect it to look like one. So we've got 'wrap' it.

Quoted 7 lines

Flushing TLBs from NMIs or irq work or any interrupt context is just a non starter.

The start/stop hardware and create gigantic gaps was also pretty bad and would completely change the perf format too

It seem to me you're trying to solve a non-problem.

Look at it this way: if the only way for it to be part of perf is by wrapping trace data in perf headers, perf framework is simply not suitable for instruction tracing. Therefore, it seems logical to have a standalone driver or, considering ETM/PTM and others, a standalone framework for exporting instruction trace data to userspace as plain mmap interface with out the overhead of overwriting userspace ptes, flushing tlbs, having inconsistent mappings in threads and that would still work for hardware that doesn't support sg lists.

What do you think?

Regards,
--
Alex

#9 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Then write the PERF_RECORD_DATA structure into the normal ring-buffer location; set data_offset to point to the first page boundary, data_size to 1mb.

Then frob things such that perf_mmap_to_page() for the next 1mb of pages points to your buffer pages and wipe the page-table entries.

Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt? mm->page_table_lock is a plain non-irq spinlock.

author reviewer
#10 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 11 lines · Dave Hansen
Quoted 6 lines · Peter Zijlstra

Then write the PERF_RECORD_DATA structure into the normal ring-buffer location; set data_offset to point to the first page boundary, data_size to 1mb.

Then frob things such that perf_mmap_to_page() for the next 1mb of pages points to your buffer pages and wipe the page-table entries.

Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt? mm->page_table_lock is a plain non-irq spinlock.

Yes, this does look more than just tricky even if we move the bulk of interrupt code to an irq_work. Peter, are you quite sure this is what we want to do just for exporting trace buffers to userspace?

Regards,
--
Alex

#11 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 7 lines
Quoted 3 lines

Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt? mm->page_table_lock is a plain non-irq spinlock.

Yes, this does look more than just tricky even if we move the bulk of interrupt code to an irq_work. Peter, are you quite sure this is what we want to do just for exporting trace buffers to userspace?

The other big problem is scalability. Even if it was somehow possible to make this scheme work the IPIs for flushing would kill performance on any multi threaded client. Given perf is not multi-threaded today, but it doesn't seem a good idea to design the interface assuming no client ever will be.

-Andi

maintainer
#12 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 8 lines · Andi Kleen
Quoted 7 lines
Quoted 3 lines

Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt? mm->page_table_lock is a plain non-irq spinlock.

Yes, this does look more than just tricky even if we move the bulk of interrupt code to an irq_work. Peter, are you quite sure this is what we want to do just for exporting trace buffers to userspace?

Quoted 5 lines

The other big problem is scalability. Even if it was somehow possible to make this scheme work the IPIs for flushing would kill performance on any multi threaded client. Given perf is not multi-threaded today, but it doesn't seem a good idea to design the interface assuming no client ever will be.

Well any mmap()ed interface that wants to swap buffers will have this same problem.

You can restrict the TLB flushing to the threads that poll() on the relevant events. This just means other threads will see old/partial data, but that shouldn't be a problem as they shouldn't be looking in the first place.

#13 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 8 lines
Quoted 5 lines

The other big problem is scalability. Even if it was somehow possible to make this scheme work the IPIs for flushing would kill performance on any multi threaded client. Given perf is not multi-threaded today, but it doesn't seem a good idea to design the interface assuming no client ever will be.

Well any mmap()ed interface that wants to swap buffers will have this same problem.

There's no need to swap buffers in a sane design. The perf ring buffer or the ftrace buffer don't need this. There's no need for a PT buffer to do so either.

Quoted 5 lines

You can restrict the TLB flushing to the threads that poll() on the relevant events. This just means other threads will see old/partial data, but that shouldn't be a problem as they shouldn't be looking in the first place.

Then we get incoherent processes. You're not serious about that are you?

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
maintainer
#14 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

In reply to Dave Hansen
Quoted 11 lines · Dave Hansen
Quoted 6 lines · Peter Zijlstra

Then write the PERF_RECORD_DATA structure into the normal ring-buffer location; set data_offset to point to the first page boundary, data_size to 1mb.

Then frob things such that perf_mmap_to_page() for the next 1mb of pages points to your buffer pages and wipe the page-table entries.

Wouldn't we have to teach a ton of code how to be IRQ safe for this to work? Just step one: how do we go modifying page tables safely from an interrupt? mm->page_table_lock is a plain non-irq spinlock.

One could modify existing page tables the same way we do the lockless lookup for GUP. But instead of doing the get_page() we do a pte modification.

But I suppose we can push all that to task context by having the polling task do it before it gets to userspace again.

maintainer
#15 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

In reply to Alexander Shishkin

How about something like this for the itrace thing?

You would mmap() the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a second mmap() with the .pgoff matching the aux_offset you gave and .length matching the aux_size you gave.

This way the mmap() content still looks like a single linear file (could be sparse if you leave a hole, although we could require the aux_offset to match the end of the data section).

And there is still the single event->rb, not more.

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

With this there should be no difficult page table tricks or anything.

The patch is way incomplete but should sketch enough of the idea..

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

---
 include/uapi/linux/perf_event.h | 19 +++++++++++++++
 kernel/events/core.c            | 51 +++++++++++++++++++++++++++++++++++++----
 kernel/events/internal.h        |  6 +++++
 kernel/events/ring_buffer.c     |  8 +------
 4 files changed, 72 insertions(+), 12 deletions(-)
include/uapi/linux/perf_event.h +19 -0
index 853bc1ccb395..adef7c0f1e7c 100644
@@ -491,6 +491,13 @@ struct perf_event_mmap_page {
 	 */
 	__u64   data_head;		/* head in the data section */
 	__u64	data_tail;		/* user-space written tail */
+	__u64	data_offset;
+	__u64	data_size;
+
+	__u64	aux_head;
+	__u64	aux_tail;
+	__u64	aux_offset;
+	__u64	aux_size;
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
@@ -705,6 +712,18 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +46 -5

75 lines not on this page · raw message

kernel/events/internal.h +1 -0
index 569b218782ad..6258aaa36097 100644
@@ -36,6 +36,7 @@ struct ring_buffer {
 	struct user_struct		*mmap_user;
 
 	struct perf_event_mmap_page	*user_page;
+	struct radix_tree_root		page_tree;
 	void				*data_pages[0];
 };
 
kernel/events/ring_buffer.c +1 -7
index 146a5792b1d2..b82505325df0 100644
@@ -251,13 +251,7 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)
 struct page *
 perf_mmap_to_page(struct ring_buffer *rb, unsigned long pgoff)
 {
-	if (pgoff > rb->nr_pages)
-		return NULL;
-
-	if (pgoff == 0)
-		return virt_to_page(rb->user_page);
-
-	return virt_to_page(rb->data_pages[pgoff - 1]);
+	return radix_tree_lookup(&rb->page_tree, pgoff);
 }
 
 static void *perf_mmap_alloc_page(int cpu)
#16 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 17 lines · Peter Zijlstra

How about something like this for the itrace thing?

You would mmap() the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a second mmap() with the .pgoff matching the aux_offset you gave and .length matching the aux_size you gave.

This way the mmap() content still looks like a single linear file (could be sparse if you leave a hole, although we could require the aux_offset to match the end of the data section).

And there is still the single event->rb, not more.

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

Quoted 7 lines

With this there should be no difficult page table tricks or anything.

The patch is way incomplete but should sketch enough of the idea..

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

This looks like a pretty good concept to me, to support the buffering quirks/constraints that itrace CPUs apparently have.

Thanks,

Ingo

#17 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

In reply to Peter Zijlstra
Quoted 3 lines

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

What happens when the aux buffer wraps? How would the client know if the data belongs to this _AUX entry or some later one?

May need some extra sequence numbers in the mmap header and the aux entry to handle this.

-Andi

maintainer
#18 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 6 lines · Andi Kleen
Quoted 3 lines

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

What happens when the aux buffer wraps? How would the client know if the data belongs to this _AUX entry or some later one?

It belongs to the last one. Rewind them from 'now' until you hit collisions in AUX space, then you're done.

Quoted 2 lines

May need some extra sequence numbers in the mmap header and the aux entry to handle this.

You're thinking of overwrite mode, right? We should update the tail in that case, I've not thought about how to do that for the AUX buffer.

There have been some patches for the normal buffer, but they stalled;

https://lkml.org/lkml/2013/7/8/154

I'm all for merging that patch (or a fixed on, since it has fail in) if we can show the current !overwrite case doesn't regress.

Also, would anybody want different mode for the data and aux parts? In that case we do need to add some extra state to the control page to indicate such.

author reviewer
#19 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Andi Kleen
Quoted 3 lines

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

What happens when the aux buffer wraps? How would the client know if the data belongs to this _AUX entry or some later one?

It belongs to the last one. Rewind them from 'now' until you hit collisions in AUX space, then you're done.

I guess the point here is that if we don't want to lose any data in aux space, we need to stop the perf_event when it fills up. Also there's a question if we need a separate wake up watermark for the AUX buffer or do we simply wake up the poller every time there's new data.

Quoted 5 lines
Quoted 2 lines

May need some extra sequence numbers in the mmap header and the aux entry to handle this.

You're thinking of overwrite mode, right? We should update the tail in that case, I've not thought about how to do that for the AUX buffer.

In the overwrite mode we don't have to write out AUX records at all before we stop the trace, we don't care how many times data in the AUX space wraps.

Quoted 10 lines

There have been some patches for the normal buffer, but they stalled;

https://lkml.org/lkml/2013/7/8/154

I'm all for merging that patch (or a fixed on, since it has fail in) if we can show the current !overwrite case doesn't regress.

Also, would anybody want different mode for the data and aux parts? In that case we do need to add some extra state to the control page to indicate such.

For the decoder to make sense of the trace, it needs all the data in the normal buffer (MMAPs, sched_switches), not just the latest bits, so it's a good idea to have it.

Regards,
--
Alex

author reviewer
#20 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

In reply to Peter Zijlstra
Quoted 1 line · Peter Zijlstra

How about something like this for the itrace thing?

It's much nicer than the page swizzling draft I was about to send you.

Quoted 4 lines

You would mmap() the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a second mmap() with the .pgoff matching the aux_offset you gave and .length matching the aux_size you gave.

Why do we need aux_{offset,size} at all, then? Userspace should know how they mmap()ed it.

Quoted 5 lines

This way the mmap() content still looks like a single linear file (could be sparse if you leave a hole, although we could require the aux_offset to match the end of the data section).

And there is still the single event->rb, not more.

Fair enough.

Quoted 5 lines

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

With this there should be no difficult page table tricks or anything.

True.

Quoted 1 line

The patch is way incomplete but should sketch enough of the idea..

Can I take it over?

Quoted 2 lines

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

With PERF_RECORD_AUX carrying offset and size, we shouldn't need aux_{head,tail} either, don't you think?

Quoted 80 lines
---
 include/uapi/linux/perf_event.h | 19 +++++++++++++++
 kernel/events/core.c            | 51 +++++++++++++++++++++++++++++++++++++----
 kernel/events/internal.h        |  6 +++++
 kernel/events/ring_buffer.c     |  8 +------
 4 files changed, 72 insertions(+), 12 deletions(-)
include/uapi/linux/perf_event.h +19 -0
index 853bc1ccb395..adef7c0f1e7c 100644
@@ -491,6 +491,13 @@ struct perf_event_mmap_page {
 	 */
 	__u64   data_head;		/* head in the data section */
 	__u64	data_tail;		/* user-space written tail */
+	__u64	data_offset;
+	__u64	data_size;
+
+	__u64	aux_head;
+	__u64	aux_tail;
+	__u64	aux_offset;
+	__u64	aux_size;
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
@@ -705,6 +712,18 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +11 -2
index 5129b1201050..993995a23b73 100644
@@ -4016,7 +4016,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)
 
 static const struct vm_operations_struct perf_mmap_vmops = {
 	.open		= perf_mmap_open,
-	.close		= perf_mmap_close,
+	.close		= perf_mmap_close, /* non mergable */
 	.fault		= perf_mmap_fault,
 	.page_mkwrite	= perf_mmap_fault,
 };
@@ -4030,6 +4030,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
 	struct ring_buffer *rb;
 	unsigned long vma_size;
 	unsigned long nr_pages;
+	unsigned long pgoff;
 	long user_extra, extra;
 	int ret = 0, flags = 0;
 
@@ -4045,7 +4046,50 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
 		return -EINVAL;
 
 	vma_size = vma->vm_end - vma->vm_start;
-	nr_pages = (vma_size / PAGE_SIZE) - 1;
+
+	if (vma->vm_pgoff == 0) {
+		nr_pages = (vma_size / PAGE_SIZE) - 1;
+	} else {
+		if (!event->rb)
+			return -EINVAL;
+
+		nr_pages = vma_size / PAGE_SIZE;
+

kernel/events/core.c static int perf_mmap(struct file *file, struct vm_area_struct *vma) · line 4067

Quoted 10 lines
kernel/events/core.c +10 -0
+		mutex_lock(&event->mmap_mutex);
+		ret = -EINVAL;
+		if (!event->rb)
+			goto err_aux_unlock;
+
+		if (!atomic_inc_not_zero(&event->rb->mmap_count))
+			goto err_aux_unlock;
+
+		if (userpg->aux_offset < userpg->data_offset + userpg->data_size)
+			goto err_aux_unlock;

The data_{offset,size} seem to be only set by userspace too, maybe we can do away with these altogether unless we want to allow for it to be a sparse file?

kernel/events/core.c static int perf_mmap(struct file *file, struct vm_area_struct *vma) · line 4069

Quoted 1 line
kernel/events/core.c +1 -0
+		pgoff = userpg->aux_offset;

..and simply do a

                pgoff = event->rb->nr_pages + 1;

?

Quoted 54 lines
kernel/events/core.c +23 -3
+		if (pgoff & ~PAGE_MASK)
+			goto err_aux_unlock;
+
+		pgoff >>= PAGE_SHIFT;
+		if (pgoff != vma->vm_pgoff)
+			goto err_aux_unlock;
+
+		/* XXX do we want to allow !power_of_2 sizes, for AUX?  */
+		if (nr_pages == 0 || !is_power_of_2(nr_pages))
+			goto err_aux_unlock;
+
+		if (vma_size != PAGE_SIZE * nr_pages)
+			goto err_aux_unlock;
+
+		if (userpg->aux_size != vma_size)
+			goto err_aux_unlock;
+			
+		ret = rb_alloc_aux(event->rb, userpg->aux_offset >> PAGE_SHIFT, nr_pages);
+
+err_aux_unlock:
+		mutex_unlock(&event->mmap_mutex);
+		return ret;
+	}
 
 	/*
 	 * If we have rb pages ensure they're a power-of-two number, so we
@@ -4057,9 +4101,6 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
 	if (vma_size != PAGE_SIZE * (1 + nr_pages))
 		return -EINVAL;
 
-	if (vma->vm_pgoff != 0)
-		return -EINVAL;
-
 	WARN_ON_ONCE(event->ctx->parent_ctx);
 again:
 	mutex_lock(&event->mmap_mutex);
kernel/events/internal.h +1 -0
index 569b218782ad..6258aaa36097 100644
@@ -36,6 +36,7 @@ struct ring_buffer {
 	struct user_struct		*mmap_user;
 
 	struct perf_event_mmap_page	*user_page;
+	struct radix_tree_root		page_tree;
 	void				*data_pages[0];
 };
 
kernel/events/ring_buffer.c +0 -0
index 146a5792b1d2..b82505325df0 100644
@@ -251,13 +251,7 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)
 struct page *

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 254

Quoted 10 lines
kernel/events/ring_buffer.c +1 -7
 perf_mmap_to_page(struct ring_buffer *rb, unsigned long pgoff)
 {
-	if (pgoff > rb->nr_pages)
-		return NULL;
-
-	if (pgoff == 0)
-		return virt_to_page(rb->user_page);
-
-	return virt_to_page(rb->data_pages[pgoff - 1]);
+	return radix_tree_lookup(&rb->page_tree, pgoff);

This can instead call into the underlying driver, which will likely maintain an array similar to data_pages[] anyway.

Regards,
--
Alex

author reviewer
#21 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 32 lines · Alexander Shishkin
Quoted 1 line · Peter Zijlstra

How about something like this for the itrace thing?

It's much nicer than the page swizzling draft I was about to send you.

Quoted 4 lines

You would mmap() the regular buffer; when write ->aux_{offset,size} in the control page. After which you can do a second mmap() with the .pgoff matching the aux_offset you gave and .length matching the aux_size you gave.

Why do we need aux_{offset,size} at all, then? Userspace should know how they mmap()ed it.

Quoted 5 lines

This way the mmap() content still looks like a single linear file (could be sparse if you leave a hole, although we could require the aux_offset to match the end of the data section).

And there is still the single event->rb, not more.

Fair enough.

Quoted 5 lines

Then, when data inside that aux data store changes they should inject an PERF_RECORD_AUX to indicate this did happen, which ties it back into the normal event flow.

With this there should be no difficult page table tricks or anything.

True.

Quoted 1 line

The patch is way incomplete but should sketch enough of the idea..

Quoted 7 lines

Can I take it over?

Quoted 2 lines

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

With PERF_RECORD_AUX carrying offset and size, we shouldn't need aux_{head,tail} either, don't you think?

I take this one back, as perf record doesn't actually parse records from the buffer, it would still need the pointers.

Regards,
--
Alex

author reviewer
#22 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

In reply to Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.

Regards,
--
Alex

maintainer
#23 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 9 lines · Alexander Shishkin
Quoted 2 lines · Peter Zijlstra

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.

Yeah, this might make it easier. So then the rule is *_offset + *_{head,tail} is the actual file offset into the buffer.

author reviewer
#24 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines · Peter Zijlstra

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

Quoted 7 lines
Quoted 4 lines

Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.

Yeah, this might make it easier. So then the rule is *_offset + *_{head,tail} is the actual file offset into the buffer.

You mean *_offset + (*_{head,tail} % *_size) ?

Regards,
--
Alex

maintainer
#25 lore raw

Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event

Quoted 13 lines · Alexander Shishkin
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
Quoted 2 lines · Peter Zijlstra

So the aux_head/tail values should also be in the file space and not start at 0 again, similar for the offsets in the AUX record.

Thinking some more about it: if we keep aux_{offset,size} in userpg, then it would make sense to have aux_{head,tail} run from 0 to infinity similar to data_{head,tail}, especially considering that neither data_* nor aux_* pointers are file offsets anyway.

Quoted 4 lines
Quoted 2 lines

Yeah, this might make it easier. So then the rule is *_offset + *_{head,tail} is the actual file offset into the buffer.

You mean *_offset + (*_{head,tail} % *_size) ?

Indeed I did..

author reviewer
#26 lore raw

PATCH 4/11 itrace: Infrastructure for instruction flow tracing units

In reply to Alexander Shishkin

Instruction tracing PMUs are capable of recording a log of instruction execution flow on a cpu core, which can be useful for profiling and crash analysis. This patch adds itrace infrastructure for perf events and the rest of the kernel to use.

Since such PMUs can produce copious amounts of trace data at the rate for hundreds of megabytes per second per core, it may be impractical to process it inside the kernel in real time, but instead export raw trace streams to userspace for subsequent analysis. Thus, itrace PMUs may export their trace buffers, which can be mmap()ed to userspace from a special file descriptor, which can be obtained from the perf_event_open() syscall by using a PERF_FLAG_FD_ITRACE flag and the original perf event descriptor.

This infrastructure should also be useful for ARM ETM/PTM and other program flow tracing units that can potentially generate a lot of trace data very fast.

---
 include/linux/itrace.h          |  89 +++++++++++++
 include/linux/perf_event.h      |   5 +
 include/uapi/linux/perf_event.h |  17 +++
 kernel/events/Makefile          |   2 +-
 kernel/events/core.c            | 130 ++++++++++++++++---
 kernel/events/itrace.c          | 271 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 495 insertions(+), 19 deletions(-)
 create mode 100644 include/linux/itrace.h
 create mode 100644 kernel/events/itrace.c
include/linux/itrace.h new file +89 -0
new file mode 100644
index 0000000..735baaf4
@@ -0,0 +1,89 @@
+/*
+ * Instruction flow trace unit infrastructure
+ * Copyright (c) 2013, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ */
+
+#ifndef _LINUX_ITRACE_H
+#define _LINUX_ITRACE_H
+
+#include <linux/perf_event.h>
+#include <linux/file.h>
+
+extern struct ring_buffer_ops itrace_rb_ops;
+
+static inline bool is_itrace_vma(struct vm_area_struct *vma)
+{
+	if (vma->vm_file) {
+		struct perf_event *event = vma->vm_file->private_data;
+		if (event->hw.itrace_file == vma->vm_file)
+			return true;
+	}
+
+	return false;
+}
+
+void *itrace_priv(struct perf_event *event);
+
+void *itrace_event_get_priv(struct perf_event *event);
+void itrace_event_put(struct perf_event *event);
+
+struct itrace_pmu {
+	struct pmu		pmu;
+	struct list_head	entry;
+	/*
+	 * Allocate/free ring_buffer backing store
+	 */
+	void			*(*alloc_buffer)(int cpu, int nr_pages, bool overwrite,
+						 void **pages,
+						 struct perf_event_mmap_page **user_page);
+	void			(*free_buffer)(void *buffer);
+
+	int			(*event_init)(struct perf_event *event);
+
+	char			*name;
+};
+
+#define to_itrace_pmu(x) container_of((x), struct itrace_pmu, pmu)
+
+#ifdef CONFIG_PERF_EVENTS
+extern int itrace_inherit_event(struct perf_event *event,
+				struct task_struct *task);
+extern void itrace_lost_data(struct perf_event *event, u64 offset);
+extern int itrace_pmu_register(struct itrace_pmu *ipmu);
+
+extern int itrace_event_installable(struct perf_event *event,
+				    struct perf_event_context *ctx);
+
+extern void itrace_wake_up(struct perf_event *event);
+
+extern bool is_itrace_event(struct perf_event *event);
+
+#else
+static int itrace_inherit_event(struct perf_event *event,
+				struct task_struct *task)	{ return 0; }
+static inline void
+itrace_lost_data(struct perf_event *event, u64 offset)		{}
+static inline int itrace_pmu_register(struct itrace_pmu *ipmu)	{ return -EINVAL; }
+
+static inline int
+itrace_event_installable(struct perf_event *event,
+			 struct perf_event_context *ctx)	{ return -EINVAL; }
+static inline void itrace_wake_up(struct perf_event *event)	{}
+static inline bool is_itrace_event(struct perf_event *event)	{ return false; }
+#endif
+
+#endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +5 -0
index 93cefb6..b0147e0 100644
@@ -126,6 +126,10 @@ struct hw_perf_event {
 			/* for tp_event->class */
 			struct list_head	tp_list;
 		};
+		struct { /* itrace */
+			struct file		*itrace_file;
+			struct task_struct	*itrace_target;
+		};
 #ifdef CONFIG_HAVE_HW_BREAKPOINT
 		struct { /* breakpoint */
 			/*
@@ -291,6 +295,7 @@ struct ring_buffer;
 
 enum perf_event_rb {
 	PERF_RB_MAIN = 0,
+	PERF_RB_ITRACE,
 	PERF_NR_RB,
 };
 
include/uapi/linux/perf_event.h +17 -0
index e244ed4..2dd57db 100644
@@ -237,6 +237,10 @@ enum perf_event_read_format {
 #define PERF_ATTR_SIZE_VER2	80	/* add: branch_sample_type */
 #define PERF_ATTR_SIZE_VER3	96	/* add: sample_regs_user */
 					/* add: sample_stack_user */
+#define PERF_ATTR_SIZE_VER4	120	/* add: itrace_config */
+					/* add: itrace_watermark */
+					/* add: itrace_sample_type */
+					/* add: itrace_sample_size */
 
 /*
  * Hardware event_id to monitor via a performance monitoring event:
@@ -333,6 +337,11 @@ struct perf_event_attr {
 
 	/* Align to u64. */
 	__u32	__reserved_2;
+
+	__u64	itrace_config;
+	__u32	itrace_watermark;	/* wakeup every n pages */
+	__u32	itrace_sample_type;	/* pmu->type of the itrace PMU */
+	__u64	itrace_sample_size;
 };
 
 #define perf_flags(attr)	(*(&(attr)->read_format + 1))
@@ -705,6 +714,13 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 
+	/*
+	 * struct {
+	 *   u64 offset;
+	 * }
+	 */
+	PERF_RECORD_ITRACE_LOST			= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -726,6 +742,7 @@ enum perf_callchain_context {
 #define PERF_FLAG_FD_OUTPUT		(1U << 1)
 #define PERF_FLAG_PID_CGROUP		(1U << 2) /* pid=cgroup id, per-cpu mode only */
 #define PERF_FLAG_FD_CLOEXEC		(1U << 3) /* O_CLOEXEC */
+#define PERF_FLAG_FD_ITRACE		(1U << 4) /* request itrace fd */
 
 union perf_mem_data_src {
 	__u64 val;
kernel/events/Makefile +1 -1
index 103f5d1..46a3770 100644
@@ -2,7 +2,7 @@ ifdef CONFIG_FUNCTION_TRACER
 CFLAGS_REMOVE_core.o = -pg
 endif
 
-obj-y := core.o ring_buffer.o callchain.o
+obj-y := core.o ring_buffer.o callchain.o itrace.o
 
 obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
 obj-$(CONFIG_UPROBES) += uprobes.o
kernel/events/core.c +112 -18

263 lines not on this page · raw message

kernel/events/itrace.c new file +271 -0

271 lines not on this page · raw message

Signature · 1 line
1.8.5.2
author reviewer
#27 lore raw

PATCH 5/11 itrace: Add functionality to include traces in perf event samples

In reply to Alexander Shishkin

Trace data from itrace PMUs can be used to annotate other perf events by including it in sample records when PERF_SAMPLE_ITRACE flag is set. In this case, a PT kernel counter is created for each such event and trace data is retrieved from it and stored in the perf data stream.

---
 include/linux/itrace.h          |  37 +++++++++
 include/linux/perf_event.h      |  15 ++++
 include/uapi/linux/perf_event.h |   5 +-
 kernel/events/core.c            |  35 +++++++++
 kernel/events/itrace.c          | 169 ++++++++++++++++++++++++++++++++++++++--
 5 files changed, 252 insertions(+), 9 deletions(-)
include/linux/itrace.h +37 -0
index 735baaf4..6adbb32 100644
@@ -54,12 +54,27 @@ struct itrace_pmu {
 
 	int			(*event_init)(struct perf_event *event);
 
+	/*
+	 * Calculate the size of a sample to be written out
+	 */
+	unsigned long		(*sample_trace)(struct perf_event *event,
+						struct perf_sample_data *data);
+
+	/*
+	 * Write out a trace sample to the given output handle
+	 */
+	void			(*sample_output)(struct perf_event *event,
+						 struct perf_output_handle *handle,
+						 struct perf_sample_data *data);
 	char			*name;
 };
 
 #define to_itrace_pmu(x) container_of((x), struct itrace_pmu, pmu)
 
 #ifdef CONFIG_PERF_EVENTS
+
+extern int itrace_kernel_event(struct perf_event *event,
+			       struct task_struct *task);
 extern int itrace_inherit_event(struct perf_event *event,
 				struct task_struct *task);
 extern void itrace_lost_data(struct perf_event *event, u64 offset);
@@ -72,7 +87,17 @@ extern void itrace_wake_up(struct perf_event *event);
 
 extern bool is_itrace_event(struct perf_event *event);
 
+extern int itrace_sampler_init(struct perf_event *event,
+			       struct task_struct *task);
+extern void itrace_sampler_fini(struct perf_event *event);
+extern unsigned long itrace_sampler_trace(struct perf_event *event,
+					  struct perf_sample_data *data);
+extern void itrace_sampler_output(struct perf_event *event,
+				  struct perf_output_handle *handle,
+				  struct perf_sample_data *data);
 #else
+static int itrace_kernel_event(struct perf_event *event,
+			       struct task_struct *task)	{ return 0; }
 static int itrace_inherit_event(struct perf_event *event,
 				struct task_struct *task)	{ return 0; }
 static inline void
@@ -84,6 +109,18 @@ itrace_event_installable(struct perf_event *event,
 			 struct perf_event_context *ctx)	{ return -EINVAL; }
 static inline void itrace_wake_up(struct perf_event *event)	{}
 static inline bool is_itrace_event(struct perf_event *event)	{ return false; }
+
+static inline int itrace_sampler_init(struct perf_event *event,
+				      struct task_struct *task)	{}
+static inline void
+itrace_sampler_fini(struct perf_event *event)			{}
+static inline unsigned long
+itrace_sampler_trace(struct perf_event *event,
+		     struct perf_sample_data *data)		{ return 0; }
+static inline void
+itrace_sampler_output(struct perf_event *event,
+		      struct perf_output_handle *handle,
+		      struct perf_sample_data *data)		{}
 #endif
 
 #endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +15 -0
index b0147e0..11eb133 100644
@@ -83,6 +83,12 @@ struct perf_regs_user {
 	struct pt_regs	*regs;
 };
 
+struct perf_trace_record {
+	u64		size;
+	unsigned long	from;
+	unsigned long	to;
+};
+
 struct task_struct;
 
 /*
@@ -97,6 +103,11 @@ struct hw_perf_event_extra {
 
 struct event_constraint;
 
+enum perf_itrace_counter_type {
+	PERF_ITRACE_USER	= BIT(1),
+	PERF_ITRACE_SAMPLING	= BIT(2),
+};
+
 /**
  * struct hw_perf_event - performance event hardware details:
  */
@@ -129,6 +140,7 @@ struct hw_perf_event {
 		struct { /* itrace */
 			struct file		*itrace_file;
 			struct task_struct	*itrace_target;
+			unsigned int		counter_type;
 		};
 #ifdef CONFIG_HAVE_HW_BREAKPOINT
 		struct { /* breakpoint */
@@ -434,6 +446,7 @@ struct perf_event {
 	perf_overflow_handler_t		overflow_handler;
 	void				*overflow_handler_context;
 
+	struct perf_event		*trace_event;
 #ifdef CONFIG_EVENT_TRACING
 	struct ftrace_event_call	*tp_event;
 	struct event_filter		*filter;
@@ -591,6 +604,7 @@ struct perf_sample_data {
 	union  perf_mem_data_src	data_src;
 	struct perf_callchain_entry	*callchain;
 	struct perf_raw_record		*raw;
+	struct perf_trace_record	trace;
 	struct perf_branch_stack	*br_stack;
 	struct perf_regs_user		regs_user;
 	u64				stack_user_size;
@@ -611,6 +625,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data,
 	data->period = period;
 	data->regs_user.abi = PERF_SAMPLE_REGS_ABI_NONE;
 	data->regs_user.regs = NULL;
+	data->trace.from = data->trace.to = data->trace.size = 0;
 	data->stack_user_size = 0;
 	data->weight = 0;
 	data->data_src.val = 0;
include/uapi/linux/perf_event.h +4 -1
index 2dd57db..a06cf4b 100644
@@ -137,8 +137,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_DATA_SRC			= 1U << 15,
 	PERF_SAMPLE_IDENTIFIER			= 1U << 16,
 	PERF_SAMPLE_TRANSACTION			= 1U << 17,
+	PERF_SAMPLE_ITRACE			= 1U << 18,
 
-	PERF_SAMPLE_MAX = 1U << 18,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 19,		/* non-ABI */
 };
 
 /*
@@ -689,6 +690,8 @@ enum perf_event_type {
 	 *	{ u64			weight;   } && PERF_SAMPLE_WEIGHT
 	 *	{ u64			data_src; } && PERF_SAMPLE_DATA_SRC
 	 *	{ u64			transaction; } && PERF_SAMPLE_TRANSACTION
+	 *	{ u64			size;
+	 *	  char			data[size]; } && PERF_SAMPLE_ITRACE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +35 -0
index ff6e286..e1388a5 100644
@@ -1576,6 +1576,9 @@ void perf_event_disable(struct perf_event *event)
 	struct perf_event_context *ctx = event->ctx;
 	struct task_struct *task = ctx->task;
 
+	if (event->trace_event)
+		perf_event_disable(event->trace_event);
+
 	if (!task) {
 		/*
 		 * Disable the event on the cpu that it's on
@@ -2070,6 +2073,8 @@ void perf_event_enable(struct perf_event *event)
 	struct perf_event_context *ctx = event->ctx;
 	struct task_struct *task = ctx->task;
 
+	if (event->trace_event)
+		perf_event_enable(event->trace_event);
 	if (!task) {
 		/*
 		 * Enable the event on the cpu that it's on
@@ -3209,6 +3214,8 @@ static void unaccount_event(struct perf_event *event)
 		static_key_slow_dec_deferred(&perf_sched_events);
 	if (has_branch_stack(event))
 		static_key_slow_dec_deferred(&perf_sched_events);
+	if ((event->attr.sample_type & PERF_SAMPLE_ITRACE) && event->trace_event)
+		itrace_sampler_fini(event);
 
 	unaccount_event_cpu(event, event->cpu);
 }
@@ -4664,6 +4671,13 @@ void perf_output_sample(struct perf_output_handle *handle,
 	if (sample_type & PERF_SAMPLE_TRANSACTION)
 		perf_output_put(handle, data->txn);
 
+	if (sample_type & PERF_SAMPLE_ITRACE) {
+		perf_output_put(handle, data->trace.size);
+
+		if (data->trace.size)
+			itrace_sampler_output(event, handle, data);
+	}
+
 	if (!event->attr.watermark) {
 		int wakeup_events = event->attr.wakeup_events;
 
@@ -4771,6 +4785,14 @@ void perf_prepare_sample(struct perf_event_header *header,
 		data->stack_user_size = stack_size;
 		header->size += size;
 	}
+
+	if (sample_type & PERF_SAMPLE_ITRACE) {
+		u64 size = sizeof(u64);
+
+		size += itrace_sampler_trace(event, data);
+
+		header->size += size;
+	}
 }
 
 static void perf_event_output(struct perf_event *event,
@@ -6795,6 +6817,15 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 			if (err)
 				goto err_pmu;
 		}
+
+		if (event->attr.sample_type & PERF_SAMPLE_ITRACE) {
+			err = itrace_sampler_init(event, task);
+			if (err) {
+				/* XXX: either clean up callchain buffers too
+				   or forbid them to go together */
+				goto err_pmu;
+			}
+		}
 	}
 
 	return event;
@@ -7369,6 +7400,10 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 
 	account_event(event);
 
+	err = itrace_kernel_event(event, task);
+	if (err)
+		goto err_free;
+
 	ctx = find_get_context(event->pmu, task, cpu);
 	if (IS_ERR(ctx)) {
 		err = PTR_ERR(ctx);
kernel/events/itrace.c +161 -8

201 lines not on this page · raw message

Signature · 1 line
1.8.5.2
author reviewer
#28 lore raw

PATCH 6/11 itrace: Add functionality to include traces in process core dumps

In reply to Alexander Shishkin

Per thread trace data that is provided by itrace PMUs can be included in process core dumps, which is controlled via a new rlimit parameter RLIMIT_ITRACE. This is done by a per-thread kernel counter that is created when this RLIMIT_ITRACE is set.

The value of RLIMIT_ITRACE indicates the size of the per-thread elf note in a core dump and the buffer size used to collect corresponding trace.

---
 fs/binfmt_elf.c                     |   6 +
 fs/proc/base.c                      |   1 +
 include/asm-generic/resource.h      |   1 +
 include/linux/itrace.h              |  36 +++++
 include/linux/perf_event.h          |   3 +
 include/uapi/asm-generic/resource.h |   3 +-
 include/uapi/linux/elf.h            |   1 +
 kernel/events/itrace.c              | 289 +++++++++++++++++++++++++++++++++++-
 kernel/exit.c                       |   3 +
 kernel/sys.c                        |   5 +
 10 files changed, 343 insertions(+), 5 deletions(-)
fs/binfmt_elf.c +6 -0
index 571a423..c7fcd49 100644
@@ -34,6 +34,7 @@
 #include <linux/utsname.h>
 #include <linux/coredump.h>
 #include <linux/sched.h>
+#include <linux/itrace.h>
 #include <asm/uaccess.h>
 #include <asm/param.h>
 #include <asm/page.h>
@@ -1576,6 +1577,8 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 		}
 	}
 
+	*total += itrace_elf_note_size(t->task);
+
 	return 1;
 }
 
@@ -1608,6 +1611,7 @@ static int fill_note_info(struct elfhdr *elf, int phdrs,
 	for (i = 0; i < view->n; ++i)
 		if (view->regsets[i].core_note_type != 0)
 			++info->thread_notes;
+	info->thread_notes++; /* ITRACE */
 
 	/*
 	 * Sanity check.  We rely on regset 0 being in NT_PRSTATUS,
@@ -1710,6 +1714,8 @@ static int write_note_info(struct elf_note_info *info,
 			    !writenote(&t->notes[i], cprm))
 				return 0;
 
+		itrace_elf_note_write(cprm, t->task);
+
 		first = 0;
 		t = t->next;
 	} while (t);
fs/proc/base.c +1 -0
index 03c8d74..69935a9 100644
@@ -471,6 +471,7 @@ static const struct limit_names lnames[RLIM_NLIMITS] = {
 	[RLIMIT_NICE] = {"Max nice priority", NULL},
 	[RLIMIT_RTPRIO] = {"Max realtime priority", NULL},
 	[RLIMIT_RTTIME] = {"Max realtime timeout", "us"},
+	[RLIMIT_ITRACE] = {"Max ITRACE buffer size", "bytes"},
 };
 
 /* Display limits for a process */
include/asm-generic/resource.h +1 -0
index b4ea8f5..e6e5657 100644
@@ -25,6 +25,7 @@
 	[RLIMIT_NICE]		= { 0, 0 },				\
 	[RLIMIT_RTPRIO]		= { 0, 0 },				\
 	[RLIMIT_RTTIME]		= {  RLIM_INFINITY,  RLIM_INFINITY },	\
+	[RLIMIT_ITRACE]		= {              0,  RLIM_INFINITY },	\
 }
 
 #endif
include/linux/itrace.h +36 -0
index 6adbb32..c1eb6d3 100644
@@ -22,6 +22,7 @@
 
 #include <linux/perf_event.h>
 #include <linux/file.h>
+#include <linux/coredump.h>
 
 extern struct ring_buffer_ops itrace_rb_ops;
 
@@ -66,6 +67,19 @@ struct itrace_pmu {
 	void			(*sample_output)(struct perf_event *event,
 						 struct perf_output_handle *handle,
 						 struct perf_sample_data *data);
+
+	/*
+	 * Get the PMU-specific part of a core dump note
+	 */
+	size_t			(*core_size)(struct perf_event *event);
+
+	/*
+	 * Write out the core dump note
+	 */
+	void			(*core_output)(struct coredump_params *cprm,
+					       struct perf_event *event,
+					       unsigned long len);
+	u64			coredump_config;
 	char			*name;
 };
 
@@ -95,6 +109,17 @@ extern unsigned long itrace_sampler_trace(struct perf_event *event,
 extern void itrace_sampler_output(struct perf_event *event,
 				  struct perf_output_handle *handle,
 				  struct perf_sample_data *data);
+
+extern int update_itrace_rlimit(struct task_struct *, unsigned long);
+extern void exit_itrace(struct task_struct *);
+
+struct itrace_note {
+	u64	itrace_config;
+};
+
+extern size_t itrace_elf_note_size(struct task_struct *tsk);
+extern void itrace_elf_note_write(struct coredump_params *cprm,
+				  struct task_struct *task);
 #else
 static int itrace_kernel_event(struct perf_event *event,
 			       struct task_struct *task)	{ return 0; }
@@ -121,6 +146,17 @@ static inline void
 itrace_sampler_output(struct perf_event *event,
 		      struct perf_output_handle *handle,
 		      struct perf_sample_data *data)		{}
+
+static inline int
+update_itrace_rlimit(struct task_struct *, unsigned long)	{ return -EINVAL; }
+static inline void exit_itrace(struct task_struct *)		{}
+
+static inline size_t
+itrace_elf_note_size(struct task_struct *tsk)			{ return 0; }
+static inline void
+itrace_elf_note_write(struct coredump_params *cprm,
+		      struct task_struct *task)			{}
+
 #endif
 
 #endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +3 -0
index 11eb133..8353d7f 100644
@@ -106,6 +106,9 @@ struct event_constraint;
 enum perf_itrace_counter_type {
 	PERF_ITRACE_USER	= BIT(1),
 	PERF_ITRACE_SAMPLING	= BIT(2),
+	PERF_ITRACE_COREDUMP	= BIT(3),
+	PERF_ITRACE_KERNEL	= (PERF_ITRACE_SAMPLING | PERF_ITRACE_COREDUMP),
+	PERF_ITRACE_ANY		= (PERF_ITRACE_KERNEL | PERF_ITRACE_USER),
 };
 
 /**
include/uapi/asm-generic/resource.h +2 -1
index f863428..073f413 100644
@@ -45,7 +45,8 @@
 					   0-39 for nice level 19 .. -20 */
 #define RLIMIT_RTPRIO		14	/* maximum realtime priority */
 #define RLIMIT_RTTIME		15	/* timeout for RT tasks in us */
-#define RLIM_NLIMITS		16
+#define RLIMIT_ITRACE		16	/* max itrace size */
+#define RLIM_NLIMITS		17
 
 /*
  * SuS says limits have to be unsigned.
include/uapi/linux/elf.h +1 -0
index ef6103b..4bfbf66 100644
@@ -369,6 +369,7 @@ typedef struct elf64_shdr {
 #define NT_PRPSINFO	3
 #define NT_TASKSTRUCT	4
 #define NT_AUXV		6
+#define NT_ITRACE	7
 /*
  * Note to userspace developers: size of NT_SIGINFO note may increase
  * in the future to accomodate more fields, don't assume it is fixed!
kernel/events/itrace.c +285 -4

345 lines not on this page · raw message

kernel/exit.c +3 -0
index a949819..28138ef 100644
@@ -48,6 +48,7 @@
 #include <linux/fs_struct.h>
 #include <linux/init_task.h>
 #include <linux/perf_event.h>
+#include <linux/itrace.h>
 #include <trace/events/sched.h>
 #include <linux/hw_breakpoint.h>
 #include <linux/oom.h>
@@ -788,6 +789,8 @@ void do_exit(long code)
 	check_stack_usage();
 	exit_thread();
 
+	exit_itrace(tsk);
+
 	/*
 	 * Flush inherited counters to the parent - before the parent
 	 * gets woken up by child-exit notifications.
kernel/sys.c +5 -0
index c723113..7651d6f 100644
@@ -14,6 +14,7 @@
 #include <linux/fs.h>
 #include <linux/kmod.h>
 #include <linux/perf_event.h>
+#include <linux/itrace.h>
 #include <linux/resource.h>
 #include <linux/kernel.h>
 #include <linux/workqueue.h>
@@ -1402,6 +1403,10 @@ int do_prlimit(struct task_struct *tsk, unsigned int resource,
 		update_rlimit_cpu(tsk, new_rlim->rlim_cur);
 out:
 	read_unlock(&tasklist_lock);
+
+	if (!retval && new_rlim && resource == RLIMIT_ITRACE)
+		retval = update_itrace_rlimit(tsk, new_rlim->rlim_cur);
+
 	return retval;
 }
 
Signature · 1 line
1.8.5.2
author reviewer
#29 lore raw

PATCH 7/11 x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

Add support for Intel Processor Trace (PT) to kernel's perf/itrace events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel. Thus, buffers containing this binary stream are zero-copy mapped to the debug tools in userspace for subsequent decoding and analysis.

---
 arch/x86/include/uapi/asm/msr-index.h     |  18 +
 arch/x86/kernel/cpu/Makefile              |   1 +
 arch/x86/kernel/cpu/intel_pt.h            | 127 ++++
 arch/x86/kernel/cpu/perf_event.c          |   4 +
 arch/x86/kernel/cpu/perf_event_intel.c    |  10 +
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 991 ++++++++++++++++++++++++++++++
 6 files changed, 1151 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
arch/x86/include/uapi/asm/msr-index.h +18 -0
index 37813b5..38979e7 100644
@@ -74,6 +74,24 @@
 #define MSR_IA32_PERF_CAPABILITIES	0x00000345
 #define MSR_PEBS_LD_LAT_THRESHOLD	0x000003f6
 
+#define MSR_IA32_RTIT_CTL		0x00000570
+#define RTIT_CTL_TRACEEN		BIT(0)
+#define RTIT_CTL_OS			BIT(2)
+#define RTIT_CTL_USR			BIT(3)
+#define RTIT_CTL_CR3EN			BIT(7)
+#define RTIT_CTL_TOPA			BIT(8)
+#define RTIT_CTL_TSC_EN			BIT(10)
+#define RTIT_CTL_DISRETC		BIT(11)
+#define RTIT_CTL_BRANCH_EN		BIT(13)
+#define MSR_IA32_RTIT_STATUS		0x00000571
+#define RTIT_STATUS_CONTEXTEN		BIT(1)
+#define RTIT_STATUS_TRIGGEREN		BIT(2)
+#define RTIT_STATUS_ERROR		BIT(4)
+#define RTIT_STATUS_STOPPED		BIT(5)
+#define MSR_IA32_RTIT_CR3_MATCH		0x00000572
+#define MSR_IA32_RTIT_OUTPUT_BASE	0x00000560
+#define MSR_IA32_RTIT_OUTPUT_MASK	0x00000561
+
 #define MSR_MTRRfix64K_00000		0x00000250
 #define MSR_MTRRfix16K_80000		0x00000258
 #define MSR_MTRRfix16K_A0000		0x00000259
arch/x86/kernel/cpu/Makefile +1 -0
index 6359506..cb69de3 100644
@@ -37,6 +37,7 @@ endif
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_p6.o perf_event_knc.o perf_event_p4.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o perf_event_intel_rapl.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o
 endif
 
 
arch/x86/kernel/cpu/intel_pt.h new file +127 -0

127 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event.c +4 -0
index 8e13293..9125797 100644
@@ -385,6 +385,10 @@ static inline int precise_br_compat(struct perf_event *event)
 
 int x86_pmu_hw_config(struct perf_event *event)
 {
+	if (event->attr.sample_type & PERF_SAMPLE_ITRACE &&
+	    event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK)
+		return -EINVAL;
+
 	if (event->attr.precise_ip) {
 		int precise = 0;
 
arch/x86/kernel/cpu/perf_event_intel.c +10 -0
index 0fa4f24..28b5023 100644
@@ -1312,6 +1312,8 @@ int intel_pmu_save_and_restart(struct perf_event *event)
 	return x86_perf_event_set_period(event);
 }
 
+void intel_pt_interrupt(void);
+
 static void intel_pmu_reset(void)
 {
 	struct debug_store *ds = __this_cpu_read(cpu_hw_events.ds);
@@ -1393,6 +1395,14 @@ again:
 	}
 
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+
+	/*
 	 * Checkpointed counters can lead to 'spurious' PMIs because the
 	 * rollback caused by the PMI will have cleared the overflow status
 	 * bit. Therefore always force probe these counters.
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +991 -0

991 lines not on this page · raw message

Signature · 1 line
1.8.5.2
#30 lore raw

Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver

arch/x86/kernel/cpu/perf_event_intel.c int intel_pmu_save_and_restart(struct perf_event *event) · line 1315

Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel.c +1 -0
index 0fa4f24..28b5023 100644
@@ -1312,6 +1312,8 @@ int intel_pmu_save_and_restart(struct perf_event *event)
 	return x86_perf_event_set_period(event);
 }
 
+void intel_pt_interrupt(void);

Should be in $(pwd)/perf_event.h

Quoted 12 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +6 -0
new file mode 100644
index 0000000..b6b1a84
@@ -0,0 +1,991 @@
+/*
+ * Intel(R) Processor Trace PMU driver for perf
+ * Copyright (c) 2013-2014, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,

arch/x86/kernel/cpu/perf_event_intel_pt.c line 16

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.

Remove the address, and add a pointer to the specification

Similar in the other files.

arch/x86/kernel/cpu/perf_event_intel_pt.c line 54

Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
+/*
+ * Capabilities of Intel PT hardware, such as number of address bits or
+ * supported output schemes, are cached and exported to userspace as "caps"
+ * attribute group of pt pmu device
+ * (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store
+ * relevant bits together with intel_pt traces.
+ *
+ * Currently, for debugging purposes, these attributes are also writable; this
+ * should be removed in the final version.

Already remove that code?

Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +4 -0
+{
+	u64 reg;
+
+	reg = RTIT_CTL_TOPA | RTIT_CTL_BRANCH_EN;

arch/x86/kernel/cpu/perf_event_intel_pt.c line 224

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+
+	if (!event->attr.exclude_kernel)
+		reg |= RTIT_CTL_OS;
+	if (!event->attr.exclude_user)
+		reg |= RTIT_CTL_USR;
+
+	reg |= (event->attr.itrace_config & PT_CONFIG_MASK);
+
+	if (wrmsr_safe(MSR_IA32_RTIT_CTL, reg, 0) < 0) {
+		pr_warn("Failed to enable PT on cpu %d\n", event->cpu);

Should rate limit this warning

arch/x86/kernel/cpu/perf_event_intel_pt.c line 235

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+		return -EINVAL;
+	}
+	return 0;
+}
+
+static void pt_config_start(bool start)
+{
+	u64 ctl;
+
+	rdmsrl(MSR_IA32_RTIT_CTL, ctl);

Should bail out here if someone else already started (e.g. hardware debugger) The read needs to be moved to before we overwrite other MSRs

Quoted 12 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+	if (start)
+		ctl |= RTIT_CTL_TRACEEN;
+	else
+		ctl &= ~RTIT_CTL_TRACEEN;
+	wrmsrl(MSR_IA32_RTIT_CTL, ctl);
 
 
+
+/**
+ * pt_handle_status - take care of possible status conditions
+ * @event: currently active PT event
+ */

arch/x86/kernel/cpu/perf_event_intel_pt.c

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+static void pt_handle_status(struct perf_event *event)
+{
+	struct pt_buffer *buf = itrace_priv(event);
+	int advance = 0;
+	u64 status;
+
+	rdmsrl(MSR_IA32_RTIT_STATUS, status);
+
+	if (status & RTIT_STATUS_ERROR) {
+		pr_err("ToPA ERROR encountered, trying to recover\n");

Add perf: prefix here (or better redefine pr_fmt at the beginning) Should be rate limited

Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
+static struct pt_buffer *pt_buffer_alloc(int cpu, size_t size,
+					 unsigned long watermark,
+					 bool snapshot, gfp_t gfp,
+					 void **pages)
+{
+	struct pt_buffer *buf;
+	int node;
+	int ret;
+

arch/x86/kernel/cpu/perf_event_intel_pt.c line 678

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+	if (!size || watermark << PAGE_SHIFT > size)
+		return NULL;
+
+	if (cpu == -1)
+		cpu = raw_smp_processor_id();
+	node = cpu_to_node(cpu);
+
+	buf = kzalloc(sizeof(struct pt_buffer), gfp);
+	if (!buf)
+		return NULL;

Should be kzalloc_node()

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
maintainer
#31 lore raw

Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 42 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +41 -0
+static int topa_insert_pages(struct pt_buffer *buf, gfp_t gfp,
+			     enum topa_sz sz)
+{
+	struct topa *topa = buf->last;
+	int node = cpu_to_node(buf->cpu);
+	int order = get_order(sizes(sz));
+	struct page *p;
+	unsigned long pn;
+
+	p = alloc_pages_node(node, gfp | GFP_USER | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY, order);
+	if (!p)
+		return -ENOMEM;
 
+
+	return 0;
+}
+
+static int pt_buffer_init_topa(struct pt_buffer *buf, size_t size, gfp_t gfp)
+{
+	struct topa *topa;
+	int err, region_size;
+
+	topa = topa_alloc(buf->cpu, gfp);
+	if (!topa)
+		return -ENOMEM;
+
+	topa_insert_table(buf, topa);
+
+	region_size = pt_get_topa_region_size(buf->snapshot, size);
+	if (region_size < 0) {
+		pt_buffer_fini_topa(buf);
+		return region_size;
+	}
+
+	while (region_size && get_order(sizes(region_size)) > MAX_ORDER)
+		region_size--;
+
+	/* fixup watermark in case of higher order allocations */
+	if (buf->watermark < (sizes(region_size) >> PAGE_SHIFT))
+		buf->watermark = sizes(region_size) >> PAGE_SHIFT;
+
+	while (buf->size < size) {

arch/x86/kernel/cpu/perf_event_intel_pt.c line 479

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+		err = topa_insert_pages(buf, gfp, region_size);
+		if (err) {
+			if (region_size) {
+				region_size--;
+				continue;
+			}
+			pt_buffer_fini_topa(buf);
+			return -ENOMEM;
+		}
+	}

So MAX_ORDER is 11, which gives us (11+12) 23bit or 8M max allocations, right?

Given this TOPA stuff is completely fucked in the first release, that's about all we can get it, right?

Now given you said 100s MB/s data rate for this itrace stuff, we're at ~0.1s traces. And that's in the very best case where we can actually get 8M.

Is that a workable amount?

#32 lore raw

Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver

Quoted 10 lines

So MAX_ORDER is 11, which gives us (11+12) 23bit or 8M max allocations, right?

Given this TOPA stuff is completely fucked in the first release, that's about all we can get it, right?

Now given you said 100s MB/s data rate for this itrace stuff, we're at ~0.1s traces. And that's in the very best case where we can actually get 8M.

Is that a workable amount

The PMI handler switches to a new buffer to allow longer traces in memory. It just costs one PMI.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
maintainer
#33 lore raw

Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 10 lines · Alexander Shishkin

Add support for Intel Processor Trace (PT) to kernel's perf/itrace events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel. Thus, buffers containing this binary stream are zero-copy mapped to the debug tools in userspace for subsequent decoding and analysis.

Quoted 10 lines
---
 arch/x86/include/uapi/asm/msr-index.h     |  18 +
 arch/x86/kernel/cpu/Makefile              |   1 +
 arch/x86/kernel/cpu/intel_pt.h            | 127 ++++
 arch/x86/kernel/cpu/perf_event.c          |   4 +
 arch/x86/kernel/cpu/perf_event_intel.c    |  10 +
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 991 ++++++++++++++++++++++++++++++
 6 files changed, 1151 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c

Andi said that when itrace is enabled the LBR is wrecked; this patch seems to fail to deal with that.

author reviewer
#34 lore raw

Re: [PATCH v1 07/11] x86: perf: intel_pt: Intel PT PMU driver

Quoted 22 lines · Peter Zijlstra
Quoted 10 lines · Alexander Shishkin

Add support for Intel Processor Trace (PT) to kernel's perf/itrace events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel. Thus, buffers containing this binary stream are zero-copy mapped to the debug tools in userspace for subsequent decoding and analysis.

Quoted 10 lines
---
 arch/x86/include/uapi/asm/msr-index.h     |  18 +
 arch/x86/kernel/cpu/Makefile              |   1 +
 arch/x86/kernel/cpu/intel_pt.h            | 127 ++++
 arch/x86/kernel/cpu/perf_event.c          |   4 +
 arch/x86/kernel/cpu/perf_event_intel.c    |  10 +
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 991 ++++++++++++++++++++++++++++++
 6 files changed, 1151 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
Quoted 2 lines

Andi said that when itrace is enabled the LBR is wrecked; this patch seems to fail to deal with that.

True, there needs to be a _safe() msr access before any configuration is done, I have a fix for that, but let's first deal with the buffer management.

Regards,
--
Alex

author reviewer
#35 lore raw

PATCH 8/11 x86: perf: intel_pt: Add sampling functionality

In reply to Alexander Shishkin

Intel Processor Trace (PT) data can be used in perf event samples to annotate other perf events. This patch implements itrace sampling related hooks that configure and output sampling data to perf stream. Users will need to include PERF_SAMPLE_ITRACE flag in the attr.sample_type mask and specify PT's PMU type in attr.itrace_sample_type field.

---
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 115 ++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_pt.c +115 -0
index b6b1a84..af1482d 100644
@@ -946,17 +946,130 @@ static void pt_event_read(struct perf_event *event)
 {
 }
 
+typedef unsigned int (*pt_copyfn)(void *data, const void *src,
+				  unsigned int len);
+
+/**
+ * pt_buffer_output - copy part of pt buffer to perf stream
+ * @buf: buffer to copy from
+ * @from: initial offset
+ * @to: final offset
+ * @copyfn: function that copies data out (like perf_output_copy())
+ * @data: data to be passed on to the copy function (like perf_output_handle)
+ */
+static int pt_buffer_output(struct pt_buffer *buf, unsigned long from,
+			    unsigned long to, pt_copyfn copyfn, void *data)
+{
+	unsigned long tocopy;
+	unsigned int len = 0, remainder;
+	void *page;
+
+	do {
+		tocopy = PAGE_SIZE - offset_in_page(from);
+		if (to > from)
+			tocopy = min(tocopy, to - from);
+		if (!tocopy)
+			break;
+
+		page = pt_buffer_get_page(buf, from >> PAGE_SHIFT);
+		if (WARN_ONCE(!page, "no data page for %lx offset\n", from))
+			break;
+
+		page += offset_in_page(from);
+
+		remainder = copyfn(data, page, tocopy);
+		if (remainder)
+			return -EFAULT;
+
+		len += tocopy;
+		from += tocopy;
+		if (from == buf->size)
+			from = 0;
+	} while (to != from);
+	return len;
+}
+
 static int pt_event_init(struct perf_event *event)
 {
 	if (event->attr.type != pt_pmu.itrace.pmu.type)
 		return -ENOENT;
 
+	/* can't be both */
+	if (event->attr.sample_type & PERF_SAMPLE_ITRACE)
+		return -ENOENT;
+
 	if (!pt_event_valid(event))
 		return -EINVAL;
 
 	return 0;
 }
 
+static unsigned long pt_trace_sampler_trace(struct perf_event *event,
+					    struct perf_sample_data *data)
+{
+	struct pt_buffer *buf;
+
+	pt_event_stop(event, 0);
+
+	buf = itrace_event_get_priv(event);
+	if (!buf) {
+		data->trace.size = 0;
+		goto out;
+	}
+
+	pt_read_offset(buf);
+	pt_update_head(buf);
+
+	data->trace.to = local64_read(&buf->head);
+
+	if (data->trace.to < event->attr.itrace_sample_size)
+		data->trace.from = buf->size + data->trace.to -
+			event->attr.itrace_sample_size;
+	else
+		data->trace.from = data->trace.to -
+			event->attr.itrace_sample_size;
+	data->trace.size = ALIGN(event->attr.itrace_sample_size, sizeof(u64));
+
+	itrace_event_put(event);
+
+out:
+	if (!data->trace.size)
+		pt_event_start(event, 0);
+
+	return data->trace.size;
+}
+
+static void pt_trace_sampler_output(struct perf_event *event,
+				    struct perf_output_handle *handle,
+				    struct perf_sample_data *data)
+{
+	unsigned long padding;
+	struct pt_buffer *buf;
+	int ret;
+
+	buf = itrace_event_get_priv(event);
+	if (!buf)
+		return;
+
+	ret = pt_buffer_output(buf, data->trace.from, data->trace.to,
+			       (pt_copyfn)perf_output_copy, handle);
+	itrace_event_put(event);
+	if (ret < 0) {
+		pr_warn("%s: failed to copy trace data\n", __func__);
+		goto out;
+	}
+
+	padding = data->trace.size - ret;
+	if (padding) {
+		u64 u = 0;
+
+		perf_output_copy(handle, &u, padding);
+	}
+
+out:
+	pt_event_start(event, 0);
+}
+
 static __init int pt_init(void)
 {
 	int ret, cpu;
@@ -982,6 +1095,8 @@ static __init int pt_init(void)
 	pt_pmu.itrace.pmu.read		= pt_event_read;
 	pt_pmu.itrace.alloc_buffer	= pt_buffer_itrace_alloc;
 	pt_pmu.itrace.free_buffer	= pt_buffer_itrace_free;
+	pt_pmu.itrace.sample_trace	= pt_trace_sampler_trace;
+	pt_pmu.itrace.sample_output	= pt_trace_sampler_output;
 	pt_pmu.itrace.name		= "intel_pt";
 	ret = itrace_pmu_register(&pt_pmu.itrace);
 
Signature · 1 line
1.8.5.2
author reviewer
#36 lore raw

PATCH 9/11 x86: perf: intel_pt: Add core dump functionality

In reply to Alexander Shishkin

Intel Processor Trace (PT) data can be used in process core dumps. This is done by implementing itrace code dump related hooks that configure and output trace data to a core file. The driver will also include the list of PT capabilities in itrace core dump notes so that the decoder can make assumptions about the binary stream.

---
 arch/x86/kernel/cpu/intel_pt.h            |  2 +
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 74 +++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
arch/x86/kernel/cpu/intel_pt.h +2 -0
index dd69092..befde1f 100644
@@ -84,6 +84,8 @@ enum pt_capabilities {
 struct pt_pmu {
 	struct itrace_pmu	itrace;
 	u32			caps[4 * PT_CPUID_LEAVES];
+	char			*capstr;
+	unsigned int		caplen;
 };
 
 /**
arch/x86/kernel/cpu/perf_event_intel_pt.c +74 -0

110 lines not on this page · raw message

Signature · 1 line
1.8.5.2
#37 lore raw

Re: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality

arch/x86/kernel/cpu/perf_event_intel_pt.c static ssize_t pt_cap_store(struct device *cdev, · line 146

Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c +1 -0
 			   char *buf)
@@ -114,6 +143,7 @@ static ssize_t pt_cap_store(struct device *cdev,
 		return -EINVAL;
 
 	pt_cap_set(cap, new);
+	pt_cap_string();

Don't we need some lock here? Otherwise it may leak memory with racing writes and become inconsistent.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
author reviewer
#38 lore raw

Re: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality

Quoted 9 lines · Andi Kleen

arch/x86/kernel/cpu/perf_event_intel_pt.c static ssize_t pt_cap_store(struct device *cdev, · line 146

Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c +1 -0
 			   char *buf)
@@ -114,6 +143,7 @@ static ssize_t pt_cap_store(struct device *cdev,
 		return -EINVAL;
 
 	pt_cap_set(cap, new);
+	pt_cap_string();

Don't we need some lock here? Otherwise it may leak memory with racing writes and become inconsistent.

Good point.

Regards,
--
Alex

#39 lore raw

Re: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality

In reply to Alexander Shishkin

arch/x86/kernel/cpu/perf_event_intel_pt.c out: · line 1131

Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c +13 -0
+
+static void pt_trace_core_output(struct coredump_params *cprm,
+				 struct perf_event *event,
+				 unsigned long len)
+{
+	struct pt_buffer *buf;
+	u64 from, to;
+	int ret;
+
+	buf = itrace_priv(event);
+
+	if (!dump_emit(cprm, pt_pmu.capstr, pt_pmu.caplen))
+		return;

It would be nicer if this was a separate note, instead of just being concatenated with the rest of the data.

Would make simpler parsing and be cleaner.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
author reviewer
#40 lore raw

Re: [PATCH v1 09/11] x86: perf: intel_pt: Add core dump functionality

Quoted 18 lines · Andi Kleen

arch/x86/kernel/cpu/perf_event_intel_pt.c out: · line 1131

Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c +13 -0
+
+static void pt_trace_core_output(struct coredump_params *cprm,
+				 struct perf_event *event,
+				 unsigned long len)
+{
+	struct pt_buffer *buf;
+	u64 from, to;
+	int ret;
+
+	buf = itrace_priv(event);
+
+	if (!dump_emit(cprm, pt_pmu.capstr, pt_pmu.caplen))
+		return;

It would be nicer if this was a separate note, instead of just being concatenated with the rest of the data.

Quoted 1 line

Would make simpler parsing and be cleaner.

So long as we won't have to include traces from two different pmus in the same core file, then matching these sections may provide another challenge. Doesn't seem like a sensible scenario, though.

Regards,
--
Alex

author reviewer
#41 lore raw

PATCH 10/11 x86: perf: intel_bts: Add BTS PMU driver

In reply to Alexander Shishkin

Add support for Branch Trace Store (BTS) via kernel perf/itrace event infrastructure. The difference with the existing implementation of BTS support is that this one is a separate PMU that exports events' trace buffers to userspace the same way as Intel PT PMU does. The immediate benefit is that the buffer size can be much bigger, resulting in fewer interrupts and no kernel side copying is involved. Also, of the kernel code is possible. Additionally, it is now possible to include BTS traces into process core dumps.

The old way of collecting BTS traces still works.

---
 arch/x86/kernel/cpu/Makefile               |   2 +-
 arch/x86/kernel/cpu/perf_event.h           |   6 +
 arch/x86/kernel/cpu/perf_event_intel.c     |   6 +-
 arch/x86/kernel/cpu/perf_event_intel_bts.c | 478 +++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |   3 +-
 5 files changed, 492 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
arch/x86/kernel/cpu/Makefile +1 -1
index cb69de3..29f7f32 100644
@@ -37,7 +37,7 @@ endif
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_p6.o perf_event_knc.o perf_event_p4.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o perf_event_intel_rapl.o
-obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o perf_event_intel_bts.o
 endif
 
 
arch/x86/kernel/cpu/perf_event.h +6 -0
index c1a8618..00b1ffb 100644
@@ -702,6 +702,12 @@ void intel_pmu_lbr_init_snb(void);
 
 int intel_pmu_setup_lbr_filter(struct perf_event *event);
 
+int intel_bts_interrupt(void);
+
+void intel_bts_enable_local(void);
+
+void intel_bts_disable_local(void);
+
 int p4_pmu_init(void);
 
 int p6_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel.c +5 -1
index 28b5023..e447972 100644
@@ -1052,6 +1052,8 @@ static void intel_pmu_disable_all(void)
 
 	if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask))
 		intel_pmu_disable_bts();
+	else
+		intel_bts_disable_local();
 
 	intel_pmu_pebs_disable_all();
 	intel_pmu_lbr_disable_all();
@@ -1074,7 +1076,8 @@ static void intel_pmu_enable_all(int added)
 			return;
 
 		intel_pmu_enable_bts(event->hw.config);
-	}
+	} else
+		intel_bts_enable_local();
 }
 
 /*
@@ -1362,6 +1365,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 		apic_write(APIC_LVTPC, APIC_DM_NMI);
 	intel_pmu_disable_all();
 	handled = intel_pmu_drain_bts_buffer();
+	handled += intel_bts_interrupt();
 	status = intel_pmu_get_status();
 	if (!status) {
 		intel_pmu_enable_all(0);
arch/x86/kernel/cpu/perf_event_intel_bts.c new file +478 -0

478 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_intel_ds.c +2 -1
index ae96cfa..21f799f 100644
@@ -444,7 +444,8 @@ void intel_pmu_enable_bts(u64 config)
 
 	debugctlmsr |= DEBUGCTLMSR_TR;
 	debugctlmsr |= DEBUGCTLMSR_BTS;
-	debugctlmsr |= DEBUGCTLMSR_BTINT;
+	if (config & ARCH_PERFMON_EVENTSEL_INT)
+		debugctlmsr |= DEBUGCTLMSR_BTINT;
 
 	if (!(config & ARCH_PERFMON_EVENTSEL_OS))
 		debugctlmsr |= DEBUGCTLMSR_BTS_OFF_OS;
Signature · 1 line
1.8.5.2
author reviewer
#42 lore raw

PATCH 11/11 x86: perf: intel_bts: Add core dump related functionality

In reply to Alexander Shishkin

BTS data can be used in process core dumps. This patch implements itrace core dump related hooks that will configure and output BTS traces into a core file.

---
 arch/x86/kernel/cpu/perf_event_intel_bts.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/x86/kernel/cpu/perf_event_intel_bts.c +22 -0
index 0a08969..20a19b2 100644
@@ -436,6 +436,26 @@ static void bts_event_read(struct perf_event *event)
 {
 }
 
+static size_t bts_trace_core_size(struct perf_event *event)
+{
+	return 0;
+}
+
+static void bts_trace_core_output(struct coredump_params *cprm,
+				  struct perf_event *event,
+				  unsigned long len)
+{
+	struct bts_buffer *buf = itrace_event_get_priv(event);
+	u64 head = local64_read(&buf->head);
+
+	if (head < len) {
+		dump_emit(cprm, buf->buf + head, buf->real_size - head);
+		dump_emit(cprm, buf->buf, len - buf->real_size + head);
+	} else
+		dump_emit(cprm, buf->buf + head - len, len);
+	itrace_event_put(event);
+}
+
 static __init int bts_init(void)
 {
 	int ret, cpu;
@@ -459,6 +479,8 @@ static __init int bts_init(void)
 	bts_pmu.pmu.read		= bts_event_read;
 	bts_pmu.alloc_buffer		= bts_buffer_itrace_alloc;
 	bts_pmu.free_buffer		= bts_buffer_itrace_free;
+	bts_pmu.core_size		= bts_trace_core_size;
+	bts_pmu.core_output		= bts_trace_core_output;
 	bts_pmu.name			= "intel_bts";
 
 	ret = itrace_pmu_register(&bts_pmu);
Signature · 1 line
1.8.5.2
#43 lore raw

Re: [PATCH v1 11/11] x86: perf: intel_bts: Add core dump related functionality

Quoted 3 lines · Alexander Shishkin

BTS data can be used in process core dumps. This patch implements itrace core dump related hooks that will configure and output BTS traces into a core file.

Don't we need a different note number here?

Otherwise how should the debugger know if it's BTS or PT.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
author reviewer
#44 lore raw

Re: [PATCH v1 11/11] x86: perf: intel_bts: Add core dump related functionality

Quoted 9 lines · Andi Kleen
Quoted 3 lines · Alexander Shishkin

BTS data can be used in process core dumps. This patch implements itrace core dump related hooks that will configure and output BTS traces into a core file.

Don't we need a different note number here?

Otherwise how should the debugger know if it's BTS or PT.

The pmu name is part of the note as well as its itrace_config.

Regards,
--
Alex