Skip to contentSkip to conversation

perf: Add infrastructure and support for Intel PT

PATCH Superseded Alexander Shishkin 7 participants 61 messages Aug 20, 2014 – Oct 7, 2014 1 message not in the archive

Subsystems (2)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Alexander Shishkin

author reviewer
#0 lore raw

v4 cover perf: Add infrastructure and support for Intel PT

Hi Peter and all,

This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.

The single most notable thing is that while PT outputs trace data in a compressed binary format, it will still generate hundreds of megabytes of trace data per second per core. Decoding this binary stream takes 2-3 orders of magnitude the cpu time that it takes to generate it. These considerations make it impossible to carry out decoding in kernel space. Therefore, the trace data is exported to userspace as a zero-copy mapping that userspace can collect and store for later decoding. To address this, this patchset extends perf ring buffer with an "AUX space", which is allocated for hardware blocks such as PT to export their trace data with minimal overhead. This space can be configured via buffer's user page and mmapped from the same file descriptor with a given offset. Data can then be collected from it by reading the aux_head (write) pointer from the user page and updating aux_tail (read) pointer similarly to data_{head,tail} of the traditional perf buffer. There is an api between perf core and pmu drivers that wish to make use of this AUX space to export their data.

For tracing blocks that don't support hardware scatter-gather tables, we provide high-order physically contiguous allocations to minimize the overhead needed for software double buffering and PMI pressure.

This way we get a normal perf data stream that provides sideband information that is required to decode the trace data, such as MMAPs, COMMs etc, plus the actual trace in its own logical space.

If the trace buffer is mapped writable, the driver will stop tracing when it fills up (aux_head approaches aux_tail), till data is read, aux_tail pointer is moved forward and an ioctl() is issued to re-enable tracing. If the trace buffer is mapped read only, the tracing will continue, overwriting older data, so that the buffer always contains the most recent data. Tracing can be stopped with an ioctl() and restarted once the data is collected.

Another use case is annotating samples of other perf events: setting PERF_SAMPLE_AUX requests attr.aux_sample_size bytes of trace to be included in each event's sample.

This patchset consists of necessary changes to the perf kernel infrastructure, and PT and BTS pmu drivers. The tooling support is not included in this series, however, it can be found in my github tree [2].

This version changes the way watermarks are handled for AUX area and gets rid of the notion of "itrace" both in the core and in the perf interface (event attribute), which makes it more logical.

[1] http://software.intel.com/en-us/intel-isa-extensions
[2] http://github.com/virtuoso/linux-perf/tree/intel_pt

Alexander Shishkin (21):
perf: Add data_{offset,size} to user_page
perf: Support high-order allocations for AUX space
perf: Add a capability for AUX_NO_SG pmus to do software double
buffering
perf: Add a pmu capability for "exclusive" events
perf: Redirect output from inherited events to parents
perf: Add api for pmus to write to AUX space
perf: Add AUX record
perf: Support overwrite mode for AUX area
perf: Add wakeup watermark control to AUX area
perf: add ITRACE_START record to indicate that tracing has started
x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
x86: perf: Intel PT and LBR/BTS are mutually exclusive
x86: perf: intel_pt: Intel PT PMU driver
x86: perf: intel_bts: Add BTS PMU driver
perf: Add rb_{alloc,free}_kernel api
perf: Add a helper to copy AUX data in the kernel
perf: Add a helper for looking up pmus by type
perf: Add infrastructure for using AUX data in perf samples
perf: Allocate ring buffers for inherited per-task kernel events
perf: Allow AUX sampling for multiple events
perf: Allow sampling of inherited events

Peter Zijlstra (1):
perf: Add AUX area to ring buffer for raw data streams

 arch/x86/include/asm/cpufeature.h          |   1 +
 arch/x86/include/uapi/asm/msr-index.h      |  18 +
 arch/x86/kernel/cpu/Makefile               |   1 +
 arch/x86/kernel/cpu/intel_pt.h             | 129 ++++
 arch/x86/kernel/cpu/perf_event.h           |  14 +
 arch/x86/kernel/cpu/perf_event_intel.c     |  14 +-
 arch/x86/kernel/cpu/perf_event_intel_bts.c | 501 +++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |  11 +-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |   9 +-
 arch/x86/kernel/cpu/perf_event_intel_pt.c  | 973 +++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/scattered.c            |   1 +
 include/linux/perf_event.h                 |  56 +-
 include/uapi/linux/perf_event.h            |  69 +-
 kernel/events/core.c                       | 545 +++++++++++++++-
 kernel/events/internal.h                   |  50 ++
 kernel/events/ring_buffer.c                | 310 ++++++++-
 16 files changed, 2658 insertions(+), 44 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
Signature · 1 line
2.1.0
author reviewer
#1 lore raw

PATCH v4 1/22 perf: Add data_{offset,size} to user_page

Currently, the actual perf ring buffer is one page into the mmap area, following the user page and the userspace follows this convention. This patch adds data_{offset,size} fields to user_page that can be used by userspace instead for locating perf data in the mmap area. This is also helpful when mapping existing or shared buffers if their size is not known in advance.

Right now, it is made to follow the existing convention that

	data_offset == PAGE_SIZE and
	data_offset + data_size == mmap_size.
---
 include/uapi/linux/perf_event.h | 5 +++++
 kernel/events/core.c            | 2 ++
 2 files changed, 7 insertions(+)
include/uapi/linux/perf_event.h +5 -0
index 9269de2548..f7d18c2cb7 100644
@@ -489,9 +489,14 @@ struct perf_event_mmap_page {
 	 * In this case the kernel will not over-write unread data.
 	 *
 	 * See perf_output_put_handle() for the data ordering.
+	 *
+	 * data_{offset,size} indicate the location and size of the perf record
+	 * buffer within the mmapped area.
 	 */
 	__u64   data_head;		/* head in the data section */
 	__u64	data_tail;		/* user-space written tail */
+	__u64	data_offset;		/* where the buffer starts */
+	__u64	data_size;		/* data buffer size */
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
kernel/events/core.c +2 -0
index 2d7363adf6..1e208bfe89 100644
@@ -3879,6 +3879,8 @@ static void perf_event_init_userpage(struct perf_event *event)
 	/* Allow new userspace to detect that bit 0 is deprecated */
 	userpg->cap_bit0_is_deprecated = 1;
 	userpg->size = offsetof(struct perf_event_mmap_page, __reserved);
+	userpg->data_offset = PAGE_SIZE;
+	userpg->data_size = perf_data_size(rb);
 
 unlock:
 	rcu_read_unlock();
Signature · 1 line
2.1.0
author reviewer
#2 lore raw

PATCH v4 2/22 perf: Add AUX area to ring buffer for raw data streams

In reply to Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---
 include/linux/perf_event.h      |  17 +++++
 include/uapi/linux/perf_event.h |  16 +++++
 kernel/events/core.c            | 140 +++++++++++++++++++++++++++++++++-------
 kernel/events/internal.h        |  21 ++++++
 kernel/events/ring_buffer.c     |  86 ++++++++++++++++++++++--
 5 files changed, 251 insertions(+), 29 deletions(-)
include/linux/perf_event.h +17 -0
index f0a1036b19..fd7b32876c 100644
@@ -263,6 +263,18 @@ struct pmu {
 	 * flush branch stack on context-switches (needed in cpu-wide mode)
 	 */
 	void (*flush_branch_stack)	(void);
+
+	/*
+	 * Set up pmu-private data structures for an AUX area
+	 */
+	void *(*setup_aux)		(int cpu, void **pages,
+					 int nr_pages, bool overwrite);
+					/* optional */
+
+	/*
+	 * Free pmu-private AUX data structures
+	 */
+	void (*free_aux)		(void *aux); /* optional */
 };
 
 /**
@@ -781,6 +793,11 @@ static inline bool has_branch_stack(struct perf_event *event)
 	return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
 }
 
+static inline bool has_aux(struct perf_event *event)
+{
+	return event->pmu->setup_aux;
+}
+
 extern int perf_output_begin(struct perf_output_handle *handle,
 			     struct perf_event *event, unsigned int size);
 extern void perf_output_end(struct perf_output_handle *handle);
include/uapi/linux/perf_event.h +16 -0
index f7d18c2cb7..7e0967c0f5 100644
@@ -497,6 +497,22 @@ struct perf_event_mmap_page {
 	__u64	data_tail;		/* user-space written tail */
 	__u64	data_offset;		/* where the buffer starts */
 	__u64	data_size;		/* data buffer size */
+
+	/*
+	 * AUX area is defined by aux_{offset,size} fields that should be set
+	 * by the userspace, so that
+	 *
+	 *   aux_offset >= data_offset + data_size
+	 *
+	 * prior to mmap()ing it. Size of the mmap()ed area should be aux_size.
+	 *
+	 * Ring buffer pointers aux_{head,tail} have the same semantics as
+	 * data_{head,tail} and same ordering rules apply.
+	 */
+	__u64	aux_head;
+	__u64	aux_tail;
+	__u64	aux_offset;
+	__u64	aux_size;
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
kernel/events/core.c +115 -25

214 lines not on this page · raw message

kernel/events/internal.h +21 -0
index 569b218782..e5374030b1 100644
@@ -35,6 +35,14 @@ struct ring_buffer {
 	unsigned long			mmap_locked;
 	struct user_struct		*mmap_user;
 
+	/* AUX area */
+	unsigned long			aux_pgoff;
+	int				aux_nr_pages;
+	atomic_t			aux_mmap_count;
+	unsigned long			aux_mmap_locked;
+	void				**aux_pages;
+	void				*aux_priv;
+
 	struct perf_event_mmap_page	*user_page;
 	void				*data_pages[0];
 };
@@ -43,6 +51,14 @@ extern void rb_free(struct ring_buffer *rb);
 extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
 extern void perf_event_wakeup(struct perf_event *event);
+extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
+			pgoff_t pgoff, int nr_pages, int flags);
+extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
+
+static inline bool rb_has_aux(struct ring_buffer *rb)
+{
+	return !!rb->aux_nr_pages;
+}
 
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
@@ -81,6 +97,11 @@ static inline unsigned long perf_data_size(struct ring_buffer *rb)
 	return rb->nr_pages << (PAGE_SHIFT + page_order(rb));
 }
 
+static inline unsigned long perf_aux_size(struct ring_buffer *rb)
+{
+	return rb->aux_nr_pages << PAGE_SHIFT;
+}
+
 #define DEFINE_OUTPUT_COPY(func_name, memcpy_func)			\
 static inline unsigned long						\
 func_name(struct perf_output_handle *handle,				\
kernel/events/ring_buffer.c +82 -4

107 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#3 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 23 lines · Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---

kernel/events/ring_buffer.c +1 -0
+void rb_free_aux(struct ring_buffer *rb, struct perf_event *event)

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 294

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+{
+	struct perf_event *iter;
+	int pg;
+
+	if (rb->aux_priv) {
+		/* disable all potential writers before freeing */
+		rcu_read_lock();
+		list_for_each_entry_rcu(iter, &rb->event_list, rb_entry)
+			perf_event_disable(iter);
+		rcu_read_unlock();

Hmm, I cannot remember this from the last time; and its not explained why this was added.

This would change semantics between munmap of a buffer with and without AUX bits in.

Quoted 11 lines
kernel/events/ring_buffer.c +11 -0
+
+		event->pmu->free_aux(rb->aux_priv);
+		rb->aux_priv = NULL;
+	}
+
+	for (pg = 0; pg < rb->aux_nr_pages; pg++)
+		free_page((unsigned long)rb->aux_pages[pg]);
+
+	kfree(rb->aux_pages);
+	rb->aux_nr_pages = 0;
+}
author reviewer
#4 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---

kernel/events/ring_buffer.c +1 -0
+void rb_free_aux(struct ring_buffer *rb, struct perf_event *event)

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 294

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+{
+	struct perf_event *iter;
+	int pg;
+
+	if (rb->aux_priv) {
+		/* disable all potential writers before freeing */
+		rcu_read_lock();
+		list_for_each_entry_rcu(iter, &rb->event_list, rb_entry)
+			perf_event_disable(iter);
+		rcu_read_unlock();

Hmm, I cannot remember this from the last time; and its not explained why this was added.

Quoted 2 lines

This would change semantics between munmap of a buffer with and without AUX bits in.

Indeed, but this seems fair: if an event is generating AUX data while we unmap the AUX buffer, there is no reason to keep it on, because chances are, AUX data is the only thing this event is generating. The alternative would be to have a separate callback for this, but then again, that would race with other pmu callbacks, since it would be called from munmap() path. I should add an elaborate comment about this.

Does this sound reasonable?

Regards,
--
Alex

maintainer
#5 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 43 lines · Alexander Shishkin
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---

kernel/events/ring_buffer.c +1 -0
+void rb_free_aux(struct ring_buffer *rb, struct perf_event *event)

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 294

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+{
+	struct perf_event *iter;
+	int pg;
+
+	if (rb->aux_priv) {
+		/* disable all potential writers before freeing */
+		rcu_read_lock();
+		list_for_each_entry_rcu(iter, &rb->event_list, rb_entry)
+			perf_event_disable(iter);
+		rcu_read_unlock();

Hmm, I cannot remember this from the last time; and its not explained why this was added.

Quoted 2 lines

This would change semantics between munmap of a buffer with and without AUX bits in.

Quoted 8 lines

Indeed, but this seems fair: if an event is generating AUX data while we unmap the AUX buffer, there is no reason to keep it on, because chances are, AUX data is the only thing this event is generating. The alternative would be to have a separate callback for this, but then again, that would race with other pmu callbacks, since it would be called from munmap() path. I should add an elaborate comment about this.

Does this sound reasonable?

Well, the same is true for regular buffers, if you munmap() them while in use the events are pointless.

Still we keep them enabled, we simply skip generating output.

I would much like not to create deviating behaviour if you don't absolutely have to. In both cases its an explicit user request, so you don't need to go hold hands. He did it, he gets to keep whatever pieces it generates.

author reviewer
#6 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 56 lines · Peter Zijlstra
Quoted 43 lines · Alexander Shishkin
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---

kernel/events/ring_buffer.c +1 -0
+void rb_free_aux(struct ring_buffer *rb, struct perf_event *event)

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 294

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+{
+	struct perf_event *iter;
+	int pg;
+
+	if (rb->aux_priv) {
+		/* disable all potential writers before freeing */
+		rcu_read_lock();
+		list_for_each_entry_rcu(iter, &rb->event_list, rb_entry)
+			perf_event_disable(iter);
+		rcu_read_unlock();

Hmm, I cannot remember this from the last time; and its not explained why this was added.

Quoted 2 lines

This would change semantics between munmap of a buffer with and without AUX bits in.

Quoted 8 lines

Indeed, but this seems fair: if an event is generating AUX data while we unmap the AUX buffer, there is no reason to keep it on, because chances are, AUX data is the only thing this event is generating. The alternative would be to have a separate callback for this, but then again, that would race with other pmu callbacks, since it would be called from munmap() path. I should add an elaborate comment about this.

Does this sound reasonable?

Well, the same is true for regular buffers, if you munmap() them while in use the events are pointless.

Quoted 6 lines

Still we keep them enabled, we simply skip generating output.

I would much like not to create deviating behaviour if you don't absolutely have to. In both cases its an explicit user request, so you don't need to go hold hands. He did it, he gets to keep whatever pieces it generates.

Fair enough. Then I'd like to disable the ACTIVE ones before freeing AUX stuff and then re-enabling them since perf_event_{en,dis}able() already provide the convenient cross-cpu calls, which would also avoid concurrency between pmu::{add,del} callbacks and this unmap path. Makes sense?

Regards,
--
Alex

maintainer
#7 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 5 lines · Alexander Shishkin

Fair enough. Then I'd like to disable the ACTIVE ones before freeing AUX stuff and then re-enabling them since perf_event_{en,dis}able() already provide the convenient cross-cpu calls, which would also avoid concurrency between pmu::{add,del} callbacks and this unmap path. Makes sense?

But why? The buffer stuff is RCU freed, so if the hardware observes pages and does get_page_unless_zero() on them its good. The memory will not be freed from underneath the hardware writer because of the get_page().

Then when the buffer is full and we 'swap', we'll find there is no next buffer. At that point we can not provide a new buffer, effectively stopping the hardware writes and release the old buffer, freeing the memory.

author reviewer
#8 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 13 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin

Fair enough. Then I'd like to disable the ACTIVE ones before freeing AUX stuff and then re-enabling them since perf_event_{en,dis}able() already provide the convenient cross-cpu calls, which would also avoid concurrency between pmu::{add,del} callbacks and this unmap path. Makes sense?

But why? The buffer stuff is RCU freed, so if the hardware observes pages and does get_page_unless_zero() on them its good. The memory will not be freed from underneath the hardware writer because of the get_page().

Quoted 4 lines

Then when the buffer is full and we 'swap', we'll find there is no next buffer. At that point we can not provide a new buffer, effectively stopping the hardware writes and release the old buffer, freeing the memory.

There are several problems with this. Firstly, aux buffers can be quite large, which means that we have to do get_page() on thousands of pages on every pmu::add, which is a hot path and free_page() again in pmu::del.

Secondly, all the sg bookkeeping that the driver keeps in aux_priv needs to be refcounted. Right now, in the mmap_close path we just free everything. But if we want to free the aux_pages in pmu::del, we need to keep a list of these pages still around after mmap_close() and same goes for the actual sg tables. I can see a way of doing that on the ring buffer side (as opposed to the driver side), but are you quite sure we should go down this road?

Regards,
--
Alex

maintainer
#9 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 25 lines · Alexander Shishkin
Quoted 13 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin

Fair enough. Then I'd like to disable the ACTIVE ones before freeing AUX stuff and then re-enabling them since perf_event_{en,dis}able() already provide the convenient cross-cpu calls, which would also avoid concurrency between pmu::{add,del} callbacks and this unmap path. Makes sense?

But why? The buffer stuff is RCU freed, so if the hardware observes pages and does get_page_unless_zero() on them its good. The memory will not be freed from underneath the hardware writer because of the get_page().

Quoted 4 lines

Then when the buffer is full and we 'swap', we'll find there is no next buffer. At that point we can not provide a new buffer, effectively stopping the hardware writes and release the old buffer, freeing the memory.

There are several problems with this. Firstly, aux buffers can be quite large, which means that we have to do get_page() on thousands of pages on every pmu::add, which is a hot path and free_page() again in pmu::del.

Quoted 7 lines

Secondly, all the sg bookkeeping that the driver keeps in aux_priv needs to be refcounted. Right now, in the mmap_close path we just free everything. But if we want to free the aux_pages in pmu::del, we need to keep a list of these pages still around after mmap_close() and same goes for the actual sg tables. I can see a way of doing that on the ring buffer side (as opposed to the driver side), but are you quite sure we should go down this road?

No, and I think I realized this after sending that email, but at the time I figured there was another way to do it. Of course, now that several weeks have passed I cannot for the life of me remember what it was.

Lemme go over these patches again to refresh my mind and maybe I'll remember.

author reviewer
#10 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 34 lines · Peter Zijlstra
Quoted 25 lines · Alexander Shishkin
Quoted 13 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin

Fair enough. Then I'd like to disable the ACTIVE ones before freeing AUX stuff and then re-enabling them since perf_event_{en,dis}able() already provide the convenient cross-cpu calls, which would also avoid concurrency between pmu::{add,del} callbacks and this unmap path. Makes sense?

But why? The buffer stuff is RCU freed, so if the hardware observes pages and does get_page_unless_zero() on them its good. The memory will not be freed from underneath the hardware writer because of the get_page().

Quoted 4 lines

Then when the buffer is full and we 'swap', we'll find there is no next buffer. At that point we can not provide a new buffer, effectively stopping the hardware writes and release the old buffer, freeing the memory.

There are several problems with this. Firstly, aux buffers can be quite large, which means that we have to do get_page() on thousands of pages on every pmu::add, which is a hot path and free_page() again in pmu::del.

Quoted 7 lines

Secondly, all the sg bookkeeping that the driver keeps in aux_priv needs to be refcounted. Right now, in the mmap_close path we just free everything. But if we want to free the aux_pages in pmu::del, we need to keep a list of these pages still around after mmap_close() and same goes for the actual sg tables. I can see a way of doing that on the ring buffer side (as opposed to the driver side), but are you quite sure we should go down this road?

Quoted 7 lines

No, and I think I realized this after sending that email, but at the time I figured there was another way to do it. Of course, now that several weeks have passed I cannot for the life of me remember what it was.

Lemme go over these patches again to refresh my mind and maybe I'll remember.

Yes, ring buffer can keep a refcount for the aux_priv object, which is grabbed once at mmap and once at perf_aux_output_begin() and released accordingly and whichever drops the refcount to zero calls pmu::free_aux. No need to grab page->_count and driver is a bit simpler.

Regards,
--
Alex

maintainer
#11 lore raw

Re: [PATCH v4 02/22] perf: Add AUX area to ring buffer for raw data streams

Quoted 7 lines · Alexander Shishkin
Quoted 5 lines · Peter Zijlstra

No, and I think I realized this after sending that email, but at the time I figured there was another way to do it. Of course, now that several weeks have passed I cannot for the life of me remember what it was.

Quoted 7 lines
Quoted 2 lines

Lemme go over these patches again to refresh my mind and maybe I'll remember.

Yes, ring buffer can keep a refcount for the aux_priv object, which is grabbed once at mmap and once at perf_aux_output_begin() and released accordingly and whichever drops the refcount to zero calls pmu::free_aux. No need to grab page->_count and driver is a bit simpler.

Yes, I suppose that'll work just fine.

author reviewer
#12 lore raw

PATCH v4 3/22 perf: Support high-order allocations for AUX space

In reply to Alexander Shishkin

Some pmus (such as BTS or Intel PT without multiple-entry ToPA capability) don't support scatter-gather and will prefer larger contiguous areas for their output regions.

This patch adds a new pmu capability to request higher order allocations.

---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 51 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 46 insertions(+), 6 deletions(-)
include/linux/perf_event.h +1 -0
index fd7b32876c..fe10bf6f94 100644
@@ -171,6 +171,7 @@ struct perf_event;
  * pmu::capabilities flags
  */
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
+#define PERF_PMU_CAP_AUX_NO_SG			0x02
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +45 -6

81 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#13 lore raw

PATCH v4 4/22 perf: Add a capability for AUX_NO_SG pmus to do software double buffering

In reply to Alexander Shishkin

For pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.

---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 15 ++++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index fe10bf6f94..1e7b659b49 100644
@@ -172,6 +172,7 @@ struct perf_event;
  */
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
+#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +14 -1
index d10919ca42..f5ee3669f8 100644
@@ -286,9 +286,22 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -ENOTSUPP;
 
-	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG)
+	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) {
 		order = get_order(nr_pages * PAGE_SIZE);
 
+		/*
+		 * PMU requests more than one contiguous chunks of memory
+		 * for SW double buffering
+		 */
+		if ((event->pmu->capabilities & PERF_PMU_CAP_AUX_SW_DOUBLEBUF) &&
+		    !overwrite) {
+			if (!order)
+				return -EINVAL;
+
+			order--;
+		}
+	}
+
 	rb->aux_pages = kzalloc_node(nr_pages * sizeof(void *), GFP_KERNEL, node);
 	if (!rb->aux_pages)
 		return -ENOMEM;
Signature · 1 line
2.1.0
maintainer
#14 lore raw

Re: [PATCH v4 04/22] perf: Add a capability for AUX_NO_SG pmus to do software double buffering

Quoted 5 lines · Alexander Shishkin

For pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.

Please expand this with more detail on how to use this.

Quoted 32 lines
---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 15 ++++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index fe10bf6f94..1e7b659b49 100644
@@ -172,6 +172,7 @@ struct perf_event;
  */
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
+#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +3 -1
index d10919ca42..f5ee3669f8 100644
@@ -286,9 +286,22 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -ENOTSUPP;
 
-	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG)
+	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) {
 		order = get_order(nr_pages * PAGE_SIZE);
 
+		/*
+		 * PMU requests more than one contiguous chunks of memory

kernel/events/ring_buffer.c int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event, · line 303

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+		 * for SW double buffering
+		 */
+		if ((event->pmu->capabilities & PERF_PMU_CAP_AUX_SW_DOUBLEBUF) &&
+		    !overwrite) {
+			if (!order)
+				return -EINVAL;
+
+			order--;
+		}
+	}

In particular this looks like it will allocate double the total amount of pages and 'loose' half of them. There is no corresponding code in the free path to collect them.

author reviewer
#15 lore raw

Re: [PATCH v4 04/22] perf: Add a capability for AUX_NO_SG pmus to do software double buffering

Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin

For pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.

Please expand this with more detail on how to use this.

Sure.

Quoted 30 lines
Quoted 30 lines
---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 15 ++++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index fe10bf6f94..1e7b659b49 100644
@@ -172,6 +172,7 @@ struct perf_event;
  */
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
+#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +1 -1
index d10919ca42..f5ee3669f8 100644
@@ -286,9 +286,22 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -ENOTSUPP;
 
-	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG)
+	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) {
 		order = get_order(nr_pages * PAGE_SIZE);
 
Quoted 16 lines

kernel/events/ring_buffer.c int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event, · line 303

Quoted 12 lines
kernel/events/ring_buffer.c +12 -0
+		/*
+		 * PMU requests more than one contiguous chunks of memory
+		 * for SW double buffering
+		 */
+		if ((event->pmu->capabilities & PERF_PMU_CAP_AUX_SW_DOUBLEBUF) &&
+		    !overwrite) {
+			if (!order)
+				return -EINVAL;
+
+			order--;
+		}
+	}

In particular this looks like it will allocate double the total amount of pages and 'loose' half of them. There is no corresponding code in the free path to collect them.

This code makes the biggest high order allocation no bigger than half of the total requested size. Then, when I allocate the high-order chunks, I do a split_page() on them and everywhere else in the code they are treated as individual pages, including the free path. So this patch has no implication on freeing. Is this your concern?

Regards,
--
Alex

maintainer
#16 lore raw

Re: [PATCH v4 04/22] perf: Add a capability for AUX_NO_SG pmus to do software double buffering

Quoted 10 lines · Alexander Shishkin
Quoted 3 lines · Peter Zijlstra

In particular this looks like it will allocate double the total amount of pages and 'loose' half of them. There is no corresponding code in the free path to collect them.

This code makes the biggest high order allocation no bigger than half of the total requested size. Then, when I allocate the high-order chunks, I do a split_page() on them and everywhere else in the code they are treated as individual pages, including the free path. So this patch has no implication on freeing. Is this your concern?

Ah right. I completely misread the patch. Sorry for the noise.

author reviewer
#17 lore raw

PATCH v4 5/22 perf: Add a pmu capability for "exclusive" events

In reply to Alexander Shishkin

Usually, pmus that do, for example, instruction tracing, would only ever be able to have one event per task per cpu (or per perf_context). For such pmus it makes sense to disallow creating conflicting events early on, so as to provide consistent behavior for the user.

This patch adds a pmu capability that indicates such constraint on event creation.

---
 include/linux/perf_event.h |  1 +
 kernel/events/core.c       | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
include/linux/perf_event.h +1 -0
index 1e7b659b49..6bd3e743b1 100644
@@ -173,6 +173,7 @@ struct perf_event;
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
 #define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
+#define PERF_PMU_CAP_EXCLUSIVE			0x08
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/core.c +45 -0

69 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#18 lore raw

PATCH v4 6/22 perf: Redirect output from inherited events to parents

In reply to Alexander Shishkin

In order to collect AUX data from an inherited event, we can redirect its output to parent's ring buffer if possible (they must be cpu affine). This patch adds set_output() to the inheritance path.

---
 kernel/events/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/events/core.c +6 -0
index 67f857ab56..e36478564c 100644
@@ -7962,6 +7962,12 @@ inherit_event(struct perf_event *parent_event,
 		= parent_event->overflow_handler_context;
 
 	/*
+	 * Direct child's output to parent's ring buffer (if any)
+	 */
+	if (parent_event->cpu != -1)
+		(void)perf_event_set_output(child_event, parent_event);
+
+	/*
 	 * Precalculate sample_data sizes
 	 */
 	perf_event__header_size(child_event);
Signature · 1 line
2.1.0
maintainer
#19 lore raw

Re: [PATCH v4 06/22] perf: Redirect output from inherited events to parents

Quoted 16 lines · Alexander Shishkin

In order to collect AUX data from an inherited event, we can redirect its output to parent's ring buffer if possible (they must be cpu affine). This patch adds set_output() to the inheritance path.

---
 kernel/events/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/events/core.c +0 -0
index 67f857ab56..e36478564c 100644
@@ -7962,6 +7962,12 @@ inherit_event(struct perf_event *parent_event,
 		= parent_event->overflow_handler_context;
 

kernel/events/core.c inherit_event(struct perf_event *parent_event, · line 7973

Quoted 10 lines
kernel/events/core.c +6 -0
 	/*
+	 * Direct child's output to parent's ring buffer (if any)
+	 */
+	if (parent_event->cpu != -1)
+		(void)perf_event_set_output(child_event, parent_event);
+
+	/*
 	 * Precalculate sample_data sizes
 	 */
 	perf_event__header_size(child_event);

Uhm, nope, see perf_output_begin(), it always redirects output to parent events.

author reviewer
#20 lore raw

Re: [PATCH v4 06/22] perf: Redirect output from inherited events to parents

Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Alexander Shishkin

In order to collect AUX data from an inherited event, we can redirect its output to parent's ring buffer if possible (they must be cpu affine). This patch adds set_output() to the inheritance path.

---
 kernel/events/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/events/core.c +0 -0
index 67f857ab56..e36478564c 100644
@@ -7962,6 +7962,12 @@ inherit_event(struct perf_event *parent_event,
 		= parent_event->overflow_handler_context;
 
Quoted 13 lines

kernel/events/core.c inherit_event(struct perf_event *parent_event, · line 7973

Quoted 10 lines
kernel/events/core.c +6 -0
 	/*
+	 * Direct child's output to parent's ring buffer (if any)
+	 */
+	if (parent_event->cpu != -1)
+		(void)perf_event_set_output(child_event, parent_event);
+
+	/*
 	 * Precalculate sample_data sizes
 	 */
 	perf_event__header_size(child_event);

Uhm, nope, see perf_output_begin(), it always redirects output to parent events.

Ouch, indeed. We'll just do the same in perf_aux_output_begin().

Regards,
--
Alex

author reviewer
#21 lore raw

PATCH v4 7/22 perf: Add api for pmus to write to AUX space

In reply to Alexander Shishkin

For pmus that wish to write data to AUX space, provide perf_aux_output_{begin,end}() calls to initiate/commit data writes, similarly to perf_output_{begin,end}. These also use the same output handle structure.

After the perf_aux_output_begin() returns successfully, handle->size is set to the maximum amount of data that can be written wrt aux_tail pointer, so that no data that the user hasn't seen will be overwritten.

PMU driver should pass the actual amount of data written as a parameter to perf_aux_output_end().

Nested writers are forbidden and guards are in place to catch such attempts.

---
 include/linux/perf_event.h  | 23 ++++++++++++-
 kernel/events/core.c        |  5 ++-
 kernel/events/internal.h    |  4 +++
 kernel/events/ring_buffer.c | 84 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 112 insertions(+), 4 deletions(-)
include/linux/perf_event.h +22 -1
index 6bd3e743b1..63016a0e32 100644
@@ -554,12 +554,22 @@ struct perf_output_handle {
 	struct ring_buffer		*rb;
 	unsigned long			wakeup;
 	unsigned long			size;
-	void				*addr;
+	union {
+		void			*addr;
+		unsigned long		head;
+	};
 	int				page;
 };
 
 #ifdef CONFIG_PERF_EVENTS
 
+extern void *perf_aux_output_begin(struct perf_output_handle *handle,
+				   struct perf_event *event);
+extern void perf_aux_output_end(struct perf_output_handle *handle,
+				unsigned long size, bool truncated);
+extern int perf_aux_output_skip(struct perf_output_handle *handle,
+				unsigned long size);
+extern void *perf_get_aux(struct perf_output_handle *handle);
 extern int perf_pmu_register(struct pmu *pmu, const char *name, int type);
 extern void perf_pmu_unregister(struct pmu *pmu);
 
@@ -816,6 +826,17 @@ extern void perf_event_disable(struct perf_event *event);
 extern int __perf_event_disable(void *info);
 extern void perf_event_task_tick(void);
 #else /* !CONFIG_PERF_EVENTS: */
+static inline void *
+perf_aux_output_begin(struct perf_output_handle *handle,
+		      struct perf_event *event)				{ return NULL; }
+static inline void
+perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+		    bool truncated)					{ }
+static inline int
+perf_aux_output_skip(struct perf_output_handle *handle,
+		     unsigned long size)				{ return -EINVAL; }
+static inline void *
+perf_get_aux(struct perf_output_handle *handle)				{ return NULL; }
 static inline void
 perf_event_task_sched_in(struct task_struct *prev,
 			 struct task_struct *task)			{ }
kernel/events/core.c +2 -3
index e36478564c..9fc9a7583b 100644
@@ -3267,7 +3267,6 @@ static void free_event_rcu(struct rcu_head *head)
 	kfree(event);
 }
 
-static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_attach(struct perf_event *event,
 			       struct ring_buffer *rb);
 
@@ -4047,7 +4046,7 @@ static void rb_free_rcu(struct rcu_head *rcu_head)
 	rb_free(rb);
 }
 
-static struct ring_buffer *ring_buffer_get(struct perf_event *event)
+struct ring_buffer *ring_buffer_get(struct perf_event *event)
 {
 	struct ring_buffer *rb;
 
@@ -4062,7 +4061,7 @@ static struct ring_buffer *ring_buffer_get(struct perf_event *event)
 	return rb;
 }
 
-static void ring_buffer_put(struct ring_buffer *rb)
+void ring_buffer_put(struct ring_buffer *rb)
 {
 	if (!atomic_dec_and_test(&rb->refcount))
 		return;
kernel/events/internal.h +4 -0
index e5374030b1..b8f6c193ea 100644
@@ -36,6 +36,8 @@ struct ring_buffer {
 	struct user_struct		*mmap_user;
 
 	/* AUX area */
+	local_t				aux_head;
+	local_t				aux_nest;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
 	atomic_t			aux_mmap_count;
@@ -54,6 +56,8 @@ extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 			pgoff_t pgoff, int nr_pages, int flags);
 extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
+extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
+extern void ring_buffer_put(struct ring_buffer *rb);
 
 static inline bool rb_has_aux(struct ring_buffer *rb)
 {
kernel/events/ring_buffer.c +84 -0

90 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#22 lore raw

Re: [PATCH v4 07/22] perf: Add api for pmus to write to AUX space

Quoted 7 lines · Alexander Shishkin
kernel/events/ring_buffer.c +0 -0
index f5ee3669f8..3b3a915767 100644
@@ -242,6 +242,90 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)
 	spin_lock_init(&rb->event_lock);
 }

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 253

Quoted 10 lines
kernel/events/ring_buffer.c +9 -0
 
+void *perf_aux_output_begin(struct perf_output_handle *handle,
+			    struct perf_event *event)
+{
+	unsigned long aux_head, aux_tail;
+	struct ring_buffer *rb;
+
+	rb = ring_buffer_get(event);
+	if (!rb)
+		return NULL;

Yeah, no need to much with ring_buffer_get() here, do as perf_output_begin()/end() and keep the RCU section over the entire output. That avoids the atomic and allows you to always use the parent event.

Quoted 19 lines
kernel/events/ring_buffer.c +19 -0
+
+	if (!rb_has_aux(rb))
+		goto err;
+
+	/*
+	 * Nesting is not supported for AUX area, make sure nested
+	 * writers are caught early
+	 */
+	if (WARN_ON_ONCE(local_xchg(&rb->aux_nest, 1)))
+		goto err;
+
+	aux_head = local_read(&rb->aux_head);
+	aux_tail = ACCESS_ONCE(rb->user_page->aux_tail);
+
+	handle->rb = rb;
+	handle->event = event;
+	handle->head = aux_head;
+	if (aux_head - aux_tail < perf_aux_size(rb))
+		handle->size = CIRC_SPACE(aux_head, aux_tail, perf_aux_size(rb));

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 282

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	else
+		handle->size = 0;
+
+	if (!handle->size) {
+		event->pending_disable = 1;
+		event->hw.state = PERF_HES_STOPPED;
+		perf_output_wakeup(handle);
+		local_set(&rb->aux_nest, 0);
+		goto err;
+	}

This needs a comment on the /* A */ barrier; see the comments in perf_output_put_handle() and perf_output_begin().

I'm not sure we can use the same control dependency that we do for the normal buffers since its the hardware doing the stores, not the regular instruction stream.

Please document the order in which the hardware writes vs this software setup and explain the ordering guarantees provided by the hardware wrt regular software.

Quoted 7 lines
kernel/events/ring_buffer.c +7 -0
+	return handle->rb->aux_priv;
+
+err:
+	ring_buffer_put(rb);
+	handle->event = NULL;
+
+	return NULL;

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 300

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+}
+
+void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+			 bool truncated)
+{
+	struct ring_buffer *rb = handle->rb;
+
+	local_add(size, &rb->aux_head);
+
+	smp_wmb();

An uncommented barrier is a bug.

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 308

Quoted 8 lines
kernel/events/ring_buffer.c +8 -0
+	rb->user_page->aux_head = local_read(&rb->aux_head);
+
+	perf_output_wakeup(handle);
+	handle->event = NULL;
+
+	local_set(&rb->aux_nest, 0);
+	ring_buffer_put(rb);
+}

Also, should perf_aux_output_end() not generate an event into the regular buffer?

maintainer
#23 lore raw

Re: [PATCH v4 07/22] perf: Add api for pmus to write to AUX space

Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Alexander Shishkin
kernel/events/ring_buffer.c +0 -0
index f5ee3669f8..3b3a915767 100644
@@ -242,6 +242,90 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)
 	spin_lock_init(&rb->event_lock);
 }
 
Quoted 14 lines

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 253

Quoted 9 lines
kernel/events/ring_buffer.c +9 -0
+void *perf_aux_output_begin(struct perf_output_handle *handle,
+			    struct perf_event *event)
+{
+	unsigned long aux_head, aux_tail;
+	struct ring_buffer *rb;
+
+	rb = ring_buffer_get(event);
+	if (!rb)
+		return NULL;

Yeah, no need to much with ring_buffer_get() here, do as perf_output_begin()/end() and keep the RCU section over the entire output. That avoids the atomic and allows you to always use the parent event.

Ah, I see what you were trying to do. You were thinking of doing that perf_aux_output_begin() when setting up the hardware buffer. Then have the hardware 'fill' it and then call perf_aux_output_end() to commit.

That way you cannot have the RCU section open across and you do indeed need that refcount -- maybe.

Quoted 33 lines
Quoted 19 lines
kernel/events/ring_buffer.c +19 -0
+
+	if (!rb_has_aux(rb))
+		goto err;
+
+	/*
+	 * Nesting is not supported for AUX area, make sure nested
+	 * writers are caught early
+	 */
+	if (WARN_ON_ONCE(local_xchg(&rb->aux_nest, 1)))
+		goto err;
+
+	aux_head = local_read(&rb->aux_head);
+	aux_tail = ACCESS_ONCE(rb->user_page->aux_tail);
+
+	handle->rb = rb;
+	handle->event = event;
+	handle->head = aux_head;
+	if (aux_head - aux_tail < perf_aux_size(rb))
+		handle->size = CIRC_SPACE(aux_head, aux_tail, perf_aux_size(rb));

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 282

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	else
+		handle->size = 0;
+
+	if (!handle->size) {
+		event->pending_disable = 1;
+		event->hw.state = PERF_HES_STOPPED;
+		perf_output_wakeup(handle);
+		local_set(&rb->aux_nest, 0);
+		goto err;
+	}

This needs a comment on the /* A */ barrier; see the comments in perf_output_put_handle() and perf_output_begin().

Quoted 7 lines

I'm not sure we can use the same control dependency that we do for the normal buffers since its the hardware doing the stores, not the regular instruction stream.

Please document the order in which the hardware writes vs this software setup and explain the ordering guarantees provided by the hardware wrt regular software.

So given that the hardware will simply not have a pointer to write to before this completes I think we can say the control dependency is good enough.

Quoted 11 lines
Quoted 11 lines
kernel/events/ring_buffer.c +11 -0
+	return handle->rb->aux_priv;
+
+err:
+	ring_buffer_put(rb);
+	handle->event = NULL;
+
+	return NULL;
+}
+
+void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+			 bool truncated)
Quoted 8 lines

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 300

Quoted 6 lines
kernel/events/ring_buffer.c +6 -0
+{
+	struct ring_buffer *rb = handle->rb;
+
+	local_add(size, &rb->aux_head);
+
+	smp_wmb();

An uncommented barrier is a bug.

Still true, and we need clarification on whether this wmb (a NOP on x86) is sufficient to order against the hardware or if we need something stronger someplace, like a full MFENCE or even SYNC.

Arguably we'd want to place that in the driver before calling this, but we need to get that clarified etc..

author reviewer
#24 lore raw

PATCH v4 8/22 perf: Add AUX record

In reply to Alexander Shishkin

When there's new data in the AUX space, output a record indicating its offset and size and weather it was truncated to fix in the ring buffer.

---
 include/uapi/linux/perf_event.h | 16 ++++++++++++++++
 kernel/events/core.c            | 39 +++++++++++++++++++++++++++++++++++++++
 kernel/events/internal.h        |  3 +++
 kernel/events/ring_buffer.c     |  1 +
 4 files changed, 59 insertions(+)
include/uapi/linux/perf_event.h +16 -0
index 7e0967c0f5..c022c3d756 100644
@@ -733,6 +733,22 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u8				truncated;
+	 *	u8				reserved[7];
+	 *	u64				id;
+	 *	u64				stream_id;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +39 -0
index 9fc9a7583b..0251983018 100644
@@ -5542,6 +5542,45 @@ void perf_event_mmap(struct vm_area_struct *vma)
 	perf_event_mmap_event(&mmap_event);
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, bool truncated)
+{
+	struct perf_output_handle handle;
+	struct perf_sample_data sample;
+	struct perf_aux_event {
+		struct perf_event_header	header;
+		u64				offset;
+		u64				size;
+		u8				truncated;
+		u8				reserved[7];
+		u64				id;
+		u64				stream_id;
+	} rec = {
+		.header = {
+			.type = PERF_RECORD_AUX,
+			.misc = 0,
+			.size = sizeof(rec),
+		},
+		.offset		= head,
+		.size		= size,
+		.truncated	= truncated,
+		.id		= primary_event_id(event),
+		.stream_id	= event->id,
+	};
+	int ret;
+
+	perf_event_header__init_id(&rec.header, &sample, event);
+	ret = perf_output_begin(&handle, event, rec.header.size);
+
+	if (ret)
+		return;
+
+	perf_output_put(&handle, rec);
+	perf_event__output_id_sample(event, &handle, &sample);
+
+	perf_output_end(&handle);
+}
+
 /*
  * IRQ throttle logging
  */
kernel/events/internal.h +3 -0
index b8f6c193ea..c6b2987afe 100644
@@ -64,6 +64,9 @@ static inline bool rb_has_aux(struct ring_buffer *rb)
 	return !!rb->aux_nr_pages;
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, bool truncated);
+
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
 			   struct perf_sample_data *data,
kernel/events/ring_buffer.c +1 -0
index 3b3a915767..925f369947 100644
@@ -296,6 +296,7 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 	struct ring_buffer *rb = handle->rb;
 
 	local_add(size, &rb->aux_head);
+	perf_event_aux_event(handle->event, aux_head, size, truncated);
 
 	smp_wmb();
 	rb->user_page->aux_head = local_read(&rb->aux_head);
Signature · 1 line
2.1.0
maintainer
#25 lore raw

Re: [PATCH v4 08/22] perf: Add AUX record

Quoted 2 lines · Alexander Shishkin

When there's new data in the AUX space, output a record indicating its offset and size and weather it was truncated to fix in the ring buffer.

This patch is too late; it should have been before the patch adding perf_aux_output_*().

I also added acme to cc, he might have wants/needs for the data format I suppose.

Quoted 16 lines
---
 include/uapi/linux/perf_event.h | 16 ++++++++++++++++
 kernel/events/core.c            | 39 +++++++++++++++++++++++++++++++++++++++
 kernel/events/internal.h        |  3 +++
 kernel/events/ring_buffer.c     |  1 +
 4 files changed, 59 insertions(+)
include/uapi/linux/perf_event.h +0 -0
index 7e0967c0f5..c022c3d756 100644
@@ -733,6 +733,22 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 

include/uapi/linux/perf_event.h enum perf_event_type { · line 745

Quoted 10 lines
include/uapi/linux/perf_event.h +10 -0
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u8				truncated;
+	 *	u8				reserved[7];

Creative.. do we want a u64 flags instead? Is there any chance at all we're going to fill out these other bits?

include/uapi/linux/perf_event.h enum perf_event_type { · line 747

Quoted 2 lines
include/uapi/linux/perf_event.h +2 -0
+	 *	u64				id;
+	 *	u64				stream_id;

You probably should have included a struct sample_id there instead.

Quoted 33 lines
include/uapi/linux/perf_event.h +4 -0
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +18 -0
index 9fc9a7583b..0251983018 100644
@@ -5542,6 +5542,45 @@ void perf_event_mmap(struct vm_area_struct *vma)
 	perf_event_mmap_event(&mmap_event);
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, bool truncated)
+{
+	struct perf_output_handle handle;
+	struct perf_sample_data sample;
+	struct perf_aux_event {
+		struct perf_event_header	header;
+		u64				offset;
+		u64				size;
+		u8				truncated;
+		u8				reserved[7];
+		u64				id;
+		u64				stream_id;
+	} rec = {
+		.header = {
+			.type = PERF_RECORD_AUX,
+			.misc = 0,
+			.size = sizeof(rec),

kernel/events/core.c void perf_event_mmap(struct vm_area_struct *vma) · line 5572

Quoted 10 lines
kernel/events/core.c +10 -0
+		},
+		.offset		= head,
+		.size		= size,
+		.truncated	= truncated,
+		.id		= primary_event_id(event),
+		.stream_id	= event->id,
+	};
+	int ret;
+
+	perf_event_header__init_id(&rec.header, &sample, event);

Oh hey, you do actually do the struct sample_id here, so why then also include the id/stream_id again?

Quoted 10 lines
kernel/events/core.c +10 -0
+	ret = perf_output_begin(&handle, event, rec.header.size);
+
+	if (ret)
+		return;
+
+	perf_output_put(&handle, rec);
+	perf_event__output_id_sample(event, &handle, &sample);
+
+	perf_output_end(&handle);
+}
author reviewer
#26 lore raw

PATCH v4 9/22 perf: Support overwrite mode for AUX area

In reply to Alexander Shishkin

This adds support for overwrite mode in the AUX area, which means "keep collecting data till you're stopped". It does not depend on data buffer's overwrite mode, so that it doesn't lose sideband data that is instrumental for processing AUX data.

---
 kernel/events/internal.h    |  1 +
 kernel/events/ring_buffer.c | 43 +++++++++++++++++++++++++++++--------------
 2 files changed, 30 insertions(+), 14 deletions(-)
kernel/events/internal.h +1 -0
index c6b2987afe..4607742be8 100644
@@ -40,6 +40,7 @@ struct ring_buffer {
 	local_t				aux_nest;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
+	int				aux_overwrite;
 	atomic_t			aux_mmap_count;
 	unsigned long			aux_mmap_locked;
 	void				**aux_pages;
kernel/events/ring_buffer.c +29 -14

66 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#27 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

Quoted 4 lines · Alexander Shishkin

This adds support for overwrite mode in the AUX area, which means "keep collecting data till you're stopped". It does not depend on data buffer's overwrite mode, so that it doesn't lose sideband data that is instrumental for processing AUX data.

This patch is out of order as well, it would've made sense right after the other aux buffer patch. Also, its not specified (and I can't find it in the patch) how it _is_ set. The Changelog above only says how it is not.

maintainer
#28 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

In reply to Alexander Shishkin
Quoted 11 lines · Alexander Shishkin
kernel/events/ring_buffer.c +1 -0
index 925f369947..5006caba63 100644
@@ -294,9 +295,22 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 			 bool truncated)
 {
 	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
 

kernel/events/ring_buffer.c void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size, · line 309

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	aux_head = local_read(&rb->aux_head);
+
+	if (rb->aux_overwrite) {
+		local_set(&rb->aux_head, size);
+
+		/*
+		 * Send a RECORD_AUX with size==0 to communicate aux_head
+		 * of this snapshot to userspace
+		 */
+		perf_event_aux_event(handle->event, size, 0, truncated);

Humm.. why not write a 'normal' AUX record?

Also, you didn't mention this in your Changelog _at_all_.

Quoted 7 lines
kernel/events/ring_buffer.c +4 -0
+	} else {
+		local_add(size, &rb->aux_head);
+		perf_event_aux_event(handle->event, aux_head, size, truncated);
+	}
 
 	smp_wmb();
 	rb->user_page->aux_head = local_read(&rb->aux_head);
author reviewer
#29 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Alexander Shishkin
kernel/events/ring_buffer.c +1 -0
index 925f369947..5006caba63 100644
@@ -294,9 +295,22 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 			 bool truncated)
 {
 	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
 
Quoted 12 lines

kernel/events/ring_buffer.c void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size, · line 309

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	aux_head = local_read(&rb->aux_head);
+
+	if (rb->aux_overwrite) {
+		local_set(&rb->aux_head, size);
+
+		/*
+		 * Send a RECORD_AUX with size==0 to communicate aux_head
+		 * of this snapshot to userspace
+		 */
+		perf_event_aux_event(handle->event, size, 0, truncated);

Humm.. why not write a 'normal' AUX record?

In this mode, the hardware is running in a circular buffer mode, overwriting old data, so we don't actually know the size of the snapshot, we have userspace figure it out later on (based on timestamps, for example). I didn't want to configure PMI for this mode to avoid overhead, but with PMI we can try to keep track of the overwrites and try to infer the actual snapshot size in the kernel. For Intel PT. As far as I can tell, ARM's scatter-gather trace-to-memory storing block does not generate interrupts at all.

Quoted 1 line

Also, you didn't mention this in your Changelog _at_all_.

Will do.

Regards,
--
Alex

maintainer
#30 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

Quoted 28 lines · Alexander Shishkin
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Alexander Shishkin
kernel/events/ring_buffer.c +1 -0
index 925f369947..5006caba63 100644
@@ -294,9 +295,22 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 			 bool truncated)
 {
 	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
 
Quoted 12 lines

kernel/events/ring_buffer.c void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size, · line 309

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	aux_head = local_read(&rb->aux_head);
+
+	if (rb->aux_overwrite) {
+		local_set(&rb->aux_head, size);
+
+		/*
+		 * Send a RECORD_AUX with size==0 to communicate aux_head
+		 * of this snapshot to userspace
+		 */
+		perf_event_aux_event(handle->event, size, 0, truncated);

Humm.. why not write a 'normal' AUX record?

Quoted 8 lines

In this mode, the hardware is running in a circular buffer mode, overwriting old data, so we don't actually know the size of the snapshot, we have userspace figure it out later on (based on timestamps, for example). I didn't want to configure PMI for this mode to avoid overhead, but with PMI we can try to keep track of the overwrites and try to infer the actual snapshot size in the kernel. For Intel PT. As far as I can tell, ARM's scatter-gather trace-to-memory storing block does not generate interrupts at all.

Well, wouldn't the 'size' be basically the entire buffer. All you have to then provide is the head pointer. Ideally you would also provide a tail pointer so you know when to stop, but I suppose you can infer that from the data stream itself? If you can provide the tail you can indeed compute the size etc.. at which point you don't have to rely on parsing the stream etc.

author reviewer
#31 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

Quoted 38 lines · Peter Zijlstra
Quoted 28 lines · Alexander Shishkin
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Alexander Shishkin
kernel/events/ring_buffer.c +1 -0
index 925f369947..5006caba63 100644
@@ -294,9 +295,22 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 			 bool truncated)
 {
 	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
 
Quoted 12 lines

kernel/events/ring_buffer.c void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size, · line 309

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	aux_head = local_read(&rb->aux_head);
+
+	if (rb->aux_overwrite) {
+		local_set(&rb->aux_head, size);
+
+		/*
+		 * Send a RECORD_AUX with size==0 to communicate aux_head
+		 * of this snapshot to userspace
+		 */
+		perf_event_aux_event(handle->event, size, 0, truncated);

Humm.. why not write a 'normal' AUX record?

Quoted 8 lines

In this mode, the hardware is running in a circular buffer mode, overwriting old data, so we don't actually know the size of the snapshot, we have userspace figure it out later on (based on timestamps, for example). I didn't want to configure PMI for this mode to avoid overhead, but with PMI we can try to keep track of the overwrites and try to infer the actual snapshot size in the kernel. For Intel PT. As far as I can tell, ARM's scatter-gather trace-to-memory storing block does not generate interrupts at all.

Quoted 2 lines

Well, wouldn't the 'size' be basically the entire buffer. All you have to then provide is the head pointer.

Yes, that's what the code above is doing. We can replace size==0 with size==$buffer_size to mean the same thing.

Quoted 3 lines

Ideally you would also provide a
tail pointer so you know when to stop, but I suppose you can infer that from the data stream itself?

The tail pointer is the problem I mentioned above, because it's either

- where we stopped the previous time
- head+1, if old data is overwritten

and in order to tell the difference, we need an interrupt. We can infer where the new data starts from the timestamps in the trace stream, so the decoder can take care of it (and that's how it's done at the moment).

Quoted 3 lines

If you can provide the tail you can indeed
compute the size etc.. at which point you don't have to rely on parsing the stream etc.

Ideally, there wouldn't be too many adjacent snapshots so it won't even be a problem. But yes, if we want to reliably know the tail/size, we need an interrupt.

Regards,
--
Alex

maintainer
#32 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

Quoted 3 lines · Alexander Shishkin

We can infer where the new data starts from the timestamps in the trace stream, so the decoder can take care of it (and that's how it's done at the moment).

So that means the data stream can be read from arbitrary locations, right? I can imagine not all data streams are always readable like that (for instance the perf datastream is not).

Does it make sense to have the driver provide head,tail for this mode? In your case you can simply provide whatever, but for those where it matters they can ensure consistent data such that the stream might be recoverable.

author reviewer
#33 lore raw

Re: [PATCH v4 09/22] perf: Support overwrite mode for AUX area

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Alexander Shishkin

We can infer where the new data starts from the timestamps in the trace stream, so the decoder can take care of it (and that's how it's done at the moment).

Quoted 8 lines

So that means the data stream can be read from arbitrary locations, right? I can imagine not all data streams are always readable like that (for instance the perf datastream is not).

Does it make sense to have the driver provide head,tail for this mode? In your case you can simply provide whatever, but for those where it matters they can ensure consistent data such that the stream might be recoverable.

Yes, there's no reason not to, and some HW might indeed benefit from this.

Regards,
--
Alex

author reviewer
#34 lore raw

PATCH v4 10/22 perf: Add wakeup watermark control to AUX area

In reply to Alexander Shishkin

When AUX area gets a certain amount of new data, we want to wake up userspace to collect it. This adds a new control to specify how much data will cause a wakeup. This is then passed down to pmu drivers via output handle's "wakeup" field, so that the driver can find the nearest point where it can generate an interrupt.

We repurpose __reserved_2 in the event attribute for this, even though it was never checked to be zero before, aux_watermark will only matter for new AUX-aware code, so the old code should still be fine.

---
 include/uapi/linux/perf_event.h |  7 +++++--
 kernel/events/core.c            |  3 ++-
 kernel/events/internal.h        |  4 +++-
 kernel/events/ring_buffer.c     | 14 +++++++++++---
 4 files changed, 21 insertions(+), 7 deletions(-)
include/uapi/linux/perf_event.h +5 -2
index c022c3d756..507b5e1f5b 100644
@@ -238,6 +238,7 @@ enum perf_event_read_format {
 #define PERF_ATTR_SIZE_VER2	80	/* add: branch_sample_type */
 #define PERF_ATTR_SIZE_VER3	96	/* add: sample_regs_user */
 					/* add: sample_stack_user */
+					/* add: aux_watermark */
 
 /*
  * Hardware event_id to monitor via a performance monitoring event:
@@ -332,8 +333,10 @@ struct perf_event_attr {
 	 */
 	__u32	sample_stack_user;
 
-	/* Align to u64. */
-	__u32	__reserved_2;
+	/*
+	 * Wakeup watermark for AUX area
+	 */
+	__u32	aux_watermark;
 };
 
 #define perf_flags(attr)	(*(&(attr)->read_format + 1))
kernel/events/core.c +2 -1
index 0251983018..c4551ac324 100644
@@ -4356,7 +4356,8 @@ accounting:
 		perf_event_init_userpage(event);
 		perf_event_update_userpage(event);
 	} else {
-		ret = rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, flags);
+		ret = rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages,
+				   event->attr.aux_watermark, flags);
 		if (ret)
 			atomic_dec(&rb->mmap_count);
 		else
kernel/events/internal.h +3 -1
index 4607742be8..4f99987bc3 100644
@@ -27,6 +27,7 @@ struct ring_buffer {
 	local_t				lost;		/* nr records lost   */
 
 	long				watermark;	/* wakeup watermark  */
+	long				aux_watermark;
 	/* poll crap */
 	spinlock_t			event_lock;
 	struct list_head		event_list;
@@ -38,6 +39,7 @@ struct ring_buffer {
 	/* AUX area */
 	local_t				aux_head;
 	local_t				aux_nest;
+	local_t				aux_wakeup;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
 	int				aux_overwrite;
@@ -55,7 +57,7 @@ extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
 extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
-			pgoff_t pgoff, int nr_pages, int flags);
+			pgoff_t pgoff, int nr_pages, long watermark, int flags);
 extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
 extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
 extern void ring_buffer_put(struct ring_buffer *rb);
kernel/events/ring_buffer.c +11 -3
index 5006caba63..27d1dd7cfa 100644
@@ -270,6 +270,7 @@ void *perf_aux_output_begin(struct perf_output_handle *handle,
 	handle->size = 0;
 	if (!rb->aux_overwrite) {
 		aux_tail = ACCESS_ONCE(rb->user_page->aux_tail);
+		handle->wakeup = local_read(&rb->aux_wakeup) + rb->aux_watermark;
 		if (aux_head - aux_tail < perf_aux_size(rb))
 			handle->size = CIRC_SPACE(aux_head, aux_tail, perf_aux_size(rb));
 
@@ -313,9 +314,12 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 	}
 
 	smp_wmb();
-	rb->user_page->aux_head = local_read(&rb->aux_head);
+	aux_head = rb->user_page->aux_head = local_read(&rb->aux_head);
 
-	perf_output_wakeup(handle);
+	if (aux_head - local_read(&rb->aux_wakeup) >= rb->aux_watermark) {
+		perf_output_wakeup(handle);
+		local_add(rb->aux_watermark, &rb->aux_wakeup);
+	}
 	handle->event = NULL;
 
 	local_set(&rb->aux_nest, 0);
@@ -376,7 +380,7 @@ static void rb_free_aux_page(struct ring_buffer *rb, int idx)
 }
 
 int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
-		 pgoff_t pgoff, int nr_pages, int flags)
+		 pgoff_t pgoff, int nr_pages, long watermark, int flags)
 {
 	bool overwrite = !(flags & RING_BUFFER_WRITABLE);
 	int node = (event->cpu == -1) ? -1 : cpu_to_node(event->cpu);
@@ -423,6 +427,10 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	if (rb->aux_priv)
 		ret = 0;
 	rb->aux_overwrite = overwrite;
+	rb->aux_watermark = watermark;
+
+	if (!rb->aux_watermark && !rb->aux_overwrite)
+		rb->aux_watermark = nr_pages << (PAGE_SHIFT - 1);
 
 out:
 	if (!ret)
Signature · 1 line
2.1.0
author reviewer
#35 lore raw

PATCH v4 11/22 perf: add ITRACE_START record to indicate that tracing has started

In reply to Alexander Shishkin

For counters such as instruction tracing, it is useful for the decoder to know which tasks are running when the event is first scheduled in, before the first sched_switch.

To single out such instruction tracing pmus, this patch alse introduces ITRACE PMU capability.

---
 include/linux/perf_event.h      |  4 ++++
 include/uapi/linux/perf_event.h | 11 +++++++++++
 kernel/events/core.c            | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 56 insertions(+)
include/linux/perf_event.h +4 -0
index 63016a0e32..bcfd7a9d84 100644
@@ -127,6 +127,9 @@ struct hw_perf_event {
 			/* for tp_event->class */
 			struct list_head	tp_list;
 		};
+		struct { /* itrace */
+			int			itrace_started;
+		};
 #ifdef CONFIG_HAVE_HW_BREAKPOINT
 		struct { /* breakpoint */
 			/*
@@ -174,6 +177,7 @@ struct perf_event;
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
 #define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
 #define PERF_PMU_CAP_EXCLUSIVE			0x08
+#define PERF_PMU_CAP_ITRACE			0x10
 
 /**
  * struct pmu - generic performance monitoring unit
include/uapi/linux/perf_event.h +11 -0
index 507b5e1f5b..349c261f93 100644
@@ -752,6 +752,17 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_AUX				= 11,
 
+	/*
+	 * Indicates that instruction trace has started
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u32				pid;
+	 *	u32				tid;
+	 * };
+	 */
+	PERF_RECORD_ITRACE_START		= 12,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +41 -0

59 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#36 lore raw

Re: [PATCH v4 11/22] perf: add ITRACE_START record to indicate that tracing has started

Quoted 6 lines · Alexander Shishkin

For counters such as instruction tracing, it is useful for the decoder to know which tasks are running when the event is first scheduled in, before the first sched_switch.

To single out such instruction tracing pmus, this patch alse introduces ITRACE PMU capability.

You forgot to tell why. Also explain why.

The next context switch event will also tell you the previous task. So this seems like superfluous information. Also if you really want this you could simply emit an empty AUX record I suppose, no need to create yet another record type.

author reviewer
#37 lore raw

Re: [PATCH v4 11/22] perf: add ITRACE_START record to indicate that tracing has started

Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Alexander Shishkin

For counters such as instruction tracing, it is useful for the decoder to know which tasks are running when the event is first scheduled in, before the first sched_switch.

To single out such instruction tracing pmus, this patch alse introduces ITRACE PMU capability.

Quoted 6 lines

You forgot to tell why. Also explain why.

The next context switch event will also tell you the previous task. So this seems like superfluous information. Also if you really want this you could simply emit an empty AUX record I suppose, no need to create yet another record type.

Before the first sched_switch we don't actually know what pid/tid is running, so we can't make sense of the beginning of the trace without some event reordering trickery. We can use AUX for that as well, I guess this might be another use case for those flags in the AUX record. Changelog, got it.

Adrian and Arnaldo cc'ed.

Cheers,
--
Alex

author reviewer
#38 lore raw

PATCH v4 12/22 x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection

In reply to Alexander Shishkin

Intel Processor Trace is an architecture extension that allows for program flow tracing.

---
 arch/x86/include/asm/cpufeature.h | 1 +
 arch/x86/kernel/cpu/scattered.c   | 1 +
 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index bb9b258d60..db2debe5bb 100644
@@ -185,6 +185,7 @@
 #define X86_FEATURE_DTHERM	( 7*32+ 7) /* Digital Thermal Sensor */
 #define X86_FEATURE_HW_PSTATE	( 7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface */
+#define X86_FEATURE_INTEL_PT	( 7*32+10) /* Intel Processor Trace */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW  ( 8*32+ 0) /* Intel TPR Shadow */
arch/x86/kernel/cpu/scattered.c +1 -0
index 4a8013d559..42f5fa953f 100644
@@ -36,6 +36,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c)
 		{ X86_FEATURE_ARAT,		CR_EAX, 2, 0x00000006, 0 },
 		{ X86_FEATURE_PLN,		CR_EAX, 4, 0x00000006, 0 },
 		{ X86_FEATURE_PTS,		CR_EAX, 6, 0x00000006, 0 },
+		{ X86_FEATURE_INTEL_PT,		CR_EBX,25, 0x00000007, 0 },
 		{ X86_FEATURE_APERFMPERF,	CR_ECX, 0, 0x00000006, 0 },
 		{ X86_FEATURE_EPB,		CR_ECX, 3, 0x00000006, 0 },
 		{ X86_FEATURE_HW_PSTATE,	CR_EDX, 7, 0x80000007, 0 },
Signature · 1 line
2.1.0
author reviewer
#39 lore raw

PATCH v4 13/22 x86: perf: Intel PT and LBR/BTS are mutually exclusive

In reply to Alexander Shishkin

Intel PT cannot be used at the same time as LBR or BTS and will cause a

general protection fault if they are used together. In order to avoid

fixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.

---
 arch/x86/kernel/cpu/perf_event.h           | 6 ++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  | 8 +++++++-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c | 9 +++++----
 3 files changed, 18 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/perf_event.h +6 -0
index fc5eb390b3..a542fa8a1d 100644
@@ -148,6 +148,7 @@ struct cpu_hw_events {
 	 * Intel DebugStore bits
 	 */
 	struct debug_store	*ds;
+	unsigned int		bts_enabled;
 	u64			pebs_enabled;
 
 	/*
@@ -161,6 +162,11 @@ struct cpu_hw_events {
 	u64				br_sel;
 
 	/*
+	 * Intel Processor Trace
+	 */
+	unsigned int			pt_enabled;
+
+	/*
 	 * Intel host/guest exclude bits
 	 */
 	u64				intel_ctrl_guest_mask;
arch/x86/kernel/cpu/perf_event_intel_ds.c +7 -1
index 9dc4199917..5ae212af23 100644
@@ -455,8 +455,13 @@ struct event_constraint bts_constraint =
 
 void intel_pmu_enable_bts(u64 config)
 {
+	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 	unsigned long debugctlmsr;
 
+	if (cpuc->pt_enabled)
+		return;
+
+	cpuc->bts_enabled = 1;
 	debugctlmsr = get_debugctlmsr();
 
 	debugctlmsr |= DEBUGCTLMSR_TR;
@@ -477,9 +482,10 @@ void intel_pmu_disable_bts(void)
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 	unsigned long debugctlmsr;
 
-	if (!cpuc->ds)
+	if (!cpuc->ds || cpuc->pt_enabled)
 		return;
 
+	cpuc->bts_enabled = 0;
 	debugctlmsr = get_debugctlmsr();
 
 	debugctlmsr &=
arch/x86/kernel/cpu/perf_event_intel_lbr.c +5 -4
index 9dd2459a4c..516e52d0ac 100644
@@ -172,7 +172,9 @@ static void intel_pmu_lbr_reset_64(void)
 
 void intel_pmu_lbr_reset(void)
 {
-	if (!x86_pmu.lbr_nr)
+	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+
+	if (!x86_pmu.lbr_nr || cpuc->pt_enabled)
 		return;
 
 	if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_32)
@@ -185,7 +187,7 @@ void intel_pmu_lbr_enable(struct perf_event *event)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 
-	if (!x86_pmu.lbr_nr)
+	if (!x86_pmu.lbr_nr || cpuc->pt_enabled)
 		return;
 
 	/*
@@ -205,11 +207,10 @@ void intel_pmu_lbr_disable(struct perf_event *event)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 
-	if (!x86_pmu.lbr_nr)
+	if (!x86_pmu.lbr_nr || !cpuc->lbr_users || cpuc->pt_enabled)
 		return;
 
 	cpuc->lbr_users--;
-	WARN_ON_ONCE(cpuc->lbr_users < 0);
 
 	if (cpuc->enabled && !cpuc->lbr_users) {
 		__intel_pmu_lbr_disable();
Signature · 1 line
2.1.0
author reviewer
#40 lore raw

PATCH v4 14/22 x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

Add support for Intel Processor Trace (PT) to kernel's perf events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel.

This driver exports trace data by through AUX space in the perf ring buffer, which is zero-copy mapped into userspace for faster data retrieval.

---
 arch/x86/include/uapi/asm/msr-index.h     |  18 +
 arch/x86/kernel/cpu/Makefile              |   1 +
 arch/x86/kernel/cpu/intel_pt.h            | 129 ++++
 arch/x86/kernel/cpu/perf_event.h          |   2 +
 arch/x86/kernel/cpu/perf_event_intel.c    |   8 +
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 973 ++++++++++++++++++++++++++++++
 6 files changed, 1131 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
arch/x86/include/uapi/asm/msr-index.h +18 -0
index eac9e92fe1..206a4c487f 100644
@@ -74,6 +74,24 @@
 #define MSR_IA32_PERF_CAPABILITIES	0x00000345
 #define MSR_PEBS_LD_LAT_THRESHOLD	0x000003f6
 
+#define MSR_IA32_RTIT_CTL		0x00000570
+#define RTIT_CTL_TRACEEN		BIT(0)
+#define RTIT_CTL_OS			BIT(2)
+#define RTIT_CTL_USR			BIT(3)
+#define RTIT_CTL_CR3EN			BIT(7)
+#define RTIT_CTL_TOPA			BIT(8)
+#define RTIT_CTL_TSC_EN			BIT(10)
+#define RTIT_CTL_DISRETC		BIT(11)
+#define RTIT_CTL_BRANCH_EN		BIT(13)
+#define MSR_IA32_RTIT_STATUS		0x00000571
+#define RTIT_STATUS_CONTEXTEN		BIT(1)
+#define RTIT_STATUS_TRIGGEREN		BIT(2)
+#define RTIT_STATUS_ERROR		BIT(4)
+#define RTIT_STATUS_STOPPED		BIT(5)
+#define MSR_IA32_RTIT_CR3_MATCH		0x00000572
+#define MSR_IA32_RTIT_OUTPUT_BASE	0x00000560
+#define MSR_IA32_RTIT_OUTPUT_MASK	0x00000561
+
 #define MSR_MTRRfix64K_00000		0x00000250
 #define MSR_MTRRfix16K_80000		0x00000258
 #define MSR_MTRRfix16K_A0000		0x00000259
arch/x86/kernel/cpu/Makefile +1 -0
index 7e1fd4e085..00d40f889d 100644
@@ -39,6 +39,7 @@ obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o per
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o perf_event_intel_uncore_snb.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore_snbep.o perf_event_intel_uncore_nhmex.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_rapl.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o
 endif
 
 
arch/x86/kernel/cpu/intel_pt.h new file +129 -0

129 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event.h +2 -0
index a542fa8a1d..2f41d0db42 100644
@@ -749,6 +749,8 @@ void intel_pmu_lbr_init_snb(void);
 
 int intel_pmu_setup_lbr_filter(struct perf_event *event);
 
+void intel_pt_interrupt(void);
+
 int p4_pmu_init(void);
 
 int p6_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel.c +8 -0
index 89bc750efc..6f8025053d 100644
@@ -1399,6 +1399,14 @@ again:
 	}
 
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+
+	/*
 	 * Checkpointed counters can lead to 'spurious' PMIs because the
 	 * rollback caused by the PMI will have cleared the overflow status
 	 * bit. Therefore always force probe these counters.
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +973 -0

973 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#41 lore raw

PATCH v4 15/22 x86: perf: intel_bts: Add BTS PMU driver

In reply to Alexander Shishkin

Add support for Branch Trace Store (BTS) via kernel perf event infrastructure. The difference with the existing implementation of BTS support is that this one is a separate PMU that exports events' trace buffers to userspace by means of AUX area of the perf buffer, which is zero-copy mapped into userspace.

The immediate benefit is that the buffer size can be much bigger, resulting in fewer interrupts and no kernel side copying is involved and little to no trace data loss. Also, kernel code can be traced with this driver.

The old way of collecting BTS traces still works.

---
 arch/x86/kernel/cpu/Makefile               |   2 +-
 arch/x86/kernel/cpu/perf_event.h           |   6 +
 arch/x86/kernel/cpu/perf_event_intel.c     |   6 +-
 arch/x86/kernel/cpu/perf_event_intel_bts.c | 501 +++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |   3 +-
 5 files changed, 515 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
arch/x86/kernel/cpu/Makefile +1 -1
index 00d40f889d..387223c716 100644
@@ -39,7 +39,7 @@ obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o per
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o perf_event_intel_uncore_snb.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore_snbep.o perf_event_intel_uncore_nhmex.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_rapl.o
-obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o perf_event_intel_bts.o
 endif
 
 
arch/x86/kernel/cpu/perf_event.h +6 -0
index 2f41d0db42..73424d3139 100644
@@ -751,6 +751,12 @@ int intel_pmu_setup_lbr_filter(struct perf_event *event);
 
 void intel_pt_interrupt(void);
 
+int intel_bts_interrupt(void);
+
+void intel_bts_enable_local(void);
+
+void intel_bts_disable_local(void);
+
 int p4_pmu_init(void);
 
 int p6_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel.c +5 -1
index 6f8025053d..2f54bcfd6f 100644
@@ -1051,6 +1051,8 @@ static void intel_pmu_disable_all(void)
 
 	if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask))
 		intel_pmu_disable_bts();
+	else
+		intel_bts_disable_local();
 
 	intel_pmu_pebs_disable_all();
 	intel_pmu_lbr_disable_all();
@@ -1073,7 +1075,8 @@ static void intel_pmu_enable_all(int added)
 			return;
 
 		intel_pmu_enable_bts(event->hw.config);
-	}
+	} else
+		intel_bts_enable_local();
 }
 
 /*
@@ -1359,6 +1362,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 		apic_write(APIC_LVTPC, APIC_DM_NMI);
 	intel_pmu_disable_all();
 	handled = intel_pmu_drain_bts_buffer();
+	handled += intel_bts_interrupt();
 	status = intel_pmu_get_status();
 	if (!status)
 		goto done;
arch/x86/kernel/cpu/perf_event_intel_bts.c new file +501 -0

501 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_intel_ds.c +2 -1
index 5ae212af23..f069f1f536 100644
@@ -466,7 +466,8 @@ void intel_pmu_enable_bts(u64 config)
 
 	debugctlmsr |= DEBUGCTLMSR_TR;
 	debugctlmsr |= DEBUGCTLMSR_BTS;
-	debugctlmsr |= DEBUGCTLMSR_BTINT;
+	if (config & ARCH_PERFMON_EVENTSEL_INT)
+		debugctlmsr |= DEBUGCTLMSR_BTINT;
 
 	if (!(config & ARCH_PERFMON_EVENTSEL_OS))
 		debugctlmsr |= DEBUGCTLMSR_BTS_OFF_OS;
Signature · 1 line
2.1.0
author reviewer
#42 lore raw

PATCH v4 16/22 perf: Add rb_{alloc,free}_kernel api

In reply to Alexander Shishkin

Events that generate AUX data can also be created by the kernel. In this case, some in-kernel infrastructure is needed to store and copy this data.

This patch adds api for ring buffer (de-)allocation that can be used by the kernel code.

---
 kernel/events/internal.h    |  3 +++
 kernel/events/ring_buffer.c | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
kernel/events/internal.h +3 -0
index 4f99987bc3..f4cfa4cabb 100644
@@ -59,6 +59,9 @@ extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 			pgoff_t pgoff, int nr_pages, long watermark, int flags);
 extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
+extern struct ring_buffer *
+rb_alloc_kernel(struct perf_event *event, int nr_pages, int aux_nr_pages);
+extern void rb_free_kernel(struct ring_buffer *rb, struct perf_event *event);
 extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
 extern void ring_buffer_put(struct ring_buffer *rb);
 
kernel/events/ring_buffer.c +33 -0
index 27d1dd7cfa..7887601aa5 100644
@@ -464,6 +464,39 @@ void rb_free_aux(struct ring_buffer *rb, struct perf_event *event)
 	rb->aux_nr_pages = 0;
 }
 
+struct ring_buffer *
+rb_alloc_kernel(struct perf_event *event, int nr_pages, int aux_nr_pages)
+{
+	struct ring_buffer *rb;
+	int ret, pgoff = nr_pages + 1;
+
+	rb = rb_alloc(nr_pages, 0, event->cpu, 0);
+	if (!rb)
+		return NULL;
+
+	ret = rb_alloc_aux(rb, event, pgoff, aux_nr_pages, 0, 0);
+	if (ret) {
+		rb_free(rb);
+		return NULL;
+	}
+
+	/*
+	 * Kernel counters don't need ring buffer wakeups, therefore we don't
+	 * use ring_buffer_attach() here and event->rb_entry stays empty
+	 */
+	rcu_assign_pointer(event->rb, rb);
+
+	return rb;
+}
+
+void rb_free_kernel(struct ring_buffer *rb, struct perf_event *event)
+{
+	BUG_ON(atomic_read(&rb->refcount) != 1);
+	rcu_assign_pointer(event->rb, NULL);
+	rb_free_aux(rb, event);
+	rb_free(rb);
+}
+
 #ifndef CONFIG_PERF_USE_VMALLOC
 
 /*
Signature · 1 line
2.1.0
maintainer
#43 lore raw

Re: [PATCH v4 16/22] perf: Add rb_{alloc,free}_kernel api

Quoted 2 lines · Alexander Shishkin

Events that generate AUX data can also be created by the kernel. In this case, some in-kernel infrastructure is needed to store and copy this data.

Oh, do tell.

You really need to work on these changelogs..

author reviewer
#44 lore raw

PATCH v4 17/22 perf: Add a helper to copy AUX data in the kernel

In reply to Alexander Shishkin

This patch adds a helper for the kernel counters that generate AUX data to copy this data around, for example, to output it to the perf ring buffer as a sample record or write it to a core dump file.

---
 kernel/events/internal.h    |  5 +++++
 kernel/events/ring_buffer.c | 31 +++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
kernel/events/internal.h +5 -0
index f4cfa4cabb..bb035dd645 100644
@@ -52,6 +52,9 @@ struct ring_buffer {
 	void				*data_pages[0];
 };
 
+typedef unsigned long (*aux_copyfn)(void *data, const void *src,
+				    unsigned long len);
+
 extern void rb_free(struct ring_buffer *rb);
 extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
@@ -59,6 +62,8 @@ extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 			pgoff_t pgoff, int nr_pages, long watermark, int flags);
 extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
+extern long rb_output_aux(struct ring_buffer *rb, unsigned long from,
+			  unsigned long to, aux_copyfn copyfn, void *data);
 extern struct ring_buffer *
 rb_alloc_kernel(struct perf_event *event, int nr_pages, int aux_nr_pages);
 extern void rb_free_kernel(struct ring_buffer *rb, struct perf_event *event);
kernel/events/ring_buffer.c +31 -0
index 7887601aa5..4cf5a1cd0a 100644
@@ -345,6 +345,37 @@ void *perf_get_aux(struct perf_output_handle *handle)
 	return handle->rb->aux_priv;
 }
 
+long rb_output_aux(struct ring_buffer *rb, unsigned long from,
+		   unsigned long to, aux_copyfn copyfn, void *data)
+{
+	unsigned long tocopy, remainder, len = 0;
+	void *addr;
+
+	from &= (rb->aux_nr_pages << PAGE_SHIFT) - 1;
+	to &= (rb->aux_nr_pages << PAGE_SHIFT) - 1;
+
+	do {
+		tocopy = PAGE_SIZE - offset_in_page(from);
+		if (to > from)
+			tocopy = min(tocopy, to - from);
+		if (!tocopy)
+			break;
+
+		addr = rb->aux_pages[from >> PAGE_SHIFT];
+		addr += offset_in_page(from);
+
+		remainder = copyfn(data, addr, tocopy);
+		if (remainder)
+			return -EFAULT;
+
+		len += tocopy;
+		from += tocopy;
+		from &= (rb->aux_nr_pages << PAGE_SHIFT) - 1;
+	} while (to != from);
+
+	return len;
+}
+
 #define PERF_AUX_GFP	(GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY)
 
 static struct page *rb_alloc_aux_page(int node, int order)
Signature · 1 line
2.1.0
author reviewer
#45 lore raw

PATCH v4 18/22 perf: Add a helper for looking up pmus by type

In reply to Alexander Shishkin

This patch adds a helper for looking up a registered pmu by its type.

---
 kernel/events/core.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
kernel/events/core.c +13 -3
index b82392911a..550c22a2b7 100644
@@ -6930,6 +6930,18 @@ void perf_pmu_unregister(struct pmu *pmu)
 }
 EXPORT_SYMBOL_GPL(perf_pmu_unregister);
 
+/* call under pmus_srcu */
+static struct pmu *__perf_find_pmu(u32 type)
+{
+	struct pmu *pmu;
+
+	rcu_read_lock();
+	pmu = idr_find(&pmu_idr, type);
+	rcu_read_unlock();
+
+	return pmu;
+}
+
 struct pmu *perf_init_event(struct perf_event *event)
 {
 	struct pmu *pmu = NULL;
@@ -6938,9 +6950,7 @@ struct pmu *perf_init_event(struct perf_event *event)
 
 	idx = srcu_read_lock(&pmus_srcu);
 
-	rcu_read_lock();
-	pmu = idr_find(&pmu_idr, event->attr.type);
-	rcu_read_unlock();
+	pmu = __perf_find_pmu(event->attr.type);
 	if (pmu) {
 		if (!try_module_get(pmu->module)) {
 			pmu = ERR_PTR(-ENODEV);
Signature · 1 line
2.1.0
author reviewer
#46 lore raw

PATCH v4 19/22 perf: Add infrastructure for using AUX data in perf samples

In reply to Alexander Shishkin

AUX data can be used to annotate other perf events by including it in sample records when PERF_SAMPLE_AUX flag is set. In this case, a kernel counter is created for each such event and trace data is retrieved from it and stored in the perf data stream.

To this end, new attribute fields are added:
* aux_sample_type: specify PMU on which the AUX data generating event

                     is created;

* aux_sample_config: event config (maps to attribute's config field),
* aux_sample_size: size of the sample to be written.

This kernel counter is configured similarly to its "main" event with regards to filtering (exclude_{hv,idle,user,kernel}) and enabled state (disabled, enable_on_exec) to make sure that we don't get out of context AUX traces.

---
 include/linux/perf_event.h      |   9 +++
 include/uapi/linux/perf_event.h |  18 ++++-
 kernel/events/core.c            | 172 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/linux/perf_event.h +9 -0
index bcfd7a9d84..8731325405 100644
@@ -84,6 +84,12 @@ struct perf_regs_user {
 	struct pt_regs	*regs;
 };
 
+struct perf_aux_record {
+	u64		size;
+	unsigned long	from;
+	unsigned long	to;
+};
+
 struct task_struct;
 
 /*
@@ -457,6 +463,7 @@ struct perf_event {
 	perf_overflow_handler_t		overflow_handler;
 	void				*overflow_handler_context;
 
+	struct perf_event		*sampler;
 #ifdef CONFIG_EVENT_TRACING
 	struct ftrace_event_call	*tp_event;
 	struct event_filter		*filter;
@@ -627,6 +634,7 @@ struct perf_sample_data {
 	union  perf_mem_data_src	data_src;
 	struct perf_callchain_entry	*callchain;
 	struct perf_raw_record		*raw;
+	struct perf_aux_record		aux;
 	struct perf_branch_stack	*br_stack;
 	struct perf_regs_user		regs_user;
 	u64				stack_user_size;
@@ -654,6 +662,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data,
 	data->period = period;
 	data->regs_user.abi = PERF_SAMPLE_REGS_ABI_NONE;
 	data->regs_user.regs = NULL;
+	data->aux.from = data->aux.to = data->aux.size = 0;
 	data->stack_user_size = 0;
 	data->weight = 0;
 	data->data_src.val = PERF_MEM_NA;
include/uapi/linux/perf_event.h +17 -1
index 349c261f93..b24f170abf 100644
@@ -137,8 +137,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_DATA_SRC			= 1U << 15,
 	PERF_SAMPLE_IDENTIFIER			= 1U << 16,
 	PERF_SAMPLE_TRANSACTION			= 1U << 17,
+	PERF_SAMPLE_AUX				= 1U << 18,
 
-	PERF_SAMPLE_MAX = 1U << 18,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 19,		/* non-ABI */
 };
 
 /*
@@ -239,6 +240,9 @@ enum perf_event_read_format {
 #define PERF_ATTR_SIZE_VER3	96	/* add: sample_regs_user */
 					/* add: sample_stack_user */
 					/* add: aux_watermark */
+#define PERF_ATTR_SIZE_VER4	120	/* add: aux_sample_config */
+					/* add: aux_sample_size */
+					/* add: aux_sample_type */
 
 /*
  * Hardware event_id to monitor via a performance monitoring event:
@@ -337,6 +341,16 @@ struct perf_event_attr {
 	 * Wakeup watermark for AUX area
 	 */
 	__u32	aux_watermark;
+
+	/*
+	 * Itrace pmus' event config
+	 */
+	__u64	aux_sample_config;	/* event config for AUX sampling */
+	__u64	aux_sample_size;	/* desired sample size */
+	__u32	aux_sample_type;	/* pmu->type of an AUX PMU */
+
+	/* Align to u64. */
+	__u32	__reserved_2;
 };
 
 #define perf_flags(attr)	(*(&(attr)->read_format + 1))
@@ -710,6 +724,8 @@ enum perf_event_type {
 	 *	{ u64			weight;   } && PERF_SAMPLE_WEIGHT
 	 *	{ u64			data_src; } && PERF_SAMPLE_DATA_SRC
 	 *	{ u64			transaction; } && PERF_SAMPLE_TRANSACTION
+	 *	{ u64			size;
+	 *	  char			data[size]; } && PERF_SAMPLE_AUX
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +172 -0

220 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#47 lore raw

Re: [PATCH v4 19/22] perf: Add infrastructure for using AUX data in perf samples

Quoted 7 lines · Alexander Shishkin

AUX data can be used to annotate other perf events by including it in sample records when PERF_SAMPLE_AUX flag is set. In this case, a kernel counter is created for each such event and trace data is retrieved from it and stored in the perf data stream.

To this end, new attribute fields are added:
* aux_sample_type: specify PMU on which the AUX data generating event

Quoted 8 lines
                     is created;

* aux_sample_config: event config (maps to attribute's config field),
* aux_sample_size: size of the sample to be written.

This kernel counter is configured similarly to its "main" event with regards to filtering (exclude_{hv,idle,user,kernel}) and enabled state (disabled, enable_on_exec) to make sure that we don't get out of context AUX traces.

WHY? Why would we want to do this? This doesn't tell me why I should want this or even consider it.

Convince me I want to read the patch.

author reviewer
#48 lore raw

PATCH v4 20/22 perf: Allocate ring buffers for inherited per-task kernel events

In reply to Alexander Shishkin

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or whatnot.

---
 kernel/events/core.c     |  9 +++++++++
 kernel/events/internal.h | 11 +++++++++++
 2 files changed, 20 insertions(+)
kernel/events/core.c +9 -0
index 3b1550fd0e..9d7f6086d5 100644
@@ -8230,6 +8230,15 @@ inherit_event(struct perf_event *parent_event,
 		(void)perf_event_set_output(child_event, parent_event);
 
 	/*
+	 * For per-task kernel events with ring buffers, set_output doesn't
+	 * make sense, but we can allocate a new buffer here.
+	 */
+	if (parent_event->cpu == -1 && kernel_rb_event(parent_event)) {
+		(void)rb_alloc_kernel(child_event, parent_event->rb->nr_pages,
+				      parent_event->rb->aux_nr_pages);
+	}
+
+	/*
 	 * Precalculate sample_data sizes
 	 */
 	perf_event__header_size(child_event);
kernel/events/internal.h +11 -0
index bb035dd645..b306bc9307 100644
@@ -120,6 +120,17 @@ static inline unsigned long perf_aux_size(struct ring_buffer *rb)
 	return rb->aux_nr_pages << PAGE_SHIFT;
 }
 
+static inline bool kernel_rb_event(struct perf_event *event)
+{
+	/*
+	 * Having a ring buffer and not being on any ring buffers' wakeup
+	 * list means it was attached by rb_alloc_kernel() and not
+	 * ring_buffer_attach(). It's the only case when these two
+	 * conditions take place at the same time.
+	 */
+	return event->rb && list_empty(&event->rb_entry);
+}
+
 #define DEFINE_OUTPUT_COPY(func_name, memcpy_func)			\
 static inline unsigned long						\
 func_name(struct perf_output_handle *handle,				\
Signature · 1 line
2.1.0
maintainer
#49 lore raw

Re: [PATCH v4 20/22] perf: Allocate ring buffers for inherited per-task kernel events

Quoted 5 lines · Alexander Shishkin

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or whatnot.

No.

author reviewer
#50 lore raw

PATCH v4 21/22 perf: Allow AUX sampling for multiple events

In reply to Alexander Shishkin

Right now, only one perf event can be annotated with AUX data, however it should be possible to annotate several events with similar configuration (wrt exclude_{hv,idle,user,kernel}, config and other event attribute fields).

So every time, before a kernel counter is created for AUX sampling, we first look for an existing counter that is suitable in terms of configuration and we use it to annotate the new event as well.

---
 kernel/events/core.c | 93 +++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 66 insertions(+), 27 deletions(-)
kernel/events/core.c +66 -27
index 9d7f6086d5..73f6f5a5b7 100644
@@ -4613,45 +4613,84 @@ static void perf_aux_sampler_destroy(struct perf_event *event)
 	ring_buffer_put(rb); /* can be last */
 }
 
+static bool exclusive_event_match(struct perf_event *e1, struct perf_event *e2);
+
+static bool perf_aux_event_match(struct perf_event *e1, struct perf_event *e2)
+{
+	return has_aux(e1) && exclusive_event_match(e1, e2);
+}
+
 static int perf_aux_sampler_init(struct perf_event *event,
 				 struct task_struct *task,
 				 struct pmu *pmu)
 {
+	struct perf_event_context *ctx;
 	struct perf_event_attr attr;
-	struct perf_event *sampler;
+	struct perf_event *sampler = NULL;
 	struct ring_buffer *rb;
-	unsigned long nr_pages;
+	unsigned long nr_pages, flags;
 
 	if (!pmu || !(pmu->setup_aux))
 		return -ENOTSUPP;
 
-	memset(&attr, 0, sizeof(attr));
-	attr.type = pmu->type;
-	attr.config = event->attr.aux_sample_config;
-	attr.sample_type = 0;
-	attr.disabled = event->attr.disabled;
-	attr.enable_on_exec = event->attr.enable_on_exec;
-	attr.exclude_hv = event->attr.exclude_hv;
-	attr.exclude_idle = event->attr.exclude_idle;
-	attr.exclude_user = event->attr.exclude_user;
-	attr.exclude_kernel = event->attr.exclude_kernel;
-	attr.aux_sample_size = event->attr.aux_sample_size;
-
-	sampler = perf_event_create_kernel_counter(&attr, event->cpu, task,
-						   NULL, NULL);
-	if (IS_ERR(sampler))
-		return PTR_ERR(sampler);
-
-	nr_pages = 1ul << __get_order(event->attr.aux_sample_size);
-
-	rb = rb_alloc_kernel(sampler, 0, nr_pages);
-	if (!rb) {
-		perf_event_release_kernel(sampler);
-		return -ENOMEM;
+	ctx = find_get_context(pmu, task, event->cpu);
+	if (ctx) {
+		raw_spin_lock_irqsave(&ctx->lock, flags);
+		list_for_each_entry(sampler, &ctx->event_list, event_entry) {
+			/*
+			 * event is not an aux event, but all the relevant
+			 * bits should match
+			 */
+			if (perf_aux_event_match(sampler, event) &&
+			    sampler->attr.type == event->attr.aux_sample_type &&
+			    sampler->attr.config == event->attr.aux_sample_config &&
+			    sampler->attr.exclude_hv == event->attr.exclude_hv &&
+			    sampler->attr.exclude_idle == event->attr.exclude_idle &&
+			    sampler->attr.exclude_user == event->attr.exclude_user &&
+			    sampler->attr.exclude_kernel == event->attr.exclude_kernel &&
+			    sampler->attr.aux_sample_size >= event->attr.aux_sample_size &&
+			    atomic_long_inc_not_zero(&sampler->refcount))
+				goto got_event;
+		}
+
+		sampler = NULL;
+
+got_event:
+		--ctx->pin_count;
+		put_ctx(ctx);
+		raw_spin_unlock_irqrestore(&ctx->lock, flags);
+	}
+
+	if (!sampler) {
+		memset(&attr, 0, sizeof(attr));
+		attr.type = pmu->type;
+		attr.config = event->attr.aux_sample_config;
+		attr.sample_type = 0;
+		attr.disabled = event->attr.disabled;
+		attr.enable_on_exec = event->attr.enable_on_exec;
+		attr.exclude_hv = event->attr.exclude_hv;
+		attr.exclude_idle = event->attr.exclude_idle;
+		attr.exclude_user = event->attr.exclude_user;
+		attr.exclude_kernel = event->attr.exclude_kernel;
+		attr.aux_sample_size = event->attr.aux_sample_size;
+
+		sampler = perf_event_create_kernel_counter(&attr, event->cpu,
+							   task, NULL, NULL);
+		if (IS_ERR(sampler))
+			return PTR_ERR(sampler);
+
+		nr_pages = 1ul << __get_order(event->attr.aux_sample_size);
+
+		rb = rb_alloc_kernel(sampler, 0, nr_pages);
+		if (!rb) {
+			perf_event_release_kernel(sampler);
+			return -ENOMEM;
+		}
+
+		sampler->destroy = perf_aux_sampler_destroy;
 	}
 
 	event->sampler = sampler;
-	sampler->destroy = perf_aux_sampler_destroy;
 
 	return 0;
 }
@@ -4664,7 +4703,7 @@ static void perf_aux_sampler_fini(struct perf_event *event)
 	if (!sampler)
 		return;
 
-	perf_event_release_kernel(sampler);
+	put_event(sampler);
 
 	event->sampler = NULL;
 }
Signature · 1 line
2.1.0
author reviewer
#51 lore raw

PATCH v4 22/22 perf: Allow sampling of inherited events

In reply to Alexander Shishkin

Try to find an AUX sampler event for the current event if none is linked. This is useful when these events are allocated by inheritance path, independently of one another.

---
 kernel/events/core.c | 94 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 62 insertions(+), 32 deletions(-)
kernel/events/core.c +62 -32
index 73f6f5a5b7..c59a596c8f 100644
@@ -4620,47 +4620,67 @@ static bool perf_aux_event_match(struct perf_event *e1, struct perf_event *e2)
 	return has_aux(e1) && exclusive_event_match(e1, e2);
 }
 
+struct perf_event *__find_sampling_counter(struct perf_event_context *ctx,
+					   struct perf_event *event,
+					   struct task_struct *task)
+{
+	struct perf_event *sampler = NULL;
+
+	list_for_each_entry(sampler, &ctx->event_list, event_entry) {
+		/*
+		 * event is not an itrace event, but all the relevant
+		 * bits should match
+		 */
+		if (perf_aux_event_match(sampler, event) &&
+		    kernel_rb_event(sampler) &&
+		    sampler->attr.type == event->attr.aux_sample_type &&
+		    sampler->attr.config == event->attr.aux_sample_config &&
+		    sampler->attr.exclude_hv == event->attr.exclude_hv &&
+		    sampler->attr.exclude_idle == event->attr.exclude_idle &&
+		    sampler->attr.exclude_user == event->attr.exclude_user &&
+		    sampler->attr.exclude_kernel == event->attr.exclude_kernel &&
+		    sampler->attr.aux_sample_size >= event->attr.aux_sample_size &&
+		    atomic_long_inc_not_zero(&sampler->refcount))
+			return sampler;
+	}
+
+	return NULL;
+}
+
+struct perf_event *find_sampling_counter(struct pmu *pmu,
+					 struct perf_event *event,
+					 struct task_struct *task)
+{
+	struct perf_event_context *ctx;
+	struct perf_event *sampler = NULL;
+	unsigned long flags;
+
+	ctx = find_get_context(pmu, task, event->cpu);
+	if (!ctx)
+		return NULL;
+
+	raw_spin_lock_irqsave(&ctx->lock, flags);
+	sampler = __find_sampling_counter(ctx, event, task);
+	--ctx->pin_count;
+	put_ctx(ctx);
+	raw_spin_unlock_irqrestore(&ctx->lock, flags);
+
+	return sampler;
+}
+
 static int perf_aux_sampler_init(struct perf_event *event,
 				 struct task_struct *task,
 				 struct pmu *pmu)
 {
-	struct perf_event_context *ctx;
 	struct perf_event_attr attr;
-	struct perf_event *sampler = NULL;
+	struct perf_event *sampler;
 	struct ring_buffer *rb;
-	unsigned long nr_pages, flags;
+	unsigned long nr_pages;
 
 	if (!pmu || !(pmu->setup_aux))
 		return -ENOTSUPP;
 
-	ctx = find_get_context(pmu, task, event->cpu);
-	if (ctx) {
-		raw_spin_lock_irqsave(&ctx->lock, flags);
-		list_for_each_entry(sampler, &ctx->event_list, event_entry) {
-			/*
-			 * event is not an aux event, but all the relevant
-			 * bits should match
-			 */
-			if (perf_aux_event_match(sampler, event) &&
-			    sampler->attr.type == event->attr.aux_sample_type &&
-			    sampler->attr.config == event->attr.aux_sample_config &&
-			    sampler->attr.exclude_hv == event->attr.exclude_hv &&
-			    sampler->attr.exclude_idle == event->attr.exclude_idle &&
-			    sampler->attr.exclude_user == event->attr.exclude_user &&
-			    sampler->attr.exclude_kernel == event->attr.exclude_kernel &&
-			    sampler->attr.aux_sample_size >= event->attr.aux_sample_size &&
-			    atomic_long_inc_not_zero(&sampler->refcount))
-				goto got_event;
-		}
-
-		sampler = NULL;
-
-got_event:
-		--ctx->pin_count;
-		put_ctx(ctx);
-		raw_spin_unlock_irqrestore(&ctx->lock, flags);
-	}
-
+	sampler = find_sampling_counter(pmu, event, task);
 	if (!sampler) {
 		memset(&attr, 0, sizeof(attr));
 		attr.type = pmu->type;
@@ -4711,9 +4731,19 @@ static void perf_aux_sampler_fini(struct perf_event *event)
 static unsigned long perf_aux_sampler_trace(struct perf_event *event,
 					    struct perf_sample_data *data)
 {
-	struct perf_event *sampler = event->sampler;
+	struct perf_event *sampler;
 	struct ring_buffer *rb;
 
+	if (!event->sampler) {
+		/*
+		 * down this path, event->ctx is already locked IF it's the
+		 * same context
+		 */
+		event->sampler = __find_sampling_counter(event->ctx, event,
+							 event->ctx->task);
+	}
+
+	sampler = event->sampler;
 	if (!sampler || sampler->state != PERF_EVENT_STATE_ACTIVE) {
 		data->aux.size = 0;
 		goto out;
Signature · 1 line
2.1.0
reviewer
#52 lore raw
In reply to Alexander Shishkin
Quoted 4 lines · Alexander Shishkin

This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.

Alex is away, so I would like to know if anyone (Peter?) plans to review or comment on these patches.

maintainer
#53 lore raw
Quoted 8 lines · Adrian Hunter
Quoted 4 lines · Alexander Shishkin

This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.

Alex is away, so I would like to know if anyone (Peter?) plans to review or comment on these patches.

I'll get to it (eventually), I had a week of down-time after KS/Linuxcon and have now returned to a copiously overflown inbox.

maintainer
#54 lore raw
In reply to Alexander Shishkin
Quoted 47 lines · Alexander Shishkin

Hi Peter and all,

This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.

The single most notable thing is that while PT outputs trace data in a compressed binary format, it will still generate hundreds of megabytes of trace data per second per core. Decoding this binary stream takes 2-3 orders of magnitude the cpu time that it takes to generate it. These considerations make it impossible to carry out decoding in kernel space. Therefore, the trace data is exported to userspace as a zero-copy mapping that userspace can collect and store for later decoding. To address this, this patchset extends perf ring buffer with an "AUX space", which is allocated for hardware blocks such as PT to export their trace data with minimal overhead. This space can be configured via buffer's user page and mmapped from the same file descriptor with a given offset. Data can then be collected from it by reading the aux_head (write) pointer from the user page and updating aux_tail (read) pointer similarly to data_{head,tail} of the traditional perf buffer. There is an api between perf core and pmu drivers that wish to make use of this AUX space to export their data.

For tracing blocks that don't support hardware scatter-gather tables, we provide high-order physically contiguous allocations to minimize the overhead needed for software double buffering and PMI pressure.

This way we get a normal perf data stream that provides sideband information that is required to decode the trace data, such as MMAPs, COMMs etc, plus the actual trace in its own logical space.

If the trace buffer is mapped writable, the driver will stop tracing when it fills up (aux_head approaches aux_tail), till data is read, aux_tail pointer is moved forward and an ioctl() is issued to re-enable tracing. If the trace buffer is mapped read only, the tracing will continue, overwriting older data, so that the buffer always contains the most recent data. Tracing can be stopped with an ioctl() and restarted once the data is collected.

Another use case is annotating samples of other perf events: setting PERF_SAMPLE_AUX requests attr.aux_sample_size bytes of trace to be included in each event's sample.

This patchset consists of necessary changes to the perf kernel infrastructure, and PT and BTS pmu drivers. The tooling support is not included in this series, however, it can be found in my github tree [2].

Quoted 6 lines

This version changes the way watermarks are handled for AUX area and gets rid of the notion of "itrace" both in the core and in the perf interface (event attribute), which makes it more logical.

[1] http://software.intel.com/en-us/intel-isa-extensions
[2] http://github.com/virtuoso/linux-perf/tree/intel_pt

It would also be good if some other archs can comment on this (the generic parts obviously). There is the ARM CoreSight stuff and ISTR that MIPS also has something like this, although I'm not entirely sure who to poke on that, Ralf?

#55 lore raw
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Alexander Shishkin

This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.

I'm particularly excited about the "to the perf kernel infrastructure" :-)

Quoted 2 lines

It would also be good if some other archs can comment on this (the generic parts obviously). There is the ARM CoreSight stuff

We've got Mathieu Poirier at Linaro working on the drivers for CoreSight components, with the latest (v5) version here:

http://thread.gmane.org/gmane.linux.ports.arm.kernel/351654

but it was mostly intended to have the processor trace decoded using a separate tool in userspace. I must say that an idea of getting it all into perf sounds really interesting.

I've copied Mathieu so he can have a look as well. As for me personally, I'd like to have 48 hours to digest it all. I've just got back from a very distant time zone and am suffering a monumental jet lag...

Paweł

A message that is not in the archive (sent off-list or not yet received): CANLsYky0vuwo7MwKbiGXypkLkrX7k6BOEf2uej3-Z3-HZHKd7w@mail.gmail.com

maintainer
#57 lore raw
Quoted 8 lines · Mathieu Poirier

Pawell, many thanks for looping me in.

I am definitely not a perf-internal guru and as such won't be able to comment on the implementation. On the flip side it is easy for me to see how the work on coresight done at Linaro can be made to tie-in what Alexander is proposing. Albeit not at the top of the priority list at this time, integration with perf (and ftrace) is definitely on the roadmap.

Quoted 8 lines

Powell is correct in his statement that Linaro's work in HW trace decoding is (currently) mainly focused on processor tracing but that will change when we have the basic infrastructure upstreamed.

Last but not least it would be interesting to have more information on the "sideband data". With coresight we have something called "metadata", also related to how the trace source was configured and instrumental to proper trace decoding. I'm pretty sure we are facing the same problems.

So we use the sideband or AUX data stream to export the 'big' data stream generated by the CPU in an opaque manner. For every AUX data block 'posted' we issue an event into the regular data buffer that describes it.

I was assuming that both ARM and MIPS would generate a single data stream as well. So please do tell more about your meta-data; is that a one time thing or a second continuous stream of data, albeit smaller than the main stream?

The way I read your explanation it is a one time blob generated once you setup the hardware. I suppose we could either dump it once into the normal data stream or maybe dump it once every time we generate an AUX buffer event into the normal data stream -- if its not too big.

In any case, can you point us to public documentation of the ARM CoreSight stuff and maybe provide a short summary for the tl;dr crowd?

#58 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Mathieu Poirier

Pawell, many thanks for looping me in.

I am definitely not a perf-internal guru and as such won't be able to comment on the implementation. On the flip side it is easy for me to see how the work on coresight done at Linaro can be made to tie-in what Alexander is proposing. Albeit not at the top of the priority list at this time, integration with perf (and ftrace) is definitely on the roadmap.

Quoted 8 lines

Powell is correct in his statement that Linaro's work in HW trace decoding is (currently) mainly focused on processor tracing but that will change when we have the basic infrastructure upstreamed.

Last but not least it would be interesting to have more information on the "sideband data". With coresight we have something called "metadata", also related to how the trace source was configured and instrumental to proper trace decoding. I'm pretty sure we are facing the same problems.

Quoted 4 lines

So we use the sideband or AUX data stream to export the 'big' data stream generated by the CPU in an opaque manner. For every AUX data block 'posted' we issue an event into the regular data buffer that describes it.

In the context of "describe it" written above, what kind of information one would typically find in that description?

Quoted 5 lines

I was assuming that both ARM and MIPS would generate a single data stream as well. So please do tell more about your meta-data; is that a one time thing or a second continuous stream of data, albeit smaller than the main stream?

Coresight does indeed generate a single steam of compressed data. Depending on the tracing specifics that were requested by the use case (trace engine configuration) the format of the packets in the compressed stream will change. Since the compressed stream itself doesn't carry clues about the formatting information, knowledge of how the trace engine was configured is mandatory for the proper decoding of the trace stream.

Metadata refer to exactly that - the configuration of the trace engine. It has to be somehow lumped with the trace stream for off target analysis.

Quoted 3 lines

The way I read your explanation it is a one time blob generated once you setup the hardware.

Correct.

Quoted 3 lines

I suppose we could either dump it once into the
normal data stream or maybe dump it once every time we generate an AUX buffer event into the normal data stream -- if its not too big.

Right, there is a set of meta-data to be generated with each trace run. With the current implementation a "trace run" pertains to all the information collected between the beginning and end of a trace scenario. Future work involve triggering a DMA transfer of the full coresight buffer to a kernel memory area, something that is probably close to the "buffer event" you are referring to.

When we get there I also agree that metadata should be lumped with each buffer event, removing the need to have the first buffer in the set to be able to decode all the other buffers.

Quoted 3 lines

In any case, can you point us to public documentation of the ARM CoreSight stuff and maybe provide a short summary for the tl;dr crowd?

Absolutely.

Information on metadata: [1]
Current coresight patchset: [2]
Quick summary on coresight: [3]
Technical details on Coresight: [4]

Best regards,
Mathieu

[1]. http://people.linaro.org/~mathieu.poirier/coresight/cs-decode.pdf
[2]. https://lkml.org/lkml/2014/8/27/479
[3]. https://lkml.org/lkml/2014/8/27/467
[4]. http://infocenter.arm.com/help/index.jsp (section "CoreSight on-chip trace and debug")

author reviewer
#59 lore raw
Quoted 19 lines · Mathieu Poirier
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Mathieu Poirier

Pawell, many thanks for looping me in.

I am definitely not a perf-internal guru and as such won't be able to comment on the implementation. On the flip side it is easy for me to see how the work on coresight done at Linaro can be made to tie-in what Alexander is proposing. Albeit not at the top of the priority list at this time, integration with perf (and ftrace) is definitely on the roadmap.

Powell is correct in his statement that Linaro's work in HW trace decoding is (currently) mainly focused on processor tracing but that will change when we have the basic infrastructure upstreamed.

Last but not least it would be interesting to have more information on the "sideband data". With coresight we have something called "metadata", also related to how the trace source was configured and instrumental to proper trace decoding. I'm pretty sure we are facing the same problems.

Quoted 7 lines
Quoted 4 lines

So we use the sideband or AUX data stream to export the 'big' data stream generated by the CPU in an opaque manner. For every AUX data block 'posted' we issue an event into the regular data buffer that describes it.

In the context of "describe it" written above, what kind of information one would typically find in that description?

It's like "got a chunk of AUX data in the AUX buffer, starting at offset $X, length $Y".

Quoted 6 lines
Quoted 5 lines

I was assuming that both ARM and MIPS would generate a single data stream as well. So please do tell more about your meta-data; is that a one time thing or a second continuous stream of data, albeit smaller than the main stream?

Quoted 7 lines

Coresight does indeed generate a single steam of compressed data. Depending on the tracing specifics that were requested by the use case (trace engine configuration) the format of the packets in the compressed stream will change. Since the compressed stream itself doesn't carry clues about the formatting information, knowledge of how the trace engine was configured is mandatory for the proper decoding of the trace stream.

Ok, in perf the trace configuration would be part of 'session' information, so the way the tracing was configured by userspace will be saved to the resulting trace file (perf.data) by the userspace. We have that with Intel PT as well.

Quoted 3 lines

Metadata refer to exactly that - the configuration of the trace engine. It has to be somehow lumped with the trace stream for off target analysis.

What we call sideband data in our code is more like runtime metadata, such as executable mappings (so that you know what is mapped to which addresses, iirc ETM/PTM also deals in virtual addresses, so you'll need this information to make sense of the trace, right?) and context switches.

One of the great things about perf here is that it provides all this information practically for free.

Quoted 10 lines
Quoted 3 lines

The way I read your explanation it is a one time blob generated once you setup the hardware.

Correct.

Quoted 3 lines

I suppose we could either dump it once into the
normal data stream or maybe dump it once every time we generate an AUX buffer event into the normal data stream -- if its not too big.

Quoted 6 lines

Right, there is a set of meta-data to be generated with each trace run. With the current implementation a "trace run" pertains to all the information collected between the beginning and end of a trace scenario. Future work involve triggering a DMA transfer of the full coresight buffer to a kernel memory area, something that is probably close to the "buffer event" you are referring to.

Again correct me if I'm wrong, but the TMC(?) controller can be configured to direct ETM/PTM output right into system memory by means of a scatter-gather table. This is what we call AUX area, it's basically a circular buffer with trace data. Trace output is sent to the system memory, which is also mmap()ed to the userspace tracing tool (perf), so that it can capture it in real time. Well, that's one of the scenarios.

Regards,
--
Alex

#60 lore raw
Quoted 83 lines · Alexander Shishkin
Quoted 11 lines · Mathieu Poirier
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Poirier

Pawell, many thanks for looping me in.

I am definitely not a perf-internal guru and as such won't be able to comment on the implementation. On the flip side it is easy for me to see how the work on coresight done at Linaro can be made to tie-in what Alexander is proposing. Albeit not at the top of the priority list at this time, integration with perf (and ftrace) is definitely on the roadmap.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Powell is correct in his statement that Linaro's work in HW trace decoding is (currently) mainly focused on processor tracing but that will change when we have the basic infrastructure upstreamed.

Last but not least it would be interesting to have more information on the "sideband data". With coresight we have something called "metadata", also related to how the trace source was configured and instrumental to proper trace decoding. I'm pretty sure we are facing

the same problems.

Quoted 8 lines
Quoted 5 lines

So we use the sideband or AUX data stream to export the 'big' data stream generated by the CPU in an opaque manner. For every AUX data block 'posted' we issue an event into the regular data buffer that describes it.

In the context of "describe it" written above, what kind of information one would typically find in that description?

It's like "got a chunk of AUX data in the AUX buffer, starting at offset $X, length $Y".

Quoted 6 lines
Quoted 5 lines

I was assuming that both ARM and MIPS would generate a single data stream as well. So please do tell more about your meta-data; is that a one time thing or a second continuous stream of data, albeit smaller than the main stream?

Quoted 7 lines

Coresight does indeed generate a single steam of compressed data. Depending on the tracing specifics that were requested by the use case (trace engine configuration) the format of the packets in the compressed stream will change. Since the compressed stream itself doesn't carry clues about the formatting information, knowledge of how the trace engine was configured is mandatory for the proper decoding of the trace stream.

Ok, in perf the trace configuration would be part of 'session' information, so the way the tracing was configured by userspace will be saved to the resulting trace file (perf.data) by the userspace. We have that with Intel PT as well.

Quoted 3 lines

Metadata refer to exactly that - the configuration of the trace engine. It has to be somehow lumped with the trace stream for off target analysis.

What we call sideband data in our code is more like runtime metadata, such as executable mappings (so that you know what is mapped to which addresses, iirc ETM/PTM also deals in virtual addresses, so you'll need this information to make sense of the trace, right?) and context switches.

One of the great things about perf here is that it provides all this information practically for free.

Quoted 10 lines
Quoted 3 lines

The way I read your explanation it is a one time blob generated once you setup the hardware.

Correct.

Quoted 3 lines

I suppose we could either dump it once into the normal data stream or maybe dump it once every time we generate an AUX buffer event into the normal data stream -- if its not too big.

Quoted 6 lines

Right, there is a set of meta-data to be generated with each trace run. With the current implementation a "trace run" pertains to all the information collected between the beginning and end of a trace scenario. Future work involve triggering a DMA transfer of the full coresight buffer to a kernel memory area, something that is probably close to the "buffer event" you are referring to.

Quoted 6 lines

Again correct me if I'm wrong, but the TMC(?) controller can be configured to direct ETM/PTM output right into system memory by means of a scatter-gather table. This is what we call AUX area, it's basically a circular buffer with trace data. Trace output is sent to the system memory, which is also mmap()ed to the userspace tracing tool (perf), so that it can capture it in real time. Well, that's one of the scenarios.

Correct. However there are two provisos:

Firstly, not all systems will have the Trace Memory Controller (TMC) in the Embedded Trace Router (ETR) configuration that can write directly to system memory. Many systems have a dedicated Embedded Trace Buffer (ETB). This is a dedicated SRAM for collecting trace that is not memory-mapped. It can only be accessed by the driver. Getting the data out is quick compared to reconstructing the trace. Systems can have a combination of multiple ETRs, ETBs, ...

Secondly, ETR/ETB etc. might contain multiple traces from different sources and different processors multiplexed together with the Trace Wrapping Protocol (TWP). For security reasons you might decide to unwrap this in privileged mode code, not in userspace. Again this is quick compared to reconstructing the trace.

So you might need to support copying the data out into a userspace buffer.

Mike.
--
Michael Williams, Principal Engineer, ARM Limited, Cambridge UK
ARM: The Architecture for the Digital World http://www.arm.com

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782

#61 lore raw
In reply to Alexander Shishkin
Quoted 5 lines

Ok, in perf the trace configuration would be part of 'session' information, so the way the tracing was configured by userspace will be
saved to the resulting trace file (perf.data) by the userspace. We have that with Intel PT as well.

For ETM, in principle, those aspects of the ETM configuration that affect packet-level decode might change during a trace session (you'd have to disable the ETM while you did that, but you might still be capturing trace). The decoder would then need to switch its own model of the configuration at exactly the right time. The most robust way I could think of for doing that (given that timestamps might not be precise enough) was to switch the ETM to use a new trace source identifier. But there also needs to be a sideband message that the decoder can listen to - a "trace configuration changed" message. That could go in perf.data somewhere.

I don't know if you have that issue for PT.

You might decide that reconfiguring a trace source during a trace capture is not a valid use case...

Quoted 4 lines
Quoted 3 lines

Metadata refer to exactly that - the configuration of the trace engine. It has to be somehow lumped with the trace stream for off target analysis.

Quoted 8 lines

What we call sideband data in our code is more like runtime metadata,
such as executable mappings (so that you know what is mapped to which
addresses, iirc ETM/PTM also deals in virtual addresses, so you'll need
this information to make sense of the trace, right?) and context switches.

Yes, that's exactly the information ETM/PTM needs too - a dynamic mapping from virtual address to opcodes.

For context switches, if the kernel is set up to use the hardware context id register (CONTEXTIDR), it might not be necessary to have a message on every context switch, only on changes to the address space maps. So you get less invasive tracing (because you're not generating a perf event every context switch) at the expense of a slight fixed increase in context switch time due to the kernel updating CONTEXTIDR.

There's a lot more about this in

http://people.linaro.org/~mathieu.poirier/coresight/cs-decode.pdf

where I've used the term "static metadata" for the basic trace configuration vs. "dynamic metadata" for the data that changes as the OS executes whatever workload you're tracing. Both are sideband data as far as ETM is concerned as they aren't in the ETM stream.

Quoted 2 lines

One of the great things about perf here is that it provides all this information practically for free.

That's good to know!

Tracing of JITted code also creates challenges and it would be interesting to hear how PT and perf solve that problem. The trace decoder needs the address->opcode mapping, but as the opcodes have only a transient lifetime now, they need to be captured somewhere - either in the perf stream itself or in a side channel. Hopefully, you only have to do that when you JIT a block, and not every time you execute the cached code block, so it's not so invasive as to negate the benefits of JIT.

Al

Quoted 34 lines
Quoted 3 lines
Quoted 3 lines

The way I read your explanation it is a one time blob generated

once you

Quoted 6 lines
Quoted 1 line

setup the hardware.

Correct.

Quoted 2 lines

I suppose we could either dump it once into the normal data stream or maybe dump it once every time we generate

an AUX

Quoted 4 lines
Quoted 1 line

buffer event into the normal data stream -- if its not too big.

Right, there is a set of meta-data to be generated with each trace run. With the current implementation a "trace run" pertains to

all

Quoted 2 lines

the information collected between the beginning and end of a trace scenario. Future work involve triggering a DMA transfer of the

full

Quoted 1 line

coresight buffer to a kernel memory area, something that is

probably

Quoted 1 line

close to the "buffer event" you are referring to.

Again correct me if I'm wrong, but the TMC(?) controller can be configured to direct ETM/PTM output right into system memory by means of
a scatter-gather table. This is what we call AUX area, it's basically a
circular buffer with trace data. Trace output is sent to the system memory, which is also mmap()ed to the userspace tracing tool (perf), so
that it can capture it in real time. Well, that's one of the scenarios.

Quoted 3 lines

Regards,
--
Alex

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782