Skip to contentSkip to conversation

perf: Add infrastructure and support for Intel PT

PATCH Superseded Alexander Shishkin 4 participants 65 messages Oct 13, 2014 – Nov 11, 2014

Subsystems (2)

v5 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Alexander Shishkin

author reviewer
#0 lore raw

v5 cover perf: Add infrastructure and support for Intel PT

Hi Peter and all,

[full description below the changelog]

This version of the patchset hopefully addresses comments from the previous (v4) version. Changelog messages should be more descriptive as well as comments in the code. Funcitonal changes:

* events are not disabled on munmap(), this got replaced with refcounting,
* explicit sfence is added to the intel_pt driver to make sure that data stores are globally visible before the aux_head is updated,
* dropped the unnecessary set_output for inherited events in favor of using parent's ring buffer,
* intel_pt needs to handle #GP from enabling WRMSR, so that a privileged user can set arbitrary RTIT_CTL bits in the range that is reserved for packet enables (see PT_BYPASS_MASK and comments around pt_config()) without potentially killing the machine.

Interface changes:

* replaced 'u8 truncated' in the PERF_RECORD_AUX with a 'u64 flags', dropped redundant id/stream_id,
* in overwrite mode, always provide offset and size even if the driver cannot tell where the snapshot begins/weather its beginning was overwritten by older data.

This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.

The single most notable thing is that while PT outputs trace data in a compressed binary format, it will still generate hundreds of megabytes of trace data per second per core. Decoding this binary stream takes 2-3 orders of magnitude the cpu time that it takes to generate it. These considerations make it impossible to carry out decoding in kernel space. Therefore, the trace data is exported to userspace as a zero-copy mapping that userspace can collect and store for later decoding. To address this, this patchset extends perf ring buffer with an "AUX space", which is allocated for hardware blocks such as PT to export their trace data with minimal overhead. This space can be configured via buffer's user page and mmapped from the same file descriptor with a given offset. Data can then be collected from it by reading the aux_head (write) pointer from the user page and updating aux_tail (read) pointer similarly to data_{head,tail} of the traditional perf buffer. There is an api between perf core and pmu drivers that wish to make use of this AUX space to export their data.

For tracing blocks that don't support hardware scatter-gather tables, we provide high-order physically contiguous allocations to minimize the overhead needed for software double buffering and PMI pressure.

This way we get a normal perf data stream that provides sideband information that is required to decode the trace data, such as MMAPs, COMMs etc, plus the actual trace in its own logical space.

If the trace buffer is mapped writable, the driver will stop tracing when it fills up (aux_head approaches aux_tail), till data is read, aux_tail pointer is moved forward and an ioctl() is issued to re-enable tracing. If the trace buffer is mapped read only, the tracing will continue, overwriting older data, so that the buffer always contains the most recent data. Tracing can be stopped with an ioctl() and restarted once the data is collected.

Another use case is annotating samples of other perf events: setting PERF_SAMPLE_AUX requests attr.aux_sample_size bytes of trace to be included in each event's sample.

This patchset consists of necessary changes to the perf kernel infrastructure, and PT and BTS pmu drivers. The tooling support is not included in this series, however, it can be found in my github tree [2].

[1] http://software.intel.com/en-us/intel-isa-extensions
[2] http://github.com/virtuoso/linux-perf/tree/intel_pt

Alexander Shishkin (19):
perf: Add data_{offset,size} to user_page
perf: Support high-order allocations for AUX space
perf: Add a capability for AUX_NO_SG pmus to do software double
buffering
perf: Add a pmu capability for "exclusive" events
perf: Add AUX record
perf: Add api for pmus to write to AUX area
perf: Support overwrite mode for AUX area
perf: Add wakeup watermark control to AUX area
x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
x86: perf: Intel PT and LBR/BTS are mutually exclusive
x86: perf: intel_pt: Intel PT PMU driver
x86: perf: intel_bts: Add BTS PMU driver
perf: add ITRACE_START record to indicate that tracing has started
perf: Add api to (de-)allocate AUX buffers for kernel counters
perf: Add a helper for looking up pmus by type
perf: Add infrastructure for using AUX data in perf samples
perf: Allocate ring buffers for inherited per-task kernel events
perf: Allow AUX sampling for multiple events
perf: Allow AUX sampling of inherited events

Peter Zijlstra (1):
perf: Add AUX area to ring buffer for raw data streams

 arch/x86/include/asm/cpufeature.h          |   1 +
 arch/x86/include/uapi/asm/msr-index.h      |  18 +
 arch/x86/kernel/cpu/Makefile               |   1 +
 arch/x86/kernel/cpu/intel_pt.h             | 129 ++++
 arch/x86/kernel/cpu/perf_event.h           |  14 +
 arch/x86/kernel/cpu/perf_event_intel.c     |  14 +-
 arch/x86/kernel/cpu/perf_event_intel_bts.c | 518 +++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |  11 +-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c |   9 +-
 arch/x86/kernel/cpu/perf_event_intel_pt.c  | 995 +++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/scattered.c            |   1 +
 include/linux/perf_event.h                 |  56 +-
 include/uapi/linux/perf_event.h            |  73 ++-
 kernel/events/core.c                       | 534 +++++++++++++++-
 kernel/events/internal.h                   |  52 ++
 kernel/events/ring_buffer.c                | 386 ++++++++++-
 16 files changed, 2768 insertions(+), 44 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
Signature · 1 line
2.1.0
author reviewer
#1 lore raw

PATCH v5 1/20 perf: Add data_{offset,size} to user_page

Currently, the actual perf ring buffer is one page into the mmap area, following the user page and the userspace follows this convention. This patch adds data_{offset,size} fields to user_page that can be used by userspace instead for locating perf data in the mmap area. This is also helpful when mapping existing or shared buffers if their size is not known in advance.

Right now, it is made to follow the existing convention that

	data_offset == PAGE_SIZE and
	data_offset + data_size == mmap_size.
---
 include/uapi/linux/perf_event.h | 5 +++++
 kernel/events/core.c            | 2 ++
 2 files changed, 7 insertions(+)
include/uapi/linux/perf_event.h +5 -0
index 9269de2548..f7d18c2cb7 100644
@@ -489,9 +489,14 @@ struct perf_event_mmap_page {
 	 * In this case the kernel will not over-write unread data.
 	 *
 	 * See perf_output_put_handle() for the data ordering.
+	 *
+	 * data_{offset,size} indicate the location and size of the perf record
+	 * buffer within the mmapped area.
 	 */
 	__u64   data_head;		/* head in the data section */
 	__u64	data_tail;		/* user-space written tail */
+	__u64	data_offset;		/* where the buffer starts */
+	__u64	data_size;		/* data buffer size */
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
kernel/events/core.c +2 -0
index b164cb07b3..23bacb8682 100644
@@ -3917,6 +3917,8 @@ static void perf_event_init_userpage(struct perf_event *event)
 	/* Allow new userspace to detect that bit 0 is deprecated */
 	userpg->cap_bit0_is_deprecated = 1;
 	userpg->size = offsetof(struct perf_event_mmap_page, __reserved);
+	userpg->data_offset = PAGE_SIZE;
+	userpg->data_size = perf_data_size(rb);
 
 unlock:
 	rcu_read_unlock();
Signature · 1 line
2.1.0
author reviewer
#2 lore raw

PATCH v5 2/20 perf: Add AUX area to ring buffer for raw data streams

In reply to Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---
 include/linux/perf_event.h      |  17 +++++
 include/uapi/linux/perf_event.h |  16 +++++
 kernel/events/core.c            | 140 +++++++++++++++++++++++++++++++++-------
 kernel/events/internal.h        |  23 +++++++
 kernel/events/ring_buffer.c     |  97 ++++++++++++++++++++++++++--
 5 files changed, 264 insertions(+), 29 deletions(-)
include/linux/perf_event.h +17 -0
index 893a0d0798..344058c71d 100644
@@ -263,6 +263,18 @@ struct pmu {
 	 * flush branch stack on context-switches (needed in cpu-wide mode)
 	 */
 	void (*flush_branch_stack)	(void);
+
+	/*
+	 * Set up pmu-private data structures for an AUX area
+	 */
+	void *(*setup_aux)		(int cpu, void **pages,
+					 int nr_pages, bool overwrite);
+					/* optional */
+
+	/*
+	 * Free pmu-private AUX data structures
+	 */
+	void (*free_aux)		(void *aux); /* optional */
 };
 
 /**
@@ -782,6 +794,11 @@ static inline bool has_branch_stack(struct perf_event *event)
 	return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
 }
 
+static inline bool has_aux(struct perf_event *event)
+{
+	return event->pmu->setup_aux;
+}
+
 extern int perf_output_begin(struct perf_output_handle *handle,
 			     struct perf_event *event, unsigned int size);
 extern void perf_output_end(struct perf_output_handle *handle);
include/uapi/linux/perf_event.h +16 -0
index f7d18c2cb7..7e0967c0f5 100644
@@ -497,6 +497,22 @@ struct perf_event_mmap_page {
 	__u64	data_tail;		/* user-space written tail */
 	__u64	data_offset;		/* where the buffer starts */
 	__u64	data_size;		/* data buffer size */
+
+	/*
+	 * AUX area is defined by aux_{offset,size} fields that should be set
+	 * by the userspace, so that
+	 *
+	 *   aux_offset >= data_offset + data_size
+	 *
+	 * prior to mmap()ing it. Size of the mmap()ed area should be aux_size.
+	 *
+	 * Ring buffer pointers aux_{head,tail} have the same semantics as
+	 * data_{head,tail} and same ordering rules apply.
+	 */
+	__u64	aux_head;
+	__u64	aux_tail;
+	__u64	aux_offset;
+	__u64	aux_size;
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
kernel/events/core.c +115 -25

214 lines not on this page · raw message

kernel/events/internal.h +23 -0
index 569b218782..1e476d2f29 100644
@@ -35,6 +35,16 @@ struct ring_buffer {
 	unsigned long			mmap_locked;
 	struct user_struct		*mmap_user;
 
+	/* AUX area */
+	unsigned long			aux_pgoff;
+	int				aux_nr_pages;
+	atomic_t			aux_mmap_count;
+	unsigned long			aux_mmap_locked;
+	void				(*free_aux)(void *);
+	struct kref			aux_refcount;
+	void				**aux_pages;
+	void				*aux_priv;
+
 	struct perf_event_mmap_page	*user_page;
 	void				*data_pages[0];
 };
@@ -43,6 +53,14 @@ extern void rb_free(struct ring_buffer *rb);
 extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
 extern void perf_event_wakeup(struct perf_event *event);
+extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
+			pgoff_t pgoff, int nr_pages, int flags);
+extern void rb_free_aux(struct ring_buffer *rb);
+
+static inline bool rb_has_aux(struct ring_buffer *rb)
+{
+	return !!rb->aux_nr_pages;
+}
 
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
@@ -81,6 +99,11 @@ static inline unsigned long perf_data_size(struct ring_buffer *rb)
 	return rb->nr_pages << (PAGE_SHIFT + page_order(rb));
 }
 
+static inline unsigned long perf_aux_size(struct ring_buffer *rb)
+{
+	return rb->aux_nr_pages << PAGE_SHIFT;
+}
+
 #define DEFINE_OUTPUT_COPY(func_name, memcpy_func)			\
 static inline unsigned long						\
 func_name(struct perf_output_handle *handle,				\
kernel/events/ring_buffer.c +93 -4

118 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#3 lore raw

Re: [PATCH v5 02/20] perf: Add AUX area to ring buffer for raw data streams

kernel/events/internal.h struct ring_buffer { · line 44

Quoted 1 line · Alexander Shishkin
kernel/events/internal.h +1 -0
+	struct kref			aux_refcount;

I'm not a fan of kref, pointless obfuscation that.

#4 lore raw

Re: [PATCH v5 02/20] perf: Add AUX area to ring buffer for raw data streams

Quoted 4 lines · Peter Zijlstra

kernel/events/internal.h struct ring_buffer { · line 44

Quoted 1 line · Alexander Shishkin
kernel/events/internal.h +1 -0
+	struct kref			aux_refcount;

I'm not a fan of kref, pointless obfuscation that.

It has a good potential for debugging though. Sure right now the get/put simple APIs only performs counting sanity checks but I've seen patches that extend it to object debugging.

Sounds quite valuable on complicated object lifecycles like perf events.

author reviewer
#5 lore raw

PATCH v5 3/20 perf: Support high-order allocations for AUX space

In reply to Alexander Shishkin

Some pmus (such as BTS or Intel PT without multiple-entry ToPA capability) don't support scatter-gather and will prefer larger contiguous areas for their output regions.

This patch adds a new pmu capability to request higher order allocations.

---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 51 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 46 insertions(+), 6 deletions(-)
include/linux/perf_event.h +1 -0
index 344058c71d..71df948b4d 100644
@@ -171,6 +171,7 @@ struct perf_event;
  * pmu::capabilities flags
  */
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
+#define PERF_PMU_CAP_AUX_NO_SG			0x02
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +45 -6

82 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#6 lore raw

PATCH v5 4/20 perf: Add a capability for AUX_NO_SG pmus to do software double buffering

In reply to Alexander Shishkin

For pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.

To make use of this feature, add PERF_PMU_CAP_AUX_SW_DOUBLEBUF to your pmu's capability mask. This will make the ring buffer AUX allocation code ensure that the biggest high order allocation for the aux buffer pages is no bigger than half of the total requested buffer size, thus making sure that the buffer has at least two high order allocations.

---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 15 ++++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index 71df948b4d..e7a15f5c3f 100644
@@ -172,6 +172,7 @@ struct perf_event;
  */
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
+#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +14 -1
index b620e94ea7..b45feb3f4a 100644
@@ -286,9 +286,22 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -ENOTSUPP;
 
-	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG)
+	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) {
 		order = get_order(nr_pages * PAGE_SIZE);
 
+		/*
+		 * PMU requests more than one contiguous chunks of memory
+		 * for SW double buffering
+		 */
+		if ((event->pmu->capabilities & PERF_PMU_CAP_AUX_SW_DOUBLEBUF) &&
+		    !overwrite) {
+			if (!order)
+				return -EINVAL;
+
+			order--;
+		}
+	}
+
 	rb->aux_pages = kzalloc_node(nr_pages * sizeof(void *), GFP_KERNEL, node);
 	if (!rb->aux_pages)
 		return -ENOMEM;
Signature · 1 line
2.1.0
author reviewer
#7 lore raw

PATCH v5 5/20 perf: Add a pmu capability for "exclusive" events

In reply to Alexander Shishkin

Usually, pmus that do, for example, instruction tracing, would only ever be able to have one event per task per cpu (or per perf_context). For such pmus it makes sense to disallow creating conflicting events early on, so as to provide consistent behavior for the user.

This patch adds a pmu capability that indicates such constraint on event creation.

---
 include/linux/perf_event.h |  1 +
 kernel/events/core.c       | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
include/linux/perf_event.h +1 -0
index e7a15f5c3f..5c3dee021c 100644
@@ -173,6 +173,7 @@ struct perf_event;
 #define PERF_PMU_CAP_NO_INTERRUPT		0x01
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
 #define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
+#define PERF_PMU_CAP_EXCLUSIVE			0x08
 
 /**
  * struct pmu - generic performance monitoring unit
kernel/events/core.c +45 -0

69 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#8 lore raw

PATCH v5 6/20 perf: Add AUX record

In reply to Alexander Shishkin

When there's new data in the AUX space, output a record indicating its offset and size and a set of flags, such as PERF_AUX_FLAG_TRUNCATED, to mean the described data was truncated to fit in the ring buffer.

Cc Arnaldo Carvalho de Melo <[email protected]>
---
 include/uapi/linux/perf_event.h | 19 +++++++++++++++++++
 kernel/events/core.c            | 34 ++++++++++++++++++++++++++++++++++
 kernel/events/internal.h        |  3 +++
 3 files changed, 56 insertions(+)
include/uapi/linux/perf_event.h +19 -0
index 7e0967c0f5..1a46627699 100644
@@ -733,6 +733,20 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u64				flags;
+	 * 	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
@@ -750,6 +764,11 @@ enum perf_callchain_context {
 	PERF_CONTEXT_MAX		= (__u64)-4095,
 };
 
+/**
+ * PERF_RECORD_AUX::flags bits
+ */
+#define PERF_AUX_FLAG_TRUNCATED		0x01	/* record was truncated to fit */
+
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
 #define PERF_FLAG_PID_CGROUP		(1UL << 2) /* pid=cgroup id, per-cpu mode only */
kernel/events/core.c +34 -0
index ced9b0819f..46a6217bbe 100644
@@ -5581,6 +5581,40 @@ void perf_event_mmap(struct vm_area_struct *vma)
 	perf_event_mmap_event(&mmap_event);
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, u64 flags)
+{
+	struct perf_output_handle handle;
+	struct perf_sample_data sample;
+	struct perf_aux_event {
+		struct perf_event_header	header;
+		u64				offset;
+		u64				size;
+		u64				flags;
+	} rec = {
+		.header = {
+			.type = PERF_RECORD_AUX,
+			.misc = 0,
+			.size = sizeof(rec),
+		},
+		.offset		= head,
+		.size		= size,
+		.flags		= flags,
+	};
+	int ret;
+
+	perf_event_header__init_id(&rec.header, &sample, event);
+	ret = perf_output_begin(&handle, event, rec.header.size);
+
+	if (ret)
+		return;
+
+	perf_output_put(&handle, rec);
+	perf_event__output_id_sample(event, &handle, &sample);
+
+	perf_output_end(&handle);
+}
+
 /*
  * IRQ throttle logging
  */
kernel/events/internal.h +3 -0
index 1e476d2f29..e0013084c5 100644
@@ -62,6 +62,9 @@ static inline bool rb_has_aux(struct ring_buffer *rb)
 	return !!rb->aux_nr_pages;
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, u64 flags);
+
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
 			   struct perf_sample_data *data,
Signature · 1 line
2.1.0
maintainer
#9 lore raw

Re: [PATCH v5 06/20] perf: Add AUX record

include/uapi/linux/perf_event.h enum perf_event_type { · line 748

Quoted 13 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +13 -0
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u64				flags;
+	 * 	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,

Given the discussion with the ARM people the last time, do we want to add the possibility of a variable data field in this event? Its easy to add now, harder to do later (although not impossible).

Also added Mathieu Poirier on CC, he asked to be included in your next postings.

author reviewer
#10 lore raw

Re: [PATCH v5 06/20] perf: Add AUX record

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +9 -0
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u64				flags;
Quoted 8 lines

include/uapi/linux/perf_event.h enum perf_event_type { · line 748

Quoted 4 lines
include/uapi/linux/perf_event.h +4 -0
+	 * 	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,

Given the discussion with the ARM people the last time, do we want to add the possibility of a variable data field in this event? Its easy to add now, harder to do later (although not impossible).

Iirc, what they want is to save a once-per-session chunk of data, which would be better synthesized by perf record than sent from a pmu driver?

We do something like that with PT right now, perf record looks at pmu's sysfs attributes and stores them in some synthesized record.

Quoted 2 lines

Also added Mathieu Poirier on CC, he asked to be included in your next postings.

Indeed, my apologies.

Regards,
--
Alex

maintainer
#11 lore raw

Re: [PATCH v5 06/20] perf: Add AUX record

Quoted 21 lines · Alexander Shishkin
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +9 -0
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u64				flags;
Quoted 8 lines

include/uapi/linux/perf_event.h enum perf_event_type { · line 748

Quoted 4 lines
include/uapi/linux/perf_event.h +4 -0
+	 * 	struct sample_id		sample_id;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,

Given the discussion with the ARM people the last time, do we want to add the possibility of a variable data field in this event? Its easy to add now, harder to do later (although not impossible).

Quoted 5 lines

Iirc, what they want is to save a once-per-session chunk of data, which would be better synthesized by perf record than sent from a pmu driver?

We do something like that with PT right now, perf record looks at pmu's sysfs attributes and stores them in some synthesized record.

Right, I just got to that. So I wasn't entirely sure if perf_event_open time conditions where sufficient for them, if they are then yes that could work.

author reviewer
#12 lore raw

PATCH v5 7/20 perf: Add api for pmus to write to AUX area

In reply to Alexander Shishkin

For pmus that wish to write data to ring buffer's AUX area, provide perf_aux_output_{begin,end}() calls to initiate/commit data writes, similarly to perf_output_{begin,end}. These also use the same output handle structure. Also, similarly to software counterparts, these will direct inherited events' output to parents' ring buffers.

After the perf_aux_output_begin() returns successfully, handle->size is set to the maximum amount of data that can be written wrt aux_tail pointer, so that no data that the user hasn't seen will be overwritten, therefore this should always be called before hardware writing is enabled. On success, this will return the pointer to pmu driver's private structure allocated for this aux area by pmu::setup_aux. Same pointer can also be retrieved using perf_get_aux() while hardware writing is enabled.

PMU driver should pass the actual amount of data written as a parameter to perf_aux_output_end(). All hardware writes should be completed and visible before this one is called.

Additionally, perf_aux_output_skip() will adjust output handle and aux_head in case some part of the buffer has to be skipped over to maintain hardware's alignment constraints.

Nested writers are forbidden and guards are in place to catch such attempts.

---
 include/linux/perf_event.h  |  23 +++++++-
 kernel/events/core.c        |   5 +-
 kernel/events/internal.h    |   4 ++
 kernel/events/ring_buffer.c | 139 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 167 insertions(+), 4 deletions(-)
include/linux/perf_event.h +22 -1
index 5c3dee021c..f126eb89e6 100644
@@ -555,12 +555,22 @@ struct perf_output_handle {
 	struct ring_buffer		*rb;
 	unsigned long			wakeup;
 	unsigned long			size;
-	void				*addr;
+	union {
+		void			*addr;
+		unsigned long		head;
+	};
 	int				page;
 };
 
 #ifdef CONFIG_PERF_EVENTS
 
+extern void *perf_aux_output_begin(struct perf_output_handle *handle,
+				   struct perf_event *event);
+extern void perf_aux_output_end(struct perf_output_handle *handle,
+				unsigned long size, bool truncated);
+extern int perf_aux_output_skip(struct perf_output_handle *handle,
+				unsigned long size);
+extern void *perf_get_aux(struct perf_output_handle *handle);
 extern int perf_pmu_register(struct pmu *pmu, const char *name, int type);
 extern void perf_pmu_unregister(struct pmu *pmu);
 
@@ -817,6 +827,17 @@ extern void perf_event_disable(struct perf_event *event);
 extern int __perf_event_disable(void *info);
 extern void perf_event_task_tick(void);
 #else /* !CONFIG_PERF_EVENTS: */
+static inline void *
+perf_aux_output_begin(struct perf_output_handle *handle,
+		      struct perf_event *event)				{ return NULL; }
+static inline void
+perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+		    bool truncated)					{ }
+static inline int
+perf_aux_output_skip(struct perf_output_handle *handle,
+		     unsigned long size)				{ return -EINVAL; }
+static inline void *
+perf_get_aux(struct perf_output_handle *handle)				{ return NULL; }
 static inline void
 perf_event_task_sched_in(struct task_struct *prev,
 			 struct task_struct *task)			{ }
kernel/events/core.c +2 -3
index 46a6217bbe..fcb61ae7a3 100644
@@ -3268,7 +3268,6 @@ static void free_event_rcu(struct rcu_head *head)
 	kfree(event);
 }
 
-static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_attach(struct perf_event *event,
 			       struct ring_buffer *rb);
 
@@ -4085,7 +4084,7 @@ static void rb_free_rcu(struct rcu_head *rcu_head)
 	rb_free(rb);
 }
 
-static struct ring_buffer *ring_buffer_get(struct perf_event *event)
+struct ring_buffer *ring_buffer_get(struct perf_event *event)
 {
 	struct ring_buffer *rb;
 
@@ -4100,7 +4099,7 @@ static struct ring_buffer *ring_buffer_get(struct perf_event *event)
 	return rb;
 }
 
-static void ring_buffer_put(struct ring_buffer *rb)
+void ring_buffer_put(struct ring_buffer *rb)
 {
 	if (!atomic_dec_and_test(&rb->refcount))
 		return;
kernel/events/internal.h +4 -0
index e0013084c5..68bbf86d8f 100644
@@ -36,6 +36,8 @@ struct ring_buffer {
 	struct user_struct		*mmap_user;
 
 	/* AUX area */
+	local_t				aux_head;
+	local_t				aux_nest;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
 	atomic_t			aux_mmap_count;
@@ -56,6 +58,8 @@ extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 			pgoff_t pgoff, int nr_pages, int flags);
 extern void rb_free_aux(struct ring_buffer *rb);
+extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
+extern void ring_buffer_put(struct ring_buffer *rb);
 
 static inline bool rb_has_aux(struct ring_buffer *rb)
 {
kernel/events/ring_buffer.c +139 -0

145 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#13 lore raw

Re: [PATCH v5 07/20] perf: Add api for pmus to write to AUX area

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 282

Quoted 6 lines · Alexander Shishkin
kernel/events/ring_buffer.c +6 -0
+	/*
+	 * Nesting is not supported for AUX area, make sure nested
+	 * writers are caught early
+	 */
+	if (WARN_ON_ONCE(local_xchg(&rb->aux_nest, 1)))
+		goto err_put;

Note that printk() from NMI is prone to making your machine die, not that I have a better option though.

author reviewer
#14 lore raw

Re: [PATCH v5 07/20] perf: Add api for pmus to write to AUX area

Quoted 10 lines · Peter Zijlstra

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 282

Quoted 6 lines · Alexander Shishkin
kernel/events/ring_buffer.c +6 -0
+	/*
+	 * Nesting is not supported for AUX area, make sure nested
+	 * writers are caught early
+	 */
+	if (WARN_ON_ONCE(local_xchg(&rb->aux_nest, 1)))
+		goto err_put;

Note that printk() from NMI is prone to making your machine die, not that I have a better option though.

Indeed, but if your driver is really attempting nested perf_aux_output_begin()s, then you had it coming to you anyway.

Regards,
--
Alex

author reviewer
#15 lore raw

PATCH v5 8/20 perf: Support overwrite mode for AUX area

In reply to Alexander Shishkin

This adds support for overwrite mode in the AUX area, which means "keep collecting data till you're stopped", turning AUX area into a circular buffer, where new data overwrites old data. It does not depend on data buffer's overwrite mode, so that it doesn't lose sideband data that is instrumental for processing AUX data.

Overwrite mode is enabled at mapping AUX area read only. Even though aux_tail in the buffer's user page might be user writable, it will be ignored in this mode.

A PERF_RECORD_AUX with PERF_AUX_FLAG_OVERWRITE set is written to the perf data stream every time an event writes new data to the AUX area. The pmu driver might not be able to infer the exact beginning of the new data in each snapshot, some drivers will only provide the tail, which is aux_offset + aux_size in the AUX record. Consumer has to be able to tell the new data from the old one, for example, by means of time stamps if such are provided in the trace.

Consumer is also responsible for disabling any events that might write to the AUX area (thus potentially racing with the consumer) before collecting the data.

---
 include/uapi/linux/perf_event.h |  1 +
 kernel/events/internal.h        |  1 +
 kernel/events/ring_buffer.c     | 48 ++++++++++++++++++++++++++++-------------
 3 files changed, 35 insertions(+), 15 deletions(-)
include/uapi/linux/perf_event.h +1 -0
index 1a46627699..a6bd31f5e0 100644
@@ -768,6 +768,7 @@ enum perf_callchain_context {
  * PERF_RECORD_AUX::flags bits
  */
 #define PERF_AUX_FLAG_TRUNCATED		0x01	/* record was truncated to fit */
+#define PERF_AUX_FLAG_OVERWRITE		0x02	/* snapshot from overwrite mode */
 
 #define PERF_FLAG_FD_NO_GROUP		(1UL << 0)
 #define PERF_FLAG_FD_OUTPUT		(1UL << 1)
kernel/events/internal.h +1 -0
index 68bbf86d8f..b1ed80f87d 100644
@@ -40,6 +40,7 @@ struct ring_buffer {
 	local_t				aux_nest;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
+	int				aux_overwrite;
 	atomic_t			aux_mmap_count;
 	unsigned long			aux_mmap_locked;
 	void				(*free_aux)(void *);
kernel/events/ring_buffer.c +33 -15

78 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#16 lore raw

PATCH v5 9/20 perf: Add wakeup watermark control to AUX area

In reply to Alexander Shishkin

When AUX area gets a certain amount of new data, we want to wake up userspace to collect it. This adds a new control to specify how much data will cause a wakeup. This is then passed down to pmu drivers via output handle's "wakeup" field, so that the driver can find the nearest point where it can generate an interrupt.

We repurpose __reserved_2 in the event attribute for this, even though it was never checked to be zero before, aux_watermark will only matter for new AUX-aware code, so the old code should still be fine.

---
 include/uapi/linux/perf_event.h |  7 +++++--
 kernel/events/core.c            |  3 ++-
 kernel/events/internal.h        |  4 +++-
 kernel/events/ring_buffer.c     | 22 +++++++++++++++++++---
 4 files changed, 29 insertions(+), 7 deletions(-)
include/uapi/linux/perf_event.h +5 -2
index a6bd31f5e0..c20aaa5af7 100644
@@ -238,6 +238,7 @@ enum perf_event_read_format {
 #define PERF_ATTR_SIZE_VER2	80	/* add: branch_sample_type */
 #define PERF_ATTR_SIZE_VER3	96	/* add: sample_regs_user */
 					/* add: sample_stack_user */
+					/* add: aux_watermark */
 
 /*
  * Hardware event_id to monitor via a performance monitoring event:
@@ -332,8 +333,10 @@ struct perf_event_attr {
 	 */
 	__u32	sample_stack_user;
 
-	/* Align to u64. */
-	__u32	__reserved_2;
+	/*
+	 * Wakeup watermark for AUX area
+	 */
+	__u32	aux_watermark;
 };
 
 #define perf_flags(attr)	(*(&(attr)->read_format + 1))
kernel/events/core.c +2 -1
index fcb61ae7a3..908dc3e63b 100644
@@ -4394,7 +4394,8 @@ accounting:
 		perf_event_init_userpage(event);
 		perf_event_update_userpage(event);
 	} else {
-		ret = rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, flags);
+		ret = rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages,
+				   event->attr.aux_watermark, flags);
 		if (ret)
 			atomic_dec(&rb->mmap_count);
 		else
kernel/events/internal.h +3 -1
index b1ed80f87d..4715aae48b 100644
@@ -27,6 +27,7 @@ struct ring_buffer {
 	local_t				lost;		/* nr records lost   */
 
 	long				watermark;	/* wakeup watermark  */
+	long				aux_watermark;
 	/* poll crap */
 	spinlock_t			event_lock;
 	struct list_head		event_list;
@@ -38,6 +39,7 @@ struct ring_buffer {
 	/* AUX area */
 	local_t				aux_head;
 	local_t				aux_nest;
+	local_t				aux_wakeup;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
 	int				aux_overwrite;
@@ -57,7 +59,7 @@ extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
 extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
-			pgoff_t pgoff, int nr_pages, int flags);
+			pgoff_t pgoff, int nr_pages, long watermark, int flags);
 extern void rb_free_aux(struct ring_buffer *rb);
 extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
 extern void ring_buffer_put(struct ring_buffer *rb);
kernel/events/ring_buffer.c +19 -3
index d0373c6d30..96ec58fb46 100644
@@ -295,6 +295,7 @@ void *perf_aux_output_begin(struct perf_output_handle *handle,
 	 */
 	if (!rb->aux_overwrite) {
 		aux_tail = ACCESS_ONCE(rb->user_page->aux_tail);
+		handle->wakeup = local_read(&rb->aux_wakeup) + rb->aux_watermark;
 		if (aux_head - aux_tail < perf_aux_size(rb))
 			handle->size = CIRC_SPACE(aux_head, aux_tail, perf_aux_size(rb));
 
@@ -358,9 +359,12 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 		perf_event_aux_event(handle->event, aux_head, size, flags);
 	}
 
-	rb->user_page->aux_head = local_read(&rb->aux_head);
+	aux_head = rb->user_page->aux_head = local_read(&rb->aux_head);
 
-	perf_output_wakeup(handle);
+	if (aux_head - local_read(&rb->aux_wakeup) >= rb->aux_watermark) {
+		perf_output_wakeup(handle);
+		local_add(rb->aux_watermark, &rb->aux_wakeup);
+	}
 	handle->event = NULL;
 
 	local_set(&rb->aux_nest, 0);
@@ -382,6 +386,14 @@ int perf_aux_output_skip(struct perf_output_handle *handle, unsigned long size)
 
 	local_add(size, &rb->aux_head);
 
+	aux_head = rb->user_page->aux_head = local_read(&rb->aux_head);
+	if (aux_head - local_read(&rb->aux_wakeup) >= rb->aux_watermark) {
+		perf_output_wakeup(handle);
+		local_add(rb->aux_watermark, &rb->aux_wakeup);
+		handle->wakeup = local_read(&rb->aux_wakeup) +
+				 rb->aux_watermark;
+	}
+
 	handle->head = aux_head;
 	handle->size -= size;
 
@@ -432,7 +444,7 @@ static void rb_free_aux_page(struct ring_buffer *rb, int idx)
 }
 
 int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
-		 pgoff_t pgoff, int nr_pages, int flags)
+		 pgoff_t pgoff, int nr_pages, long watermark, int flags)
 {
 	bool overwrite = !(flags & RING_BUFFER_WRITABLE);
 	int node = (event->cpu == -1) ? -1 : cpu_to_node(event->cpu);
@@ -491,6 +503,10 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	kref_init(&rb->aux_refcount);
 
 	rb->aux_overwrite = overwrite;
+	rb->aux_watermark = watermark;
+
+	if (!rb->aux_watermark && !rb->aux_overwrite)
+		rb->aux_watermark = nr_pages << (PAGE_SHIFT - 1);
 
 out:
 	if (!ret)
Signature · 1 line
2.1.0
author reviewer
#17 lore raw

PATCH v5 10/20 x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection

In reply to Alexander Shishkin

Intel Processor Trace is an architecture extension that allows for program flow tracing.

---
 arch/x86/include/asm/cpufeature.h | 1 +
 arch/x86/kernel/cpu/scattered.c   | 1 +
 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index bb9b258d60..db2debe5bb 100644
@@ -185,6 +185,7 @@
 #define X86_FEATURE_DTHERM	( 7*32+ 7) /* Digital Thermal Sensor */
 #define X86_FEATURE_HW_PSTATE	( 7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface */
+#define X86_FEATURE_INTEL_PT	( 7*32+10) /* Intel Processor Trace */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW  ( 8*32+ 0) /* Intel TPR Shadow */
arch/x86/kernel/cpu/scattered.c +1 -0
index 4a8013d559..42f5fa953f 100644
@@ -36,6 +36,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c)
 		{ X86_FEATURE_ARAT,		CR_EAX, 2, 0x00000006, 0 },
 		{ X86_FEATURE_PLN,		CR_EAX, 4, 0x00000006, 0 },
 		{ X86_FEATURE_PTS,		CR_EAX, 6, 0x00000006, 0 },
+		{ X86_FEATURE_INTEL_PT,		CR_EBX,25, 0x00000007, 0 },
 		{ X86_FEATURE_APERFMPERF,	CR_ECX, 0, 0x00000006, 0 },
 		{ X86_FEATURE_EPB,		CR_ECX, 3, 0x00000006, 0 },
 		{ X86_FEATURE_HW_PSTATE,	CR_EDX, 7, 0x80000007, 0 },
Signature · 1 line
2.1.0
author reviewer
#18 lore raw

PATCH v5 11/20 x86: perf: Intel PT and LBR/BTS are mutually exclusive

In reply to Alexander Shishkin

Intel PT cannot be used at the same time as LBR or BTS and will cause a

general protection fault if they are used together. In order to avoid

fixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.

---
 arch/x86/kernel/cpu/perf_event.h           | 6 ++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  | 8 +++++++-
 arch/x86/kernel/cpu/perf_event_intel_lbr.c | 9 +++++----
 3 files changed, 18 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/perf_event.h +6 -0
index d98a34d435..8eee92be78 100644
@@ -148,6 +148,7 @@ struct cpu_hw_events {
 	 * Intel DebugStore bits
 	 */
 	struct debug_store	*ds;
+	unsigned int		bts_enabled;
 	u64			pebs_enabled;
 
 	/*
@@ -161,6 +162,11 @@ struct cpu_hw_events {
 	u64				br_sel;
 
 	/*
+	 * Intel Processor Trace
+	 */
+	unsigned int			pt_enabled;
+
+	/*
 	 * Intel host/guest exclude bits
 	 */
 	u64				intel_ctrl_guest_mask;
arch/x86/kernel/cpu/perf_event_intel_ds.c +7 -1
index b1553d05a5..865ee1268a 100644
@@ -455,8 +455,13 @@ struct event_constraint bts_constraint =
 
 void intel_pmu_enable_bts(u64 config)
 {
+	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 	unsigned long debugctlmsr;
 
+	if (cpuc->pt_enabled)
+		return;
+
+	cpuc->bts_enabled = 1;
 	debugctlmsr = get_debugctlmsr();
 
 	debugctlmsr |= DEBUGCTLMSR_TR;
@@ -477,9 +482,10 @@ void intel_pmu_disable_bts(void)
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 	unsigned long debugctlmsr;
 
-	if (!cpuc->ds)
+	if (!cpuc->ds || cpuc->pt_enabled)
 		return;
 
+	cpuc->bts_enabled = 0;
 	debugctlmsr = get_debugctlmsr();
 
 	debugctlmsr &=
arch/x86/kernel/cpu/perf_event_intel_lbr.c +5 -4
index 4af10617de..1b09c4b91a 100644
@@ -172,7 +172,9 @@ static void intel_pmu_lbr_reset_64(void)
 
 void intel_pmu_lbr_reset(void)
 {
-	if (!x86_pmu.lbr_nr)
+	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+
+	if (!x86_pmu.lbr_nr || cpuc->pt_enabled)
 		return;
 
 	if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_32)
@@ -185,7 +187,7 @@ void intel_pmu_lbr_enable(struct perf_event *event)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 
-	if (!x86_pmu.lbr_nr)
+	if (!x86_pmu.lbr_nr || cpuc->pt_enabled)
 		return;
 
 	/*
@@ -205,11 +207,10 @@ void intel_pmu_lbr_disable(struct perf_event *event)
 {
 	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
 
-	if (!x86_pmu.lbr_nr)
+	if (!x86_pmu.lbr_nr || !cpuc->lbr_users || cpuc->pt_enabled)
 		return;
 
 	cpuc->lbr_users--;
-	WARN_ON_ONCE(cpuc->lbr_users < 0);
 
 	if (cpuc->enabled && !cpuc->lbr_users) {
 		__intel_pmu_lbr_disable();
Signature · 1 line
2.1.0
maintainer
#19 lore raw

Re: [PATCH v5 11/20] x86: perf: Intel PT and LBR/BTS are mutually exclusive

Quoted 5 lines · Alexander Shishkin

Intel PT cannot be used at the same time as LBR or BTS and will cause a

general protection fault if they are used together. In order to avoid

fixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.

Yeah, don't like this. Like I've said many times before we should simply disallow creating PT events when there are LBR events and vice versa.

author reviewer
#20 lore raw

Re: [PATCH v5 11/20] x86: perf: Intel PT and LBR/BTS are mutually exclusive

Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin

Intel PT cannot be used at the same time as LBR or BTS and will cause a

general protection fault if they are used together. In order to avoid

fixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.

Yeah, don't like this. Like I've said many times before we should simply disallow creating PT events when there are LBR events and vice versa.

Ok, I must have misunderstood you before. This makes things a bit easier.

Regards,
--
Alex

author reviewer
#21 lore raw

PATCH v5 12/20 x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

Add support for Intel Processor Trace (PT) to kernel's perf events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel.

This driver exports trace data by through AUX space in the perf ring buffer, which is zero-copy mapped into userspace for faster data retrieval.

---
 arch/x86/include/uapi/asm/msr-index.h     |  18 +
 arch/x86/kernel/cpu/Makefile              |   1 +
 arch/x86/kernel/cpu/intel_pt.h            | 129 ++++
 arch/x86/kernel/cpu/perf_event.h          |   2 +
 arch/x86/kernel/cpu/perf_event_intel.c    |   8 +
 arch/x86/kernel/cpu/perf_event_intel_pt.c | 995 ++++++++++++++++++++++++++++++
 6 files changed, 1153 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/intel_pt.h
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
arch/x86/include/uapi/asm/msr-index.h +18 -0
index e21331ce36..caae296ab7 100644
@@ -74,6 +74,24 @@
 #define MSR_IA32_PERF_CAPABILITIES	0x00000345
 #define MSR_PEBS_LD_LAT_THRESHOLD	0x000003f6
 
+#define MSR_IA32_RTIT_CTL		0x00000570
+#define RTIT_CTL_TRACEEN		BIT(0)
+#define RTIT_CTL_OS			BIT(2)
+#define RTIT_CTL_USR			BIT(3)
+#define RTIT_CTL_CR3EN			BIT(7)
+#define RTIT_CTL_TOPA			BIT(8)
+#define RTIT_CTL_TSC_EN			BIT(10)
+#define RTIT_CTL_DISRETC		BIT(11)
+#define RTIT_CTL_BRANCH_EN		BIT(13)
+#define MSR_IA32_RTIT_STATUS		0x00000571
+#define RTIT_STATUS_CONTEXTEN		BIT(1)
+#define RTIT_STATUS_TRIGGEREN		BIT(2)
+#define RTIT_STATUS_ERROR		BIT(4)
+#define RTIT_STATUS_STOPPED		BIT(5)
+#define MSR_IA32_RTIT_CR3_MATCH		0x00000572
+#define MSR_IA32_RTIT_OUTPUT_BASE	0x00000560
+#define MSR_IA32_RTIT_OUTPUT_MASK	0x00000561
+
 #define MSR_MTRRfix64K_00000		0x00000250
 #define MSR_MTRRfix16K_80000		0x00000258
 #define MSR_MTRRfix16K_A0000		0x00000259
arch/x86/kernel/cpu/Makefile +1 -0
index 7e1fd4e085..00d40f889d 100644
@@ -39,6 +39,7 @@ obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o per
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o perf_event_intel_uncore_snb.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore_snbep.o perf_event_intel_uncore_nhmex.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_rapl.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o
 endif
 
 
arch/x86/kernel/cpu/intel_pt.h new file +129 -0

129 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event.h +2 -0
index 8eee92be78..9f9b02fea6 100644
@@ -750,6 +750,8 @@ void intel_pmu_lbr_init_snb(void);
 
 int intel_pmu_setup_lbr_filter(struct perf_event *event);
 
+void intel_pt_interrupt(void);
+
 int p4_pmu_init(void);
 
 int p6_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel.c +8 -0
index 3851def505..6e04da1b04 100644
@@ -1528,6 +1528,14 @@ again:
 	}
 
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+
+	/*
 	 * Checkpointed counters can lead to 'spurious' PMIs because the
 	 * rollback caused by the PMI will have cleared the overflow status
 	 * bit. Therefore always force probe these counters.
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +995 -0

995 lines not on this page · raw message

Signature · 1 line
2.1.0
maintainer
#22 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 2 lines · Alexander Shishkin

+/**
+ * struct pt_buffer - buffer configuration; one buffer per task_struct or

- * cpu, depending on perf event configuration
+ * cpu, depending on perf event configuration

Quoted 14 lines

+ * @tables: list of ToPA tables in this buffer
+ * @first, @last: shorthands for first and last topa tables
+ * @cur: current topa table
+ * @nr_pages: buffer size in pages
+ * @cur_idx: current output region's index within @cur table
+ * @output_off: offset within the current output region
+ * @data_size: running total of the amount of data in this buffer
+ * @lost: if data was lost/truncated
+ * @head: logical write offset inside the buffer
+ * @snapshot: if this is for a snapshot/overwrite counter
+ * @stop_pos, @intr_pos: STOP and INT topa entries in the buffer
+ * @data_pages: array of pages from perf
+ * @topa_index: table of topa entries indexed by page offset
+ */

maintainer
#23 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537

Quoted 12 lines · Alexander Shishkin

+++ b/arch/x86/kernel/cpu/perf_event_intel.c

arch/x86/kernel/cpu/perf_event_intel.c +7 -0
@@ -1528,6 +1528,14 @@ again:
 	}
 
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+

How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like FREEZE_ON_PMI etc?

author reviewer
#24 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Alexander Shishkin

+++ b/arch/x86/kernel/cpu/perf_event_intel.c

(quoted diff) +0 -0
@@ -1528,6 +1528,14 @@ again:
 	}
 
Quoted 11 lines

arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537

Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c +7 -0
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+

How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like FREEZE_ON_PMI etc?

It ignores the FREEZE_ON_PMI bit. I stop it by hand inside the PMI handler, so you can see parts of the handler in the trace if you're tracing the kernel.

Regards,
--
Alex

maintainer
#25 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 19 lines · Alexander Shishkin
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Alexander Shishkin

+++ b/arch/x86/kernel/cpu/perf_event_intel.c

(quoted diff) +0 -0
@@ -1528,6 +1528,14 @@ again:
 	}
 
Quoted 11 lines

arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537

Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c +7 -0
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+

How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like FREEZE_ON_PMI etc?

Quoted 3 lines

It ignores the FREEZE_ON_PMI bit. I stop it by hand inside the PMI handler, so you can see parts of the handler in the trace if you're tracing the kernel.

Urgh, horrid that. Routing something to the same interrupt, sharing status registers but not observing the same semantics for the interrupt is a massive fail.

IIRC Andi was planning to start using FREEZE_ON_PMI to avoid the MSR writes in intel_pmu_{disable,enable}_all(), this interrupt not actually respecting that makes that non-trivial.

We already use FREEZE_ON_PMI for LBR, but for now PT and LBR are mutually exclusive so that's not a problem, if we ever get those working together this needs to get fixed.

author reviewer
#26 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Alexander Shishkin
Quoted 16 lines · Peter Zijlstra

arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537

Quoted 12 lines · Alexander Shishkin

+++ b/arch/x86/kernel/cpu/perf_event_intel.c

arch/x86/kernel/cpu/perf_event_intel.c +7 -0
@@ -1528,6 +1528,14 @@ again:
 	}
 
 	/*
+	 * Intel PT
+	 */
+	if (__test_and_clear_bit(55, (unsigned long *)&status)) {
+		handled++;
+		intel_pt_interrupt();
+	}
+

How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like FREEZE_ON_PMI etc?

Quoted 7 lines
Quoted 3 lines

It ignores the FREEZE_ON_PMI bit. I stop it by hand inside the PMI handler, so you can see parts of the handler in the trace if you're tracing the kernel.

Urgh, horrid that. Routing something to the same interrupt, sharing status registers but not observing the same semantics for the interrupt is a massive fail.

I can't pretend to understand the logic behind this either.

Quoted 7 lines

IIRC Andi was planning to start using FREEZE_ON_PMI to avoid the MSR writes in intel_pmu_{disable,enable}_all(), this interrupt not actually respecting that makes that non-trivial.

We already use FREEZE_ON_PMI for LBR, but for now PT and LBR are mutually exclusive so that's not a problem, if we ever get those working together this needs to get fixed.

Agreed.

Regards,
--
Alex

maintainer
#27 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 7 lines · Alexander Shishkin

+/*
+ * Capabilities of Intel PT hardware, such as number of address bits or
+ * supported output schemes, are cached and exported to userspace as "caps"
+ * attribute group of pt pmu device
+ * (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store
+ * relevant bits together with intel_pt traces.
+ */

Does it make sense to use that AUX record variable data thing for this instead -- much like what the ARM folks wanted?

maintainer
#28 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

arch/x86/kernel/cpu/perf_event_intel_pt.c line 127

Quoted 9 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
+	if (test_cpu_cap(&boot_cpu_data, X86_FEATURE_INTEL_PT)) {
+		for (i = 0; i < PT_CPUID_LEAVES; i++)
+			cpuid_count(20, i,
+				    &pt_pmu.caps[CR_EAX + i * 4],
+				    &pt_pmu.caps[CR_EBX + i * 4],
+				    &pt_pmu.caps[CR_ECX + i * 4],
+				    &pt_pmu.caps[CR_EDX + i * 4]);
+	} else
+		return -ENODEV;

Please use {} for every multi-line stmt, even if its a single stmt, also use {} in both branches of a condition, even if only one strictly requires it.

author reviewer
#29 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 14 lines · Peter Zijlstra

arch/x86/kernel/cpu/perf_event_intel_pt.c line 127

Quoted 9 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
+	if (test_cpu_cap(&boot_cpu_data, X86_FEATURE_INTEL_PT)) {
+		for (i = 0; i < PT_CPUID_LEAVES; i++)
+			cpuid_count(20, i,
+				    &pt_pmu.caps[CR_EAX + i * 4],
+				    &pt_pmu.caps[CR_EBX + i * 4],
+				    &pt_pmu.caps[CR_ECX + i * 4],
+				    &pt_pmu.caps[CR_EDX + i * 4]);
+	} else
+		return -ENODEV;

Please use {} for every multi-line stmt, even if its a single stmt, also use {} in both branches of a condition, even if only one strictly requires it.

Ok.

maintainer
#30 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 40 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +40 -0
+
+enum cpuid_regs {
+	CR_EAX = 0,
+	CR_ECX,
+	CR_EDX,
+	CR_EBX
+};
+
+/*
+ * Capabilities of Intel PT hardware, such as number of address bits or
+ * supported output schemes, are cached and exported to userspace as "caps"
+ * attribute group of pt pmu device
+ * (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store
+ * relevant bits together with intel_pt traces.
+ */
+#define PT_CAP(_n, _l, _r, _m)						\
+	[PT_CAP_ ## _n] = { .name = __stringify(_n), .leaf = _l,	\
+			    .reg = _r, .mask = _m }
+
+static struct pt_cap_desc {
+	const char	*name;
+	u32		leaf;
+	u8		reg;
+	u32		mask;
+} pt_caps[] = {
+	PT_CAP(max_subleaf,		0, CR_EAX, 0xffffffff),
+	PT_CAP(cr3_filtering,		0, CR_EBX, BIT(0)),
+	PT_CAP(topa_output,		0, CR_ECX, BIT(0)),
+	PT_CAP(topa_multiple_entries,	0, CR_ECX, BIT(1)),
+	PT_CAP(payloads_lip,		0, CR_ECX, BIT(31)),
+};
+
+static u32 pt_cap_get(enum pt_capabilities cap)
+{
+	struct pt_cap_desc *cd = &pt_caps[cap];
+	u32 c = pt_pmu.caps[cd->leaf * 4 + cd->reg];
+	unsigned int shift = __ffs(cd->mask);
+
+	return (c & cd->mask) >> shift;
+}

arch/x86/kernel/cpu/perf_event_intel_pt.c line 127

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
 
+	if (test_cpu_cap(&boot_cpu_data, X86_FEATURE_INTEL_PT)) {
+		for (i = 0; i < PT_CPUID_LEAVES; i++)
+			cpuid_count(20, i,
+				    &pt_pmu.caps[CR_EAX + i * 4],
+				    &pt_pmu.caps[CR_EBX + i * 4],
+				    &pt_pmu.caps[CR_ECX + i * 4],
+				    &pt_pmu.caps[CR_EDX + i * 4]);
+	} else
+		return -ENODEV;

I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().

author reviewer
#31 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 42 lines · Peter Zijlstra
Quoted 40 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +40 -0
+
+enum cpuid_regs {
+	CR_EAX = 0,
+	CR_ECX,
+	CR_EDX,
+	CR_EBX
+};
+
+/*
+ * Capabilities of Intel PT hardware, such as number of address bits or
+ * supported output schemes, are cached and exported to userspace as "caps"
+ * attribute group of pt pmu device
+ * (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store
+ * relevant bits together with intel_pt traces.
+ */
+#define PT_CAP(_n, _l, _r, _m)						\
+	[PT_CAP_ ## _n] = { .name = __stringify(_n), .leaf = _l,	\
+			    .reg = _r, .mask = _m }
+
+static struct pt_cap_desc {
+	const char	*name;
+	u32		leaf;
+	u8		reg;
+	u32		mask;
+} pt_caps[] = {
+	PT_CAP(max_subleaf,		0, CR_EAX, 0xffffffff),
+	PT_CAP(cr3_filtering,		0, CR_EBX, BIT(0)),
+	PT_CAP(topa_output,		0, CR_ECX, BIT(0)),
+	PT_CAP(topa_multiple_entries,	0, CR_ECX, BIT(1)),
+	PT_CAP(payloads_lip,		0, CR_ECX, BIT(31)),
+};
+
+static u32 pt_cap_get(enum pt_capabilities cap)
+{
+	struct pt_cap_desc *cd = &pt_caps[cap];
+	u32 c = pt_pmu.caps[cd->leaf * 4 + cd->reg];
+	unsigned int shift = __ffs(cd->mask);
+
+	return (c & cd->mask) >> shift;
+}
Quoted 13 lines

arch/x86/kernel/cpu/perf_event_intel_pt.c line 127

Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
+	if (test_cpu_cap(&boot_cpu_data, X86_FEATURE_INTEL_PT)) {
+		for (i = 0; i < PT_CPUID_LEAVES; i++)
+			cpuid_count(20, i,
+				    &pt_pmu.caps[CR_EAX + i * 4],
+				    &pt_pmu.caps[CR_EBX + i * 4],
+				    &pt_pmu.caps[CR_ECX + i * 4],
+				    &pt_pmu.caps[CR_EDX + i * 4]);
+	} else
+		return -ENODEV;

I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().

It looks like it would only work for the first cpuid leaf, but we'll need more than that. And the array makes it easier to allocate attributes for sysfs en masse.

I guess it doesn't really matter if we use unions unless these bits need to be exported to other parts of the kernel? But *that* is hardly a good idea. What do you think?

Regards,
--
Alex

maintainer
#32 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 14 lines · Alexander Shishkin
Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5 -0
+	if (test_cpu_cap(&boot_cpu_data, X86_FEATURE_INTEL_PT)) {
+		for (i = 0; i < PT_CPUID_LEAVES; i++)
+			cpuid_count(20, i,
+				    &pt_pmu.caps[CR_EAX + i * 4],
+				    &pt_pmu.caps[CR_EBX + i * 4],
Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_pt.c line 127

Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +4 -0
+				    &pt_pmu.caps[CR_ECX + i * 4],
+				    &pt_pmu.caps[CR_EDX + i * 4]);
+	} else
+		return -ENODEV;

I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().

Quoted 7 lines

It looks like it would only work for the first cpuid leaf, but we'll need more than that. And the array makes it easier to allocate attributes for sysfs en masse.

I guess it doesn't really matter if we use unions unless these bits need to be exported to other parts of the kernel? But *that* is hardly a good idea. What do you think?

Ah yes, the generation. C is lacking there isn't it :/

Now I'm not sure we want to export all the bits you're using though. Like the topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.

author reviewer
#33 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 23 lines · Peter Zijlstra
Quoted 14 lines · Alexander Shishkin
Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5 -0
+	if (test_cpu_cap(&boot_cpu_data, X86_FEATURE_INTEL_PT)) {
+		for (i = 0; i < PT_CPUID_LEAVES; i++)
+			cpuid_count(20, i,
+				    &pt_pmu.caps[CR_EAX + i * 4],
+				    &pt_pmu.caps[CR_EBX + i * 4],
Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_pt.c line 127

Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +4 -0
+				    &pt_pmu.caps[CR_ECX + i * 4],
+				    &pt_pmu.caps[CR_EDX + i * 4]);
+	} else
+		return -ENODEV;

I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().

Quoted 7 lines

It looks like it would only work for the first cpuid leaf, but we'll need more than that. And the array makes it easier to allocate attributes for sysfs en masse.

I guess it doesn't really matter if we use unions unless these bits need to be exported to other parts of the kernel? But *that* is hardly a good idea. What do you think?

Quoted 5 lines

Ah yes, the generation. C is lacking there isn't it :/

Now I'm not sure we want to export all the bits you're using though. Like the topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.

Actually, userspace can make assumptions about lost data from this bit. But there are others, such as encoded address length. In future there will also be several "caps" with allowed values for certain bit fields in RTIT_CTL, such as timing packet frequencies.

Regards,
--
Alex

maintainer
#34 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 8 lines · Alexander Shishkin
Quoted 3 lines · Peter Zijlstra

Now I'm not sure we want to export all the bits you're using though. Like the topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.

Actually, userspace can make assumptions about lost data from this bit.

Do explain; it feels entirely wrong to expose something like this, esp. since this crappy single TOPA thing is going away.

Quoted 3 lines

But there are others, such as encoded address length. In future there will also be several "caps" with allowed values for certain bit fields in RTIT_CTL, such as timing packet frequencies.

No, those should not be caps, those should be in the format description.

author reviewer
#35 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Alexander Shishkin
Quoted 3 lines · Peter Zijlstra

Now I'm not sure we want to export all the bits you're using though. Like the topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.

Quoted 5 lines
Quoted 2 lines

Actually, userspace can make assumptions about lost data from this bit.

Do explain; it feels entirely wrong to expose something like this, esp. since this crappy single TOPA thing is going away.

At the very least, it gives you the minimal amount of memory you can ask for the aux buffer. With multiple entry topa it is one page, with single entry topa it is two pages. You can leave it up to the userspace to find out by trial and error, of course.

Quoted 5 lines
Quoted 3 lines

But there are others, such as encoded address length. In future there will also be several "caps" with allowed values for certain bit fields in RTIT_CTL, such as timing packet frequencies.

No, those should not be caps, those should be in the format description.

Format description gives a bit field, within which not all combinations may be valid. That is,

format/blah_freq: config:16-19
caps/blah_freq: 81

which means that on this cpu you can only set blah_freq to either 0 or
7. Figuring out these things by trial and error from userspace is not really nice.

Regards,
--
Alex

maintainer
#36 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

arch/x86/kernel/cpu/perf_event_intel_pt.c line 176

Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +13 -0
+static bool pt_event_valid(struct perf_event *event)
+{
+	u64 config = event->attr.config;
+
+	/* admin can set any packet generation parameters */
+	if (capable(CAP_SYS_ADMIN) && (config & PT_BYPASS_MASK) == config)
+		return true;
+
+	if ((config & PT_CONFIG_MASK) != config)
+		return false;
+
+	return true;
+}

This seems to suggest PT is available to !priv users, is this right?

author reviewer
#37 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 16 lines · Peter Zijlstra

arch/x86/kernel/cpu/perf_event_intel_pt.c line 176

Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +13 -0
+static bool pt_event_valid(struct perf_event *event)
+{
+	u64 config = event->attr.config;
+
+	/* admin can set any packet generation parameters */
+	if (capable(CAP_SYS_ADMIN) && (config & PT_BYPASS_MASK) == config)
+		return true;
+
+	if ((config & PT_CONFIG_MASK) != config)
+		return false;
+
+	return true;
+}

This seems to suggest PT is available to !priv users, is this right?

Yes, that's the intention. PT does CPL-based filtering in hardware and unlike previous attempts at branch tracing, does not leak out-of-context addresses (or any other information for that matter).

Regards,
--
Alex

maintainer
#38 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 12 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +12 -0
+static int pt_config(struct perf_event *event)
+{
+	u64 reg;
+
+	reg = RTIT_CTL_TOPA | RTIT_CTL_BRANCH_EN;
+
+	if (!event->attr.exclude_kernel)
+		reg |= RTIT_CTL_OS;
+	if (!event->attr.exclude_user)
+		reg |= RTIT_CTL_USR;
+
+	reg |= (event->attr.config & PT_CONFIG_MASK);

arch/x86/kernel/cpu/perf_event_intel_pt.c line 213

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+
+	/*
+	 * User can try to set bits in RTIT_CTL through PT_BYPASS_MASK,
+	 * that aren't supported by the hardware. Weather or not a
+	 * particular bitmask is supported by a cpu can't be determined
+	 * via cpuid or otherwise, so we have to rely on #GP handling
+	 * to catch these cases.
+	 */
+	return wrmsrl_safe(MSR_IA32_RTIT_CTL, reg);
+}

Whether the weather is nice or not :-)

But no, this cannot be, once we've accepted the event is must be programmable. Failing at the time of programming is vile; pmu::start() is a void return, failure is not an option there.

The fact that the hardware cannot even tell you the supported mask is further fail.

IIRC I think Andi once suggested probing each of the 64 bits in that MSR to determine the supported mask at device init time.

BTW, what's that RTIT thing? Did someone forget to propagate the latest name change of the thing or so?

author reviewer
#39 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 26 lines · Peter Zijlstra
Quoted 12 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +12 -0
+static int pt_config(struct perf_event *event)
+{
+	u64 reg;
+
+	reg = RTIT_CTL_TOPA | RTIT_CTL_BRANCH_EN;
+
+	if (!event->attr.exclude_kernel)
+		reg |= RTIT_CTL_OS;
+	if (!event->attr.exclude_user)
+		reg |= RTIT_CTL_USR;
+
+	reg |= (event->attr.config & PT_CONFIG_MASK);

arch/x86/kernel/cpu/perf_event_intel_pt.c line 213

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+
+	/*
+	 * User can try to set bits in RTIT_CTL through PT_BYPASS_MASK,
+	 * that aren't supported by the hardware. Weather or not a
+	 * particular bitmask is supported by a cpu can't be determined
+	 * via cpuid or otherwise, so we have to rely on #GP handling
+	 * to catch these cases.
+	 */
+	return wrmsrl_safe(MSR_IA32_RTIT_CTL, reg);
+}

Whether the weather is nice or not :-)

Quoted 3 lines

But no, this cannot be, once we've accepted the event is must be programmable. Failing at the time of programming is vile; pmu::start() is a void return, failure is not an option there.

This is called from pmu::add(), which can fail. If this wrmsrl throws a gp, pmu::add() will fail and we won't even get to pmu::start(). Of course, the problem with such event faulting every time it is added is still there. Maybe we can simply disable such events after the first fault. Good news is, only a CAP_SYS_ADMIN can set arbitrary bits, so the damage is limited.

Quoted 5 lines

The fact that the hardware cannot even tell you the supported mask is further fail.

IIRC I think Andi once suggested probing each of the 64 bits in that MSR to determine the supported mask at device init time.

The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.

Regards,
--
Alex

maintainer
#40 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 5 lines · Alexander Shishkin
Quoted 2 lines

The fact that the hardware cannot even tell you the supported mask is further fail.

The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.

OK, so put the magic number in the big model array.

author reviewer
#41 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines

The fact that the hardware cannot even tell you the supported mask is further fail.

The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.

OK, so put the magic number in the big model array.

I'm not sure I follow. These bits are reserved for the future, they can potentially be whatever combinations of whatever. If we want to probe around for valid combinations is to check everything in the range of 0..2^43 (or something like that, the region reserved for packet enables) and store all the valid ones, which sounds crazy.

maintainer
#42 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 9 lines · Alexander Shishkin
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines

The fact that the hardware cannot even tell you the supported mask is further fail.

The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.

Quoted 7 lines
Quoted 1 line

OK, so put the magic number in the big model array.

I'm not sure I follow. These bits are reserved for the future, they can potentially be whatever combinations of whatever. If we want to probe around for valid combinations is to check everything in the range of 0..2^43 (or something like that, the region reserved for packet enables) and store all the valid ones, which sounds crazy.

I was assuming that the accepted bits are model specific, and we have this big model switch statement in perf_event_intel.c:intel_pmu_init(), so why not have something like x86_pmu.pt_magic_bitmask = 0xf00d in there?

No need to probe in that case. That is the same thing we do for all unenumerated model specific things.

author reviewer
#43 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 18 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines

The fact that the hardware cannot even tell you the supported mask is further fail.

The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.

Quoted 7 lines
Quoted 1 line

OK, so put the magic number in the big model array.

I'm not sure I follow. These bits are reserved for the future, they can potentially be whatever combinations of whatever. If we want to probe around for valid combinations is to check everything in the range of 0..2^43 (or something like that, the region reserved for packet enables) and store all the valid ones, which sounds crazy.

Quoted 4 lines

I was assuming that the accepted bits are model specific, and we have this big model switch statement in perf_event_intel.c:intel_pmu_init(), so why not have something like x86_pmu.pt_magic_bitmask = 0xf00d in there?

Ah, I see what you mean. The main point of this whole reserved region proposal is that one shouldn't have to update one's kernel to enable new PT packets by doing -e intel_pt/config=0xf00d/, if one is CAP_SYS_ADMIN.

Quoted 2 lines

No need to probe in that case. That is the same thing we do for all unenumerated model specific things.

They are, actually, enumerated, we just want to be able to enable them before they are. If the driver is aware of feature X, it can test for it in CPUID and allow/disallow it based on that.

Regards,
--
Alex

maintainer
#44 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 6 lines · Alexander Shishkin

Peter Zijlstra <[email protected]> writes:

Ah, I see what you mean. The main point of this whole reserved region proposal is that one shouldn't have to update one's kernel to enable new PT packets by doing -e intel_pt/config=0xf00d/, if one is CAP_SYS_ADMIN.

Quoted 6 lines
Quoted 2 lines

No need to probe in that case. That is the same thing we do for all unenumerated model specific things.

They are, actually, enumerated, we just want to be able to enable them before they are. If the driver is aware of feature X, it can test for it in CPUID and allow/disallow it based on that.

If you've got hardware that 'exposes' functionality not enumerated in CPUID you have the capability of modifying the kernel.

And while it might seem like a cute feature when you have early hardware, that is no reason to make the code horrible. Either just patch your firmware such that CPUID reports the right bits for your hardware or frob the kernel.

maintainer
#45 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 12 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +12 -0
+static void pt_config_start(bool start)
+{
+	u64 ctl;
+
+	rdmsrl(MSR_IA32_RTIT_CTL, ctl);
+	if (start)
+		ctl |= RTIT_CTL_TRACEEN;
+	else
+		ctl &= ~RTIT_CTL_TRACEEN;
+	wrmsrl(MSR_IA32_RTIT_CTL, ctl);
+
+	/*

arch/x86/kernel/cpu/perf_event_intel_pt.c line 236

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+	 * A wrmsr that disables trace generation serializes other PT
+	 * registers and causes all data packets to be written to memory,
+	 * but a fence is required for the data to become globally visible.
+	 *
+	 * The below WMB, separating data store and aux_head store matches
+	 * the consumer's RMB that separates aux_head load and data load.
+	 */
+	if (!start)
+		wmb();
+}

wmb is sfence, is that sufficient? One would have expected an mfence since that would also orders later reads.

maintainer
#46 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +17 -0
+static void pt_config_start(bool start)
+{
+	u64 ctl;
+
+	rdmsrl(MSR_IA32_RTIT_CTL, ctl);
+	if (start)
+		ctl |= RTIT_CTL_TRACEEN;
+	else
+		ctl &= ~RTIT_CTL_TRACEEN;
+	wrmsrl(MSR_IA32_RTIT_CTL, ctl);
+
+	/*
+	 * A wrmsr that disables trace generation serializes other PT
+	 * registers and causes all data packets to be written to memory,
+	 * but a fence is required for the data to become globally visible.
+	 *
+	 * The below WMB, separating data store and aux_head store matches
Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_pt.c line 236

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5 -0
+	 * the consumer's RMB that separates aux_head load and data load.
+	 */
+	if (!start)
+		wmb();
+}

wmb is sfence, is that sufficient? One would have expected an mfence since that would also orders later reads.

Silly me, we're separating two stores here. Ignore that.

maintainer
#47 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

arch/x86/kernel/cpu/perf_event_intel_pt.c line 267

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +2 -0
+/* make negative table index stand for the last table entry */
+#define TOPA_ENTRY(t, i) ((i) == -1 ? &(t)->table[(t)->last] : &(t)->table[(i)])

code does not match comment; negative would be: i < 0, not i == -1.

Something like: ({ if (i < 0) i += t->size; t->table[i]; }), might work, of course that goes bang when: i !e [-size,size)

author reviewer
#48 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 4 lines · Peter Zijlstra

arch/x86/kernel/cpu/perf_event_intel_pt.c line 267

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +2 -0
+/* make negative table index stand for the last table entry */
+#define TOPA_ENTRY(t, i) ((i) == -1 ? &(t)->table[(t)->last] : &(t)->table[(i)])

code does not match comment; negative would be: i < 0, not i == -1.

Indeed.

Quoted 2 lines

Something like: ({ if (i < 0) i += t->size; t->table[i]; }), might work, of course that goes bang when: i !e [-size,size)

Table indices and sizes are measured in different units. Using -1 to mean the last is just a small convenience, in the code it would be used explicitly as TOPA_ENTRY(topa, -1).end = 1, for example.

Regards,
--
Alex

maintainer
#49 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin

arch/x86/kernel/cpu/perf_event_intel_pt.c line 485

Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +11 -0
+			if ((pt_cap_get(PT_CAP_topa_multiple_entries)
+			     && topa->table[i].stop)
+			    || topa->table[i].end)
+				break;
 
+		old = (local64_xchg(&buf->head, base) &
+		       ((buf->nr_pages << PAGE_SHIFT) - 1));
 
+		if (!pt_cap_get(PT_CAP_topa_multiple_entries) ||
+		    buf->output_off == sizes(TOPA_ENTRY(buf->cur, buf->cur_idx)->size)) {
+			local_inc(&buf->lost);
+			advance++;
+		}

Inconsistent coding style. Please put operators at the end of a line, not at the start of one.

author reviewer
#50 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +6 -0
+			if ((pt_cap_get(PT_CAP_topa_multiple_entries)
+			     && topa->table[i].stop)
+			    || topa->table[i].end)
+				break;
 
+		old = (local64_xchg(&buf->head, base) &
+		       ((buf->nr_pages << PAGE_SHIFT) - 1));
Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_pt.c line 485

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5 -0
+		if (!pt_cap_get(PT_CAP_topa_multiple_entries) ||
+		    buf->output_off == sizes(TOPA_ENTRY(buf->cur, buf->cur_idx)->size)) {
+			local_inc(&buf->lost);
+			advance++;
+		}

Inconsistent coding style. Please put operators at the end of a line, not at the start of one.

Ok.

maintainer
#51 lore raw

Re: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver

In reply to Alexander Shishkin
Quoted 9 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9 -0
+static void pt_event_start(struct perf_event *event, int mode)
+{
+	struct pt *pt = this_cpu_ptr(&pt_ctx);
+	struct pt_buffer *buf = perf_get_aux(&pt->handle);
+
+	if (pt_is_running() || !buf || pt_buffer_is_full(buf, pt) ||
+	    pt_config(event)) {
+		event->hw.state = PERF_HES_STOPPED;
+		return;

arch/x86/kernel/cpu/perf_event_intel_pt.c line 823

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10 -0
+	}
+
+	ACCESS_ONCE(pt->handle_nmi) = 1;
+	event->hw.state = 0;
+
+	pt_config_buffer(buf->cur->table, buf->cur_idx,
+			 buf->output_off);
+	wrmsrl(MSR_IA32_RTIT_STATUS, 0);
+	pt_config_start(true);
+}

That's two RTIT_CTL writes, should we not try and merge those?

author reviewer
#52 lore raw

PATCH v5 13/20 x86: perf: intel_bts: Add BTS PMU driver

In reply to Alexander Shishkin

Add support for Branch Trace Store (BTS) via kernel perf event infrastructure. The difference with the existing implementation of BTS support is that this one is a separate PMU that exports events' trace buffers to userspace by means of AUX area of the perf buffer, which is zero-copy mapped into userspace.

The immediate benefit is that the buffer size can be much bigger, resulting in fewer interrupts and no kernel side copying is involved and little to no trace data loss. Also, kernel code can be traced with this driver.

The old way of collecting BTS traces still works.

---
 arch/x86/kernel/cpu/Makefile               |   2 +-
 arch/x86/kernel/cpu/perf_event.h           |   6 +
 arch/x86/kernel/cpu/perf_event_intel.c     |   6 +-
 arch/x86/kernel/cpu/perf_event_intel_bts.c | 518 +++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_ds.c  |   3 +-
 5 files changed, 532 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
arch/x86/kernel/cpu/Makefile +1 -1
index 00d40f889d..387223c716 100644
@@ -39,7 +39,7 @@ obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o per
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore.o perf_event_intel_uncore_snb.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_uncore_snbep.o perf_event_intel_uncore_nhmex.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_rapl.o
-obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o perf_event_intel_bts.o
 endif
 
 
arch/x86/kernel/cpu/perf_event.h +6 -0
index 9f9b02fea6..155721e54d 100644
@@ -752,6 +752,12 @@ int intel_pmu_setup_lbr_filter(struct perf_event *event);
 
 void intel_pt_interrupt(void);
 
+int intel_bts_interrupt(void);
+
+void intel_bts_enable_local(void);
+
+void intel_bts_disable_local(void);
+
 int p4_pmu_init(void);
 
 int p6_pmu_init(void);
arch/x86/kernel/cpu/perf_event_intel.c +5 -1
index 6e04da1b04..0124c0d868 100644
@@ -1180,6 +1180,8 @@ static void intel_pmu_disable_all(void)
 
 	if (test_bit(INTEL_PMC_IDX_FIXED_BTS, cpuc->active_mask))
 		intel_pmu_disable_bts();
+	else
+		intel_bts_disable_local();
 
 	intel_pmu_pebs_disable_all();
 	intel_pmu_lbr_disable_all();
@@ -1202,7 +1204,8 @@ static void intel_pmu_enable_all(int added)
 			return;
 
 		intel_pmu_enable_bts(event->hw.config);
-	}
+	} else
+		intel_bts_enable_local();
 }
 
 /*
@@ -1488,6 +1491,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 		apic_write(APIC_LVTPC, APIC_DM_NMI);
 	intel_pmu_disable_all();
 	handled = intel_pmu_drain_bts_buffer();
+	handled += intel_bts_interrupt();
 	status = intel_pmu_get_status();
 	if (!status)
 		goto done;
arch/x86/kernel/cpu/perf_event_intel_bts.c new file +518 -0

518 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_intel_ds.c +2 -1
index 865ee1268a..ed3cb75327 100644
@@ -466,7 +466,8 @@ void intel_pmu_enable_bts(u64 config)
 
 	debugctlmsr |= DEBUGCTLMSR_TR;
 	debugctlmsr |= DEBUGCTLMSR_BTS;
-	debugctlmsr |= DEBUGCTLMSR_BTINT;
+	if (config & ARCH_PERFMON_EVENTSEL_INT)
+		debugctlmsr |= DEBUGCTLMSR_BTINT;
 
 	if (!(config & ARCH_PERFMON_EVENTSEL_OS))
 		debugctlmsr |= DEBUGCTLMSR_BTS_OFF_OS;
Signature · 1 line
2.1.0
author reviewer
#53 lore raw

PATCH v5 14/20 perf: add ITRACE_START record to indicate that tracing has started

In reply to Alexander Shishkin

For counters that generate AUX data that is bound to the context of a running task, such as instruction tracing, the decoder needs to know exactly which task is running when the event is first scheduled in, before the first sched_switch. The decoder's need to know this stems from the fact that instruction flow trace decoding will almost always require program's object code in order to reconstruct said flow and for that we need at least its pid/tid in the perf stream.

To single out such instruction tracing pmus, this patch introduces ITRACE PMU capability. The reason this is not part of RECORD_AUX record is that not all pmus capable of generating AUX data need this, and the opposite is *probably* also true.

While sched_switch covers for most cases, there are two problems with it:
the consumer will need to process events out of order (that is, having found RECORD_AUX, it will have to skip forward to the nearest sched_switch to figure out which task it was, then go back to the actual trace to decode it) and it completely misses the case when the tracing is enabled and disabled before sched_switch, for example, via PERF_EVENT_IOC_DISABLE.

---
 include/linux/perf_event.h      |  4 ++++
 include/uapi/linux/perf_event.h | 11 +++++++++++
 kernel/events/core.c            | 41 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 56 insertions(+)
include/linux/perf_event.h +4 -0
index f126eb89e6..282721b2df 100644
@@ -127,6 +127,9 @@ struct hw_perf_event {
 			/* for tp_event->class */
 			struct list_head	tp_list;
 		};
+		struct { /* itrace */
+			int			itrace_started;
+		};
 #ifdef CONFIG_HAVE_HW_BREAKPOINT
 		struct { /* breakpoint */
 			/*
@@ -174,6 +177,7 @@ struct perf_event;
 #define PERF_PMU_CAP_AUX_NO_SG			0x02
 #define PERF_PMU_CAP_AUX_SW_DOUBLEBUF		0x04
 #define PERF_PMU_CAP_EXCLUSIVE			0x08
+#define PERF_PMU_CAP_ITRACE			0x10
 
 /**
  * struct pmu - generic performance monitoring unit
include/uapi/linux/perf_event.h +11 -0
index c20aaa5af7..a0cafbdc1c 100644
@@ -750,6 +750,17 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_AUX				= 11,
 
+	/*
+	 * Indicates that instruction trace has started
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u32				pid;
+	 *	u32				tid;
+	 * };
+	 */
+	PERF_RECORD_ITRACE_START		= 12,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +41 -0

59 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#54 lore raw

PATCH v5 15/20 perf: Add api to (de-)allocate AUX buffers for kernel counters

In reply to Alexander Shishkin

Several use cases for AUX data, namely event sampling (including a piece of AUX data in some perf event sample, so that the user can get, for example, instruction traces leading up to a certain event like a breakpoint or a hardware event), process core dumps (providing user with a history of a process' instruction flow leading up to a crash), system crash dumps and storing AUX data in pstore across reboot (to facilitate post-mortem investigation of a system crash) require different parts of the kernel code to be able to configure hardware to produce AUX data and collect it when it is needed.

Luckily, there is already an api for in-kernel perf events, which has several users. This proposal is to extend that api to allow in-kernel users to allocate AUX buffers for kernel counters. Such users will call rb_alloc_kernel() to allocate what they want and later copy the data out to other backends, e.g. a sample in another event's ring buffer or a core dump file. These buffers are never mapped to userspace.

There are no additional constraints or requirements on the pmu drivers.

A typical user of this interface will first create a kernel counter with a call to perf_event_create_kernel_counter() and then allocate a ring buffer for it with rb_alloc_kernel(). Data can then be copied out from the AUX buffer using rb_output_aux(), which is passed a callback that will write chunks of AUX data into the desired destination, such as perf_output_copy() or dump_emit(). Caller needs to use perf_event_disable to make sure that the counter is not active while it copies data out.

---
 kernel/events/internal.h    |  8 ++++++
 kernel/events/ring_buffer.c | 64 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)
kernel/events/internal.h +8 -0
index 4715aae48b..81cb7afec4 100644
@@ -54,6 +54,9 @@ struct ring_buffer {
 	void				*data_pages[0];
 };
 
+typedef unsigned long (*aux_copyfn)(void *data, const void *src,
+				    unsigned long len);
+
 extern void rb_free(struct ring_buffer *rb);
 extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
@@ -61,6 +64,11 @@ extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 			pgoff_t pgoff, int nr_pages, long watermark, int flags);
 extern void rb_free_aux(struct ring_buffer *rb);
+extern long rb_output_aux(struct ring_buffer *rb, unsigned long from,
+			  unsigned long to, aux_copyfn copyfn, void *data);
+extern struct ring_buffer *
+rb_alloc_kernel(struct perf_event *event, int nr_pages, int aux_nr_pages);
+extern void rb_free_kernel(struct ring_buffer *rb, struct perf_event *event);
 extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
 extern void ring_buffer_put(struct ring_buffer *rb);
 
kernel/events/ring_buffer.c +64 -0

76 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#55 lore raw

PATCH v5 16/20 perf: Add a helper for looking up pmus by type

In reply to Alexander Shishkin

This patch adds a helper for looking up a registered pmu by its type.

---
 kernel/events/core.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
kernel/events/core.c +13 -3
index d4057dff27..92da1aecc7 100644
@@ -6963,6 +6963,18 @@ void perf_pmu_unregister(struct pmu *pmu)
 }
 EXPORT_SYMBOL_GPL(perf_pmu_unregister);
 
+/* call under pmus_srcu */
+static struct pmu *__perf_find_pmu(u32 type)
+{
+	struct pmu *pmu;
+
+	rcu_read_lock();
+	pmu = idr_find(&pmu_idr, type);
+	rcu_read_unlock();
+
+	return pmu;
+}
+
 struct pmu *perf_init_event(struct perf_event *event)
 {
 	struct pmu *pmu = NULL;
@@ -6971,9 +6983,7 @@ struct pmu *perf_init_event(struct perf_event *event)
 
 	idx = srcu_read_lock(&pmus_srcu);
 
-	rcu_read_lock();
-	pmu = idr_find(&pmu_idr, event->attr.type);
-	rcu_read_unlock();
+	pmu = __perf_find_pmu(event->attr.type);
 	if (pmu) {
 		if (!try_module_get(pmu->module)) {
 			pmu = ERR_PTR(-ENODEV);
Signature · 1 line
2.1.0
author reviewer
#56 lore raw

PATCH v5 17/20 perf: Add infrastructure for using AUX data in perf samples

In reply to Alexander Shishkin

AUX data can be used to annotate perf events such as performance counters or tracepoints/breakpoints by including it in sample records when PERF_SAMPLE_AUX flag is set. Such samples would be instrumental in debugging and profiling by providing, for example, a history of instruction flow leading up to the event's overflow.

To facilitate this, this patch adds code to create a kernel counter with a ring buffer using rb_{alloc,free}_kernel() interface for each event that needs to include AUX samples and to copy AUX data from it into the perf data stream.

The user interface is extended to allow for this, new attribute fields are added:

* aux_sample_type: specify PMU on which the AUX data generating event

                     is created;

* aux_sample_config: event config (maps to attribute's config field),
* aux_sample_size: size of the sample to be written.

This kernel counter is configured similarly to the event that is being annotated with regards to filtering (exclude_{hv,idle,user,kernel}) and enabled state (disabled, enable_on_exec) to make sure that the two events are scheduled at the same time and that no out of context activity is sampled.

---
 include/linux/perf_event.h      |   9 +++
 include/uapi/linux/perf_event.h |  18 ++++-
 kernel/events/core.c            | 172 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 198 insertions(+), 1 deletion(-)
include/linux/perf_event.h +9 -0
index 282721b2df..de8cc714e9 100644
@@ -84,6 +84,12 @@ struct perf_regs_user {
 	struct pt_regs	*regs;
 };
 
+struct perf_aux_record {
+	u64		size;
+	unsigned long	from;
+	unsigned long	to;
+};
+
 struct task_struct;
 
 /*
@@ -458,6 +464,7 @@ struct perf_event {
 	perf_overflow_handler_t		overflow_handler;
 	void				*overflow_handler_context;
 
+	struct perf_event		*sampler;
 #ifdef CONFIG_EVENT_TRACING
 	struct ftrace_event_call	*tp_event;
 	struct event_filter		*filter;
@@ -628,6 +635,7 @@ struct perf_sample_data {
 	union  perf_mem_data_src	data_src;
 	struct perf_callchain_entry	*callchain;
 	struct perf_raw_record		*raw;
+	struct perf_aux_record		aux;
 	struct perf_branch_stack	*br_stack;
 	struct perf_regs_user		regs_user;
 	u64				stack_user_size;
@@ -655,6 +663,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data,
 	data->period = period;
 	data->regs_user.abi = PERF_SAMPLE_REGS_ABI_NONE;
 	data->regs_user.regs = NULL;
+	data->aux.from = data->aux.to = data->aux.size = 0;
 	data->stack_user_size = 0;
 	data->weight = 0;
 	data->data_src.val = PERF_MEM_NA;
include/uapi/linux/perf_event.h +17 -1
index a0cafbdc1c..ed2e21fa13 100644
@@ -137,8 +137,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_DATA_SRC			= 1U << 15,
 	PERF_SAMPLE_IDENTIFIER			= 1U << 16,
 	PERF_SAMPLE_TRANSACTION			= 1U << 17,
+	PERF_SAMPLE_AUX				= 1U << 18,
 
-	PERF_SAMPLE_MAX = 1U << 18,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 19,		/* non-ABI */
 };
 
 /*
@@ -239,6 +240,9 @@ enum perf_event_read_format {
 #define PERF_ATTR_SIZE_VER3	96	/* add: sample_regs_user */
 					/* add: sample_stack_user */
 					/* add: aux_watermark */
+#define PERF_ATTR_SIZE_VER4	120	/* add: aux_sample_config */
+					/* add: aux_sample_size */
+					/* add: aux_sample_type */
 
 /*
  * Hardware event_id to monitor via a performance monitoring event:
@@ -337,6 +341,16 @@ struct perf_event_attr {
 	 * Wakeup watermark for AUX area
 	 */
 	__u32	aux_watermark;
+
+	/*
+	 * Itrace pmus' event config
+	 */
+	__u64	aux_sample_config;	/* event config for AUX sampling */
+	__u64	aux_sample_size;	/* desired sample size */
+	__u32	aux_sample_type;	/* pmu->type of an AUX PMU */
+
+	/* Align to u64. */
+	__u32	__reserved_2;
 };
 
 #define perf_flags(attr)	(*(&(attr)->read_format + 1))
@@ -710,6 +724,8 @@ enum perf_event_type {
 	 *	{ u64			weight;   } && PERF_SAMPLE_WEIGHT
 	 *	{ u64			data_src; } && PERF_SAMPLE_DATA_SRC
 	 *	{ u64			transaction; } && PERF_SAMPLE_TRANSACTION
+	 *	{ u64			size;
+	 *	  char			data[size]; } && PERF_SAMPLE_AUX
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +172 -0

220 lines not on this page · raw message

Signature · 1 line
2.1.0
author reviewer
#57 lore raw

PATCH v5 18/20 perf: Allocate ring buffers for inherited per-task kernel events

In reply to Alexander Shishkin

Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.

However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.

---
 kernel/events/core.c     |  9 +++++++++
 kernel/events/internal.h | 11 +++++++++++
 2 files changed, 20 insertions(+)
kernel/events/core.c +9 -0
index 5da1bc403f..60e354d668 100644
@@ -8267,6 +8267,15 @@ inherit_event(struct perf_event *parent_event,
 		= parent_event->overflow_handler_context;
 
 	/*
+	 * For per-task kernel events with ring buffers, set_output doesn't
+	 * make sense, but we can allocate a new buffer here.
+	 */
+	if (parent_event->cpu == -1 && kernel_rb_event(parent_event)) {
+		(void)rb_alloc_kernel(child_event, parent_event->rb->nr_pages,
+				      parent_event->rb->aux_nr_pages);
+	}
+
+	/*
 	 * Precalculate sample_data sizes
 	 */
 	perf_event__header_size(child_event);
kernel/events/internal.h +11 -0
index 81cb7afec4..373ac012f5 100644
@@ -122,6 +122,17 @@ static inline unsigned long perf_aux_size(struct ring_buffer *rb)
 	return rb->aux_nr_pages << PAGE_SHIFT;
 }
 
+static inline bool kernel_rb_event(struct perf_event *event)
+{
+	/*
+	 * Having a ring buffer and not being on any ring buffers' wakeup
+	 * list means it was attached by rb_alloc_kernel() and not
+	 * ring_buffer_attach(). It's the only case when these two
+	 * conditions take place at the same time.
+	 */
+	return event->rb && list_empty(&event->rb_entry);
+}
+
 #define DEFINE_OUTPUT_COPY(func_name, memcpy_func)			\
 static inline unsigned long						\
 func_name(struct perf_output_handle *handle,				\
Signature · 1 line
2.1.0
maintainer
#58 lore raw

Re: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events

Quoted 10 lines · Alexander Shishkin

Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.

However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.

Quoted 6 lines

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.

This causes a pinned memory explosion, not at all nice that.

I think I see why and all, but it would be ever so good to not have to allocate so much memory.

author reviewer
#59 lore raw

Re: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events

Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Alexander Shishkin

Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.

However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.

Quoted 6 lines

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.

Quoted 4 lines

This causes a pinned memory explosion, not at all nice that.

I think I see why and all, but it would be ever so good to not have to allocate so much memory.

Are there any controls we could use to limit such memory usage? Theoretically, the buffers that we'd allocate for this are way smaller than, for example, what we use if we try to capture a complete trace, since we'd only be interested in the most recent trace data. We already have RLIMIT_NPROC, which implicitly limits the number of these buffers, for example. Or maybe we can introduce a new rlimit/sysctl/whatnot that would limit the maximum amount of such memory per-cpu/system/user. What do you think?

Per-cpu buffers with inheritance would solve this problem, but raises other issues: we'd need sched_switch again to tell the traces apart and since those buffers run in overwrite mode, a cpu hog task can potentially overwrite any useful trace data.

Regards,
--
Alex

maintainer
#60 lore raw

Re: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events

Quoted 20 lines · Alexander Shishkin
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Alexander Shishkin

Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.

However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.

Quoted 6 lines
Quoted 4 lines

This causes a pinned memory explosion, not at all nice that.

I think I see why and all, but it would be ever so good to not have to allocate so much memory.

Are there any controls we could use to limit such memory usage?

I'd say the same limit we're already accounting the mmap()s against. But the question is; what do we do when we run out?

Will we fail clone()? That might 'surprise' quite a few people, that their application won't work when profiled.

In any case, lets focus on the other parts of this work and delay this feature till later.

author reviewer
#61 lore raw

Re: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events

Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Alexander Shishkin
Quoted 22 lines · Peter Zijlstra
Quoted 10 lines · Alexander Shishkin

Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.

However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.

Quoted 6 lines

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.

This causes a pinned memory explosion, not at all nice that.

I think I see why and all, but it would be ever so good to not have to allocate so much memory.

Quoted 7 lines
Quoted 1 line

Are there any controls we could use to limit such memory usage?

I'd say the same limit we're already accounting the mmap()s against. But the question is; what do we do when we run out?

Will we fail clone()? That might 'surprise' quite a few people, that their application won't work when profiled.

Or we just don't allocate any more buffers for this user; if there's a perf stream involved, we can output a record saying that.

Quoted 2 lines

In any case, lets focus on the other parts of this work and delay this feature till later.

Agreed.

Regards,
--
Alex

#62 lore raw

Re: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events

In reply to Peter Zijlstra

Em Thu, Oct 30, 2014 at 09:43:59AM +0100, Peter Zijlstra escreveu:

Quoted 14 lines · Alexander Shishkin
Quoted 12 lines
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin

When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.

This causes a pinned memory explosion, not at all nice that.

I think I see why and all, but it would be ever so good to not have to allocate so much memory.

Quoted 8 lines
Quoted 1 line

Are there any controls we could use to limit such memory usage?

I'd say the same limit we're already accounting the mmap()s against. But the question is; what do we do when we run out?

Will we fail clone()? That might 'surprise' quite a few people, that

their application won't work when profiled.

Can't we just emit PERF_RECORD_THROTTLE or similar stuff then? I.e. somehow mark the cloned process as not being profiled due to ENOMEM?

- Arnaldo

author reviewer
#63 lore raw

PATCH v5 19/20 perf: Allow AUX sampling for multiple events

In reply to Alexander Shishkin

Right now, only one perf event in a context at a time can be annotated with AUX data, because AUX pmus will normally have only one event scheduled at a time, however it should be possible to annotate several events with similar configuration (wrt exclude_{hv,idle,user,kernel}, config and other event attribute fields) using the same AUX event.

This patch changes the behavior so that every time, before a kernel counter is created for AUX sampling, we first look for an existing counter that is suitable in terms of configuration and we use it to annotate the new event as well. If no such event is found, we fall back to the old path and try to allocate a new kernel counter.

---
 kernel/events/core.c | 93 +++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 66 insertions(+), 27 deletions(-)
kernel/events/core.c +66 -27
index 60e354d668..ad7b1e92dd 100644
@@ -4651,45 +4651,84 @@ static void perf_aux_sampler_destroy(struct perf_event *event)
 	ring_buffer_put(rb); /* can be last */
 }
 
+static bool exclusive_event_match(struct perf_event *e1, struct perf_event *e2);
+
+static bool perf_aux_event_match(struct perf_event *e1, struct perf_event *e2)
+{
+	return has_aux(e1) && exclusive_event_match(e1, e2);
+}
+
 static int perf_aux_sampler_init(struct perf_event *event,
 				 struct task_struct *task,
 				 struct pmu *pmu)
 {
+	struct perf_event_context *ctx;
 	struct perf_event_attr attr;
-	struct perf_event *sampler;
+	struct perf_event *sampler = NULL;
 	struct ring_buffer *rb;
-	unsigned long nr_pages;
+	unsigned long nr_pages, flags;
 
 	if (!pmu || !(pmu->setup_aux))
 		return -ENOTSUPP;
 
-	memset(&attr, 0, sizeof(attr));
-	attr.type = pmu->type;
-	attr.config = event->attr.aux_sample_config;
-	attr.sample_type = 0;
-	attr.disabled = event->attr.disabled;
-	attr.enable_on_exec = event->attr.enable_on_exec;
-	attr.exclude_hv = event->attr.exclude_hv;
-	attr.exclude_idle = event->attr.exclude_idle;
-	attr.exclude_user = event->attr.exclude_user;
-	attr.exclude_kernel = event->attr.exclude_kernel;
-	attr.aux_sample_size = event->attr.aux_sample_size;
-
-	sampler = perf_event_create_kernel_counter(&attr, event->cpu, task,
-						   NULL, NULL);
-	if (IS_ERR(sampler))
-		return PTR_ERR(sampler);
-
-	nr_pages = 1ul << __get_order(event->attr.aux_sample_size);
-
-	rb = rb_alloc_kernel(sampler, 0, nr_pages);
-	if (!rb) {
-		perf_event_release_kernel(sampler);
-		return -ENOMEM;
+	ctx = find_get_context(pmu, task, event->cpu);
+	if (ctx) {
+		raw_spin_lock_irqsave(&ctx->lock, flags);
+		list_for_each_entry(sampler, &ctx->event_list, event_entry) {
+			/*
+			 * event is not an aux event, but all the relevant
+			 * bits should match
+			 */
+			if (perf_aux_event_match(sampler, event) &&
+			    sampler->attr.type == event->attr.aux_sample_type &&
+			    sampler->attr.config == event->attr.aux_sample_config &&
+			    sampler->attr.exclude_hv == event->attr.exclude_hv &&
+			    sampler->attr.exclude_idle == event->attr.exclude_idle &&
+			    sampler->attr.exclude_user == event->attr.exclude_user &&
+			    sampler->attr.exclude_kernel == event->attr.exclude_kernel &&
+			    sampler->attr.aux_sample_size >= event->attr.aux_sample_size &&
+			    atomic_long_inc_not_zero(&sampler->refcount))
+				goto got_event;
+		}
+
+		sampler = NULL;
+
+got_event:
+		--ctx->pin_count;
+		put_ctx(ctx);
+		raw_spin_unlock_irqrestore(&ctx->lock, flags);
+	}
+
+	if (!sampler) {
+		memset(&attr, 0, sizeof(attr));
+		attr.type = pmu->type;
+		attr.config = event->attr.aux_sample_config;
+		attr.sample_type = 0;
+		attr.disabled = event->attr.disabled;
+		attr.enable_on_exec = event->attr.enable_on_exec;
+		attr.exclude_hv = event->attr.exclude_hv;
+		attr.exclude_idle = event->attr.exclude_idle;
+		attr.exclude_user = event->attr.exclude_user;
+		attr.exclude_kernel = event->attr.exclude_kernel;
+		attr.aux_sample_size = event->attr.aux_sample_size;
+
+		sampler = perf_event_create_kernel_counter(&attr, event->cpu,
+							   task, NULL, NULL);
+		if (IS_ERR(sampler))
+			return PTR_ERR(sampler);
+
+		nr_pages = 1ul << __get_order(event->attr.aux_sample_size);
+
+		rb = rb_alloc_kernel(sampler, 0, nr_pages);
+		if (!rb) {
+			perf_event_release_kernel(sampler);
+			return -ENOMEM;
+		}
+
+		sampler->destroy = perf_aux_sampler_destroy;
 	}
 
 	event->sampler = sampler;
-	sampler->destroy = perf_aux_sampler_destroy;
 
 	return 0;
 }
@@ -4702,7 +4741,7 @@ static void perf_aux_sampler_fini(struct perf_event *event)
 	if (!sampler)
 		return;
 
-	perf_event_release_kernel(sampler);
+	put_event(sampler);
 
 	event->sampler = NULL;
 }
Signature · 1 line
2.1.0
author reviewer
#64 lore raw

PATCH v5 20/20 perf: Allow AUX sampling of inherited events

In reply to Alexander Shishkin

Try to find an AUX sampler event for the current event if none is linked via event::sampler.

This is useful when these events (the one that is being sampled and the one providing sample annotation) are allocated by inheritance path, independently of one another and the latter is not yet referenced by the former's event::sampler.

---
 kernel/events/core.c | 94 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 62 insertions(+), 32 deletions(-)
kernel/events/core.c +62 -32
index ad7b1e92dd..02fcd84b0f 100644
@@ -4658,47 +4658,67 @@ static bool perf_aux_event_match(struct perf_event *e1, struct perf_event *e2)
 	return has_aux(e1) && exclusive_event_match(e1, e2);
 }
 
+struct perf_event *__find_sampling_counter(struct perf_event_context *ctx,
+					   struct perf_event *event,
+					   struct task_struct *task)
+{
+	struct perf_event *sampler = NULL;
+
+	list_for_each_entry(sampler, &ctx->event_list, event_entry) {
+		/*
+		 * event is not an itrace event, but all the relevant
+		 * bits should match
+		 */
+		if (perf_aux_event_match(sampler, event) &&
+		    kernel_rb_event(sampler) &&
+		    sampler->attr.type == event->attr.aux_sample_type &&
+		    sampler->attr.config == event->attr.aux_sample_config &&
+		    sampler->attr.exclude_hv == event->attr.exclude_hv &&
+		    sampler->attr.exclude_idle == event->attr.exclude_idle &&
+		    sampler->attr.exclude_user == event->attr.exclude_user &&
+		    sampler->attr.exclude_kernel == event->attr.exclude_kernel &&
+		    sampler->attr.aux_sample_size >= event->attr.aux_sample_size &&
+		    atomic_long_inc_not_zero(&sampler->refcount))
+			return sampler;
+	}
+
+	return NULL;
+}
+
+struct perf_event *find_sampling_counter(struct pmu *pmu,
+					 struct perf_event *event,
+					 struct task_struct *task)
+{
+	struct perf_event_context *ctx;
+	struct perf_event *sampler = NULL;
+	unsigned long flags;
+
+	ctx = find_get_context(pmu, task, event->cpu);
+	if (!ctx)
+		return NULL;
+
+	raw_spin_lock_irqsave(&ctx->lock, flags);
+	sampler = __find_sampling_counter(ctx, event, task);
+	--ctx->pin_count;
+	put_ctx(ctx);
+	raw_spin_unlock_irqrestore(&ctx->lock, flags);
+
+	return sampler;
+}
+
 static int perf_aux_sampler_init(struct perf_event *event,
 				 struct task_struct *task,
 				 struct pmu *pmu)
 {
-	struct perf_event_context *ctx;
 	struct perf_event_attr attr;
-	struct perf_event *sampler = NULL;
+	struct perf_event *sampler;
 	struct ring_buffer *rb;
-	unsigned long nr_pages, flags;
+	unsigned long nr_pages;
 
 	if (!pmu || !(pmu->setup_aux))
 		return -ENOTSUPP;
 
-	ctx = find_get_context(pmu, task, event->cpu);
-	if (ctx) {
-		raw_spin_lock_irqsave(&ctx->lock, flags);
-		list_for_each_entry(sampler, &ctx->event_list, event_entry) {
-			/*
-			 * event is not an aux event, but all the relevant
-			 * bits should match
-			 */
-			if (perf_aux_event_match(sampler, event) &&
-			    sampler->attr.type == event->attr.aux_sample_type &&
-			    sampler->attr.config == event->attr.aux_sample_config &&
-			    sampler->attr.exclude_hv == event->attr.exclude_hv &&
-			    sampler->attr.exclude_idle == event->attr.exclude_idle &&
-			    sampler->attr.exclude_user == event->attr.exclude_user &&
-			    sampler->attr.exclude_kernel == event->attr.exclude_kernel &&
-			    sampler->attr.aux_sample_size >= event->attr.aux_sample_size &&
-			    atomic_long_inc_not_zero(&sampler->refcount))
-				goto got_event;
-		}
-
-		sampler = NULL;
-
-got_event:
-		--ctx->pin_count;
-		put_ctx(ctx);
-		raw_spin_unlock_irqrestore(&ctx->lock, flags);
-	}
-
+	sampler = find_sampling_counter(pmu, event, task);
 	if (!sampler) {
 		memset(&attr, 0, sizeof(attr));
 		attr.type = pmu->type;
@@ -4749,9 +4769,19 @@ static void perf_aux_sampler_fini(struct perf_event *event)
 static unsigned long perf_aux_sampler_trace(struct perf_event *event,
 					    struct perf_sample_data *data)
 {
-	struct perf_event *sampler = event->sampler;
+	struct perf_event *sampler;
 	struct ring_buffer *rb;
 
+	if (!event->sampler) {
+		/*
+		 * down this path, event->ctx is already locked IF it's the
+		 * same context
+		 */
+		event->sampler = __find_sampling_counter(event->ctx, event,
+							 event->ctx->task);
+	}
+
+	sampler = event->sampler;
 	if (!sampler || sampler->state != PERF_EVENT_STATE_ACTIVE) {
 		data->aux.size = 0;
 		goto out;
Signature · 1 line
2.1.0