Skip to contentSkip to conversation

perf: perf: add AUX space to ring_buffer

RFC Superseded Alexander Shishkin 3 participants 17 messages Jun 11, 2014 – Jun 25, 2014 perf events

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Alexander Shishkin

author reviewer
#0 lore raw

v2 cover perf: perf: add AUX space to ring_buffer

Hi Peter,

Here's the 2nd go at the AUX area. This covers pretty much what I need for Intel PT buffer management.

Alexander Shishkin (6):
perf: add data_{offset,size} to user_page
perf: support high-order allocations for AUX space
perf: add a capability for AUX_NO_SG pmus to do software double
buffering
perf: add a pmu capability for "exclusive" events
perf: add api for pmus to write to AUX space
perf: add AUX record

Peter Zijlstra (1):
perf: add AUX area to ring buffer for raw data streams

 include/linux/perf_event.h      |  39 ++++++-
 include/uapi/linux/perf_event.h |  36 +++++++
 kernel/events/core.c            | 225 ++++++++++++++++++++++++++++++++++------
 kernel/events/internal.h        |  27 +++++
 kernel/events/ring_buffer.c     | 182 +++++++++++++++++++++++++++++++-
 5 files changed, 475 insertions(+), 34 deletions(-)
Signature · 1 line
2.0.0
author reviewer
#1 lore raw

RFC v2 1/7 perf: add data_{offset,size} to user_page

Currently, the actual perf ring buffer is one page into the mmap area, following the user page and the userspace follows this convention. This patch adds data_{offset,size} fields to user_page that can be used by userspace instead for locating perf data in the mmap area. This is also helpful when mapping existing or shared buffers if their size is not known in advance.

Right now, it is made to follow the existing convention that

	data_offset == PAGE_SIZE and
	data_offset + data_size == mmap_size.
---
 include/uapi/linux/perf_event.h | 5 +++++
 kernel/events/core.c            | 2 ++
 2 files changed, 7 insertions(+)
include/uapi/linux/perf_event.h +5 -0
index 853bc1c..ef95af4 100644
@@ -488,9 +488,14 @@ struct perf_event_mmap_page {
 	 * In this case the kernel will not over-write unread data.
 	 *
 	 * See perf_output_put_handle() for the data ordering.
+	 *
+	 * data_{offset,size} indicate the location and size of the perf record
+	 * buffer within the mmapped area.
 	 */
 	__u64   data_head;		/* head in the data section */
 	__u64	data_tail;		/* user-space written tail */
+	__u64	data_offset;		/* where the buffer starts */
+	__u64	data_size;		/* data buffer size */
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
kernel/events/core.c +2 -0
index 2f9db66..6c1b030 100644
@@ -3768,6 +3768,8 @@ static void perf_event_init_userpage(struct perf_event *event)
 	/* Allow new userspace to detect that bit 0 is deprecated */
 	userpg->cap_bit0_is_deprecated = 1;
 	userpg->size = offsetof(struct perf_event_mmap_page, __reserved);
+	userpg->data_offset = PAGE_SIZE;
+	userpg->data_size = perf_data_size(rb);
 
 unlock:
 	rcu_read_unlock();
Signature · 1 line
2.0.0
#2 lore raw

Re: [RFC v2 1/7] perf: add data_{offset,size} to user_page

Quoted 8 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +0 -0
index 853bc1c..ef95af4 100644
@@ -488,9 +488,14 @@ struct perf_event_mmap_page {
 	 * In this case the kernel will not over-write unread data.
 	 *
 	 * See perf_output_put_handle() for the data ordering.

include/uapi/linux/perf_event.h struct perf_event_mmap_page { · line 500

Quoted 10 lines
include/uapi/linux/perf_event.h +5 -0
+	 *
+	 * data_{offset,size} indicate the location and size of the perf record
+	 * buffer within the mmapped area.
 	 */
 	__u64   data_head;		/* head in the data section */
 	__u64	data_tail;		/* user-space written tail */
+	__u64	data_offset;		/* where the buffer starts */
+	__u64	data_size;		/* data buffer size */
 };
 

Yes, right. This change is useful to determine the buffer size for existing buffers to be able to share them (e.g. persistent events).

If the buffer size is not known by userspace The mechanism would be to mmap the header page only and later remap the buffer with the correct buffer size determined from the header page.

If there are no objections on this patch it would be good if this patch could be applied independently of the status of the follow on patches.

Thanks,

-Robert

author reviewer
#3 lore raw

RFC v2 2/7 perf: add AUX area to ring buffer for raw data streams

In reply to Alexander Shishkin

From: Peter Zijlstra <[email protected]>

This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.

AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.

In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.

Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.

---
 include/linux/perf_event.h      |  17 +++++
 include/uapi/linux/perf_event.h |  16 +++++
 kernel/events/core.c            | 142 ++++++++++++++++++++++++++++++++--------
 kernel/events/internal.h        |  21 ++++++
 kernel/events/ring_buffer.c     |  77 ++++++++++++++++++++--
 5 files changed, 243 insertions(+), 30 deletions(-)
include/linux/perf_event.h +17 -0
index a6b6ab1..75b9e6d 100644
@@ -257,6 +257,18 @@ struct pmu {
 	 * flush branch stack on context-switches (needed in cpu-wide mode)
 	 */
 	void (*flush_branch_stack)	(void);
+
+	/*
+	 * Set up pmu-private data structures for an AUX area
+	 */
+	void *(*setup_aux)		(int cpu, void **pages,
+					 int nr_pages, bool overwrite);
+					/* optional */
+
+	/*
+	 * Free pmu-private AUX data structures
+	 */
+	void (*free_aux)		(void *aux); /* optional */
 };
 
 /**
@@ -767,6 +779,11 @@ static inline bool has_branch_stack(struct perf_event *event)
 	return event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK;
 }
 
+static inline bool has_aux(struct perf_event *event)
+{
+	return event->pmu->setup_aux;
+}
+
 extern int perf_output_begin(struct perf_output_handle *handle,
 			     struct perf_event *event, unsigned int size);
 extern void perf_output_end(struct perf_output_handle *handle);
include/uapi/linux/perf_event.h +16 -0
index ef95af4..dd10f16 100644
@@ -496,6 +496,22 @@ struct perf_event_mmap_page {
 	__u64	data_tail;		/* user-space written tail */
 	__u64	data_offset;		/* where the buffer starts */
 	__u64	data_size;		/* data buffer size */
+
+	/*
+	 * AUX area is defined by aux_{offset,size} fields that should be set
+	 * by the userspace, so that
+	 *
+	 *   aux_offset >= data_offset + data_size
+	 *
+	 * prior to mmap()ing it. Size of the mmap()ed area should be aux_size.
+	 *
+	 * Ring buffer pointers aux_{head,tail} have the same semantics as
+	 * data_{head,tail} and same ordering rules apply.
+	 */
+	__u64	aux_head;
+	__u64	aux_tail;
+	__u64	aux_offset;
+	__u64	aux_size;
 };
 
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
kernel/events/core.c +116 -26

216 lines not on this page · raw message

kernel/events/internal.h +21 -0
index 569b2187..e537403 100644
@@ -35,6 +35,14 @@ struct ring_buffer {
 	unsigned long			mmap_locked;
 	struct user_struct		*mmap_user;
 
+	/* AUX area */
+	unsigned long			aux_pgoff;
+	int				aux_nr_pages;
+	atomic_t			aux_mmap_count;
+	unsigned long			aux_mmap_locked;
+	void				**aux_pages;
+	void				*aux_priv;
+
 	struct perf_event_mmap_page	*user_page;
 	void				*data_pages[0];
 };
@@ -43,6 +51,14 @@ extern void rb_free(struct ring_buffer *rb);
 extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
 extern void perf_event_wakeup(struct perf_event *event);
+extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
+			pgoff_t pgoff, int nr_pages, int flags);
+extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
+
+static inline bool rb_has_aux(struct ring_buffer *rb)
+{
+	return !!rb->aux_nr_pages;
+}
 
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
@@ -81,6 +97,11 @@ static inline unsigned long perf_data_size(struct ring_buffer *rb)
 	return rb->nr_pages << (PAGE_SHIFT + page_order(rb));
 }
 
+static inline unsigned long perf_aux_size(struct ring_buffer *rb)
+{
+	return rb->aux_nr_pages << PAGE_SHIFT;
+}
+
 #define DEFINE_OUTPUT_COPY(func_name, memcpy_func)			\
 static inline unsigned long						\
 func_name(struct perf_output_handle *handle,				\
kernel/events/ring_buffer.c +73 -4

98 lines not on this page · raw message

Signature · 1 line
2.0.0
maintainer
#4 lore raw

Re: [RFC v2 2/7] perf: add AUX area to ring buffer for raw data streams

include/linux/perf_event.h struct pmu { · line 271

Quoted 11 lines · Alexander Shishkin
include/linux/perf_event.h +11 -0
+	/*
+	 * Set up pmu-private data structures for an AUX area
+	 */
+	void *(*setup_aux)		(int cpu, void **pages,
+					 int nr_pages, bool overwrite);
+					/* optional */
+
+	/*
+	 * Free pmu-private AUX data structures
+	 */
+	void (*free_aux)		(void *aux); /* optional */

I was hoping you could replace those with a PERF_CAP_AUX or something and then have one generic allocation routine like you provide in the subsequent patches.

author reviewer
#5 lore raw

Re: [RFC v2 2/7] perf: add AUX area to ring buffer for raw data streams

Quoted 16 lines · Peter Zijlstra

include/linux/perf_event.h struct pmu { · line 271

Quoted 11 lines · Alexander Shishkin
include/linux/perf_event.h +11 -0
+	/*
+	 * Set up pmu-private data structures for an AUX area
+	 */
+	void *(*setup_aux)		(int cpu, void **pages,
+					 int nr_pages, bool overwrite);
+					/* optional */
+
+	/*
+	 * Free pmu-private AUX data structures
+	 */
+	void (*free_aux)		(void *aux); /* optional */

I was hoping you could replace those with a PERF_CAP_AUX or something and then have one generic allocation routine like you provide in the subsequent patches.

I need these to allocate pmu-specific SG tables now, which I don't see how to generalize nicely. User-visible aux_pages are allocated in the generic rb_alloc_aux().

As for sg tables, the alternative would be to allocate them in the event::pmu::add() path, which is probably not very good for performance, unless I'm missing something? Like, I can allocate stuff in the first add() and then free it in event::destroy().

Or, we can assume a generic sg table format and allocate them in the generic code, but then we'll need more capabilities to indicate at least the size of a table entry. PT needs 64 bits per entry, ARM's TMC needs 32 and the non-SG PT will still need some tables. But I'd rather leave this part to the pmu drivers. What do you think?

Regards,
--
Alex

maintainer
#6 lore raw

Re: [RFC v2 2/7] perf: add AUX area to ring buffer for raw data streams

Quoted 28 lines · Alexander Shishkin
Quoted 16 lines · Peter Zijlstra

include/linux/perf_event.h struct pmu { · line 271

Quoted 11 lines · Alexander Shishkin
include/linux/perf_event.h +11 -0
+	/*
+	 * Set up pmu-private data structures for an AUX area
+	 */
+	void *(*setup_aux)		(int cpu, void **pages,
+					 int nr_pages, bool overwrite);
+					/* optional */
+
+	/*
+	 * Free pmu-private AUX data structures
+	 */
+	void (*free_aux)		(void *aux); /* optional */

I was hoping you could replace those with a PERF_CAP_AUX or something and then have one generic allocation routine like you provide in the subsequent patches.

I need these to allocate pmu-specific SG tables now, which I don't see how to generalize nicely. User-visible aux_pages are allocated in the generic rb_alloc_aux().

As for sg tables, the alternative would be to allocate them in the event::pmu::add() path, which is probably not very good for performance, unless I'm missing something? Like, I can allocate stuff in the first add() and then free it in event::destroy().

Quoted 5 lines

Or, we can assume a generic sg table format and allocate them in the generic code, but then we'll need more capabilities to indicate at least the size of a table entry. PT needs 64 bits per entry, ARM's TMC needs 32 and the non-SG PT will still need some tables. But I'd rather leave this part to the pmu drivers. What do you think?

Ah, I wasn't aware.. too bad. Maybe add this to the changelog.

author reviewer
#7 lore raw

RFC v2 3/7 perf: support high-order allocations for AUX space

In reply to Alexander Shishkin

Some pmus (such as BTS or Intel PT without multiple-entry ToPA capability) don't support scatter-gather and will prefer larger contiguous areas for their output regions.

This patch adds a new pmu capability to request higher order allocations.

---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 51 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 46 insertions(+), 6 deletions(-)
include/linux/perf_event.h +1 -0
index 75b9e6d..b316971 100644
@@ -275,6 +275,7 @@ struct pmu {
  * struct pmu::capabilities flags
  */
 #define PERF_PMU_CAP_NO_INTERRUPT	1
+#define PERF_PMU_CAP_AUX_NO_SG		2
 
 /**
  * enum perf_event_active_state - the states of a event
kernel/events/ring_buffer.c +45 -6

81 lines not on this page · raw message

Signature · 1 line
2.0.0
author reviewer
#8 lore raw

RFC v2 4/7 perf: add a capability for AUX_NO_SG pmus to do software double buffering

In reply to Alexander Shishkin

For pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.

---
 include/linux/perf_event.h  |  1 +
 kernel/events/ring_buffer.c | 14 +++++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index b316971..3d68411 100644
@@ -276,6 +276,7 @@ struct pmu {
  */
 #define PERF_PMU_CAP_NO_INTERRUPT	1
 #define PERF_PMU_CAP_AUX_NO_SG		2
+#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF	4
 
 /**
  * enum perf_event_active_state - the states of a event
kernel/events/ring_buffer.c +13 -1
index e66ed76..43571ac 100644
@@ -286,9 +286,21 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -ENOTSUPP;
 
-	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG)
+	if (event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) {
 		order = get_order(nr_pages * PAGE_SIZE);
 
+		/*
+		 * PMU requests more than one contiguous chunks of memory
+		 * for SW double buffering
+		 */
+		if (event->pmu->capabilities & PERF_PMU_CAP_AUX_SW_DOUBLEBUF) {
+			if (!order)
+				return -EINVAL;
+
+			order--;
+		}
+	}
+
 	rb->aux_pages = kzalloc_node(nr_pages * sizeof(void *), GFP_KERNEL, node);
 	if (!rb->aux_pages)
 		return -ENOMEM;
Signature · 1 line
2.0.0
author reviewer
#9 lore raw

RFC v2 5/7 perf: add a pmu capability for "exclusive" events

In reply to Alexander Shishkin

Usually, pmus that do, for example, instruction tracing, would only ever be able to have one event per task per cpu (or per perf_context). For such pmus it makes sense to disallow creating conflicting events early on, so as to provide consistent behavior for the user.

This patch adds a pmu capability that indicates such constraint on event creation.

---
 include/linux/perf_event.h |  1 +
 kernel/events/core.c       | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/linux/perf_event.h +1 -0
index 3d68411..b6f7408 100644
@@ -277,6 +277,7 @@ struct pmu {
 #define PERF_PMU_CAP_NO_INTERRUPT	1
 #define PERF_PMU_CAP_AUX_NO_SG		2
 #define PERF_PMU_CAP_AUX_SW_DOUBLEBUF	4
+#define PERF_PMU_CAP_EXCLUSIVE		8
 
 /**
  * enum perf_event_active_state - the states of a event
kernel/events/core.c +38 -0
index 48ad31b..9783c60 100644
@@ -7113,6 +7113,32 @@ out:
 	return ret;
 }
 
+static bool exclusive_event_match(struct perf_event *e1, struct perf_event *e2)
+{
+	if ((e1->pmu->capabilities & PERF_PMU_CAP_EXCLUSIVE) &&
+	    (e1->cpu == e2->cpu ||
+	     e1->cpu == -1 ||
+	     e2->cpu == -1))
+		return true;
+	return false;
+}
+
+static bool exclusive_event_ok(struct perf_event *event,
+			      struct perf_event_context *ctx)
+{
+	struct perf_event *iter_event;
+
+	if (!(event->pmu->capabilities & PERF_PMU_CAP_EXCLUSIVE))
+		return true;
+
+	list_for_each_entry(iter_event, &ctx->event_list, event_entry) {
+		if (exclusive_event_match(iter_event, event))
+			return false;
+	}
+
+	return true;
+}
+
 /**
  * sys_perf_event_open - open a performance event, associate it to a task/cpu
  *
@@ -7261,6 +7287,11 @@ SYSCALL_DEFINE5(perf_event_open,
 		goto err_alloc;
 	}
 
+	if (!exclusive_event_ok(event, ctx)) {
+		err = -EBUSY;
+		goto err_context;
+	}
+
 	if (task) {
 		put_task_struct(task);
 		task = NULL;
@@ -7427,6 +7458,13 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 		goto err_free;
 	}
 
+	if (!exclusive_event_ok(event, ctx)) {
+		perf_unpin_context(ctx);
+		put_ctx(ctx);
+		err = -EBUSY;
+		goto err_free;
+	}
+
 	WARN_ON_ONCE(ctx->parent_ctx);
 	mutex_lock(&ctx->mutex);
 	perf_install_in_context(ctx, event, cpu);
Signature · 1 line
2.0.0
maintainer
#10 lore raw

Re: [RFC v2 5/7] perf: add a pmu capability for "exclusive" events

kernel/events/core.c out: · line 7137

Quoted 12 lines · Alexander Shishkin
kernel/events/core.c +12 -0
+static bool exclusive_event_ok(struct perf_event *event,
+			      struct perf_event_context *ctx)
+{
+	struct perf_event *iter_event;
+
+	if (!(event->pmu->capabilities & PERF_PMU_CAP_EXCLUSIVE))
+		return true;
+
+	list_for_each_entry(iter_event, &ctx->event_list, event_entry) {
+		if (exclusive_event_match(iter_event, event))
+			return false;
+	}

This list iteration needs either rcu or ctx->lock or ctx->mutex, and the two callsites below don't have either afaict.

Quoted 36 lines
kernel/events/core.c +16 -0
+
+	return true;
+}
+
 /**
  * sys_perf_event_open - open a performance event, associate it to a task/cpu
  *
@@ -7261,6 +7287,11 @@ SYSCALL_DEFINE5(perf_event_open,
 		goto err_alloc;
 	}
 
+	if (!exclusive_event_ok(event, ctx)) {
+		err = -EBUSY;
+		goto err_context;
+	}
+
 	if (task) {
 		put_task_struct(task);
 		task = NULL;
@@ -7427,6 +7458,13 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 		goto err_free;
 	}
 
+	if (!exclusive_event_ok(event, ctx)) {
+		perf_unpin_context(ctx);
+		put_ctx(ctx);
+		err = -EBUSY;
+		goto err_free;
+	}
+
 	WARN_ON_ONCE(ctx->parent_ctx);
 	mutex_lock(&ctx->mutex);
 	perf_install_in_context(ctx, event, cpu);
Signature · 1 line
2.0.0
author reviewer
#11 lore raw

Re: [RFC v2 5/7] perf: add a pmu capability for "exclusive" events

Quoted 16 lines · Peter Zijlstra

kernel/events/core.c out: · line 7137

Quoted 12 lines · Alexander Shishkin
kernel/events/core.c +12 -0
+static bool exclusive_event_ok(struct perf_event *event,
+			      struct perf_event_context *ctx)
+{
+	struct perf_event *iter_event;
+
+	if (!(event->pmu->capabilities & PERF_PMU_CAP_EXCLUSIVE))
+		return true;
+
+	list_for_each_entry(iter_event, &ctx->event_list, event_entry) {
+		if (exclusive_event_match(iter_event, event))
+			return false;
+	}

This list iteration needs either rcu or ctx->lock or ctx->mutex, and the two callsites below don't have either afaict.

True, this needs to be done before perf_install_in_context() under the same ctx->mutex. Thanks!

Regards,
--
Alex

author reviewer
#12 lore raw

RFC v2 6/7 perf: add api for pmus to write to AUX space

In reply to Alexander Shishkin

For pmus that wish to write data to AUX space, provide perf_aux_output_{begin,end}() calls to initiate/commit data writes, similarly to perf_output_{begin,end}. These also use the same output handle structure.

After the perf_aux_output_begin() returns successfully, handle->size is set to the maximum amount of data that can be written wrt aux_tail pointer, so that no data that the user hasn't seen will be overwritten.

PMU driver should pass the actual amount of data written as a parameter to perf_aux_output_end().

---
 include/linux/perf_event.h  | 19 +++++++++++++++-
 kernel/events/core.c        |  5 ++---
 kernel/events/internal.h    |  3 +++
 kernel/events/ring_buffer.c | 53 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+), 4 deletions(-)
include/linux/perf_event.h +18 -1
index b6f7408..e85d134 100644
@@ -548,12 +548,20 @@ struct perf_output_handle {
 	struct ring_buffer		*rb;
 	unsigned long			wakeup;
 	unsigned long			size;
-	void				*addr;
+	union {
+		void			*addr;
+		unsigned long		head;
+	};
 	int				page;
 };
 
 #ifdef CONFIG_PERF_EVENTS
 
+extern void *perf_aux_output_begin(struct perf_output_handle *handle,
+				   struct perf_event *event);
+extern void perf_aux_output_end(struct perf_output_handle *handle,
+				unsigned long size, bool truncated);
+extern void *perf_get_aux(struct perf_output_handle *handle);
 extern int perf_pmu_register(struct pmu *pmu, const char *name, int type);
 extern void perf_pmu_unregister(struct pmu *pmu);
 
@@ -802,6 +810,15 @@ extern void perf_event_disable(struct perf_event *event);
 extern int __perf_event_disable(void *info);
 extern void perf_event_task_tick(void);
 #else
+static inline void *
+perf_aux_output_begin(struct perf_output_handle *handle,
+		      struct perf_event *event)				{ return NULL; }
+static inline void
+perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+		    bool truncated)					{ }
+static inline void *
+perf_get_aux(struct perf_output_handle *handle)				{ return NULL; }
+
 static inline void
 perf_event_task_sched_in(struct task_struct *prev,
 			 struct task_struct *task)			{ }
kernel/events/core.c +2 -3
index 9783c60..44d6d01 100644
@@ -3194,7 +3194,6 @@ static void free_event_rcu(struct rcu_head *head)
 	kfree(event);
 }
 
-static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
 static void unaccount_event_cpu(struct perf_event *event, int cpu)
@@ -3919,7 +3918,7 @@ static void rb_free_rcu(struct rcu_head *rcu_head)
 	rb_free(rb);
 }
 
-static struct ring_buffer *ring_buffer_get(struct perf_event *event)
+struct ring_buffer *ring_buffer_get(struct perf_event *event)
 {
 	struct ring_buffer *rb;
 
@@ -3934,7 +3933,7 @@ static struct ring_buffer *ring_buffer_get(struct perf_event *event)
 	return rb;
 }
 
-static void ring_buffer_put(struct ring_buffer *rb)
+void ring_buffer_put(struct ring_buffer *rb)
 {
 	if (!atomic_dec_and_test(&rb->refcount))
 		return;
kernel/events/internal.h +3 -0
index e537403..e20d073 100644
@@ -36,6 +36,7 @@ struct ring_buffer {
 	struct user_struct		*mmap_user;
 
 	/* AUX area */
+	local_t				aux_head;
 	unsigned long			aux_pgoff;
 	int				aux_nr_pages;
 	atomic_t			aux_mmap_count;
@@ -54,6 +55,8 @@ extern void perf_event_wakeup(struct perf_event *event);
 extern int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,
 			pgoff_t pgoff, int nr_pages, int flags);
 extern void rb_free_aux(struct ring_buffer *rb, struct perf_event *event);
+extern struct ring_buffer *ring_buffer_get(struct perf_event *event);
+extern void ring_buffer_put(struct ring_buffer *rb);
 
 static inline bool rb_has_aux(struct ring_buffer *rb)
 {
kernel/events/ring_buffer.c +53 -0

59 lines not on this page · raw message

Signature · 1 line
2.0.0
maintainer
#13 lore raw

Re: [RFC v2 6/7] perf: add api for pmus to write to AUX space

Quoted 7 lines · Alexander Shishkin
kernel/events/ring_buffer.c +7 -0
+void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+			 bool truncated)
+{
+	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
+
+	aux_head = local_read(&rb->aux_head);

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 291

Quoted 10 lines
kernel/events/ring_buffer.c +10 -0
+	local_add(size, &rb->aux_head);
+
+	rb->user_page->aux_head = local_read(&rb->aux_head);
+	smp_wmb();
+
+	perf_output_wakeup(handle);
+	handle->event = NULL;
+
+	ring_buffer_put(rb);
+}

This thing is distinctly less complex than perf_output_put_handle().. ?

author reviewer
#14 lore raw

Re: [RFC v2 6/7] perf: add api for pmus to write to AUX space

Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Alexander Shishkin
kernel/events/ring_buffer.c +11 -0
+void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+			 bool truncated)
+{
+	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
+
+	aux_head = local_read(&rb->aux_head);
+	local_add(size, &rb->aux_head);
+
+	rb->user_page->aux_head = local_read(&rb->aux_head);
+	smp_wmb();
Quoted 8 lines

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 291

Quoted 6 lines
kernel/events/ring_buffer.c +6 -0
+
+	perf_output_wakeup(handle);
+	handle->event = NULL;
+
+	ring_buffer_put(rb);
+}

This thing is distinctly less complex than perf_output_put_handle().. ?

This one doesn't support nested writers, but there shouldn't be any either. But I notice now that the barrier is misplaced. Wakeup watermark is not yet taken care of, but will be. Anything else I'm missing?

Regards,
--
Alex

maintainer
#15 lore raw

Re: [RFC v2 6/7] perf: add api for pmus to write to AUX space

Quoted 23 lines · Alexander Shishkin
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Alexander Shishkin
kernel/events/ring_buffer.c +11 -0
+void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
+			 bool truncated)
+{
+	struct ring_buffer *rb = handle->rb;
+	unsigned long aux_head;
+
+	aux_head = local_read(&rb->aux_head);
+	local_add(size, &rb->aux_head);
+
+	rb->user_page->aux_head = local_read(&rb->aux_head);
+	smp_wmb();
Quoted 8 lines

kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 291

Quoted 6 lines
kernel/events/ring_buffer.c +6 -0
+
+	perf_output_wakeup(handle);
+	handle->event = NULL;
+
+	ring_buffer_put(rb);
+}

This thing is distinctly less complex than perf_output_put_handle().. ?

Quoted 3 lines

This one doesn't support nested writers, but there shouldn't be any either. But I notice now that the barrier is misplaced. Wakeup watermark is not yet taken care of, but will be. Anything else I'm missing?

Maybe put a comment in about explicitly not supporting nesting etc. Maybe also a debug check to make sure we catch anybody doing it anyway.

author reviewer
#16 lore raw

RFC v2 7/7 perf: add AUX record

In reply to Alexander Shishkin

When there's new data in the AUX space, output a record indicating its offset and size and weather it was truncated to fix in the ring buffer.

---
 include/uapi/linux/perf_event.h | 15 +++++++++++++++
 kernel/events/core.c            | 38 ++++++++++++++++++++++++++++++++++++++
 kernel/events/internal.h        |  3 +++
 kernel/events/ring_buffer.c     |  1 +
 4 files changed, 57 insertions(+)
include/uapi/linux/perf_event.h +15 -0
index dd10f16..7f8621f 100644
@@ -726,6 +726,21 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_MMAP2			= 10,
 
+	/*
+	 * Records that new data landed in the AUX buffer part.
+	 *
+	 * struct {
+	 * 	struct perf_event_header	header;
+	 *
+	 * 	u64				aux_offset;
+	 * 	u64				aux_size;
+	 *	u8				truncated;
+	 *	u64				id;
+	 *	u64				stream_id;
+	 * };
+	 */
+	PERF_RECORD_AUX				= 11,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +38 -0
index 44d6d01..cd64b75 100644
@@ -5394,6 +5394,44 @@ void perf_event_mmap(struct vm_area_struct *vma)
 	perf_event_mmap_event(&mmap_event);
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, bool truncated)
+{
+	struct perf_output_handle handle;
+	struct perf_sample_data sample;
+	struct perf_aux_event {
+		struct perf_event_header	header;
+		u64				offset;
+		u64				size;
+		u8				truncated;
+		u64				id;
+		u64				stream_id;
+	} rec = {
+		.header = {
+			.type = PERF_RECORD_AUX,
+			.misc = 0,
+			.size = sizeof(rec),
+		},
+		.offset		= head,
+		.size		= size,
+		.truncated	= truncated,
+		.id		= primary_event_id(event),
+		.stream_id	= event->id,
+	};
+	int ret;
+
+	perf_event_header__init_id(&rec.header, &sample, event);
+	ret = perf_output_begin(&handle, event, rec.header.size);
+
+	if (ret)
+		return;
+
+	perf_output_put(&handle, rec);
+	perf_event__output_id_sample(event, &handle, &sample);
+
+	perf_output_end(&handle);
+}
+
 /*
  * IRQ throttle logging
  */
kernel/events/internal.h +3 -0
index e20d073..8bc4547 100644
@@ -63,6 +63,9 @@ static inline bool rb_has_aux(struct ring_buffer *rb)
 	return !!rb->aux_nr_pages;
 }
 
+void perf_event_aux_event(struct perf_event *event, unsigned long head,
+			  unsigned long size, bool truncated);
+
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
 			   struct perf_sample_data *data,
kernel/events/ring_buffer.c +1 -0
index 8f76e5c..4e6f815 100644
@@ -280,6 +280,7 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
 
 	aux_head = local_read(&rb->aux_head);
 	local_add(size, &rb->aux_head);
+	perf_event_aux_event(handle->event, aux_head, size, truncated);
 
 	rb->user_page->aux_head = local_read(&rb->aux_head);
 	smp_wmb();
Signature · 1 line
2.0.0