perf: Add infrastructure and support for Intel PT
Subsystems (2)
Alexander Shishkin
reviewerv3 cover perf: Add infrastructure and support for Intel PT
Hi Peter and all,
Here's a new version of PT support patchset, this time including PT and BTS drivers and a few more tweaks to the core. I still left out some bits like core dump support for now. Tooling support is not included in this series so that it's easier to review the kernel bits, I suppose it's best to send it separately, since it's quite a huge patchset of its own.
Alexander Shishkin (22):
perf: Add data_{offset,size} to user_page
perf: Support high-order allocations for AUX space
perf: Add a capability for AUX_NO_SG pmus to do software double
buffering
perf: Add a pmu capability for "exclusive" events
perf: Redirect output from inherited events to parents
perf: Add api for pmus to write to AUX space
perf: Add AUX record
perf: Support overwrite mode for AUX area
perf: Add wakeup watermark control to AUX area
perf: Add itrace_config to the event attribute
perf: add ITRACE_START record to indicate that tracing has started
x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
x86: perf: Intel PT and LBR/BTS are mutually exclusive
x86: perf: intel_pt: Intel PT PMU driver
x86: perf: intel_bts: Add BTS PMU driver
perf: Add rb_{alloc,free}_kernel api
perf: Add a helper to copy AUX data in the kernel
perf: Add a helper for looking up pmus by type
perf: itrace: Infrastructure for sampling instruction flow traces
perf: Allocate ring buffers for inherited per-task kernel events
perf: itrace: Allow itrace sampling for multiple events
perf: itrace: Allow sampling of inherited events
Peter Zijlstra (1):
perf: Add AUX area to ring buffer for raw data streams
arch/x86/include/asm/cpufeature.h | 1 + arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 129 ++++ arch/x86/kernel/cpu/perf_event.h | 14 + arch/x86/kernel/cpu/perf_event_intel.c | 14 +- arch/x86/kernel/cpu/perf_event_intel_bts.c | 496 +++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 11 +- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 9 +- arch/x86/kernel/cpu/perf_event_intel_pt.c | 952 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/scattered.c | 1 + include/linux/itrace.h | 45 ++ include/linux/perf_event.h | 61 +- include/uapi/linux/perf_event.h | 69 ++- kernel/events/Makefile | 2 +- kernel/events/core.c | 348 +++++++++-- kernel/events/internal.h | 54 ++ kernel/events/itrace.c | 234 +++++++ kernel/events/ring_buffer.c | 313 +++++++++- 19 files changed, 2724 insertions(+), 48 deletions(-) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c create mode 100644 include/linux/itrace.h create mode 100644 kernel/events/itrace.c
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 1/23 perf: Add data_{offset,size} to user_page
Currently, the actual perf ring buffer is one page into the mmap area, following the user page and the userspace follows this convention. This patch adds data_{offset,size} fields to user_page that can be used by userspace instead for locating perf data in the mmap area. This is also helpful when mapping existing or shared buffers if their size is not known in advance.
Right now, it is made to follow the existing convention that
data_offset == PAGE_SIZE and data_offset + data_size == mmap_size.
--- include/uapi/linux/perf_event.h | 5 +++++ kernel/events/core.c | 2 ++ 2 files changed, 7 insertions(+)
include/uapi/linux/perf_event.h +5 -0
@@ -489,9 +489,14 @@ struct perf_event_mmap_page {* In this case the kernel will not over-write unread data. * * See perf_output_put_handle() for the data ordering. +* +* data_{offset,size} indicate the location and size of the perf record +* buffer within the mmapped area. */ __u64 data_head;/* head in the data section */ __u64 data_tail;/* user-space written tail */ +__u64 data_offset;/* where the buffer starts */ +__u64 data_size;/* data buffer size */ };#define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)
kernel/events/core.c +2 -0
@@ -3780,6 +3780,8 @@ static void perf_event_init_userpage(struct perf_event *event)/* Allow new userspace to detect that bit 0 is deprecated */ userpg->cap_bit0_is_deprecated =1 ; userpg->size = offsetof(struct perf_event_mmap_page , __reserved); + userpg->data_offset =PAGE_SIZE ; + userpg->data_size =perf_data_size (rb); unlock:rcu_read_unlock ();
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 2/23 perf: Add AUX area to ring buffer for raw data streams
In reply to Alexander ShishkinFrom: Peter Zijlstra <[email protected]>
This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.
AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.
In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.
Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.
--- include/linux/perf_event.h | 17 +++++ include/uapi/linux/perf_event.h | 16 +++++ kernel/events/core.c | 140 +++++++++++++++++++++++++++++++++------- kernel/events/internal.h | 21 ++++++ kernel/events/ring_buffer.c | 86 ++++++++++++++++++++++-- 5 files changed, 251 insertions(+), 29 deletions(-)
include/linux/perf_event.h +17 -0
@@ -262,6 +262,18 @@ struct pmu {* flush branch stack on context-switches (needed in cpu-wide mode) */ void (*flush_branch_stack) (void ); + +/* +* Set up pmu-private data structures for an AUX area +*/ +void *(*setup_aux) (int cpu,void **pages, +int nr_pages,bool overwrite); +/* optional */ + +/* +* Free pmu-private AUX data structures +*/ +void (*free_aux) (void *aux);/* optional */ };/**
@@ -770,6 +782,11 @@ static inline bool has_branch_stack(struct perf_event *event)return event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ; } +static inline bool has_aux (struct perf_event *event) +{ +return event->pmu->setup_aux; +} +extern int perf_output_begin (struct perf_output_handle *handle,struct perf_event *event,unsigned int size);extern void perf_output_end (struct perf_output_handle *handle);
include/uapi/linux/perf_event.h +16 -0
@@ -497,6 +497,22 @@ struct perf_event_mmap_page {__u64 data_tail;/* user-space written tail */ __u64 data_offset;/* where the buffer starts */ __u64 data_size;/* data buffer size */ + +/* +* AUX area is defined by aux_{offset,size} fields that should be set +* by the userspace, so that +* +* aux_offset >= data_offset + data_size +* +* prior to mmap()ing it. Size of the mmap()ed area should be aux_size. +* +* Ring buffer pointers aux_{head,tail} have the same semantics as +* data_{head,tail} and same ordering rules apply. +*/ +__u64 aux_head; +__u64 aux_tail; +__u64 aux_offset; +__u64 aux_size; };#define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)
kernel/events/core.c +115 -25
214 lines not on this page · raw message
kernel/events/internal.h +21 -0
@@ -35,6 +35,14 @@ struct ring_buffer {unsigned long mmap_locked;struct user_struct *mmap_user; +/* AUX area */ +unsigned long aux_pgoff; +int aux_nr_pages; +atomic_t aux_mmap_count; +unsigned long aux_mmap_locked; +void **aux_pages; +void *aux_priv; +struct perf_event_mmap_page *user_page;void *data_pages[0 ]; };
@@ -43,6 +51,14 @@ extern void rb_free(struct ring_buffer *rb);extern struct ring_buffer *rb_alloc (int nr_pages,long watermark,int cpu,int flags);extern void perf_event_wakeup (struct perf_event *event); +extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event, +pgoff_t pgoff,int nr_pages,int flags); +extern void rb_free_aux (struct ring_buffer *rb,struct perf_event *event); + +static inline bool rb_has_aux (struct ring_buffer *rb) +{ +return !!rb->aux_nr_pages; +} extern void perf_event_header__init_id(struct perf_event_header *header,
@@ -81,6 +97,11 @@ static inline unsigned long perf_data_size(struct ring_buffer *rb)return rb->nr_pages << (PAGE_SHIFT +page_order (rb)); } +static inline unsigned long perf_aux_size (struct ring_buffer *rb) +{ +return rb->aux_nr_pages <<PAGE_SHIFT ; +} +#define DEFINE_OUTPUT_COPY (func_name, memcpy_func) \ static inline unsigned long \ func_name(struct perf_output_handle *handle, \
kernel/events/ring_buffer.c +82 -4
107 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 3/23 perf: Support high-order allocations for AUX space
In reply to Alexander ShishkinSome pmus (such as BTS or Intel PT without multiple-entry ToPA capability) don't support scatter-gather and will prefer larger contiguous areas for their output regions.
This patch adds a new pmu capability to request higher order allocations.
--- include/linux/perf_event.h | 1 + kernel/events/ring_buffer.c | 51 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 46 insertions(+), 6 deletions(-)
include/linux/perf_event.h +1 -0
@@ -170,6 +170,7 @@ struct perf_event;* pmu::capabilities flags */ #define PERF_PMU_CAP_NO_INTERRUPT 0x01 +#define PERF_PMU_CAP_AUX_NO_SG 0x02/** * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +45 -6
81 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 4/23 perf: Add a capability for AUX_NO_SG pmus to do software double buffering
In reply to Alexander ShishkinFor pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.
--- include/linux/perf_event.h | 1 + kernel/events/ring_buffer.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
@@ -171,6 +171,7 @@ struct perf_event;*/ #define PERF_PMU_CAP_NO_INTERRUPT 0x01#define PERF_PMU_CAP_AUX_NO_SG 0x02 +#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF 0x04/** * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +14 -1
@@ -286,9 +286,22 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,if (!has_aux (event))return -ENOTSUPP ; -if (event->pmu->capabilities &PERF_PMU_CAP_AUX_NO_SG ) +if (event->pmu->capabilities &PERF_PMU_CAP_AUX_NO_SG ) { order =get_order (nr_pages *PAGE_SIZE ); +/* +* PMU requests more than one contiguous chunks of memory +* for SW double buffering +*/ +if ((event->pmu->capabilities &PERF_PMU_CAP_AUX_SW_DOUBLEBUF ) && + !overwrite) { +if (!order) +return -EINVAL ; + + order--; + } + } + rb->aux_pages =kzalloc_node (nr_pages *sizeof (void *),GFP_KERNEL , node);if (!rb->aux_pages)return -ENOMEM ;
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 5/23 perf: Add a pmu capability for "exclusive" events
In reply to Alexander ShishkinUsually, pmus that do, for example, instruction tracing, would only ever be able to have one event per task per cpu (or per perf_context). For such pmus it makes sense to disallow creating conflicting events early on, so as to provide consistent behavior for the user.
This patch adds a pmu capability that indicates such constraint on event creation.
--- include/linux/perf_event.h | 1 + kernel/events/core.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
include/linux/perf_event.h +1 -0
@@ -172,6 +172,7 @@ struct perf_event;#define PERF_PMU_CAP_NO_INTERRUPT 0x01#define PERF_PMU_CAP_AUX_NO_SG 0x02#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF 0x04 +#define PERF_PMU_CAP_EXCLUSIVE 0x08/** * struct pmu - generic performance monitoring unit
kernel/events/core.c +45 -0
69 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 6/23 perf: Redirect output from inherited events to parents
In reply to Alexander ShishkinIn order to collect AUX data from an inherited event, we can redirect its output to parent's ring buffer if possible (they must be cpu affine). This patch adds set_output() to the inheritance path.
--- kernel/events/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/events/core.c +6 -0
@@ -7859,6 +7859,12 @@ inherit_event(struct perf_event *parent_event, = parent_event->overflow_handler_context;/* +* Direct child's output to parent's ring buffer (if any) +*/ +if (parent_event->cpu !=-1 ) + (void )perf_event_set_output (child_event, parent_event); + +/* * Precalculate sample_data sizes */ perf_event__header_size (child_event);
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 7/23 perf: Add api for pmus to write to AUX space
In reply to Alexander ShishkinFor pmus that wish to write data to AUX space, provide perf_aux_output_{begin,end}() calls to initiate/commit data writes, similarly to perf_output_{begin,end}. These also use the same output handle structure.
After the perf_aux_output_begin() returns successfully, handle->size is set to the maximum amount of data that can be written wrt aux_tail pointer, so that no data that the user hasn't seen will be overwritten.
PMU driver should pass the actual amount of data written as a parameter to perf_aux_output_end().
Nested writers are forbidden and guards are in place to catch such attempts.
--- include/linux/perf_event.h | 23 ++++++++++++- kernel/events/core.c | 5 ++- kernel/events/internal.h | 4 +++ kernel/events/ring_buffer.c | 81 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 109 insertions(+), 4 deletions(-)
include/linux/perf_event.h +22 -1
@@ -550,12 +550,22 @@ struct perf_output_handle {struct ring_buffer *rb;unsigned long wakeup;unsigned long size; -void *addr; +union { +void *addr; +unsigned long head; + };int page; };#ifdef CONFIG_PERF_EVENTS +extern void *perf_aux_output_begin (struct perf_output_handle *handle, +struct perf_event *event); +extern void perf_aux_output_end (struct perf_output_handle *handle, +unsigned long size,bool truncated); +extern int perf_aux_output_skip (struct perf_output_handle *handle, +unsigned long size); +extern void *perf_get_aux (struct perf_output_handle *handle);extern int perf_pmu_register (struct pmu *pmu,const char *name,int type);extern void perf_pmu_unregister (struct pmu *pmu);
@@ -805,6 +815,17 @@ extern void perf_event_disable(struct perf_event *event);extern int __perf_event_disable (void *info);extern void perf_event_task_tick (void );#else /* !CONFIG_PERF_EVENTS: */ +static inline void * +perf_aux_output_begin (struct perf_output_handle *handle, +struct perf_event *event) {return NULL ; } +static inline void +perf_aux_output_end (struct perf_output_handle *handle,unsigned long size, +bool truncated) { } +static inline int +perf_aux_output_skip (struct perf_output_handle *handle, +unsigned long size) {return -EINVAL ; } +static inline void * +perf_get_aux (struct perf_output_handle *handle) {return NULL ; }static inline void perf_event_task_sched_in (struct task_struct *prev,struct task_struct *task) { }
kernel/events/core.c +2 -3
@@ -3212,7 +3212,6 @@ static void free_event_rcu(struct rcu_head *head)kfree (event); } -static void ring_buffer_put (struct ring_buffer *rb);static void ring_buffer_attach (struct perf_event *event,struct ring_buffer *rb);
@@ -3948,7 +3947,7 @@ static void rb_free_rcu(struct rcu_head *rcu_head)rb_free (rb); } -static struct ring_buffer *ring_buffer_get (struct perf_event *event) +struct ring_buffer *ring_buffer_get (struct perf_event *event) {struct ring_buffer *rb;
@@ -3963,7 +3962,7 @@ static struct ring_buffer *ring_buffer_get(struct perf_event *event)return rb; } -static void ring_buffer_put (struct ring_buffer *rb) +void ring_buffer_put (struct ring_buffer *rb) {if (!atomic_dec_and_test (&rb->refcount))return ;
kernel/events/internal.h +4 -0
@@ -36,6 +36,8 @@ struct ring_buffer {struct user_struct *mmap_user;/* AUX area */ +local_t aux_head; +local_t aux_nest;unsigned long aux_pgoff;int aux_nr_pages;atomic_t aux_mmap_count;
@@ -54,6 +56,8 @@ extern void perf_event_wakeup(struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event,pgoff_t pgoff,int nr_pages,int flags);extern void rb_free_aux (struct ring_buffer *rb,struct perf_event *event); +extern struct ring_buffer *ring_buffer_get (struct perf_event *event); +extern void ring_buffer_put (struct ring_buffer *rb);static inline bool rb_has_aux (struct ring_buffer *rb) {
kernel/events/ring_buffer.c +81 -0
87 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 8/23 perf: Add AUX record
In reply to Alexander ShishkinWhen there's new data in the AUX space, output a record indicating its offset and size and weather it was truncated to fix in the ring buffer.
--- include/uapi/linux/perf_event.h | 16 ++++++++++++++++ kernel/events/core.c | 39 +++++++++++++++++++++++++++++++++++++++ kernel/events/internal.h | 3 +++ kernel/events/ring_buffer.c | 1 + 4 files changed, 59 insertions(+)
include/uapi/linux/perf_event.h +16 -0
@@ -733,6 +733,22 @@ enum perf_event_type {*/ PERF_RECORD_MMAP2 =10 , +/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* u8 truncated; +* u8 reserved[7]; +* u64 id; +* u64 stream_id; +* }; +*/ +PERF_RECORD_AUX =11 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +39 -0
@@ -5443,6 +5443,45 @@ void perf_event_mmap(struct vm_area_struct *vma)perf_event_mmap_event (&mmap_event); } +void perf_event_aux_event (struct perf_event *event,unsigned long head, +unsigned long size,bool truncated) +{ +struct perf_output_handle handle; +struct perf_sample_data sample; +struct perf_aux_event { +struct perf_event_header header; +u64 offset; +u64 size; +u8 truncated; +u8 reserved[7 ]; +u64 id; +u64 stream_id; + } rec = { + .header = { + .type =PERF_RECORD_AUX , + .misc =0 , + .size =sizeof (rec), + }, + .offset = head, + .size = size, + .truncated = truncated, + .id =primary_event_id (event), + .stream_id = event->id, + }; +int ret; + +perf_event_header__init_id (&rec.header, &sample, event); + ret =perf_output_begin (&handle, event, rec.header.size); + +if (ret) +return ; + +perf_output_put (&handle, rec); +perf_event__output_id_sample (event, &handle, &sample); + +perf_output_end (&handle); +} +/* * IRQ throttle logging */
kernel/events/internal.h +3 -0
@@ -64,6 +64,9 @@ static inline bool rb_has_aux(struct ring_buffer *rb)return !!rb->aux_nr_pages; } +void perf_event_aux_event (struct perf_event *event,unsigned long head, +unsigned long size,bool truncated); + extern void perf_event_header__init_id(struct perf_event_header *header, struct perf_sample_data *data,
kernel/events/ring_buffer.c +1 -0
@@ -293,6 +293,7 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,struct ring_buffer *rb = handle->rb;local_add (size, &rb->aux_head); +perf_event_aux_event (handle->event, aux_head, size, truncated);smp_wmb (); rb->user_page->aux_head =local_read (&rb->aux_head);
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 9/23 perf: Support overwrite mode for AUX area
In reply to Alexander ShishkinThis adds support for overwrite mode in the AUX area, which means "keep collecting data till you're stopped". It does not depend on data buffer's overwrite mode, so that it doesn't lose sideband data that is instrumental for processing AUX data.
--- kernel/events/internal.h | 1 + kernel/events/ring_buffer.c | 40 +++++++++++++++++++++++++++++----------- 2 files changed, 30 insertions(+), 11 deletions(-)
kernel/events/internal.h +1 -0
@@ -40,6 +40,7 @@ struct ring_buffer {local_t aux_nest;unsigned long aux_pgoff;int aux_nr_pages; +int aux_overwrite;atomic_t aux_mmap_count;unsigned long aux_mmap_locked;void **aux_pages;
kernel/events/ring_buffer.c +29 -11
@@ -263,19 +263,23 @@ void *perf_aux_output_begin(struct perf_output_handle *handle, goto err; aux_head =local_read (&rb->aux_head); - aux_tail =ACCESS_ONCE (rb->user_page->aux_tail); handle->rb = rb; handle->event = event; handle->head = aux_head; - handle->size =CIRC_SPACE (aux_head, aux_tail,perf_aux_size (rb)); - -if (!handle->size) { - event->pending_disable =1 ; - event->hw.state =PERF_HES_STOPPED ; -perf_output_wakeup (handle); -local_set (&rb->aux_nest,0 ); - goto err; +if (!rb->aux_overwrite) { + aux_tail =ACCESS_ONCE (rb->user_page->aux_tail); + handle->size =CIRC_SPACE (aux_head, aux_tail,perf_aux_size (rb)); + +if (!handle->size) { + event->pending_disable =1 ; + event->hw.state =PERF_HES_STOPPED ; +perf_output_wakeup (handle); +local_set (&rb->aux_nest,0 ); + goto err; + } + }else { + handle->size =0 ; }return handle->rb->aux_priv;
@@ -291,9 +295,22 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size, bool truncated) {struct ring_buffer *rb = handle->rb; +unsigned long aux_head; -local_add (size, &rb->aux_head); -perf_event_aux_event (handle->event, aux_head, size, truncated); + aux_head =local_read (&rb->aux_head); + +if (rb->aux_overwrite) { +local_set (&rb->aux_head, size); + +/* +* Send a RECORD_AUX with size==0 to communicate aux_head +* of this snapshot to userspace +*/ +perf_event_aux_event (handle->event, size,0 , truncated); + }else { +local_add (size, &rb->aux_head); +perf_event_aux_event (handle->event, aux_head, size, truncated); + }smp_wmb (); rb->user_page->aux_head =local_read (&rb->aux_head);
@@ -405,6 +422,7 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event, overwrite);if (rb->aux_priv) ret =0 ; + rb->aux_overwrite = overwrite; out:if (!ret)
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 10/23 perf: Add wakeup watermark control to AUX area
In reply to Alexander ShishkinWhen AUX area gets a certain amount of new data, we want to wake up userspace to collect it. This adds a new control to specify how much data will cause a wakeup.
We repurpose __reserved_2 in the event attribute for this, even though it was never checked to be zero before, aux_watermark will only matter for new AUX-aware code, so the old code should still be fine.
--- include/uapi/linux/perf_event.h | 7 +++++-- kernel/events/core.c | 3 ++- kernel/events/internal.h | 4 +++- kernel/events/ring_buffer.c | 17 ++++++++++++++--- 4 files changed, 24 insertions(+), 7 deletions(-)
include/uapi/linux/perf_event.h +5 -2
@@ -238,6 +238,7 @@ enum perf_event_read_format {#define PERF_ATTR_SIZE_VER2 80/* add: branch_sample_type */ #define PERF_ATTR_SIZE_VER3 96/* add: sample_regs_user */ /* add: sample_stack_user */ +/* add: aux_watermark */ /* * Hardware event_id to monitor via a performance monitoring event:
@@ -332,8 +333,10 @@ struct perf_event_attr {*/ __u32 sample_stack_user; -/* Align to u64. */ -__u32 __reserved_2; +/* +* Wakeup watermark for AUX area +*/ +__u32 aux_watermark; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
kernel/events/core.c +2 -1
@@ -4257,7 +4257,8 @@ accounting:perf_event_init_userpage (event);perf_event_update_userpage (event); }else { - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, flags); + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags);if (ret)atomic_dec (&rb->mmap_count);else
kernel/events/internal.h +3 -1
@@ -27,6 +27,7 @@ struct ring_buffer {local_t lost;/* nr records lost */ long watermark;/* wakeup watermark */ +long aux_watermark;/* poll crap */ spinlock_t event_lock;struct list_head event_list;
@@ -38,6 +39,7 @@ struct ring_buffer {/* AUX area */ local_t aux_head;local_t aux_nest; +local_t aux_wakeup;unsigned long aux_pgoff;int aux_nr_pages;int aux_overwrite;
@@ -55,7 +57,7 @@ extern struct ring_buffer * rb_alloc(int nr_pages, long watermark, int cpu, int flags);extern void perf_event_wakeup (struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event, -pgoff_t pgoff,int nr_pages,int flags); +pgoff_t pgoff,int nr_pages,long watermark,int flags);extern void rb_free_aux (struct ring_buffer *rb,struct perf_event *event);extern struct ring_buffer *ring_buffer_get (struct perf_event *event);extern void ring_buffer_put (struct ring_buffer *rb);
kernel/events/ring_buffer.c +14 -3
@@ -269,8 +269,12 @@ void *perf_aux_output_begin(struct perf_output_handle *handle, handle->head = aux_head;if (!rb->aux_overwrite) { aux_tail =ACCESS_ONCE (rb->user_page->aux_tail); + handle->wakeup =local_read (&rb->aux_wakeup); handle->size =CIRC_SPACE (aux_head, aux_tail,perf_aux_size (rb)); +if (rb->aux_watermark && handle->size > rb->aux_watermark) + handle->size = rb->aux_watermark; +if (!handle->size) { event->pending_disable =1 ; event->hw.state =PERF_HES_STOPPED ;
@@ -313,9 +317,12 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size, }smp_wmb (); - rb->user_page->aux_head =local_read (&rb->aux_head); + aux_head = rb->user_page->aux_head =local_read (&rb->aux_head); -perf_output_wakeup (handle); +if (aux_head -local_read (&rb->aux_wakeup) > rb->aux_watermark) { +perf_output_wakeup (handle); +local_add (rb->aux_watermark, &rb->aux_wakeup); + } handle->event =NULL ;local_set (&rb->aux_nest,0 );
@@ -376,7 +383,7 @@ static void rb_free_aux_page(struct ring_buffer *rb, int idx) }int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event, -pgoff_t pgoff,int nr_pages,int flags) +pgoff_t pgoff,int nr_pages,long watermark,int flags) {bool overwrite = !(flags &RING_BUFFER_WRITABLE );int node = (event->cpu ==-1 ) ?-1 :cpu_to_node (event->cpu);
@@ -423,6 +430,10 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,if (rb->aux_priv) ret =0 ; rb->aux_overwrite = overwrite; + rb->aux_watermark = watermark; + +if (!rb->aux_watermark && !rb->aux_overwrite) + rb->aux_watermark = nr_pages << (PAGE_SHIFT -1 ); out:if (!ret)
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 11/23 perf: Add itrace_config to the event attribute
In reply to Alexander ShishkinTo configure itrace events, we use a separate config field in the attribute, which can be used by normal and sampling counters. The latter will use config to specify the actual event and itrace_config for itrace configuration that will be used for sampling.
--- include/linux/perf_event.h | 1 + include/uapi/linux/perf_event.h | 6 ++++++ 2 files changed, 7 insertions(+)
include/linux/perf_event.h +1 -0
@@ -173,6 +173,7 @@ struct perf_event;#define PERF_PMU_CAP_AUX_NO_SG 0x02#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF 0x04#define PERF_PMU_CAP_EXCLUSIVE 0x08 +#define PERF_PMU_CAP_ITRACE 0x10/** * struct pmu - generic performance monitoring unit
include/uapi/linux/perf_event.h +6 -0
@@ -239,6 +239,7 @@ enum perf_event_read_format {#define PERF_ATTR_SIZE_VER3 96/* add: sample_regs_user */ /* add: sample_stack_user */ /* add: aux_watermark */ +#define PERF_ATTR_SIZE_VER4 104/* add: itrace_config */ /* * Hardware event_id to monitor via a performance monitoring event:
@@ -337,6 +338,11 @@ struct perf_event_attr {* Wakeup watermark for AUX area */ __u32 aux_watermark; + +/* +* Itrace pmus' event config +*/ +__u64 itrace_config; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 12/23 perf: add ITRACE_START record to indicate that tracing has started
In reply to Alexander ShishkinFor counters such as instruction tracing, it is useful for the decoder to know which tasks are running when the event is first scheduled in, before the first sched_switch.
--- include/linux/perf_event.h | 3 +++ include/uapi/linux/perf_event.h | 11 +++++++++++ kernel/events/core.c | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 55 insertions(+)
include/linux/perf_event.h +3 -0
@@ -126,6 +126,9 @@ struct hw_perf_event {/* for tp_event->class */ struct list_head tp_list; }; +struct {/* itrace */ +int itrace_started; + }; #ifdef CONFIG_HAVE_HW_BREAKPOINT struct { /* breakpoint */ /*
include/uapi/linux/perf_event.h +11 -0
@@ -758,6 +758,17 @@ enum perf_event_type {*/ PERF_RECORD_AUX =11 , +/* +* Indicates that instruction trace has started +* +* struct { +* struct perf_event_header header; +* u32 pid; +* u32 tid; +* }; +*/ +PERF_RECORD_ITRACE_START =12 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +41 -0
59 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 13/23 x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
In reply to Alexander ShishkinIntel Processor Trace is an architecture extension that allows for program flow tracing.
--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
@@ -186,6 +186,7 @@#define X86_FEATURE_DTHERM (7*32+ 7)/* Digital Thermal Sensor */ #define X86_FEATURE_HW_PSTATE (7*32+ 8)/* AMD HW-PState */ #define X86_FEATURE_PROC_FEEDBACK (7*32+ 9)/* AMD ProcFeedbackInterface */ +#define X86_FEATURE_INTEL_PT (7*32+10)/* Intel Processor Trace */ /* Virtualization flags: Linux defined, word 8 */ #define X86_FEATURE_TPR_SHADOW (8*32+ 0)/* Intel TPR Shadow */
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -36,6 +36,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c) { X86_FEATURE_ARAT, CR_EAX, 2, 0x00000006, 0 }, { X86_FEATURE_PLN, CR_EAX, 4, 0x00000006, 0 }, { X86_FEATURE_PTS, CR_EAX, 6, 0x00000006, 0 }, + { X86_FEATURE_INTEL_PT, CR_EBX,25, 0x00000007, 0 }, { X86_FEATURE_APERFMPERF, CR_ECX, 0, 0x00000006, 0 }, { X86_FEATURE_EPB, CR_ECX, 3, 0x00000006, 0 }, { X86_FEATURE_XSAVEOPT, CR_EAX, 0, 0x0000000d, 1 },
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 14/23 x86: perf: Intel PT and LBR/BTS are mutually exclusive
In reply to Alexander ShishkinIntel PT cannot be used at the same time as LBR or BTS and will cause a
general protection fault if they are used together. In order to avoid
fixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.
--- arch/x86/kernel/cpu/perf_event.h | 6 ++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 8 +++++++- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 9 +++++---- 3 files changed, 18 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/perf_event.h +6 -0
@@ -146,6 +146,7 @@ struct cpu_hw_events {* Intel DebugStore bits */ struct debug_store *ds; +unsigned int bts_enabled;u64 pebs_enabled;/*
@@ -159,6 +160,11 @@ struct cpu_hw_events {u64 br_sel;/* +* Intel Processor Trace +*/ +unsigned int pt_enabled; + +/* * Intel host/guest exclude bits */ u64 intel_ctrl_guest_mask;
arch/x86/kernel/cpu/perf_event_intel_ds.c +7 -1
@@ -456,8 +456,13 @@ struct event_constraint bts_constraint =void intel_pmu_enable_bts (u64 config) { +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);unsigned long debugctlmsr; +if (cpuc->pt_enabled) +return ; + + cpuc->bts_enabled =1 ; debugctlmsr =get_debugctlmsr (); debugctlmsr |=DEBUGCTLMSR_TR ;
@@ -478,9 +483,10 @@ void intel_pmu_disable_bts(void)struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);unsigned long debugctlmsr; -if (!cpuc->ds) +if (!cpuc->ds || cpuc->pt_enabled)return ; + cpuc->bts_enabled =0 ; debugctlmsr =get_debugctlmsr (); debugctlmsr &=
arch/x86/kernel/cpu/perf_event_intel_lbr.c +5 -4
@@ -172,7 +172,9 @@ static void intel_pmu_lbr_reset_64(void)void intel_pmu_lbr_reset (void ) { -if (!x86_pmu.lbr_nr) +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); + +if (!x86_pmu.lbr_nr || cpuc->pt_enabled)return ;if (x86_pmu.intel_cap.lbr_format ==LBR_FORMAT_32 )
@@ -185,7 +187,7 @@ void intel_pmu_lbr_enable(struct perf_event *event) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); -if (!x86_pmu.lbr_nr) +if (!x86_pmu.lbr_nr || cpuc->pt_enabled)return ;/*
@@ -205,11 +207,10 @@ void intel_pmu_lbr_disable(struct perf_event *event) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); -if (!x86_pmu.lbr_nr) +if (!x86_pmu.lbr_nr || !cpuc->lbr_users || cpuc->pt_enabled)return ; cpuc->lbr_users--; -WARN_ON_ONCE (cpuc->lbr_users <0 );if (cpuc->enabled && !cpuc->lbr_users) {__intel_pmu_lbr_disable ();
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 15/23 x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinAdd support for Intel Processor Trace (PT) to kernel's perf events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel.
This driver exports trace data by through AUX space in the perf ring buffer, which is zero-copy mapped into userspace for faster data retrieval.
--- arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 129 ++++ arch/x86/kernel/cpu/perf_event.h | 2 + arch/x86/kernel/cpu/perf_event_intel.c | 8 + arch/x86/kernel/cpu/perf_event_intel_pt.c | 952 ++++++++++++++++++++++++++++++ 6 files changed, 1110 insertions(+) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
arch/x86/include/uapi/asm/msr-index.h +18 -0
@@ -74,6 +74,24 @@#define MSR_IA32_PERF_CAPABILITIES 0x00000345#define MSR_PEBS_LD_LAT_THRESHOLD 0x000003f6 +#define MSR_IA32_RTIT_CTL 0x00000570 +#define RTIT_CTL_TRACEEN BIT(0) +#define RTIT_CTL_OS BIT(2) +#define RTIT_CTL_USR BIT(3) +#define RTIT_CTL_CR3EN BIT(7) +#define RTIT_CTL_TOPA BIT(8) +#define RTIT_CTL_TSC_EN BIT(10) +#define RTIT_CTL_DISRETC BIT(11) +#define RTIT_CTL_BRANCH_EN BIT(13) +#define MSR_IA32_RTIT_STATUS 0x00000571 +#define RTIT_STATUS_CONTEXTEN BIT(1) +#define RTIT_STATUS_TRIGGEREN BIT(2) +#define RTIT_STATUS_ERROR BIT(4) +#define RTIT_STATUS_STOPPED BIT(5) +#define MSR_IA32_RTIT_CR3_MATCH 0x00000572 +#define MSR_IA32_RTIT_OUTPUT_BASE 0x00000560 +#define MSR_IA32_RTIT_OUTPUT_MASK 0x00000561 +#define MSR_MTRRfix64K_00000 0x00000250#define MSR_MTRRfix16K_80000 0x00000258#define MSR_MTRRfix16K_A0000 0x00000259
arch/x86/kernel/cpu/Makefile +1 -0
@@ -37,6 +37,7 @@ endif obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_p6.o perf_event_knc.o perf_event_p4.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore.o perf_event_intel_rapl.o +obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o endif
arch/x86/kernel/cpu/intel_pt.h new file +129 -0
129 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event.h +2 -0
@@ -713,6 +713,8 @@ void intel_pmu_lbr_init_snb(void);int intel_pmu_setup_lbr_filter (struct perf_event *event); +void intel_pt_interrupt (void ); +int p4_pmu_init (void );int p6_pmu_init (void );
arch/x86/kernel/cpu/perf_event_intel.c +8 -0
@@ -1399,6 +1399,14 @@ again: }/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } + +/* * Checkpointed counters can lead to 'spurious' PMIs because the * rollback caused by the PMI will have cleared the overflow status * bit. Therefore always force probe these counters.
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +952 -0
952 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 16/23 x86: perf: intel_bts: Add BTS PMU driver
In reply to Alexander ShishkinAdd support for Branch Trace Store (BTS) via kernel perf event infrastructure. The difference with the existing implementation of BTS support is that this one is a separate PMU that exports events' trace buffers to userspace by means of AUX area of the perf buffer, which is zero-copy mapped into userspace.
The immediate benefit is that the buffer size can be much bigger, resulting in fewer interrupts and no kernel side copying is involved and little to no trace data loss. Also, kernel code can be traced with this driver.
The old way of collecting BTS traces still works.
--- arch/x86/kernel/cpu/Makefile | 2 +- arch/x86/kernel/cpu/perf_event.h | 6 + arch/x86/kernel/cpu/perf_event_intel.c | 6 +- arch/x86/kernel/cpu/perf_event_intel_bts.c | 496 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 3 +- 5 files changed, 510 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
arch/x86/kernel/cpu/Makefile +1 -1
@@ -37,7 +37,7 @@ endif obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_p6.o perf_event_knc.o perf_event_p4.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore.o perf_event_intel_rapl.o -obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o +obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o perf_event_intel_bts.o endif
arch/x86/kernel/cpu/perf_event.h +6 -0
@@ -715,6 +715,12 @@ int intel_pmu_setup_lbr_filter(struct perf_event *event);void intel_pt_interrupt (void ); +int intel_bts_interrupt (void ); + +void intel_bts_enable_local (void ); + +void intel_bts_disable_local (void ); +int p4_pmu_init (void );int p6_pmu_init (void );
arch/x86/kernel/cpu/perf_event_intel.c +5 -1
@@ -1051,6 +1051,8 @@ static void intel_pmu_disable_all(void)if (test_bit (INTEL_PMC_IDX_FIXED_BTS , cpuc->active_mask))intel_pmu_disable_bts (); +else +intel_bts_disable_local ();intel_pmu_pebs_disable_all ();intel_pmu_lbr_disable_all ();
@@ -1073,7 +1075,8 @@ static void intel_pmu_enable_all(int added)return ;intel_pmu_enable_bts (event->hw.config); - } + }else +intel_bts_enable_local (); }/*
@@ -1359,6 +1362,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)apic_write (APIC_LVTPC ,APIC_DM_NMI );intel_pmu_disable_all (); handled =intel_pmu_drain_bts_buffer (); + handled +=intel_bts_interrupt (); status =intel_pmu_get_status ();if (!status) goto done;
arch/x86/kernel/cpu/perf_event_intel_bts.c new file +496 -0
496 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event_intel_ds.c +2 -1
@@ -467,7 +467,8 @@ void intel_pmu_enable_bts(u64 config) debugctlmsr |=DEBUGCTLMSR_TR ; debugctlmsr |=DEBUGCTLMSR_BTS ; - debugctlmsr |=DEBUGCTLMSR_BTINT ; +if (config &ARCH_PERFMON_EVENTSEL_INT ) + debugctlmsr |=DEBUGCTLMSR_BTINT ;if (!(config &ARCH_PERFMON_EVENTSEL_OS )) debugctlmsr |=DEBUGCTLMSR_BTS_OFF_OS ;
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 17/23 perf: Add rb_{alloc,free}_kernel api
In reply to Alexander ShishkinEvents that generate AUX data can also be created by the kernel. In this case, some in-kernel infrastructure is needed to store and copy this data.
This patch adds api for ring buffer (de-)allocation that can be used by the kernel code.
--- kernel/events/internal.h | 3 +++ kernel/events/ring_buffer.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+)
kernel/events/internal.h +3 -0
@@ -59,6 +59,9 @@ extern void perf_event_wakeup(struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event,pgoff_t pgoff,int nr_pages,long watermark,int flags);extern void rb_free_aux (struct ring_buffer *rb,struct perf_event *event); +extern struct ring_buffer * +rb_alloc_kernel (struct perf_event *event,int nr_pages,int aux_nr_pages); +extern void rb_free_kernel (struct ring_buffer *rb,struct perf_event *event);extern struct ring_buffer *ring_buffer_get (struct perf_event *event);extern void ring_buffer_put (struct ring_buffer *rb);
kernel/events/ring_buffer.c +33 -0
@@ -467,6 +467,39 @@ void rb_free_aux(struct ring_buffer *rb, struct perf_event *event) rb->aux_nr_pages =0 ; } +struct ring_buffer * +rb_alloc_kernel (struct perf_event *event,int nr_pages,int aux_nr_pages) +{ +struct ring_buffer *rb; +int ret, pgoff = nr_pages +1 ; + + rb =rb_alloc (nr_pages,0 , event->cpu,0 ); +if (!rb) +return NULL ; + + ret =rb_alloc_aux (rb, event, pgoff, aux_nr_pages,0 ,0 ); +if (ret) { +rb_free (rb); +return NULL ; + } + +/* +* Kernel counters don't need ring buffer wakeups, therefore we don't +* use ring_buffer_attach() here and event->rb_entry stays empty +*/ +rcu_assign_pointer (event->rb, rb); + +return rb; +} + +void rb_free_kernel (struct ring_buffer *rb,struct perf_event *event) +{ +BUG_ON (atomic_read (&rb->refcount) !=1 ); +rcu_assign_pointer (event->rb,NULL ); +rb_free_aux (rb, event); +rb_free (rb); +} +#ifndef CONFIG_PERF_USE_VMALLOC /*
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 18/23 perf: Add a helper to copy AUX data in the kernel
In reply to Alexander ShishkinThis patch adds a helper for the kernel counters that generate AUX data to copy this data around, for example, to output it to the perf ring buffer as a sample record or write it to a core dump file.
--- kernel/events/internal.h | 5 +++++ kernel/events/ring_buffer.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+)
kernel/events/internal.h +5 -0
@@ -52,6 +52,9 @@ struct ring_buffer {void *data_pages[0 ]; }; +typedef unsigned long (*aux_copyfn )(void *data,const void *src, +unsigned long len); +extern void rb_free (struct ring_buffer *rb);extern struct ring_buffer *rb_alloc (int nr_pages,long watermark,int cpu,int flags);
@@ -59,6 +62,8 @@ extern void perf_event_wakeup(struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event,pgoff_t pgoff,int nr_pages,long watermark,int flags);extern void rb_free_aux (struct ring_buffer *rb,struct perf_event *event); +extern long rb_output_aux (struct ring_buffer *rb,unsigned long from, +unsigned long to,aux_copyfn copyfn,void *data);extern struct ring_buffer *rb_alloc_kernel (struct perf_event *event,int nr_pages,int aux_nr_pages);extern void rb_free_kernel (struct ring_buffer *rb,struct perf_event *event);
kernel/events/ring_buffer.c +31 -0
@@ -348,6 +348,37 @@ void *perf_get_aux(struct perf_output_handle *handle)return handle->rb->aux_priv; } +long rb_output_aux (struct ring_buffer *rb,unsigned long from, +unsigned long to,aux_copyfn copyfn,void *data) +{ +unsigned long tocopy, remainder, len =0 ; +void *addr; + + from &= (rb->aux_nr_pages <<PAGE_SHIFT ) -1 ; + to &= (rb->aux_nr_pages <<PAGE_SHIFT ) -1 ; + +do { + tocopy =PAGE_SIZE -offset_in_page (from); +if (to > from) + tocopy =min (tocopy, to - from); +if (!tocopy) +break ; + + addr = rb->aux_pages[from >>PAGE_SHIFT ]; + addr +=offset_in_page (from); + + remainder =copyfn (data, addr, tocopy); +if (remainder) +return -EFAULT ; + + len += tocopy; + from += tocopy; + from &= (rb->aux_nr_pages <<PAGE_SHIFT ) -1 ; + }while (to != from); + +return len; +} +#define PERF_AUX_GFP (GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY)static struct page *rb_alloc_aux_page (int node,int order)
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 19/23 perf: Add a helper for looking up pmus by type
In reply to Alexander ShishkinThis patch adds a helper for looking up a registered pmu by its type.
--- kernel/events/core.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
kernel/events/core.c +13 -3
@@ -6831,6 +6831,18 @@ void perf_pmu_unregister(struct pmu *pmu) }EXPORT_SYMBOL_GPL (perf_pmu_unregister); +/* call under pmus_srcu */ +static struct pmu *__perf_find_pmu (u32 type) +{ +struct pmu *pmu; + +rcu_read_lock (); + pmu =idr_find (&pmu_idr, type); +rcu_read_unlock (); + +return pmu; +} +struct pmu *perf_init_event (struct perf_event *event) {struct pmu *pmu =NULL ;
@@ -6839,9 +6851,7 @@ struct pmu *perf_init_event(struct perf_event *event) idx =srcu_read_lock (&pmus_srcu); -rcu_read_lock (); - pmu =idr_find (&pmu_idr, event->attr.type); -rcu_read_unlock (); + pmu =__perf_find_pmu (event->attr.type);if (pmu) {if (!try_module_get (pmu->module)) { pmu =ERR_PTR (-ENODEV );
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 20/23 perf: itrace: Infrastructure for sampling instruction flow traces
In reply to Alexander ShishkinInstruction tracing PMUs are capable of recording a log of instruction execution flow on a cpu core, which can be useful for profiling and crash analysis. This patch adds itrace infrastructure for perf events and the rest of the kernel to use.
This trace data can be used to annotate other perf events by including it in sample records when PERF_SAMPLE_ITRACE flag is set. In this case, a kernel counter is created for each such event and trace data is retrieved from it and stored in the perf data stream.
--- include/linux/itrace.h | 45 ++++++++++++ include/linux/perf_event.h | 14 ++++ include/uapi/linux/perf_event.h | 14 +++- kernel/events/Makefile | 2 +- kernel/events/core.c | 38 ++++++++++ kernel/events/itrace.c | 159 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 269 insertions(+), 3 deletions(-) create mode 100644 include/linux/itrace.h create mode 100644 kernel/events/itrace.c
include/linux/itrace.h new file +45 -0
@@ -0,0 +1,45 @@ +/* +* Instruction flow trace unit infrastructure +* Copyright (c) 2013-2014, Intel Corporation. +* +* This program is free software; you can redistribute it and/or modify it +* under the terms and conditions of the GNU General Public License, +* version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope it will be useful, but WITHOUT +* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +* more details. +*/ + +#ifndef _LINUX_ITRACE_H +#define _LINUX_ITRACE_H + +#include <linux/perf_event.h> + +#ifdef CONFIG_PERF_EVENTS +extern int itrace_sampler_init (struct perf_event *event, +struct task_struct *task, +struct pmu *pmu); +extern void itrace_sampler_fini (struct perf_event *event); +extern unsigned long itrace_sampler_trace (struct perf_event *event, +struct perf_sample_data *data); +extern void itrace_sampler_output (struct perf_event *event, +struct perf_output_handle *handle, +struct perf_sample_data *data); +#else +static inline int itrace_sampler_init (struct perf_event *event, +struct task_struct *task, +struct pmu *pmu) {return -EINVAL ; } +static inline void +itrace_sampler_fini (struct perf_event *event) {} +static inline unsigned long +itrace_sampler_trace (struct perf_event *event, +struct perf_sample_data *data) {return 0 ; } +static inline void +itrace_sampler_output (struct perf_event *event, +struct perf_output_handle *handle, +struct perf_sample_data *data) {} +#endif + +#endif /* _LINUX_PERF_EVENT_H */
include/linux/perf_event.h +14 -0
@@ -83,6 +83,12 @@ struct perf_regs_user {struct pt_regs *regs; }; +struct perf_trace_record { +u64 size; +unsigned long from; +unsigned long to; +}; +struct task_struct ;/*
@@ -456,6 +462,7 @@ struct perf_event {perf_overflow_handler_t overflow_handler;void *overflow_handler_context; +struct perf_event *trace_event;#ifdef CONFIG_EVENT_TRACING struct ftrace_event_call *tp_event;struct event_filter *filter;
@@ -623,6 +630,7 @@ struct perf_sample_data {union perf_mem_data_src data_src;struct perf_callchain_entry *callchain;struct perf_raw_record *raw; +struct perf_trace_record trace;struct perf_branch_stack *br_stack;struct perf_regs_user regs_user;u64 stack_user_size;
@@ -643,6 +651,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data, data->period = period; data->regs_user.abi =PERF_SAMPLE_REGS_ABI_NONE ; data->regs_user.regs =NULL ; + data->trace.from = data->trace.to = data->trace.size =0 ; data->stack_user_size =0 ; data->weight =0 ; data->data_src.val =0 ;
@@ -804,6 +813,11 @@ static inline bool has_aux(struct perf_event *event)return event->pmu->setup_aux; } +static inline bool is_itrace_event (struct perf_event *event) +{ +return !!(event->pmu->capabilities &PERF_PMU_CAP_ITRACE ); +} +extern int perf_output_begin (struct perf_output_handle *handle,struct perf_event *event,unsigned int size);extern void perf_output_end (struct perf_output_handle *handle);
include/uapi/linux/perf_event.h +12 -2
@@ -137,8 +137,9 @@ enum perf_event_sample_format {PERF_SAMPLE_DATA_SRC =1U <<15 ,PERF_SAMPLE_IDENTIFIER =1U <<16 ,PERF_SAMPLE_TRANSACTION =1U <<17 , +PERF_SAMPLE_ITRACE =1U <<18 , - PERF_SAMPLE_MAX = 1U << 18, /* non-ABI */ + PERF_SAMPLE_MAX = 1U << 19, /* non-ABI */ };/*
@@ -239,7 +240,9 @@ enum perf_event_read_format {#define PERF_ATTR_SIZE_VER3 96/* add: sample_regs_user */ /* add: sample_stack_user */ /* add: aux_watermark */ -#define PERF_ATTR_SIZE_VER4 104/* add: itrace_config */ +#define PERF_ATTR_SIZE_VER4 120/* add: itrace_config */ +/* add: itrace_sample_size */ +/* add: itrace_sample_type */ /* * Hardware event_id to monitor via a performance monitoring event:
@@ -343,6 +346,11 @@ struct perf_event_attr {* Itrace pmus' event config */ __u64 itrace_config; +__u64 itrace_sample_size; +__u32 itrace_sample_type;/* pmu->type of the itrace PMU */ + +/* Align to u64. */ +__u32 __reserved_2; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
@@ -716,6 +724,8 @@ enum perf_event_type {* { u64 weight; } && PERF_SAMPLE_WEIGHT * { u64 data_src; } && PERF_SAMPLE_DATA_SRC * { u64 transaction; } && PERF_SAMPLE_TRANSACTION +* { u64 size; +* char data[size]; } && PERF_SAMPLE_ITRACE * }; */ PERF_RECORD_SAMPLE = 9,
kernel/events/Makefile +1 -1
@@ -2,7 +2,7 @@ ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_core.o = -pg endif -obj-y := core.o ring_buffer.o callchain.o +obj-y := core.o ring_buffer.o callchain.o itrace.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o obj-$(CONFIG_UPROBES) += uprobes.o
kernel/events/core.c +38 -0
@@ -41,6 +41,7 @@#include <linux/cgroup.h> #include <linux/module.h> #include <linux/mman.h> +#include <linux/itrace.h> #include "internal.h"
@@ -1595,6 +1596,9 @@ void perf_event_disable(struct perf_event *event)struct perf_event_context *ctx = event->ctx;struct task_struct *task = ctx->task; +if (event->trace_event) +perf_event_disable (event->trace_event); +if (!task) {/* * Disable the event on the cpu that it's on
@@ -2094,6 +2098,8 @@ void perf_event_enable(struct perf_event *event)struct perf_event_context *ctx = event->ctx;struct task_struct *task = ctx->task; +if (event->trace_event) +perf_event_enable (event->trace_event);if (!task) {/* * Enable the event on the cpu that it's on
@@ -3250,6 +3256,8 @@ static void unaccount_event(struct perf_event *event)static_key_slow_dec_deferred (&perf_sched_events);if (has_branch_stack (event))static_key_slow_dec_deferred (&perf_sched_events); +if ((event->attr.sample_type &PERF_SAMPLE_ITRACE )) +itrace_sampler_fini (event);unaccount_event_cpu (event, event->cpu); }
@@ -4781,6 +4789,13 @@ void perf_output_sample(struct perf_output_handle *handle,if (sample_type &PERF_SAMPLE_TRANSACTION )perf_output_put (handle, data->txn); +if (sample_type &PERF_SAMPLE_ITRACE ) { +perf_output_put (handle, data->trace.size); + +if (data->trace.size) +itrace_sampler_output (event, handle, data); + } +if (!event->attr.watermark) {int wakeup_events = event->attr.wakeup_events;
@@ -4888,6 +4903,14 @@ void perf_prepare_sample(struct perf_event_header *header, data->stack_user_size = stack_size; header->size += size; } + +if (sample_type &PERF_SAMPLE_ITRACE ) { +u64 size =sizeof (u64); + + size +=itrace_sampler_trace (event, data); + + header->size += size; + } } static void perf_event_output(struct perf_event *event,
@@ -7040,6 +7063,21 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,if (err) goto err_pmu; } + +if (event->attr.sample_type &PERF_SAMPLE_ITRACE ) { +struct pmu *itrace_pmu; +int idx; + + idx =srcu_read_lock (&pmus_srcu); + itrace_pmu =__perf_find_pmu (event->attr.itrace_sample_type); + err =itrace_sampler_init (event, task, itrace_pmu); +srcu_read_unlock (&pmus_srcu, idx); + +if (err) { +put_callchain_buffers (); + goto err_pmu; + } + } }return event;
kernel/events/itrace.c new file +159 -0
159 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 21/23 perf: Allocate ring buffers for inherited per-task kernel events
In reply to Alexander ShishkinWhen a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or whatnot.
--- kernel/events/core.c | 9 +++++++++ kernel/events/internal.h | 11 +++++++++++ 2 files changed, 20 insertions(+)
kernel/events/core.c +9 -0
@@ -7993,6 +7993,15 @@ inherit_event(struct perf_event *parent_event, (void )perf_event_set_output (child_event, parent_event);/* +* For per-task kernel events with ring buffers, set_output doesn't +* make sense, but we can allocate a new buffer here. +*/ +if (parent_event->cpu ==-1 &&kernel_rb_event (parent_event)) { + (void )rb_alloc_kernel (child_event, parent_event->rb->nr_pages, + parent_event->rb->aux_nr_pages); + } + +/* * Precalculate sample_data sizes */ perf_event__header_size (child_event);
kernel/events/internal.h +11 -0
@@ -120,6 +120,17 @@ static inline unsigned long perf_aux_size(struct ring_buffer *rb)return rb->aux_nr_pages <<PAGE_SHIFT ; } +static inline bool kernel_rb_event (struct perf_event *event) +{ +/* +* Having a ring buffer and not being on any ring buffers' wakeup +* list means it was attached by rb_alloc_kernel() and not +* ring_buffer_attach(). It's the only case when these two +* conditions take place at the same time. +*/ +return event->rb &&list_empty (&event->rb_entry); +} +#define DEFINE_OUTPUT_COPY (func_name, memcpy_func) \ static inline unsigned long \ func_name(struct perf_output_handle *handle, \
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 22/23 perf: itrace: Allow itrace sampling for multiple events
In reply to Alexander ShishkinRight now, only one perf event can be annotated with itrace data, however it should be possible to annotate several events with similar configuration (wrt exclude_{hv,idle,user,kernel}, itrace_config, etc).
So every time, before a kernel counter is created for itrace sampling, we first look for an existing counter that is suitable in terms of configuration and we use it to annotate the new event as well.
--- kernel/events/core.c | 6 ++-- kernel/events/internal.h | 4 +++ kernel/events/itrace.c | 94 +++++++++++++++++++++++++++++++++++------------- 3 files changed, 77 insertions(+), 27 deletions(-)
kernel/events/core.c +3 -3
@@ -891,7 +891,7 @@ static void get_ctx(struct perf_event_context *ctx)WARN_ON (!atomic_inc_not_zero (&ctx->refcount)); } -static void put_ctx (struct perf_event_context *ctx) +void put_ctx (struct perf_event_context *ctx) {if (atomic_dec_and_test (&ctx->refcount)) {if (ctx->parent_ctx)
@@ -3130,7 +3130,7 @@ errout:/* * Returns a matching context with refcount and pincount. */ -static struct perf_event_context * +struct perf_event_context *find_get_context (struct pmu *pmu,struct task_struct *task,int cpu) {struct perf_event_context *ctx;
@@ -3324,7 +3324,7 @@ static void free_event(struct perf_event *event)/* * Called when the last reference to the file is gone. */ -static void put_event (struct perf_event *event) +void put_event (struct perf_event *event) {struct perf_event_context *ctx = event->ctx;struct task_struct *owner;
kernel/events/internal.h +4 -0
@@ -55,6 +55,10 @@ struct ring_buffer {typedef unsigned long (*aux_copyfn )(void *data,const void *src,unsigned long len); +extern struct perf_event_context * +find_get_context (struct pmu *pmu,struct task_struct *task,int cpu); +extern void put_ctx (struct perf_event_context *ctx); +void put_event (struct perf_event *event);extern void rb_free (struct ring_buffer *rb);extern struct ring_buffer *rb_alloc (int nr_pages,long watermark,int cpu,int flags);
kernel/events/itrace.c +70 -24
128 lines not on this page · raw message
Signature · 1 line
2.1.0.rc1
Alexander Shishkin
reviewerPATCH v3 23/23 perf: itrace: Allow sampling of inherited events
In reply to Alexander ShishkinTry to find an itrace sampler event for the current event if none is linked. This is useful when these events are allocated by inheritance path, independently of one another.
--- kernel/events/itrace.c | 91 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 60 insertions(+), 31 deletions(-)
kernel/events/itrace.c +60 -31
@@ -48,46 +48,66 @@ static bool itrace_event_match(struct perf_event *e1, struct perf_event *e2)* to configure and obtain itrace samples. */ +struct perf_event *__find_sampling_counter (struct perf_event_context *ctx, +struct perf_event *event, +struct task_struct *task) +{ +struct perf_event *sampler =NULL ; + +list_for_each_entry (sampler, &ctx->event_list, event_entry) { +/* +* event is not an itrace event, but all the relevant +* bits should match +*/ +if (itrace_event_match (sampler, event) && +kernel_rb_event (sampler) && + sampler->attr.exclude_hv == event->attr.exclude_hv && + sampler->attr.exclude_idle == event->attr.exclude_idle && + sampler->attr.exclude_user == event->attr.exclude_user && + sampler->attr.exclude_kernel == event->attr.exclude_kernel && + sampler->attr.itrace_config == event->attr.itrace_config && + sampler->attr.type == event->attr.itrace_sample_type && + sampler->attr.itrace_sample_size >= event->attr.itrace_sample_size && +atomic_long_inc_not_zero (&sampler->refcount)) +return sampler; + } + +return NULL ; +} + +struct perf_event *find_sampling_counter (struct pmu *pmu, +struct perf_event *event, +struct task_struct *task) +{ +struct perf_event_context *ctx; +struct perf_event *sampler =NULL ; +unsigned long flags; + + ctx =find_get_context (pmu, task, event->cpu); +if (!ctx) +return NULL ; + +raw_spin_lock_irqsave (&ctx->lock, flags); + sampler =__find_sampling_counter (ctx, event, task); + --ctx->pin_count; +put_ctx (ctx); +raw_spin_unlock_irqrestore (&ctx->lock, flags); + +return sampler; +} +int itrace_sampler_init (struct perf_event *event,struct task_struct *task,struct pmu *pmu) { -struct perf_event_context *ctx;struct perf_event_attr attr;struct perf_event *tevt =NULL ;struct ring_buffer *rb; -unsigned long nr_pages, flags; +unsigned long nr_pages;if (!pmu || !(pmu->capabilities &PERF_PMU_CAP_ITRACE ))return -ENOTSUPP ; - ctx =find_get_context (pmu, task, event->cpu); -if (ctx) { -raw_spin_lock_irqsave (&ctx->lock, flags); -list_for_each_entry (tevt, &ctx->event_list, event_entry) { -/* -* event is not an itrace event, but all the relevant -* bits should match -*/ -if (itrace_event_match (tevt, event) && - tevt->attr.exclude_hv == event->attr.exclude_hv && - tevt->attr.exclude_idle == event->attr.exclude_idle && - tevt->attr.exclude_user == event->attr.exclude_user && - tevt->attr.exclude_kernel == event->attr.exclude_kernel && - tevt->attr.itrace_config == event->attr.itrace_config && - tevt->attr.type == event->attr.itrace_sample_type && - tevt->attr.itrace_sample_size >= event->attr.itrace_sample_size && -atomic_long_inc_not_zero (&tevt->refcount)) - goto got_event; - } - - tevt =NULL ; - -got_event: - --ctx->pin_count; -put_ctx (ctx); -raw_spin_unlock_irqrestore (&ctx->lock, flags); - } - + tevt =find_sampling_counter (pmu, event, task);if (!tevt) {memset (&attr,0 ,sizeof (attr)); attr.type = pmu->type;
@@ -139,9 +159,18 @@ void itrace_sampler_fini(struct perf_event *event)unsigned long itrace_sampler_trace (struct perf_event *event,struct perf_sample_data *data) { -struct perf_event *tevt = event->trace_event; +struct perf_event *tevt;struct ring_buffer *rb; +if (!event->trace_event) { +/* +* down this path, event->ctx is already locked IF it's the +* same context +*/ + event->trace_event =__find_sampling_counter (event->ctx, event, event->ctx->task); + } + + tevt = event->trace_event;/* Don't go further if the event is being scheduled out */ if (!tevt || tevt->state !=PERF_EVENT_STATE_ACTIVE ) { data->trace.size =0 ;
Signature · 1 line
2.1.0.rc1
Quoted 7 lines · Alexander Shishkin
Hi Peter and all,
Here's a new version of PT support patchset, this time including PT and BTS drivers and a few more tweaks to the core. I still left out some bits like core dump support for now. Tooling support is not included in this series so that it's easier to review the kernel bits, I suppose it's best to send it separately, since it's quite a huge patchset of its own.
I know what this series is about, but pretty please, always include a complete introduction, or at least a link to a good introdcution, which spells out the whole problem space, the proposed solution, its various design trade-offs (if any), links to tooling for people who want to have a look at both sides, list of items not yet properly implemented [or a clear statement that it's all ready to go in your view], etc., in as much detail as possible!
The reason is that many reviewers will skip early versions of patch series, especially if they see something trivially objectionable in it. Why spend effort on reviewing something that might never see the light of the day? But if later re-sends of the series skip essential information it's harder to 'jump in' and offer useful feedback...
If you worry about being overly verbose in your patch series announcement, in my experience as a kernel maintainer it's literally impossible for kernel developers to over-do the description of a new feature. (Steve Rostedt tries on occasion but even he never succeeded in the past. I claim this is due to software developer brain structure, but I digress.)
It's also absolutely fine to cut & paste an old 0/N description over and over again, and to update it only as far as the patches have changed.
A good rule of thumb is to have at least as many sentences in the 0/N description as there are patches in the series, i.e. 23 sentences in this instance. Preferably scaled up by complexity.
Thanks,
Ingo