perf: Add infrastructure and support for Intel PT
Subsystems (2)
Alexander Shishkin
reviewerv5 cover perf: Add infrastructure and support for Intel PT
Hi Peter and all,
[full description below the changelog]
This version of the patchset hopefully addresses comments from the previous (v4) version. Changelog messages should be more descriptive as well as comments in the code. Funcitonal changes:
* events are not disabled on munmap(), this got replaced with refcounting,
* explicit sfence is added to the intel_pt driver to make sure that data stores are globally visible before the aux_head is updated,
* dropped the unnecessary set_output for inherited events in favor of using parent's ring buffer,
* intel_pt needs to handle #GP from enabling WRMSR, so that a privileged user can set arbitrary RTIT_CTL bits in the range that is reserved for packet enables (see PT_BYPASS_MASK and comments around pt_config()) without potentially killing the machine.
Interface changes:
* replaced 'u8 truncated' in the PERF_RECORD_AUX with a 'u64 flags', dropped redundant id/stream_id,
* in overwrite mode, always provide offset and size even if the driver cannot tell where the snapshot begins/weather its beginning was overwritten by older data.
This patchset adds support for Intel Processor Trace (PT) extension [1] of Intel Architecture that allows the capture of information about software execution flow, to the perf kernel infrastructure.
The single most notable thing is that while PT outputs trace data in a compressed binary format, it will still generate hundreds of megabytes of trace data per second per core. Decoding this binary stream takes 2-3 orders of magnitude the cpu time that it takes to generate it. These considerations make it impossible to carry out decoding in kernel space. Therefore, the trace data is exported to userspace as a zero-copy mapping that userspace can collect and store for later decoding. To address this, this patchset extends perf ring buffer with an "AUX space", which is allocated for hardware blocks such as PT to export their trace data with minimal overhead. This space can be configured via buffer's user page and mmapped from the same file descriptor with a given offset. Data can then be collected from it by reading the aux_head (write) pointer from the user page and updating aux_tail (read) pointer similarly to data_{head,tail} of the traditional perf buffer. There is an api between perf core and pmu drivers that wish to make use of this AUX space to export their data.
For tracing blocks that don't support hardware scatter-gather tables, we provide high-order physically contiguous allocations to minimize the overhead needed for software double buffering and PMI pressure.
This way we get a normal perf data stream that provides sideband information that is required to decode the trace data, such as MMAPs, COMMs etc, plus the actual trace in its own logical space.
If the trace buffer is mapped writable, the driver will stop tracing when it fills up (aux_head approaches aux_tail), till data is read, aux_tail pointer is moved forward and an ioctl() is issued to re-enable tracing. If the trace buffer is mapped read only, the tracing will continue, overwriting older data, so that the buffer always contains the most recent data. Tracing can be stopped with an ioctl() and restarted once the data is collected.
Another use case is annotating samples of other perf events: setting PERF_SAMPLE_AUX requests attr.aux_sample_size bytes of trace to be included in each event's sample.
This patchset consists of necessary changes to the perf kernel infrastructure, and PT and BTS pmu drivers. The tooling support is not included in this series, however, it can be found in my github tree [2].
[1] http://software.intel.com/en-us/intel-isa-extensions
[2] http://github.com/virtuoso/linux-perf/tree/intel_pt
Alexander Shishkin (19):
perf: Add data_{offset,size} to user_page
perf: Support high-order allocations for AUX space
perf: Add a capability for AUX_NO_SG pmus to do software double
buffering
perf: Add a pmu capability for "exclusive" events
perf: Add AUX record
perf: Add api for pmus to write to AUX area
perf: Support overwrite mode for AUX area
perf: Add wakeup watermark control to AUX area
x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
x86: perf: Intel PT and LBR/BTS are mutually exclusive
x86: perf: intel_pt: Intel PT PMU driver
x86: perf: intel_bts: Add BTS PMU driver
perf: add ITRACE_START record to indicate that tracing has started
perf: Add api to (de-)allocate AUX buffers for kernel counters
perf: Add a helper for looking up pmus by type
perf: Add infrastructure for using AUX data in perf samples
perf: Allocate ring buffers for inherited per-task kernel events
perf: Allow AUX sampling for multiple events
perf: Allow AUX sampling of inherited events
Peter Zijlstra (1):
perf: Add AUX area to ring buffer for raw data streams
arch/x86/include/asm/cpufeature.h | 1 + arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 129 ++++ arch/x86/kernel/cpu/perf_event.h | 14 + arch/x86/kernel/cpu/perf_event_intel.c | 14 +- arch/x86/kernel/cpu/perf_event_intel_bts.c | 518 +++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 11 +- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 9 +- arch/x86/kernel/cpu/perf_event_intel_pt.c | 995 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/scattered.c | 1 + include/linux/perf_event.h | 56 +- include/uapi/linux/perf_event.h | 73 ++- kernel/events/core.c | 534 +++++++++++++++- kernel/events/internal.h | 52 ++ kernel/events/ring_buffer.c | 386 ++++++++++- 16 files changed, 2768 insertions(+), 44 deletions(-) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 1/20 perf: Add data_{offset,size} to user_page
Currently, the actual perf ring buffer is one page into the mmap area, following the user page and the userspace follows this convention. This patch adds data_{offset,size} fields to user_page that can be used by userspace instead for locating perf data in the mmap area. This is also helpful when mapping existing or shared buffers if their size is not known in advance.
Right now, it is made to follow the existing convention that
data_offset == PAGE_SIZE and data_offset + data_size == mmap_size.
--- include/uapi/linux/perf_event.h | 5 +++++ kernel/events/core.c | 2 ++ 2 files changed, 7 insertions(+)
include/uapi/linux/perf_event.h +5 -0
@@ -489,9 +489,14 @@ struct perf_event_mmap_page {* In this case the kernel will not over-write unread data. * * See perf_output_put_handle() for the data ordering. +* +* data_{offset,size} indicate the location and size of the perf record +* buffer within the mmapped area. */ __u64 data_head;/* head in the data section */ __u64 data_tail;/* user-space written tail */ +__u64 data_offset;/* where the buffer starts */ +__u64 data_size;/* data buffer size */ };#define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)
kernel/events/core.c +2 -0
@@ -3917,6 +3917,8 @@ static void perf_event_init_userpage(struct perf_event *event)/* Allow new userspace to detect that bit 0 is deprecated */ userpg->cap_bit0_is_deprecated =1 ; userpg->size = offsetof(struct perf_event_mmap_page , __reserved); + userpg->data_offset =PAGE_SIZE ; + userpg->data_size =perf_data_size (rb); unlock:rcu_read_unlock ();
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 2/20 perf: Add AUX area to ring buffer for raw data streams
In reply to Alexander ShishkinFrom: Peter Zijlstra <[email protected]>
This patch introduces "AUX space" in the perf mmap buffer, intended for exporting high bandwidth data streams to userspace, such as instruction flow traces.
AUX space is a ring buffer, defined by aux_{offset,size} fields in the user_page structure, and read/write pointers aux_{head,tail}, which abide by the same rules as data_* counterparts of the main perf buffer.
In order to allocate/mmap AUX, userspace needs to set up aux_offset to such an offset that will be greater than data_offset+data_size and aux_size to be the desired buffer size. Both need to be page aligned. Then, same aux_offset and aux_size should be passed to mmap() call and if everything adds up, you should have an AUX buffer as a result.
Pages that are mapped into this buffer also come out of user's mlock rlimit plus perf_event_mlock_kb allowance.
--- include/linux/perf_event.h | 17 +++++ include/uapi/linux/perf_event.h | 16 +++++ kernel/events/core.c | 140 +++++++++++++++++++++++++++++++++------- kernel/events/internal.h | 23 +++++++ kernel/events/ring_buffer.c | 97 ++++++++++++++++++++++++++-- 5 files changed, 264 insertions(+), 29 deletions(-)
include/linux/perf_event.h +17 -0
@@ -263,6 +263,18 @@ struct pmu {* flush branch stack on context-switches (needed in cpu-wide mode) */ void (*flush_branch_stack) (void ); + +/* +* Set up pmu-private data structures for an AUX area +*/ +void *(*setup_aux) (int cpu,void **pages, +int nr_pages,bool overwrite); +/* optional */ + +/* +* Free pmu-private AUX data structures +*/ +void (*free_aux) (void *aux);/* optional */ };/**
@@ -782,6 +794,11 @@ static inline bool has_branch_stack(struct perf_event *event)return event->attr.sample_type &PERF_SAMPLE_BRANCH_STACK ; } +static inline bool has_aux (struct perf_event *event) +{ +return event->pmu->setup_aux; +} +extern int perf_output_begin (struct perf_output_handle *handle,struct perf_event *event,unsigned int size);extern void perf_output_end (struct perf_output_handle *handle);
include/uapi/linux/perf_event.h +16 -0
@@ -497,6 +497,22 @@ struct perf_event_mmap_page {__u64 data_tail;/* user-space written tail */ __u64 data_offset;/* where the buffer starts */ __u64 data_size;/* data buffer size */ + +/* +* AUX area is defined by aux_{offset,size} fields that should be set +* by the userspace, so that +* +* aux_offset >= data_offset + data_size +* +* prior to mmap()ing it. Size of the mmap()ed area should be aux_size. +* +* Ring buffer pointers aux_{head,tail} have the same semantics as +* data_{head,tail} and same ordering rules apply. +*/ +__u64 aux_head; +__u64 aux_tail; +__u64 aux_offset; +__u64 aux_size; };#define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)
kernel/events/core.c +115 -25
214 lines not on this page · raw message
kernel/events/internal.h +23 -0
@@ -35,6 +35,16 @@ struct ring_buffer {unsigned long mmap_locked;struct user_struct *mmap_user; +/* AUX area */ +unsigned long aux_pgoff; +int aux_nr_pages; +atomic_t aux_mmap_count; +unsigned long aux_mmap_locked; +void (*free_aux)(void *); +struct kref aux_refcount; +void **aux_pages; +void *aux_priv; +struct perf_event_mmap_page *user_page;void *data_pages[0 ]; };
@@ -43,6 +53,14 @@ extern void rb_free(struct ring_buffer *rb);extern struct ring_buffer *rb_alloc (int nr_pages,long watermark,int cpu,int flags);extern void perf_event_wakeup (struct perf_event *event); +extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event, +pgoff_t pgoff,int nr_pages,int flags); +extern void rb_free_aux (struct ring_buffer *rb); + +static inline bool rb_has_aux (struct ring_buffer *rb) +{ +return !!rb->aux_nr_pages; +} extern void perf_event_header__init_id(struct perf_event_header *header,
@@ -81,6 +99,11 @@ static inline unsigned long perf_data_size(struct ring_buffer *rb)return rb->nr_pages << (PAGE_SHIFT +page_order (rb)); } +static inline unsigned long perf_aux_size (struct ring_buffer *rb) +{ +return rb->aux_nr_pages <<PAGE_SHIFT ; +} +#define DEFINE_OUTPUT_COPY (func_name, memcpy_func) \ static inline unsigned long \ func_name(struct perf_output_handle *handle, \
kernel/events/ring_buffer.c +93 -4
118 lines not on this page · raw message
Signature · 1 line
2.1.0
Peter Zijlstra
maintainerRe: [PATCH v5 02/20] perf: Add AUX area to ring buffer for raw data streams
kernel/events/internal.h struct ring_buffer { · line 44
Quoted 1 line · Alexander Shishkin
kernel/events/internal.h +1
-0+struct kref aux_refcount;
I'm not a fan of kref, pointless obfuscation that.
Re: [PATCH v5 02/20] perf: Add AUX area to ring buffer for raw data streams
Quoted 4 lines · Peter Zijlstra
kernel/events/internal.h struct ring_buffer { · line 44
Quoted 1 line · Alexander Shishkin
kernel/events/internal.h +1
-0+struct kref aux_refcount;I'm not a fan of kref, pointless obfuscation that.
It has a good potential for debugging though. Sure right now the get/put simple APIs only performs counting sanity checks but I've seen patches that extend it to object debugging.
Sounds quite valuable on complicated object lifecycles like perf events.
Alexander Shishkin
reviewerPATCH v5 3/20 perf: Support high-order allocations for AUX space
In reply to Alexander ShishkinSome pmus (such as BTS or Intel PT without multiple-entry ToPA capability) don't support scatter-gather and will prefer larger contiguous areas for their output regions.
This patch adds a new pmu capability to request higher order allocations.
--- include/linux/perf_event.h | 1 + kernel/events/ring_buffer.c | 51 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 46 insertions(+), 6 deletions(-)
include/linux/perf_event.h +1 -0
@@ -171,6 +171,7 @@ struct perf_event;* pmu::capabilities flags */ #define PERF_PMU_CAP_NO_INTERRUPT 0x01 +#define PERF_PMU_CAP_AUX_NO_SG 0x02/** * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +45 -6
82 lines not on this page · raw message
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 4/20 perf: Add a capability for AUX_NO_SG pmus to do software double buffering
In reply to Alexander ShishkinFor pmus that don't support scatter-gather for AUX data in hardware, it might still make sense to implement software double buffering to avoid losing data while the user is reading data out. For this purpose, add a pmu capability that guarantees multiple high-order chunks for AUX buffer, so that the pmu driver can do switchover tricks.
To make use of this feature, add PERF_PMU_CAP_AUX_SW_DOUBLEBUF to your pmu's capability mask. This will make the ring buffer AUX allocation code ensure that the biggest high order allocation for the aux buffer pages is no bigger than half of the total requested buffer size, thus making sure that the buffer has at least two high order allocations.
--- include/linux/perf_event.h | 1 + kernel/events/ring_buffer.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
@@ -172,6 +172,7 @@ struct perf_event;*/ #define PERF_PMU_CAP_NO_INTERRUPT 0x01#define PERF_PMU_CAP_AUX_NO_SG 0x02 +#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF 0x04/** * struct pmu - generic performance monitoring unit
kernel/events/ring_buffer.c +14 -1
@@ -286,9 +286,22 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,if (!has_aux (event))return -ENOTSUPP ; -if (event->pmu->capabilities &PERF_PMU_CAP_AUX_NO_SG ) +if (event->pmu->capabilities &PERF_PMU_CAP_AUX_NO_SG ) { order =get_order (nr_pages *PAGE_SIZE ); +/* +* PMU requests more than one contiguous chunks of memory +* for SW double buffering +*/ +if ((event->pmu->capabilities &PERF_PMU_CAP_AUX_SW_DOUBLEBUF ) && + !overwrite) { +if (!order) +return -EINVAL ; + + order--; + } + } + rb->aux_pages =kzalloc_node (nr_pages *sizeof (void *),GFP_KERNEL , node);if (!rb->aux_pages)return -ENOMEM ;
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 5/20 perf: Add a pmu capability for "exclusive" events
In reply to Alexander ShishkinUsually, pmus that do, for example, instruction tracing, would only ever be able to have one event per task per cpu (or per perf_context). For such pmus it makes sense to disallow creating conflicting events early on, so as to provide consistent behavior for the user.
This patch adds a pmu capability that indicates such constraint on event creation.
--- include/linux/perf_event.h | 1 + kernel/events/core.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
include/linux/perf_event.h +1 -0
@@ -173,6 +173,7 @@ struct perf_event;#define PERF_PMU_CAP_NO_INTERRUPT 0x01#define PERF_PMU_CAP_AUX_NO_SG 0x02#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF 0x04 +#define PERF_PMU_CAP_EXCLUSIVE 0x08/** * struct pmu - generic performance monitoring unit
kernel/events/core.c +45 -0
69 lines not on this page · raw message
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 6/20 perf: Add AUX record
In reply to Alexander ShishkinWhen there's new data in the AUX space, output a record indicating its offset and size and a set of flags, such as PERF_AUX_FLAG_TRUNCATED, to mean the described data was truncated to fit in the ring buffer.
--- include/uapi/linux/perf_event.h | 19 +++++++++++++++++++ kernel/events/core.c | 34 ++++++++++++++++++++++++++++++++++ kernel/events/internal.h | 3 +++ 3 files changed, 56 insertions(+)
include/uapi/linux/perf_event.h +19 -0
@@ -733,6 +733,20 @@ enum perf_event_type {*/ PERF_RECORD_MMAP2 =10 , +/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* u64 flags; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_AUX =11 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -750,6 +764,11 @@ enum perf_callchain_context { PERF_CONTEXT_MAX = (__u64)-4095, }; +/** +* PERF_RECORD_AUX::flags bits +*/ +#define PERF_AUX_FLAG_TRUNCATED 0x01/* record was truncated to fit */ +#define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)#define PERF_FLAG_PID_CGROUP (1UL << 2)/* pid=cgroup id, per-cpu mode only */
kernel/events/core.c +34 -0
@@ -5581,6 +5581,40 @@ void perf_event_mmap(struct vm_area_struct *vma)perf_event_mmap_event (&mmap_event); } +void perf_event_aux_event (struct perf_event *event,unsigned long head, +unsigned long size,u64 flags) +{ +struct perf_output_handle handle; +struct perf_sample_data sample; +struct perf_aux_event { +struct perf_event_header header; +u64 offset; +u64 size; +u64 flags; + } rec = { + .header = { + .type =PERF_RECORD_AUX , + .misc =0 , + .size =sizeof (rec), + }, + .offset = head, + .size = size, + .flags = flags, + }; +int ret; + +perf_event_header__init_id (&rec.header, &sample, event); + ret =perf_output_begin (&handle, event, rec.header.size); + +if (ret) +return ; + +perf_output_put (&handle, rec); +perf_event__output_id_sample (event, &handle, &sample); + +perf_output_end (&handle); +} +/* * IRQ throttle logging */
kernel/events/internal.h +3 -0
@@ -62,6 +62,9 @@ static inline bool rb_has_aux(struct ring_buffer *rb)return !!rb->aux_nr_pages; } +void perf_event_aux_event (struct perf_event *event,unsigned long head, +unsigned long size,u64 flags); + extern void perf_event_header__init_id(struct perf_event_header *header, struct perf_sample_data *data,
Signature · 1 line
2.1.0
Peter Zijlstra
maintainerRe: [PATCH v5 06/20] perf: Add AUX record
include/uapi/linux/perf_event.h enum perf_event_type { · line 748
Quoted 13 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +13
-0+/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* u64 flags; +* struct sample_id sample_id; +* }; +*/ + PERF_RECORD_AUX = 11,
Given the discussion with the ARM people the last time, do we want to add the possibility of a variable data field in this event? Its easy to add now, harder to do later (although not impossible).
Also added Mathieu Poirier on CC, he asked to be included in your next postings.
Alexander Shishkin
reviewerRe: [PATCH v5 06/20] perf: Add AUX record
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +9
-0+/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* u64 flags;
Quoted 8 lines
include/uapi/linux/perf_event.h enum perf_event_type { · line 748
Quoted 4 lines
include/uapi/linux/perf_event.h +4
-0+* struct sample_id sample_id; +* }; +*/ + PERF_RECORD_AUX = 11,Given the discussion with the ARM people the last time, do we want to add the possibility of a variable data field in this event? Its easy to add now, harder to do later (although not impossible).
Iirc, what they want is to save a once-per-session chunk of data, which would be better synthesized by perf record than sent from a pmu driver?
We do something like that with PT right now, perf record looks at pmu's sysfs attributes and stores them in some synthesized record.
Quoted 2 lines
Also added Mathieu Poirier on CC, he asked to be included in your next postings.
Indeed, my apologies.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 06/20] perf: Add AUX record
Quoted 21 lines · Alexander Shishkin
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
include/uapi/linux/perf_event.h +9
-0+/* +* Records that new data landed in the AUX buffer part. +* +* struct { +* struct perf_event_header header; +* +* u64 aux_offset; +* u64 aux_size; +* u64 flags; Quoted 8 lines
include/uapi/linux/perf_event.h enum perf_event_type { · line 748
Quoted 4 lines
include/uapi/linux/perf_event.h +4
-0+* struct sample_id sample_id; +* }; +*/ + PERF_RECORD_AUX = 11,Given the discussion with the ARM people the last time, do we want to add the possibility of a variable data field in this event? Its easy to add now, harder to do later (although not impossible).
Quoted 5 lines
Iirc, what they want is to save a once-per-session chunk of data, which would be better synthesized by perf record than sent from a pmu driver?
We do something like that with PT right now, perf record looks at pmu's sysfs attributes and stores them in some synthesized record.
Right, I just got to that. So I wasn't entirely sure if perf_event_open time conditions where sufficient for them, if they are then yes that could work.
Alexander Shishkin
reviewerPATCH v5 7/20 perf: Add api for pmus to write to AUX area
In reply to Alexander ShishkinFor pmus that wish to write data to ring buffer's AUX area, provide perf_aux_output_{begin,end}() calls to initiate/commit data writes, similarly to perf_output_{begin,end}. These also use the same output handle structure. Also, similarly to software counterparts, these will direct inherited events' output to parents' ring buffers.
After the perf_aux_output_begin() returns successfully, handle->size is set to the maximum amount of data that can be written wrt aux_tail pointer, so that no data that the user hasn't seen will be overwritten, therefore this should always be called before hardware writing is enabled. On success, this will return the pointer to pmu driver's private structure allocated for this aux area by pmu::setup_aux. Same pointer can also be retrieved using perf_get_aux() while hardware writing is enabled.
PMU driver should pass the actual amount of data written as a parameter to perf_aux_output_end(). All hardware writes should be completed and visible before this one is called.
Additionally, perf_aux_output_skip() will adjust output handle and aux_head in case some part of the buffer has to be skipped over to maintain hardware's alignment constraints.
Nested writers are forbidden and guards are in place to catch such attempts.
--- include/linux/perf_event.h | 23 +++++++- kernel/events/core.c | 5 +- kernel/events/internal.h | 4 ++ kernel/events/ring_buffer.c | 139 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 167 insertions(+), 4 deletions(-)
include/linux/perf_event.h +22 -1
@@ -555,12 +555,22 @@ struct perf_output_handle {struct ring_buffer *rb;unsigned long wakeup;unsigned long size; -void *addr; +union { +void *addr; +unsigned long head; + };int page; };#ifdef CONFIG_PERF_EVENTS +extern void *perf_aux_output_begin (struct perf_output_handle *handle, +struct perf_event *event); +extern void perf_aux_output_end (struct perf_output_handle *handle, +unsigned long size,bool truncated); +extern int perf_aux_output_skip (struct perf_output_handle *handle, +unsigned long size); +extern void *perf_get_aux (struct perf_output_handle *handle);extern int perf_pmu_register (struct pmu *pmu,const char *name,int type);extern void perf_pmu_unregister (struct pmu *pmu);
@@ -817,6 +827,17 @@ extern void perf_event_disable(struct perf_event *event);extern int __perf_event_disable (void *info);extern void perf_event_task_tick (void );#else /* !CONFIG_PERF_EVENTS: */ +static inline void * +perf_aux_output_begin (struct perf_output_handle *handle, +struct perf_event *event) {return NULL ; } +static inline void +perf_aux_output_end (struct perf_output_handle *handle,unsigned long size, +bool truncated) { } +static inline int +perf_aux_output_skip (struct perf_output_handle *handle, +unsigned long size) {return -EINVAL ; } +static inline void * +perf_get_aux (struct perf_output_handle *handle) {return NULL ; }static inline void perf_event_task_sched_in (struct task_struct *prev,struct task_struct *task) { }
kernel/events/core.c +2 -3
@@ -3268,7 +3268,6 @@ static void free_event_rcu(struct rcu_head *head)kfree (event); } -static void ring_buffer_put (struct ring_buffer *rb);static void ring_buffer_attach (struct perf_event *event,struct ring_buffer *rb);
@@ -4085,7 +4084,7 @@ static void rb_free_rcu(struct rcu_head *rcu_head)rb_free (rb); } -static struct ring_buffer *ring_buffer_get (struct perf_event *event) +struct ring_buffer *ring_buffer_get (struct perf_event *event) {struct ring_buffer *rb;
@@ -4100,7 +4099,7 @@ static struct ring_buffer *ring_buffer_get(struct perf_event *event)return rb; } -static void ring_buffer_put (struct ring_buffer *rb) +void ring_buffer_put (struct ring_buffer *rb) {if (!atomic_dec_and_test (&rb->refcount))return ;
kernel/events/internal.h +4 -0
@@ -36,6 +36,8 @@ struct ring_buffer {struct user_struct *mmap_user;/* AUX area */ +local_t aux_head; +local_t aux_nest;unsigned long aux_pgoff;int aux_nr_pages;atomic_t aux_mmap_count;
@@ -56,6 +58,8 @@ extern void perf_event_wakeup(struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event,pgoff_t pgoff,int nr_pages,int flags);extern void rb_free_aux (struct ring_buffer *rb); +extern struct ring_buffer *ring_buffer_get (struct perf_event *event); +extern void ring_buffer_put (struct ring_buffer *rb);static inline bool rb_has_aux (struct ring_buffer *rb) {
kernel/events/ring_buffer.c +139 -0
145 lines not on this page · raw message
Signature · 1 line
2.1.0
Peter Zijlstra
maintainerRe: [PATCH v5 07/20] perf: Add api for pmus to write to AUX area
kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 282
Quoted 6 lines · Alexander Shishkin
kernel/events/ring_buffer.c +6
-0+/* +* Nesting is not supported for AUX area, make sure nested +* writers are caught early +*/ +if (WARN_ON_ONCE (local_xchg (&rb->aux_nest,1 ))) + goto err_put;
Note that printk() from NMI is prone to making your machine die, not that I have a better option though.
Alexander Shishkin
reviewerRe: [PATCH v5 07/20] perf: Add api for pmus to write to AUX area
Quoted 10 lines · Peter Zijlstra
kernel/events/ring_buffer.c ring_buffer_init(struct ring_buffer *rb, long watermark, int flags) · line 282
Quoted 6 lines · Alexander Shishkin
kernel/events/ring_buffer.c +6
-0+/* +* Nesting is not supported for AUX area, make sure nested +* writers are caught early +*/ +if (WARN_ON_ONCE (local_xchg (&rb->aux_nest,1 ))) + goto err_put;Note that
printk()from NMI is prone to making your machine die, not that I have a better option though.
Indeed, but if your driver is really attempting nested perf_aux_output_begin()s, then you had it coming to you anyway.
Regards,
--
Alex
Alexander Shishkin
reviewerPATCH v5 8/20 perf: Support overwrite mode for AUX area
In reply to Alexander ShishkinThis adds support for overwrite mode in the AUX area, which means "keep collecting data till you're stopped", turning AUX area into a circular buffer, where new data overwrites old data. It does not depend on data buffer's overwrite mode, so that it doesn't lose sideband data that is instrumental for processing AUX data.
Overwrite mode is enabled at mapping AUX area read only. Even though aux_tail in the buffer's user page might be user writable, it will be ignored in this mode.
A PERF_RECORD_AUX with PERF_AUX_FLAG_OVERWRITE set is written to the perf data stream every time an event writes new data to the AUX area. The pmu driver might not be able to infer the exact beginning of the new data in each snapshot, some drivers will only provide the tail, which is aux_offset + aux_size in the AUX record. Consumer has to be able to tell the new data from the old one, for example, by means of time stamps if such are provided in the trace.
Consumer is also responsible for disabling any events that might write to the AUX area (thus potentially racing with the consumer) before collecting the data.
--- include/uapi/linux/perf_event.h | 1 + kernel/events/internal.h | 1 + kernel/events/ring_buffer.c | 48 ++++++++++++++++++++++++++++------------- 3 files changed, 35 insertions(+), 15 deletions(-)
include/uapi/linux/perf_event.h +1 -0
@@ -768,6 +768,7 @@ enum perf_callchain_context {* PERF_RECORD_AUX::flags bits */ #define PERF_AUX_FLAG_TRUNCATED 0x01/* record was truncated to fit */ +#define PERF_AUX_FLAG_OVERWRITE 0x02/* snapshot from overwrite mode */ #define PERF_FLAG_FD_NO_GROUP (1UL << 0)#define PERF_FLAG_FD_OUTPUT (1UL << 1)
kernel/events/internal.h +1 -0
@@ -40,6 +40,7 @@ struct ring_buffer {local_t aux_nest;unsigned long aux_pgoff;int aux_nr_pages; +int aux_overwrite;atomic_t aux_mmap_count;unsigned long aux_mmap_locked;void (*free_aux)(void *);
kernel/events/ring_buffer.c +33 -15
78 lines not on this page · raw message
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 9/20 perf: Add wakeup watermark control to AUX area
In reply to Alexander ShishkinWhen AUX area gets a certain amount of new data, we want to wake up userspace to collect it. This adds a new control to specify how much data will cause a wakeup. This is then passed down to pmu drivers via output handle's "wakeup" field, so that the driver can find the nearest point where it can generate an interrupt.
We repurpose __reserved_2 in the event attribute for this, even though it was never checked to be zero before, aux_watermark will only matter for new AUX-aware code, so the old code should still be fine.
--- include/uapi/linux/perf_event.h | 7 +++++-- kernel/events/core.c | 3 ++- kernel/events/internal.h | 4 +++- kernel/events/ring_buffer.c | 22 +++++++++++++++++++--- 4 files changed, 29 insertions(+), 7 deletions(-)
include/uapi/linux/perf_event.h +5 -2
@@ -238,6 +238,7 @@ enum perf_event_read_format {#define PERF_ATTR_SIZE_VER2 80/* add: branch_sample_type */ #define PERF_ATTR_SIZE_VER3 96/* add: sample_regs_user */ /* add: sample_stack_user */ +/* add: aux_watermark */ /* * Hardware event_id to monitor via a performance monitoring event:
@@ -332,8 +333,10 @@ struct perf_event_attr {*/ __u32 sample_stack_user; -/* Align to u64. */ -__u32 __reserved_2; +/* +* Wakeup watermark for AUX area +*/ +__u32 aux_watermark; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
kernel/events/core.c +2 -1
@@ -4394,7 +4394,8 @@ accounting:perf_event_init_userpage (event);perf_event_update_userpage (event); }else { - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, flags); + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags);if (ret)atomic_dec (&rb->mmap_count);else
kernel/events/internal.h +3 -1
@@ -27,6 +27,7 @@ struct ring_buffer {local_t lost;/* nr records lost */ long watermark;/* wakeup watermark */ +long aux_watermark;/* poll crap */ spinlock_t event_lock;struct list_head event_list;
@@ -38,6 +39,7 @@ struct ring_buffer {/* AUX area */ local_t aux_head;local_t aux_nest; +local_t aux_wakeup;unsigned long aux_pgoff;int aux_nr_pages;int aux_overwrite;
@@ -57,7 +59,7 @@ extern struct ring_buffer * rb_alloc(int nr_pages, long watermark, int cpu, int flags);extern void perf_event_wakeup (struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event, -pgoff_t pgoff,int nr_pages,int flags); +pgoff_t pgoff,int nr_pages,long watermark,int flags);extern void rb_free_aux (struct ring_buffer *rb);extern struct ring_buffer *ring_buffer_get (struct perf_event *event);extern void ring_buffer_put (struct ring_buffer *rb);
kernel/events/ring_buffer.c +19 -3
@@ -295,6 +295,7 @@ void *perf_aux_output_begin(struct perf_output_handle *handle,*/ if (!rb->aux_overwrite) { aux_tail =ACCESS_ONCE (rb->user_page->aux_tail); + handle->wakeup =local_read (&rb->aux_wakeup) + rb->aux_watermark;if (aux_head - aux_tail <perf_aux_size (rb)) handle->size =CIRC_SPACE (aux_head, aux_tail,perf_aux_size (rb));
@@ -358,9 +359,12 @@ void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,perf_event_aux_event (handle->event, aux_head, size, flags); } - rb->user_page->aux_head =local_read (&rb->aux_head); + aux_head = rb->user_page->aux_head =local_read (&rb->aux_head); -perf_output_wakeup (handle); +if (aux_head -local_read (&rb->aux_wakeup) >= rb->aux_watermark) { +perf_output_wakeup (handle); +local_add (rb->aux_watermark, &rb->aux_wakeup); + } handle->event =NULL ;local_set (&rb->aux_nest,0 );
@@ -382,6 +386,14 @@ int perf_aux_output_skip(struct perf_output_handle *handle, unsigned long size)local_add (size, &rb->aux_head); + aux_head = rb->user_page->aux_head =local_read (&rb->aux_head); +if (aux_head -local_read (&rb->aux_wakeup) >= rb->aux_watermark) { +perf_output_wakeup (handle); +local_add (rb->aux_watermark, &rb->aux_wakeup); + handle->wakeup =local_read (&rb->aux_wakeup) + + rb->aux_watermark; + } + handle->head = aux_head; handle->size -= size;
@@ -432,7 +444,7 @@ static void rb_free_aux_page(struct ring_buffer *rb, int idx) }int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event, -pgoff_t pgoff,int nr_pages,int flags) +pgoff_t pgoff,int nr_pages,long watermark,int flags) {bool overwrite = !(flags &RING_BUFFER_WRITABLE );int node = (event->cpu ==-1 ) ?-1 :cpu_to_node (event->cpu);
@@ -491,6 +503,10 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event,kref_init (&rb->aux_refcount); rb->aux_overwrite = overwrite; + rb->aux_watermark = watermark; + +if (!rb->aux_watermark && !rb->aux_overwrite) + rb->aux_watermark = nr_pages << (PAGE_SHIFT -1 ); out:if (!ret)
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 10/20 x86: Add Intel Processor Trace (INTEL_PT) cpu feature detection
In reply to Alexander ShishkinIntel Processor Trace is an architecture extension that allows for program flow tracing.
--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
@@ -185,6 +185,7 @@#define X86_FEATURE_DTHERM ( 7*32+ 7)/* Digital Thermal Sensor */ #define X86_FEATURE_HW_PSTATE ( 7*32+ 8)/* AMD HW-PState */ #define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9)/* AMD ProcFeedbackInterface */ +#define X86_FEATURE_INTEL_PT ( 7*32+10)/* Intel Processor Trace */ /* Virtualization flags: Linux defined, word 8 */ #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0)/* Intel TPR Shadow */
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -36,6 +36,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c) { X86_FEATURE_ARAT, CR_EAX, 2, 0x00000006, 0 }, { X86_FEATURE_PLN, CR_EAX, 4, 0x00000006, 0 }, { X86_FEATURE_PTS, CR_EAX, 6, 0x00000006, 0 }, + { X86_FEATURE_INTEL_PT, CR_EBX,25, 0x00000007, 0 }, { X86_FEATURE_APERFMPERF, CR_ECX, 0, 0x00000006, 0 }, { X86_FEATURE_EPB, CR_ECX, 3, 0x00000006, 0 }, { X86_FEATURE_HW_PSTATE, CR_EDX, 7, 0x80000007, 0 },
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 11/20 x86: perf: Intel PT and LBR/BTS are mutually exclusive
In reply to Alexander ShishkinIntel PT cannot be used at the same time as LBR or BTS and will cause a
general protection fault if they are used together. In order to avoid
fixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.
--- arch/x86/kernel/cpu/perf_event.h | 6 ++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 8 +++++++- arch/x86/kernel/cpu/perf_event_intel_lbr.c | 9 +++++---- 3 files changed, 18 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/perf_event.h +6 -0
@@ -148,6 +148,7 @@ struct cpu_hw_events {* Intel DebugStore bits */ struct debug_store *ds; +unsigned int bts_enabled;u64 pebs_enabled;/*
@@ -161,6 +162,11 @@ struct cpu_hw_events {u64 br_sel;/* +* Intel Processor Trace +*/ +unsigned int pt_enabled; + +/* * Intel host/guest exclude bits */ u64 intel_ctrl_guest_mask;
arch/x86/kernel/cpu/perf_event_intel_ds.c +7 -1
@@ -455,8 +455,13 @@ struct event_constraint bts_constraint =void intel_pmu_enable_bts (u64 config) { +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);unsigned long debugctlmsr; +if (cpuc->pt_enabled) +return ; + + cpuc->bts_enabled =1 ; debugctlmsr =get_debugctlmsr (); debugctlmsr |=DEBUGCTLMSR_TR ;
@@ -477,9 +482,10 @@ void intel_pmu_disable_bts(void)struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events);unsigned long debugctlmsr; -if (!cpuc->ds) +if (!cpuc->ds || cpuc->pt_enabled)return ; + cpuc->bts_enabled =0 ; debugctlmsr =get_debugctlmsr (); debugctlmsr &=
arch/x86/kernel/cpu/perf_event_intel_lbr.c +5 -4
@@ -172,7 +172,9 @@ static void intel_pmu_lbr_reset_64(void)void intel_pmu_lbr_reset (void ) { -if (!x86_pmu.lbr_nr) +struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); + +if (!x86_pmu.lbr_nr || cpuc->pt_enabled)return ;if (x86_pmu.intel_cap.lbr_format ==LBR_FORMAT_32 )
@@ -185,7 +187,7 @@ void intel_pmu_lbr_enable(struct perf_event *event) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); -if (!x86_pmu.lbr_nr) +if (!x86_pmu.lbr_nr || cpuc->pt_enabled)return ;/*
@@ -205,11 +207,10 @@ void intel_pmu_lbr_disable(struct perf_event *event) {struct cpu_hw_events *cpuc = &__get_cpu_var (cpu_hw_events); -if (!x86_pmu.lbr_nr) +if (!x86_pmu.lbr_nr || !cpuc->lbr_users || cpuc->pt_enabled)return ; cpuc->lbr_users--; -WARN_ON_ONCE (cpuc->lbr_users <0 );if (cpuc->enabled && !cpuc->lbr_users) {__intel_pmu_lbr_disable ();
Signature · 1 line
2.1.0
Peter Zijlstra
maintainerRe: [PATCH v5 11/20] x86: perf: Intel PT and LBR/BTS are mutually exclusive
Quoted 5 lines · Alexander Shishkin
Intel PT cannot be used at the same time as LBR or BTS and will cause a
general protection fault if they are used together. In order to avoidfixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.
Yeah, don't like this. Like I've said many times before we should simply disallow creating PT events when there are LBR events and vice versa.
Alexander Shishkin
reviewerRe: [PATCH v5 11/20] x86: perf: Intel PT and LBR/BTS are mutually exclusive
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Intel PT cannot be used at the same time as LBR or BTS and will cause a
general protection fault if they are used together. In order to avoidfixing up GPs in the fast path, instead we use flags to indicate that that one of these is in use so that the other avoids MSR access altogether.
Yeah, don't like this. Like I've said many times before we should simply disallow creating PT events when there are LBR events and vice versa.
Ok, I must have misunderstood you before. This makes things a bit easier.
Regards,
--
Alex
Alexander Shishkin
reviewerPATCH v5 12/20 x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinAdd support for Intel Processor Trace (PT) to kernel's perf events. PT is an extension of Intel Architecture that collects information about software execuction such as control flow, execution modes and timings and formats it into highly compressed binary packets. Even being compressed, these packets are generated at hundreds of megabytes per second per core, which makes it impractical to decode them on the fly in the kernel.
This driver exports trace data by through AUX space in the perf ring buffer, which is zero-copy mapped into userspace for faster data retrieval.
--- arch/x86/include/uapi/asm/msr-index.h | 18 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/intel_pt.h | 129 ++++ arch/x86/kernel/cpu/perf_event.h | 2 + arch/x86/kernel/cpu/perf_event_intel.c | 8 + arch/x86/kernel/cpu/perf_event_intel_pt.c | 995 ++++++++++++++++++++++++++++++ 6 files changed, 1153 insertions(+) create mode 100644 arch/x86/kernel/cpu/intel_pt.h create mode 100644 arch/x86/kernel/cpu/perf_event_intel_pt.c
arch/x86/include/uapi/asm/msr-index.h +18 -0
@@ -74,6 +74,24 @@#define MSR_IA32_PERF_CAPABILITIES 0x00000345#define MSR_PEBS_LD_LAT_THRESHOLD 0x000003f6 +#define MSR_IA32_RTIT_CTL 0x00000570 +#define RTIT_CTL_TRACEEN BIT(0) +#define RTIT_CTL_OS BIT(2) +#define RTIT_CTL_USR BIT(3) +#define RTIT_CTL_CR3EN BIT(7) +#define RTIT_CTL_TOPA BIT(8) +#define RTIT_CTL_TSC_EN BIT(10) +#define RTIT_CTL_DISRETC BIT(11) +#define RTIT_CTL_BRANCH_EN BIT(13) +#define MSR_IA32_RTIT_STATUS 0x00000571 +#define RTIT_STATUS_CONTEXTEN BIT(1) +#define RTIT_STATUS_TRIGGEREN BIT(2) +#define RTIT_STATUS_ERROR BIT(4) +#define RTIT_STATUS_STOPPED BIT(5) +#define MSR_IA32_RTIT_CR3_MATCH 0x00000572 +#define MSR_IA32_RTIT_OUTPUT_BASE 0x00000560 +#define MSR_IA32_RTIT_OUTPUT_MASK 0x00000561 +#define MSR_MTRRfix64K_00000 0x00000250#define MSR_MTRRfix16K_80000 0x00000258#define MSR_MTRRfix16K_A0000 0x00000259
arch/x86/kernel/cpu/Makefile +1 -0
@@ -39,6 +39,7 @@ obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_lbr.o perf_event_intel_ds.o per obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore.o perf_event_intel_uncore_snb.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore_snbep.o perf_event_intel_uncore_nhmex.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_rapl.o +obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o endif
arch/x86/kernel/cpu/intel_pt.h new file +129 -0
129 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event.h +2 -0
@@ -750,6 +750,8 @@ void intel_pmu_lbr_init_snb(void);int intel_pmu_setup_lbr_filter (struct perf_event *event); +void intel_pt_interrupt (void ); +int p4_pmu_init (void );int p6_pmu_init (void );
arch/x86/kernel/cpu/perf_event_intel.c +8 -0
@@ -1528,6 +1528,14 @@ again: }/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } + +/* * Checkpointed counters can lead to 'spurious' PMIs because the * rollback caused by the PMI will have cleared the overflow status * bit. Therefore always force probe these counters.
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +995 -0
995 lines not on this page · raw message
Signature · 1 line
2.1.0
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 2 lines · Alexander Shishkin
+/**
+ * structpt_buffer- buffer configuration; one buffer pertask_structor
- * cpu, depending on perf event configuration
+ * cpu, depending on perf event configuration
Quoted 14 lines
+ * @tables: list of ToPA tables in this buffer
+ * @first, @last: shorthands for first and last topa tables
+ * @cur: current topa table
+ * @nr_pages: buffer size in pages
+ * @cur_idx: current output region's index within @cur table
+ * @output_off: offset within the current output region
+ * @data_size: running total of the amount of data in this buffer
+ * @lost: if data was lost/truncated
+ * @head: logical write offset inside the buffer
+ * @snapshot: if this is for a snapshot/overwrite counter
+ * @stop_pos, @intr_pos: STOP and INT topa entries in the buffer
+ * @data_pages: array of pages from perf
+ * @topa_index: table of topa entries indexed by page offset
+ */
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander Shishkinarch/x86/kernel/cpu/perf_event_intel.c again: · line 1537
Quoted 12 lines · Alexander Shishkin
+++
b/arch/x86/kernel/cpu/perf_event_intel.carch/x86/kernel/cpu/perf_event_intel.c +7
-0@@ -1528,6 +1528,14 @@ again: }/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } +
How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like FREEZE_ON_PMI etc?
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Alexander Shishkin
+++
b/arch/x86/kernel/cpu/perf_event_intel.c(quoted diff) +0
-0@@ -1528,6 +1528,14 @@ again: }
Quoted 11 lines
arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537
Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c +7
-0/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } +How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like
FREEZE_ON_PMIetc?
It ignores the FREEZE_ON_PMI bit. I stop it by hand inside the PMI handler, so you can see parts of the handler in the trace if you're tracing the kernel.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 19 lines · Alexander Shishkin
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Alexander Shishkin
+++
b/arch/x86/kernel/cpu/perf_event_intel.c(quoted diff) +0
-0@@ -1528,6 +1528,14 @@ again: }Quoted 11 lines
arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537
Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c +7
-0/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } +How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like
FREEZE_ON_PMIetc?
Quoted 3 lines
It ignores the
FREEZE_ON_PMIbit. I stop it by hand inside the PMI handler, so you can see parts of the handler in the trace if you're tracing the kernel.
Urgh, horrid that. Routing something to the same interrupt, sharing status registers but not observing the same semantics for the interrupt is a massive fail.
IIRC Andi was planning to start using FREEZE_ON_PMI to avoid the MSR writes in intel_pmu_{disable,enable}_all(), this interrupt not actually respecting that makes that non-trivial.
We already use FREEZE_ON_PMI for LBR, but for now PT and LBR are mutually exclusive so that's not a problem, if we ever get those working together this needs to get fixed.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Alexander Shishkin
Quoted 16 lines · Peter Zijlstra
arch/x86/kernel/cpu/perf_event_intel.c again: · line 1537
Quoted 12 lines · Alexander Shishkin
+++
b/arch/x86/kernel/cpu/perf_event_intel.carch/x86/kernel/cpu/perf_event_intel.c +7
-0@@ -1528,6 +1528,14 @@ again: }/* +* Intel PT +*/ +if (__test_and_clear_bit (55 , (unsigned long *)&status)) { + handled++; +intel_pt_interrupt (); + } +How does the PT interrupt interact with the regular PMI? In particular does it respect stuff like
FREEZE_ON_PMIetc?
Quoted 7 lines
Quoted 3 lines
It ignores the
FREEZE_ON_PMIbit. I stop it by hand inside the PMI handler, so you can see parts of the handler in the trace if you're tracing the kernel.Urgh, horrid that. Routing something to the same interrupt, sharing status registers but not observing the same semantics for the interrupt is a massive fail.
I can't pretend to understand the logic behind this either.
Quoted 7 lines
IIRC Andi was planning to start using
FREEZE_ON_PMIto avoid the MSR writes in intel_pmu_{disable,enable}_all(), this interrupt not actually respecting that makes that non-trivial.We already use
FREEZE_ON_PMIfor LBR, but for now PT and LBR are mutually exclusive so that's not a problem, if we ever get those working together this needs to get fixed.
Agreed.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 7 lines · Alexander Shishkin
+/*
+ * Capabilities of Intel PT hardware, such as number of address bits or
+ * supported output schemes, are cached and exported to userspace as "caps"
+ * attribute group of pt pmu device
+ * (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store
+ * relevant bits together withintel_pttraces.
+ */
Does it make sense to use that AUX record variable data thing for this instead -- much like what the ARM folks wanted?
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander Shishkinarch/x86/kernel/cpu/perf_event_intel_pt.c line 127
Quoted 9 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+if (test_cpu_cap (&boot_cpu_data,X86_FEATURE_INTEL_PT )) { +for (i =0 ; i <PT_CPUID_LEAVES ; i++) +cpuid_count (20 , i, + &pt_pmu.caps[CR_EAX + i *4 ], + &pt_pmu.caps[CR_EBX + i *4 ], + &pt_pmu.caps[CR_ECX + i *4 ], + &pt_pmu.caps[CR_EDX + i *4 ]); + }else +return -ENODEV ;
Please use {} for every multi-line stmt, even if its a single stmt, also use {} in both branches of a condition, even if only one strictly requires it.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 14 lines · Peter Zijlstra
arch/x86/kernel/cpu/perf_event_intel_pt.c line 127
Quoted 9 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+if (test_cpu_cap (&boot_cpu_data,X86_FEATURE_INTEL_PT )) { +for (i =0 ; i <PT_CPUID_LEAVES ; i++) +cpuid_count (20 , i, + &pt_pmu.caps[CR_EAX + i *4 ], + &pt_pmu.caps[CR_EBX + i *4 ], + &pt_pmu.caps[CR_ECX + i *4 ], + &pt_pmu.caps[CR_EDX + i *4 ]); + }else +return -ENODEV ;Please use {} for every multi-line stmt, even if its a single stmt, also use {} in both branches of a condition, even if only one strictly requires it.
Ok.
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 40 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +40
-0+ +enum cpuid_regs { +CR_EAX =0 , +CR_ECX , +CR_EDX , +CR_EBX +}; + +/* +* Capabilities of Intel PT hardware, such as number of address bits or +* supported output schemes, are cached and exported to userspace as "caps" +* attribute group of pt pmu device +* (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store +* relevant bits together with intel_pt traces. +*/ +#define PT_CAP (_n, _l, _r, _m) \ + [PT_CAP_ ## _n] = { .name = __stringify(_n), .leaf = _l, \ + .reg = _r, .mask = _m } + +static struct pt_cap_desc { +const char *name; +u32 leaf; +u8 reg; +u32 mask; +} pt_caps[] = { +PT_CAP (max_subleaf,0 ,CR_EAX ,0xffffffff ), +PT_CAP (cr3_filtering,0 ,CR_EBX ,BIT (0 )), +PT_CAP (topa_output,0 ,CR_ECX ,BIT (0 )), +PT_CAP (topa_multiple_entries,0 ,CR_ECX ,BIT (1 )), +PT_CAP (payloads_lip,0 ,CR_ECX ,BIT (31 )), +}; + +static u32 pt_cap_get (enum pt_capabilities cap) +{ +struct pt_cap_desc *cd = &pt_caps[cap]; +u32 c = pt_pmu.caps[cd->leaf *4 + cd->reg]; +unsigned int shift =__ffs (cd->mask); + +return (c & cd->mask) >> shift; +}
arch/x86/kernel/cpu/perf_event_intel_pt.c line 127
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+if (test_cpu_cap (&boot_cpu_data,X86_FEATURE_INTEL_PT )) { +for (i =0 ; i <PT_CPUID_LEAVES ; i++) +cpuid_count (20 , i, + &pt_pmu.caps[CR_EAX + i *4 ], + &pt_pmu.caps[CR_EBX + i *4 ], + &pt_pmu.caps[CR_ECX + i *4 ], + &pt_pmu.caps[CR_EDX + i *4 ]); + }else +return -ENODEV ;
I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 42 lines · Peter Zijlstra
Quoted 40 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +40
-0+ +enum cpuid_regs { +CR_EAX =0 , +CR_ECX , +CR_EDX , +CR_EBX +}; + +/* +* Capabilities of Intel PT hardware, such as number of address bits or +* supported output schemes, are cached and exported to userspace as "caps" +* attribute group of pt pmu device +* (/sys/bus/event_source/devices/intel_pt/caps/) so that userspace can store +* relevant bits together with intel_pt traces. +*/ +#define PT_CAP (_n, _l, _r, _m) \ + [PT_CAP_ ## _n] = { .name = __stringify(_n), .leaf = _l, \ + .reg = _r, .mask = _m } + +static struct pt_cap_desc { +const char *name; +u32 leaf; +u8 reg; +u32 mask; +} pt_caps[] = { +PT_CAP (max_subleaf,0 ,CR_EAX ,0xffffffff ), +PT_CAP (cr3_filtering,0 ,CR_EBX ,BIT (0 )), +PT_CAP (topa_output,0 ,CR_ECX ,BIT (0 )), +PT_CAP (topa_multiple_entries,0 ,CR_ECX ,BIT (1 )), +PT_CAP (payloads_lip,0 ,CR_ECX ,BIT (31 )), +}; + +static u32 pt_cap_get (enum pt_capabilities cap) +{ +struct pt_cap_desc *cd = &pt_caps[cap]; +u32 c = pt_pmu.caps[cd->leaf *4 + cd->reg]; +unsigned int shift =__ffs (cd->mask); + +return (c & cd->mask) >> shift; +}
Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c line 127
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+if (test_cpu_cap (&boot_cpu_data,X86_FEATURE_INTEL_PT )) { +for (i =0 ; i <PT_CPUID_LEAVES ; i++) +cpuid_count (20 , i, + &pt_pmu.caps[CR_EAX + i *4 ], + &pt_pmu.caps[CR_EBX + i *4 ], + &pt_pmu.caps[CR_ECX + i *4 ], + &pt_pmu.caps[CR_EDX + i *4 ]); + }else +return -ENODEV ;I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().
It looks like it would only work for the first cpuid leaf, but we'll need more than that. And the array makes it easier to allocate attributes for sysfs en masse.
I guess it doesn't really matter if we use unions unless these bits need to be exported to other parts of the kernel? But *that* is hardly a good idea. What do you think?
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 14 lines · Alexander Shishkin
Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5
-0+if (test_cpu_cap (&boot_cpu_data,X86_FEATURE_INTEL_PT )) { +for (i =0 ; i <PT_CPUID_LEAVES ; i++) + cpuid_count(20, i, + &pt_pmu.caps[CR_EAX + i * 4], + &pt_pmu.caps[CR_EBX + i * 4],Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c line 127
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +4
-0+ &pt_pmu.caps[CR_ECX + i *4 ], + &pt_pmu.caps[CR_EDX + i * 4]); + }else +return -ENODEV ;I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().
Quoted 7 lines
It looks like it would only work for the first cpuid leaf, but we'll need more than that. And the array makes it easier to allocate attributes for sysfs en masse.
I guess it doesn't really matter if we use unions unless these bits need to be exported to other parts of the kernel? But *that* is hardly a good idea. What do you think?
Ah yes, the generation. C is lacking there isn't it :/
Now I'm not sure we want to export all the bits you're using though. Like the topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 23 lines · Peter Zijlstra
Quoted 14 lines · Alexander Shishkin
Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5
-0+if (test_cpu_cap (&boot_cpu_data,X86_FEATURE_INTEL_PT )) { +for (i =0 ; i <PT_CPUID_LEAVES ; i++) + cpuid_count(20, i, + &pt_pmu.caps[CR_EAX + i * 4], + &pt_pmu.caps[CR_EBX + i * 4],Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c line 127
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +4
-0+ &pt_pmu.caps[CR_ECX + i *4 ], + &pt_pmu.caps[CR_EDX + i * 4]); + }else +return -ENODEV ;I would really rather you use bitfield unions for cpuid stuff, have a look at union cpuid10_e[abd]x as used in perf_event_intel.c:intel_pmu_init().
Quoted 7 lines
It looks like it would only work for the first cpuid leaf, but we'll need more than that. And the array makes it easier to allocate attributes for sysfs en masse.
I guess it doesn't really matter if we use unions unless these bits need to be exported to other parts of the kernel? But *that* is hardly a good idea. What do you think?
Quoted 5 lines
Ah yes, the generation. C is lacking there isn't it :/
Now I'm not sure we want to export all the bits you're using though. Like the
topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.
Actually, userspace can make assumptions about lost data from this bit. But there are others, such as encoded address length. In future there will also be several "caps" with allowed values for certain bit fields in RTIT_CTL, such as timing packet frequencies.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 8 lines · Alexander Shishkin
Quoted 3 lines · Peter Zijlstra
Now I'm not sure we want to export all the bits you're using though. Like the
topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.Actually, userspace can make assumptions about lost data from this bit.
Do explain; it feels entirely wrong to expose something like this, esp. since this crappy single TOPA thing is going away.
Quoted 3 lines
But there are others, such as encoded address length. In future there will also be several "caps" with allowed values for certain bit fields in
RTIT_CTL, such as timing packet frequencies.
No, those should not be caps, those should be in the format description.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Alexander Shishkin
Quoted 3 lines · Peter Zijlstra
Now I'm not sure we want to export all the bits you're using though. Like the
topa_multiple_entires, that appears an implementation detail userspace should not really care about either way.
Quoted 5 lines
Quoted 2 lines
Actually, userspace can make assumptions about lost data from this bit.
Do explain; it feels entirely wrong to expose something like this, esp. since this crappy single TOPA thing is going away.
At the very least, it gives you the minimal amount of memory you can ask for the aux buffer. With multiple entry topa it is one page, with single entry topa it is two pages. You can leave it up to the userspace to find out by trial and error, of course.
Quoted 5 lines
Quoted 3 lines
But there are others, such as encoded address length. In future there will also be several "caps" with allowed values for certain bit fields in
RTIT_CTL, such as timing packet frequencies.No, those should not be caps, those should be in the format description.
Format description gives a bit field, within which not all combinations may be valid. That is,
format/blah_freq: config:16-19
caps/blah_freq: 81
which means that on this cpu you can only set blah_freq to either 0 or
7. Figuring out these things by trial and error from userspace is not really nice.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander Shishkinarch/x86/kernel/cpu/perf_event_intel_pt.c line 176
Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +13
-0+static bool pt_event_valid (struct perf_event *event) +{ +u64 config = event->attr.config; + +/* admin can set any packet generation parameters */ +if (capable (CAP_SYS_ADMIN ) && (config &PT_BYPASS_MASK ) == config) +return true; + +if ((config &PT_CONFIG_MASK ) != config) +return false; + +return true; +}
This seems to suggest PT is available to !priv users, is this right?
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 16 lines · Peter Zijlstra
arch/x86/kernel/cpu/perf_event_intel_pt.c line 176
Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +13
-0+static bool pt_event_valid (struct perf_event *event) +{ +u64 config = event->attr.config; + +/* admin can set any packet generation parameters */ +if (capable (CAP_SYS_ADMIN ) && (config &PT_BYPASS_MASK ) == config) +return true; + +if ((config &PT_CONFIG_MASK ) != config) +return false; + +return true; +}This seems to suggest PT is available to !priv users, is this right?
Yes, that's the intention. PT does CPL-based filtering in hardware and unlike previous attempts at branch tracing, does not leak out-of-context addresses (or any other information for that matter).
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 12 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +12
-0+static int pt_config (struct perf_event *event) +{ +u64 reg; + + reg =RTIT_CTL_TOPA |RTIT_CTL_BRANCH_EN ; + +if (!event->attr.exclude_kernel) + reg |=RTIT_CTL_OS ; +if (!event->attr.exclude_user) + reg |=RTIT_CTL_USR ; + + reg |= (event->attr.config &PT_CONFIG_MASK );
arch/x86/kernel/cpu/perf_event_intel_pt.c line 213
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+ +/* +* User can try to set bits in RTIT_CTL through PT_BYPASS_MASK, +* that aren't supported by the hardware. Weather or not a +* particular bitmask is supported by a cpu can't be determined +* via cpuid or otherwise, so we have to rely on #GP handling +* to catch these cases. +*/ +return wrmsrl_safe (MSR_IA32_RTIT_CTL , reg); +}
Whether the weather is nice or not :-)
But no, this cannot be, once we've accepted the event is must be programmable. Failing at the time of programming is vile; pmu::start() is a void return, failure is not an option there.
The fact that the hardware cannot even tell you the supported mask is further fail.
IIRC I think Andi once suggested probing each of the 64 bits in that MSR to determine the supported mask at device init time.
BTW, what's that RTIT thing? Did someone forget to propagate the latest name change of the thing or so?
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 26 lines · Peter Zijlstra
Quoted 12 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +12
-0+static int pt_config (struct perf_event *event) +{ +u64 reg; + + reg =RTIT_CTL_TOPA |RTIT_CTL_BRANCH_EN ; + +if (!event->attr.exclude_kernel) + reg |=RTIT_CTL_OS ; +if (!event->attr.exclude_user) + reg |=RTIT_CTL_USR ; + + reg |= (event->attr.config &PT_CONFIG_MASK );arch/x86/kernel/cpu/perf_event_intel_pt.c line 213
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+ +/* +* User can try to set bits in RTIT_CTL through PT_BYPASS_MASK, +* that aren't supported by the hardware. Weather or not a +* particular bitmask is supported by a cpu can't be determined +* via cpuid or otherwise, so we have to rely on #GP handling +* to catch these cases. +*/ +return wrmsrl_safe (MSR_IA32_RTIT_CTL , reg); +}Whether the weather is nice or not :-)
Quoted 3 lines
But no, this cannot be, once we've accepted the event is must be programmable. Failing at the time of programming is vile;
pmu::start()is a void return, failure is not an option there.
This is called from pmu::add(), which can fail. If this wrmsrl throws a gp, pmu::add() will fail and we won't even get to pmu::start(). Of course, the problem with such event faulting every time it is added is still there. Maybe we can simply disable such events after the first fault. Good news is, only a CAP_SYS_ADMIN can set arbitrary bits, so the damage is limited.
Quoted 5 lines
The fact that the hardware cannot even tell you the supported mask is further fail.
IIRC I think Andi once suggested probing each of the 64 bits in that MSR to determine the supported mask at device init time.
The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines
The fact that the hardware cannot even tell you the supported mask is further fail.
The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.
OK, so put the magic number in the big model array.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines
The fact that the hardware cannot even tell you the supported mask is further fail.
The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.
OK, so put the magic number in the big model array.
I'm not sure I follow. These bits are reserved for the future, they can potentially be whatever combinations of whatever. If we want to probe around for valid combinations is to check everything in the range of 0..2^43 (or something like that, the region reserved for packet enables) and store all the valid ones, which sounds crazy.
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 9 lines · Alexander Shishkin
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines
The fact that the hardware cannot even tell you the supported mask is further fail.
The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.
Quoted 7 lines
Quoted 1 line
OK, so put the magic number in the big model array.
I'm not sure I follow. These bits are reserved for the future, they can potentially be whatever combinations of whatever. If we want to probe around for valid combinations is to check everything in the range of 0..2^43 (or something like that, the region reserved for packet enables) and store all the valid ones, which sounds crazy.
I was assuming that the accepted bits are model specific, and we have this big model switch statement in perf_event_intel.c:intel_pmu_init(), so why not have something like x86_pmu.pt_magic_bitmask = 0xf00d in there?
No need to probe in that case. That is the same thing we do for all unenumerated model specific things.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 18 lines · Peter Zijlstra
Quoted 9 lines · Alexander Shishkin
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
Quoted 2 lines
The fact that the hardware cannot even tell you the supported mask is further fail.
The problem with this is that some bits go in groups, there'd be 2..3..4 bit fields encoding desired packet frequency, for example.
Quoted 7 lines
Quoted 1 line
OK, so put the magic number in the big model array.
I'm not sure I follow. These bits are reserved for the future, they can potentially be whatever combinations of whatever. If we want to probe around for valid combinations is to check everything in the range of 0..2^43 (or something like that, the region reserved for packet enables) and store all the valid ones, which sounds crazy.
Quoted 4 lines
I was assuming that the accepted bits are model specific, and we have this big model switch statement in perf_event_intel.c:intel_pmu_init(), so why not have something like x86_pmu.pt_magic_bitmask =
0xf00din there?
Ah, I see what you mean. The main point of this whole reserved region proposal is that one shouldn't have to update one's kernel to enable new PT packets by doing -e intel_pt/config=0xf00d/, if one is CAP_SYS_ADMIN.
Quoted 2 lines
No need to probe in that case. That is the same thing we do for all unenumerated model specific things.
They are, actually, enumerated, we just want to be able to enable them before they are. If the driver is aware of feature X, it can test for it in CPUID and allow/disallow it based on that.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 6 lines · Alexander Shishkin
Peter Zijlstra <[email protected]> writes:
Ah, I see what you mean. The main point of this whole reserved region proposal is that one shouldn't have to update one's kernel to enable new PT packets by doing -e intel_pt/config=0xf00d/, if one is
CAP_SYS_ADMIN.
Quoted 6 lines
Quoted 2 lines
No need to probe in that case. That is the same thing we do for all unenumerated model specific things.
They are, actually, enumerated, we just want to be able to enable them before they are. If the driver is aware of feature X, it can test for it in CPUID and allow/disallow it based on that.
If you've got hardware that 'exposes' functionality not enumerated in CPUID you have the capability of modifying the kernel.
And while it might seem like a cute feature when you have early hardware, that is no reason to make the code horrible. Either just patch your firmware such that CPUID reports the right bits for your hardware or frob the kernel.
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 12 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +12
-0+static void pt_config_start (bool start) +{ +u64 ctl; + +rdmsrl (MSR_IA32_RTIT_CTL , ctl); +if (start) + ctl |=RTIT_CTL_TRACEEN ; +else + ctl &= ~RTIT_CTL_TRACEEN ; +wrmsrl (MSR_IA32_RTIT_CTL , ctl); + +/*
arch/x86/kernel/cpu/perf_event_intel_pt.c line 236
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+* A wrmsr that disables trace generation serializes other PT +* registers and causes all data packets to be written to memory, +* but a fence is required for the data to become globally visible. +* +* The below WMB, separating data store and aux_head store matches +* the consumer's RMB that separates aux_head load and data load. +*/ +if (!start) +wmb (); +}
wmb is sfence, is that sufficient? One would have expected an mfence since that would also orders later reads.
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +17
-0+static void pt_config_start (bool start) +{ +u64 ctl; + +rdmsrl (MSR_IA32_RTIT_CTL , ctl); +if (start) + ctl |=RTIT_CTL_TRACEEN ; +else + ctl &= ~RTIT_CTL_TRACEEN ; +wrmsrl (MSR_IA32_RTIT_CTL , ctl); + +/* +* A wrmsr that disables trace generation serializes other PT +* registers and causes all data packets to be written to memory, +* but a fence is required for the data to become globally visible. +* +* The below WMB, separating data store and aux_head store matches
Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c line 236
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5
-0+* the consumer's RMB that separates aux_head load and data load. +*/ +if (!start) +wmb (); +}wmb is sfence, is that sufficient? One would have expected an mfence since that would also orders later reads.
Silly me, we're separating two stores here. Ignore that.
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander Shishkinarch/x86/kernel/cpu/perf_event_intel_pt.c line 267
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +2
-0+/* make negative table index stand for the last table entry */ +#define TOPA_ENTRY (t, i) ((i) == -1 ? &(t)->table[(t)->last] : &(t)->table[(i)])
code does not match comment; negative would be: i < 0, not i == -1.
Something like: ({ if (i < 0) i += t->size; t->table[i]; }), might work, of course that goes bang when: i !e [-size,size)
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 4 lines · Peter Zijlstra
arch/x86/kernel/cpu/perf_event_intel_pt.c line 267
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +2
-0+/* make negative table index stand for the last table entry */ +#define TOPA_ENTRY (t, i) ((i) == -1 ? &(t)->table[(t)->last] : &(t)->table[(i)])code does not match comment; negative would be: i < 0, not i == -1.
Indeed.
Quoted 2 lines
Something like: ({ if (i < 0) i +=
t->size; t->table[i]; }), might work, of course that goes bang when: i !e [-size,size)
Table indices and sizes are measured in different units. Using -1 to mean the last is just a small convenience, in the code it would be used explicitly as TOPA_ENTRY(topa, -1).end = 1, for example.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander Shishkinarch/x86/kernel/cpu/perf_event_intel_pt.c line 485
Quoted 13 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +11
-0+if ((pt_cap_get (PT_CAP_topa_multiple_entries) + && topa->table[i].stop) + || topa->table[i].end) +break ; + old = (local64_xchg (&buf->head, base) & + ((buf->nr_pages <<PAGE_SHIFT ) -1 )); +if (!pt_cap_get (PT_CAP_topa_multiple_entries) || + buf->output_off ==sizes (TOPA_ENTRY (buf->cur, buf->cur_idx)->size)) { +local_inc (&buf->lost); + advance++; + }
Inconsistent coding style. Please put operators at the end of a line, not at the start of one.
Alexander Shishkin
reviewerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +6
-0+if ((pt_cap_get (PT_CAP_topa_multiple_entries) + && topa->table[i].stop) + || topa->table[i].end) +break ; + old = (local64_xchg (&buf->head, base) & + ((buf->nr_pages <<PAGE_SHIFT ) -1 ));
Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c line 485
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +5
-0+if (!pt_cap_get (PT_CAP_topa_multiple_entries) || + buf->output_off ==sizes (TOPA_ENTRY (buf->cur, buf->cur_idx)->size)) { +local_inc (&buf->lost); + advance++; + }Inconsistent coding style. Please put operators at the end of a line, not at the start of one.
Ok.
Peter Zijlstra
maintainerRe: [PATCH v5 12/20] x86: perf: intel_pt: Intel PT PMU driver
In reply to Alexander ShishkinQuoted 9 lines · Alexander Shishkin
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +9
-0+static void pt_event_start (struct perf_event *event,int mode) +{ +struct pt *pt =this_cpu_ptr (&pt_ctx); +struct pt_buffer *buf =perf_get_aux (&pt->handle); + +if (pt_is_running () || !buf ||pt_buffer_is_full (buf, pt) || +pt_config (event)) { + event->hw.state =PERF_HES_STOPPED ; +return ;
arch/x86/kernel/cpu/perf_event_intel_pt.c line 823
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_pt.c new file +10
-0+ } + +ACCESS_ONCE (pt->handle_nmi) =1 ; + event->hw.state =0 ; + +pt_config_buffer (buf->cur->table, buf->cur_idx, + buf->output_off); +wrmsrl (MSR_IA32_RTIT_STATUS ,0 ); +pt_config_start (true); +}
That's two RTIT_CTL writes, should we not try and merge those?
Alexander Shishkin
reviewerPATCH v5 13/20 x86: perf: intel_bts: Add BTS PMU driver
In reply to Alexander ShishkinAdd support for Branch Trace Store (BTS) via kernel perf event infrastructure. The difference with the existing implementation of BTS support is that this one is a separate PMU that exports events' trace buffers to userspace by means of AUX area of the perf buffer, which is zero-copy mapped into userspace.
The immediate benefit is that the buffer size can be much bigger, resulting in fewer interrupts and no kernel side copying is involved and little to no trace data loss. Also, kernel code can be traced with this driver.
The old way of collecting BTS traces still works.
--- arch/x86/kernel/cpu/Makefile | 2 +- arch/x86/kernel/cpu/perf_event.h | 6 + arch/x86/kernel/cpu/perf_event_intel.c | 6 +- arch/x86/kernel/cpu/perf_event_intel_bts.c | 518 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_ds.c | 3 +- 5 files changed, 532 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cpu/perf_event_intel_bts.c
arch/x86/kernel/cpu/Makefile +1 -1
@@ -39,7 +39,7 @@ obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_lbr.o perf_event_intel_ds.o per obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore.o perf_event_intel_uncore_snb.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_uncore_snbep.o perf_event_intel_uncore_nhmex.o obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_rapl.o -obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o +obj-$(CONFIG_CPU_SUP_INTEL) += perf_event_intel_pt.o perf_event_intel_bts.o endif
arch/x86/kernel/cpu/perf_event.h +6 -0
@@ -752,6 +752,12 @@ int intel_pmu_setup_lbr_filter(struct perf_event *event);void intel_pt_interrupt (void ); +int intel_bts_interrupt (void ); + +void intel_bts_enable_local (void ); + +void intel_bts_disable_local (void ); +int p4_pmu_init (void );int p6_pmu_init (void );
arch/x86/kernel/cpu/perf_event_intel.c +5 -1
@@ -1180,6 +1180,8 @@ static void intel_pmu_disable_all(void)if (test_bit (INTEL_PMC_IDX_FIXED_BTS , cpuc->active_mask))intel_pmu_disable_bts (); +else +intel_bts_disable_local ();intel_pmu_pebs_disable_all ();intel_pmu_lbr_disable_all ();
@@ -1202,7 +1204,8 @@ static void intel_pmu_enable_all(int added)return ;intel_pmu_enable_bts (event->hw.config); - } + }else +intel_bts_enable_local (); }/*
@@ -1488,6 +1491,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)apic_write (APIC_LVTPC ,APIC_DM_NMI );intel_pmu_disable_all (); handled =intel_pmu_drain_bts_buffer (); + handled +=intel_bts_interrupt (); status =intel_pmu_get_status ();if (!status) goto done;
arch/x86/kernel/cpu/perf_event_intel_bts.c new file +518 -0
518 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event_intel_ds.c +2 -1
@@ -466,7 +466,8 @@ void intel_pmu_enable_bts(u64 config) debugctlmsr |=DEBUGCTLMSR_TR ; debugctlmsr |=DEBUGCTLMSR_BTS ; - debugctlmsr |=DEBUGCTLMSR_BTINT ; +if (config &ARCH_PERFMON_EVENTSEL_INT ) + debugctlmsr |=DEBUGCTLMSR_BTINT ;if (!(config &ARCH_PERFMON_EVENTSEL_OS )) debugctlmsr |=DEBUGCTLMSR_BTS_OFF_OS ;
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 14/20 perf: add ITRACE_START record to indicate that tracing has started
In reply to Alexander ShishkinFor counters that generate AUX data that is bound to the context of a running task, such as instruction tracing, the decoder needs to know exactly which task is running when the event is first scheduled in, before the first sched_switch. The decoder's need to know this stems from the fact that instruction flow trace decoding will almost always require program's object code in order to reconstruct said flow and for that we need at least its pid/tid in the perf stream.
To single out such instruction tracing pmus, this patch introduces ITRACE PMU capability. The reason this is not part of RECORD_AUX record is that not all pmus capable of generating AUX data need this, and the opposite is *probably* also true.
While sched_switch covers for most cases, there are two problems with it:
the consumer will need to process events out of order (that is, having found RECORD_AUX, it will have to skip forward to the nearest sched_switch to figure out which task it was, then go back to the actual trace to decode it) and it completely misses the case when the tracing is enabled and disabled before sched_switch, for example, via PERF_EVENT_IOC_DISABLE.
--- include/linux/perf_event.h | 4 ++++ include/uapi/linux/perf_event.h | 11 +++++++++++ kernel/events/core.c | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+)
include/linux/perf_event.h +4 -0
@@ -127,6 +127,9 @@ struct hw_perf_event {/* for tp_event->class */ struct list_head tp_list; }; +struct {/* itrace */ +int itrace_started; + }; #ifdef CONFIG_HAVE_HW_BREAKPOINT struct { /* breakpoint */ /*
@@ -174,6 +177,7 @@ struct perf_event;#define PERF_PMU_CAP_AUX_NO_SG 0x02#define PERF_PMU_CAP_AUX_SW_DOUBLEBUF 0x04#define PERF_PMU_CAP_EXCLUSIVE 0x08 +#define PERF_PMU_CAP_ITRACE 0x10/** * struct pmu - generic performance monitoring unit
include/uapi/linux/perf_event.h +11 -0
@@ -750,6 +750,17 @@ enum perf_event_type {*/ PERF_RECORD_AUX =11 , +/* +* Indicates that instruction trace has started +* +* struct { +* struct perf_event_header header; +* u32 pid; +* u32 tid; +* }; +*/ +PERF_RECORD_ITRACE_START =12 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +41 -0
59 lines not on this page · raw message
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 15/20 perf: Add api to (de-)allocate AUX buffers for kernel counters
In reply to Alexander ShishkinSeveral use cases for AUX data, namely event sampling (including a piece of AUX data in some perf event sample, so that the user can get, for example, instruction traces leading up to a certain event like a breakpoint or a hardware event), process core dumps (providing user with a history of a process' instruction flow leading up to a crash), system crash dumps and storing AUX data in pstore across reboot (to facilitate post-mortem investigation of a system crash) require different parts of the kernel code to be able to configure hardware to produce AUX data and collect it when it is needed.
Luckily, there is already an api for in-kernel perf events, which has several users. This proposal is to extend that api to allow in-kernel users to allocate AUX buffers for kernel counters. Such users will call rb_alloc_kernel() to allocate what they want and later copy the data out to other backends, e.g. a sample in another event's ring buffer or a core dump file. These buffers are never mapped to userspace.
There are no additional constraints or requirements on the pmu drivers.
A typical user of this interface will first create a kernel counter with a call to perf_event_create_kernel_counter() and then allocate a ring buffer for it with rb_alloc_kernel(). Data can then be copied out from the AUX buffer using rb_output_aux(), which is passed a callback that will write chunks of AUX data into the desired destination, such as perf_output_copy() or dump_emit(). Caller needs to use perf_event_disable to make sure that the counter is not active while it copies data out.
--- kernel/events/internal.h | 8 ++++++ kernel/events/ring_buffer.c | 64 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+)
kernel/events/internal.h +8 -0
@@ -54,6 +54,9 @@ struct ring_buffer {void *data_pages[0 ]; }; +typedef unsigned long (*aux_copyfn )(void *data,const void *src, +unsigned long len); +extern void rb_free (struct ring_buffer *rb);extern struct ring_buffer *rb_alloc (int nr_pages,long watermark,int cpu,int flags);
@@ -61,6 +64,11 @@ extern void perf_event_wakeup(struct perf_event *event);extern int rb_alloc_aux (struct ring_buffer *rb,struct perf_event *event,pgoff_t pgoff,int nr_pages,long watermark,int flags);extern void rb_free_aux (struct ring_buffer *rb); +extern long rb_output_aux (struct ring_buffer *rb,unsigned long from, +unsigned long to,aux_copyfn copyfn,void *data); +extern struct ring_buffer * +rb_alloc_kernel (struct perf_event *event,int nr_pages,int aux_nr_pages); +extern void rb_free_kernel (struct ring_buffer *rb,struct perf_event *event);extern struct ring_buffer *ring_buffer_get (struct perf_event *event);extern void ring_buffer_put (struct ring_buffer *rb);
kernel/events/ring_buffer.c +64 -0
76 lines not on this page · raw message
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 16/20 perf: Add a helper for looking up pmus by type
In reply to Alexander ShishkinThis patch adds a helper for looking up a registered pmu by its type.
--- kernel/events/core.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
kernel/events/core.c +13 -3
@@ -6963,6 +6963,18 @@ void perf_pmu_unregister(struct pmu *pmu) }EXPORT_SYMBOL_GPL (perf_pmu_unregister); +/* call under pmus_srcu */ +static struct pmu *__perf_find_pmu (u32 type) +{ +struct pmu *pmu; + +rcu_read_lock (); + pmu =idr_find (&pmu_idr, type); +rcu_read_unlock (); + +return pmu; +} +struct pmu *perf_init_event (struct perf_event *event) {struct pmu *pmu =NULL ;
@@ -6971,9 +6983,7 @@ struct pmu *perf_init_event(struct perf_event *event) idx =srcu_read_lock (&pmus_srcu); -rcu_read_lock (); - pmu =idr_find (&pmu_idr, event->attr.type); -rcu_read_unlock (); + pmu =__perf_find_pmu (event->attr.type);if (pmu) {if (!try_module_get (pmu->module)) { pmu =ERR_PTR (-ENODEV );
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 17/20 perf: Add infrastructure for using AUX data in perf samples
In reply to Alexander ShishkinAUX data can be used to annotate perf events such as performance counters or tracepoints/breakpoints by including it in sample records when PERF_SAMPLE_AUX flag is set. Such samples would be instrumental in debugging and profiling by providing, for example, a history of instruction flow leading up to the event's overflow.
To facilitate this, this patch adds code to create a kernel counter with a ring buffer using rb_{alloc,free}_kernel() interface for each event that needs to include AUX samples and to copy AUX data from it into the perf data stream.
The user interface is extended to allow for this, new attribute fields are added:
* aux_sample_type: specify PMU on which the AUX data generating event
is created;
* aux_sample_config: event config (maps to attribute's config field),
* aux_sample_size: size of the sample to be written.
This kernel counter is configured similarly to the event that is being annotated with regards to filtering (exclude_{hv,idle,user,kernel}) and enabled state (disabled, enable_on_exec) to make sure that the two events are scheduled at the same time and that no out of context activity is sampled.
--- include/linux/perf_event.h | 9 +++ include/uapi/linux/perf_event.h | 18 ++++- kernel/events/core.c | 172 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 198 insertions(+), 1 deletion(-)
include/linux/perf_event.h +9 -0
@@ -84,6 +84,12 @@ struct perf_regs_user {struct pt_regs *regs; }; +struct perf_aux_record { +u64 size; +unsigned long from; +unsigned long to; +}; +struct task_struct ;/*
@@ -458,6 +464,7 @@ struct perf_event {perf_overflow_handler_t overflow_handler;void *overflow_handler_context; +struct perf_event *sampler;#ifdef CONFIG_EVENT_TRACING struct ftrace_event_call *tp_event;struct event_filter *filter;
@@ -628,6 +635,7 @@ struct perf_sample_data {union perf_mem_data_src data_src;struct perf_callchain_entry *callchain;struct perf_raw_record *raw; +struct perf_aux_record aux;struct perf_branch_stack *br_stack;struct perf_regs_user regs_user;u64 stack_user_size;
@@ -655,6 +663,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data, data->period = period; data->regs_user.abi =PERF_SAMPLE_REGS_ABI_NONE ; data->regs_user.regs =NULL ; + data->aux.from = data->aux.to = data->aux.size =0 ; data->stack_user_size =0 ; data->weight =0 ; data->data_src.val =PERF_MEM_NA ;
include/uapi/linux/perf_event.h +17 -1
@@ -137,8 +137,9 @@ enum perf_event_sample_format {PERF_SAMPLE_DATA_SRC =1U <<15 ,PERF_SAMPLE_IDENTIFIER =1U <<16 ,PERF_SAMPLE_TRANSACTION =1U <<17 , +PERF_SAMPLE_AUX =1U <<18 , - PERF_SAMPLE_MAX = 1U << 18, /* non-ABI */ + PERF_SAMPLE_MAX = 1U << 19, /* non-ABI */ };/*
@@ -239,6 +240,9 @@ enum perf_event_read_format {#define PERF_ATTR_SIZE_VER3 96/* add: sample_regs_user */ /* add: sample_stack_user */ /* add: aux_watermark */ +#define PERF_ATTR_SIZE_VER4 120/* add: aux_sample_config */ +/* add: aux_sample_size */ +/* add: aux_sample_type */ /* * Hardware event_id to monitor via a performance monitoring event:
@@ -337,6 +341,16 @@ struct perf_event_attr {* Wakeup watermark for AUX area */ __u32 aux_watermark; + +/* +* Itrace pmus' event config +*/ +__u64 aux_sample_config;/* event config for AUX sampling */ +__u64 aux_sample_size;/* desired sample size */ +__u32 aux_sample_type;/* pmu->type of an AUX PMU */ + +/* Align to u64. */ +__u32 __reserved_2; };#define perf_flags (attr) (*(&(attr)->read_format + 1))
@@ -710,6 +724,8 @@ enum perf_event_type {* { u64 weight; } && PERF_SAMPLE_WEIGHT * { u64 data_src; } && PERF_SAMPLE_DATA_SRC * { u64 transaction; } && PERF_SAMPLE_TRANSACTION +* { u64 size; +* char data[size]; } && PERF_SAMPLE_AUX * }; */ PERF_RECORD_SAMPLE = 9,
kernel/events/core.c +172 -0
220 lines not on this page · raw message
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 18/20 perf: Allocate ring buffers for inherited per-task kernel events
In reply to Alexander ShishkinNormally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.
However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.
When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.
--- kernel/events/core.c | 9 +++++++++ kernel/events/internal.h | 11 +++++++++++ 2 files changed, 20 insertions(+)
kernel/events/core.c +9 -0
@@ -8267,6 +8267,15 @@ inherit_event(struct perf_event *parent_event, = parent_event->overflow_handler_context;/* +* For per-task kernel events with ring buffers, set_output doesn't +* make sense, but we can allocate a new buffer here. +*/ +if (parent_event->cpu ==-1 &&kernel_rb_event (parent_event)) { + (void )rb_alloc_kernel (child_event, parent_event->rb->nr_pages, + parent_event->rb->aux_nr_pages); + } + +/* * Precalculate sample_data sizes */ perf_event__header_size (child_event);
kernel/events/internal.h +11 -0
@@ -122,6 +122,17 @@ static inline unsigned long perf_aux_size(struct ring_buffer *rb)return rb->aux_nr_pages <<PAGE_SHIFT ; } +static inline bool kernel_rb_event (struct perf_event *event) +{ +/* +* Having a ring buffer and not being on any ring buffers' wakeup +* list means it was attached by rb_alloc_kernel() and not +* ring_buffer_attach(). It's the only case when these two +* conditions take place at the same time. +*/ +return event->rb &&list_empty (&event->rb_entry); +} +#define DEFINE_OUTPUT_COPY (func_name, memcpy_func) \ static inline unsigned long \ func_name(struct perf_output_handle *handle, \
Signature · 1 line
2.1.0
Peter Zijlstra
maintainerRe: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events
Quoted 10 lines · Alexander Shishkin
Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.
However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.
Quoted 6 lines
When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.
This causes a pinned memory explosion, not at all nice that.
I think I see why and all, but it would be ever so good to not have to allocate so much memory.
Alexander Shishkin
reviewerRe: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Alexander Shishkin
Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.
However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.
Quoted 6 lines
When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.
Quoted 4 lines
This causes a pinned memory explosion, not at all nice that.
I think I see why and all, but it would be ever so good to not have to allocate so much memory.
Are there any controls we could use to limit such memory usage? Theoretically, the buffers that we'd allocate for this are way smaller than, for example, what we use if we try to capture a complete trace, since we'd only be interested in the most recent trace data. We already have RLIMIT_NPROC, which implicitly limits the number of these buffers, for example. Or maybe we can introduce a new rlimit/sysctl/whatnot that would limit the maximum amount of such memory per-cpu/system/user. What do you think?
Per-cpu buffers with inheritance would solve this problem, but raises other issues: we'd need sched_switch again to tell the traces apart and since those buffers run in overwrite mode, a cpu hog task can potentially overwrite any useful trace data.
Regards,
--
Alex
Peter Zijlstra
maintainerRe: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events
Quoted 20 lines · Alexander Shishkin
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · Alexander Shishkin
Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.
However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.
When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.
Quoted 6 lines
Quoted 4 lines
This causes a pinned memory explosion, not at all nice that.
I think I see why and all, but it would be ever so good to not have to allocate so much memory.
Are there any controls we could use to limit such memory usage?
I'd say the same limit we're already accounting the mmap()s against. But the question is; what do we do when we run out?
Will we fail clone()? That might 'surprise' quite a few people, that their application won't work when profiled.
In any case, lets focus on the other parts of this work and delay this feature till later.
Alexander Shishkin
reviewerRe: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Alexander Shishkin
Quoted 22 lines · Peter Zijlstra
Quoted 10 lines · Alexander Shishkin
Normally, per-task events can't be inherited parents' ring buffers to avoid multiple events contending for the same buffer. And since buffer allocation is typically done by the userspace consumer, there is no practical interface to allocate new buffers for inherited counters.
However, for kernel users we can allocate new buffers for inherited events as soon as they are created (and also reap them on event destruction). This pattern has a number of use cases, such as event sample annotation and process core dump annotation.
Quoted 6 lines
When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.
This causes a pinned memory explosion, not at all nice that.
I think I see why and all, but it would be ever so good to not have to allocate so much memory.
Quoted 7 lines
Quoted 1 line
Are there any controls we could use to limit such memory usage?
I'd say the same limit we're already accounting the mmap()s against. But the question is; what do we do when we run out?
Will we fail
clone()? That might 'surprise' quite a few people, that their application won't work when profiled.
Or we just don't allocate any more buffers for this user; if there's a perf stream involved, we can output a record saying that.
Quoted 2 lines
In any case, lets focus on the other parts of this work and delay this feature till later.
Agreed.
Regards,
--
Alex
Re: [PATCH v5 18/20] perf: Allocate ring buffers for inherited per-task kernel events
In reply to Peter ZijlstraEm Thu, Oct 30, 2014 at 09:43:59AM +0100, Peter Zijlstra escreveu:
Quoted 14 lines · Alexander Shishkin
Quoted 12 lines
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines · Alexander Shishkin
When a new event is inherited from a per-task kernel event that has a ring buffer, allocate a new buffer for this event so that data from the child task is collected and can later be retrieved for sample annotation or core dump inclusion. This ring buffer is released when the event is freed, for example, when the child task exits.
This causes a pinned memory explosion, not at all nice that.
I think I see why and all, but it would be ever so good to not have to allocate so much memory.
Quoted 8 lines
Quoted 1 line
Are there any controls we could use to limit such memory usage?
I'd say the same limit we're already accounting the mmap()s against. But the question is; what do we do when we run out?
Will we fail
clone()? That might 'surprise' quite a few people, thattheir application won't work when profiled.
Can't we just emit PERF_RECORD_THROTTLE or similar stuff then? I.e. somehow mark the cloned process as not being profiled due to ENOMEM?
- Arnaldo
Alexander Shishkin
reviewerPATCH v5 19/20 perf: Allow AUX sampling for multiple events
In reply to Alexander ShishkinRight now, only one perf event in a context at a time can be annotated with AUX data, because AUX pmus will normally have only one event scheduled at a time, however it should be possible to annotate several events with similar configuration (wrt exclude_{hv,idle,user,kernel}, config and other event attribute fields) using the same AUX event.
This patch changes the behavior so that every time, before a kernel counter is created for AUX sampling, we first look for an existing counter that is suitable in terms of configuration and we use it to annotate the new event as well. If no such event is found, we fall back to the old path and try to allocate a new kernel counter.
--- kernel/events/core.c | 93 +++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 66 insertions(+), 27 deletions(-)
kernel/events/core.c +66 -27
@@ -4651,45 +4651,84 @@ static void perf_aux_sampler_destroy(struct perf_event *event)ring_buffer_put (rb);/* can be last */ } +static bool exclusive_event_match (struct perf_event *e1,struct perf_event *e2); + +static bool perf_aux_event_match (struct perf_event *e1,struct perf_event *e2) +{ +return has_aux (e1) &&exclusive_event_match (e1, e2); +} +static int perf_aux_sampler_init (struct perf_event *event,struct task_struct *task,struct pmu *pmu) { +struct perf_event_context *ctx;struct perf_event_attr attr; -struct perf_event *sampler; +struct perf_event *sampler =NULL ;struct ring_buffer *rb; -unsigned long nr_pages; +unsigned long nr_pages, flags;if (!pmu || !(pmu->setup_aux))return -ENOTSUPP ; -memset (&attr,0 ,sizeof (attr)); - attr.type = pmu->type; - attr.config = event->attr.aux_sample_config; - attr.sample_type =0 ; - attr.disabled = event->attr.disabled; - attr.enable_on_exec = event->attr.enable_on_exec; - attr.exclude_hv = event->attr.exclude_hv; - attr.exclude_idle = event->attr.exclude_idle; - attr.exclude_user = event->attr.exclude_user; - attr.exclude_kernel = event->attr.exclude_kernel; - attr.aux_sample_size = event->attr.aux_sample_size; - - sampler =perf_event_create_kernel_counter (&attr, event->cpu, task, -NULL ,NULL ); -if (IS_ERR (sampler)) -return PTR_ERR (sampler); - - nr_pages =1ul <<__get_order (event->attr.aux_sample_size); - - rb =rb_alloc_kernel (sampler,0 , nr_pages); -if (!rb) { -perf_event_release_kernel (sampler); -return -ENOMEM ; + ctx =find_get_context (pmu, task, event->cpu); +if (ctx) { +raw_spin_lock_irqsave (&ctx->lock, flags); +list_for_each_entry (sampler, &ctx->event_list, event_entry) { +/* +* event is not an aux event, but all the relevant +* bits should match +*/ +if (perf_aux_event_match (sampler, event) && + sampler->attr.type == event->attr.aux_sample_type && + sampler->attr.config == event->attr.aux_sample_config && + sampler->attr.exclude_hv == event->attr.exclude_hv && + sampler->attr.exclude_idle == event->attr.exclude_idle && + sampler->attr.exclude_user == event->attr.exclude_user && + sampler->attr.exclude_kernel == event->attr.exclude_kernel && + sampler->attr.aux_sample_size >= event->attr.aux_sample_size && +atomic_long_inc_not_zero (&sampler->refcount)) + goto got_event; + } + + sampler =NULL ; + +got_event: + --ctx->pin_count; +put_ctx (ctx); +raw_spin_unlock_irqrestore (&ctx->lock, flags); + } + +if (!sampler) { +memset (&attr,0 ,sizeof (attr)); + attr.type = pmu->type; + attr.config = event->attr.aux_sample_config; + attr.sample_type =0 ; + attr.disabled = event->attr.disabled; + attr.enable_on_exec = event->attr.enable_on_exec; + attr.exclude_hv = event->attr.exclude_hv; + attr.exclude_idle = event->attr.exclude_idle; + attr.exclude_user = event->attr.exclude_user; + attr.exclude_kernel = event->attr.exclude_kernel; + attr.aux_sample_size = event->attr.aux_sample_size; + + sampler =perf_event_create_kernel_counter (&attr, event->cpu, + task,NULL ,NULL ); +if (IS_ERR (sampler)) +return PTR_ERR (sampler); + + nr_pages =1ul <<__get_order (event->attr.aux_sample_size); + + rb =rb_alloc_kernel (sampler,0 , nr_pages); +if (!rb) { +perf_event_release_kernel (sampler); +return -ENOMEM ; + } + + sampler->destroy = perf_aux_sampler_destroy; } event->sampler = sampler; - sampler->destroy = perf_aux_sampler_destroy;return 0 ; }
@@ -4702,7 +4741,7 @@ static void perf_aux_sampler_fini(struct perf_event *event)if (!sampler)return ; -perf_event_release_kernel (sampler); +put_event (sampler); event->sampler =NULL ; }
Signature · 1 line
2.1.0
Alexander Shishkin
reviewerPATCH v5 20/20 perf: Allow AUX sampling of inherited events
In reply to Alexander ShishkinTry to find an AUX sampler event for the current event if none is linked via event::sampler.
This is useful when these events (the one that is being sampled and the one providing sample annotation) are allocated by inheritance path, independently of one another and the latter is not yet referenced by the former's event::sampler.
--- kernel/events/core.c | 94 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 62 insertions(+), 32 deletions(-)
kernel/events/core.c +62 -32
@@ -4658,47 +4658,67 @@ static bool perf_aux_event_match(struct perf_event *e1, struct perf_event *e2)return has_aux (e1) &&exclusive_event_match (e1, e2); } +struct perf_event *__find_sampling_counter (struct perf_event_context *ctx, +struct perf_event *event, +struct task_struct *task) +{ +struct perf_event *sampler =NULL ; + +list_for_each_entry (sampler, &ctx->event_list, event_entry) { +/* +* event is not an itrace event, but all the relevant +* bits should match +*/ +if (perf_aux_event_match (sampler, event) && +kernel_rb_event (sampler) && + sampler->attr.type == event->attr.aux_sample_type && + sampler->attr.config == event->attr.aux_sample_config && + sampler->attr.exclude_hv == event->attr.exclude_hv && + sampler->attr.exclude_idle == event->attr.exclude_idle && + sampler->attr.exclude_user == event->attr.exclude_user && + sampler->attr.exclude_kernel == event->attr.exclude_kernel && + sampler->attr.aux_sample_size >= event->attr.aux_sample_size && +atomic_long_inc_not_zero (&sampler->refcount)) +return sampler; + } + +return NULL ; +} + +struct perf_event *find_sampling_counter (struct pmu *pmu, +struct perf_event *event, +struct task_struct *task) +{ +struct perf_event_context *ctx; +struct perf_event *sampler =NULL ; +unsigned long flags; + + ctx =find_get_context (pmu, task, event->cpu); +if (!ctx) +return NULL ; + +raw_spin_lock_irqsave (&ctx->lock, flags); + sampler =__find_sampling_counter (ctx, event, task); + --ctx->pin_count; +put_ctx (ctx); +raw_spin_unlock_irqrestore (&ctx->lock, flags); + +return sampler; +} +static int perf_aux_sampler_init (struct perf_event *event,struct task_struct *task,struct pmu *pmu) { -struct perf_event_context *ctx;struct perf_event_attr attr; -struct perf_event *sampler =NULL ; +struct perf_event *sampler;struct ring_buffer *rb; -unsigned long nr_pages, flags; +unsigned long nr_pages;if (!pmu || !(pmu->setup_aux))return -ENOTSUPP ; - ctx =find_get_context (pmu, task, event->cpu); -if (ctx) { -raw_spin_lock_irqsave (&ctx->lock, flags); -list_for_each_entry (sampler, &ctx->event_list, event_entry) { -/* -* event is not an aux event, but all the relevant -* bits should match -*/ -if (perf_aux_event_match (sampler, event) && - sampler->attr.type == event->attr.aux_sample_type && - sampler->attr.config == event->attr.aux_sample_config && - sampler->attr.exclude_hv == event->attr.exclude_hv && - sampler->attr.exclude_idle == event->attr.exclude_idle && - sampler->attr.exclude_user == event->attr.exclude_user && - sampler->attr.exclude_kernel == event->attr.exclude_kernel && - sampler->attr.aux_sample_size >= event->attr.aux_sample_size && -atomic_long_inc_not_zero (&sampler->refcount)) - goto got_event; - } - - sampler =NULL ; - -got_event: - --ctx->pin_count; -put_ctx (ctx); -raw_spin_unlock_irqrestore (&ctx->lock, flags); - } - + sampler =find_sampling_counter (pmu, event, task);if (!sampler) {memset (&attr,0 ,sizeof (attr)); attr.type = pmu->type;
@@ -4749,9 +4769,19 @@ static void perf_aux_sampler_fini(struct perf_event *event)static unsigned long perf_aux_sampler_trace (struct perf_event *event,struct perf_sample_data *data) { -struct perf_event *sampler = event->sampler; +struct perf_event *sampler;struct ring_buffer *rb; +if (!event->sampler) { +/* +* down this path, event->ctx is already locked IF it's the +* same context +*/ + event->sampler =__find_sampling_counter (event->ctx, event, + event->ctx->task); + } + + sampler = event->sampler;if (!sampler || sampler->state !=PERF_EVENT_STATE_ACTIVE ) { data->aux.size =0 ; goto out;
Signature · 1 line
2.1.0