perf: Add build id parsing fault detection/fix
cover perf: Add build id parsing fault detection/fix
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.
It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ...
...
[ perf record: Woken up 1 times to write data ]
[ perf record: Failed to parse 4 build ids]
[ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only
...
# build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --stat
Aggregated stats:
TOTAL events: 104
....
BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.data
And verify:
# perf report -i perf-fixed.data --stats
Aggregated stats:
TOTAL events: 104
....
Good, let's see how many we fixed:
# perf report --header-only -i perf-fixed.data
...
# build id mmap stats: FAULTS 4, LOST 0, FIXED(4)I don't have a good way to test it, just by artificially adding the faults in kernel code, but Ian and Namhyung might have setup that could generate that.. would be great to have a perf test for this.
Also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/buildid_stats
thoughts?
thanks,
jirka
---
Jiri Olsa (10):
perf: Track build id faults for mmap2 event
perf: Move build_id_parse to check only regular files
perf: Add new read_format bit to read build id faults
perf: Add new read_format bit to read lost events
tools: Sync perf_event.h uapi
libperf: Do not allow PERF_FORMAT_GROUP in perf_evsel__read perf record: Add support to read build id fails
perf record: Add new HEADER_BUILD_ID_MMAP feature perf report: Display build id fails stats
perf inject: Add --buildid-mmap2 option to fix failed build ids
include/linux/perf_event.h | 2 ++ include/uapi/linux/perf_event.h | 20 +++++++++++++------- kernel/events/core.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ kernel/events/ring_buffer.c | 3 +++ tools/include/uapi/linux/perf_event.h | 20 +++++++++++++------- tools/lib/perf/evsel.c | 10 ++++++++++ tools/lib/perf/include/perf/evsel.h | 11 ++++++++++- tools/perf/Documentation/perf-inject.txt | 3 +++ tools/perf/Documentation/perf.data-file-format.txt | 19 +++++++++++++++++++ tools/perf/builtin-inject.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- tools/perf/builtin-record.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin-report.c | 35 +++++++++++++++++++++++++++++++++++ tools/perf/util/env.h | 6 ++++++ tools/perf/util/evsel.c | 12 ++++++++++++ tools/perf/util/header.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 1 + tools/perf/util/map.h | 15 +++++++++++++++ tools/perf/util/perf_event_attr_fprintf.c | 3 ++- 18 files changed, 407 insertions(+), 24 deletions(-)
PATCH 1/10 perf: Track build id faults for mmap2 event
Keep track of build id parsing faults per event,
so we can report it in following patches.
--- include/linux/perf_event.h | 1 + kernel/events/core.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
include/linux/perf_event.h +1 -0
@@ -669,6 +669,7 @@ struct perf_event {unsigned int attach_state;local64_t count;atomic64_t child_count; +local64_t build_id_faults;/* * These are the total time in nanoseconds that the event
kernel/events/core.c +9 -2
@@ -8183,6 +8183,7 @@ struct perf_mmap_event {u32 prot, flags;u8 build_id[BUILD_ID_SIZE_MAX ];u32 build_id_size; +bool build_id_fault;struct {struct perf_event_header header;
@@ -8244,6 +8245,9 @@ static void perf_event_mmap_output(struct perf_event *event,if (event->attr.mmap2 && use_build_id) mmap_event->event_id.header.misc |=PERF_RECORD_MISC_MMAP_BUILD_ID ; +if (mmap_event->build_id_fault) +local64_inc (&event->build_id_faults); +perf_output_put (&handle, mmap_event->event_id);if (event->attr.mmap2) {
@@ -8386,8 +8390,11 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event) mmap_event->event_id.header.size =sizeof (mmap_event->event_id) + size; -if (atomic_read (&nr_build_id_events)) -build_id_parse (vma, mmap_event->build_id, &mmap_event->build_id_size); +if (atomic_read (&nr_build_id_events)) { +int err =build_id_parse (vma, mmap_event->build_id, + &mmap_event->build_id_size); + mmap_event->build_id_fault = err == -EFAULT ; + } perf_iterate_sb(perf_event_mmap_output, mmap_event,
Signature · 1 line
2.31.1
PATCH 2/10 perf: Move build_id_parse to check only regular files
In reply to Jiri OlsaMoving the build_id_parse function to place where it parses only regular files.
There's no change wrt maps we generate the build id for, because we get parsed build ids for only for regular files (non-special files) anyway.
--- kernel/events/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/events/core.c +6 -6
@@ -8336,6 +8336,12 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event) maj =MAJOR (dev); min =MINOR (dev); +if (atomic_read (&nr_build_id_events) && *name =='/' ) { +int err =build_id_parse (vma, mmap_event->build_id, + &mmap_event->build_id_size); + mmap_event->build_id_fault = err == -EFAULT ; + } + goto got_name; }else {if (vma->vm_ops && vma->vm_ops->name) {
@@ -8390,12 +8396,6 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event) mmap_event->event_id.header.size =sizeof (mmap_event->event_id) + size; -if (atomic_read (&nr_build_id_events)) { -int err =build_id_parse (vma, mmap_event->build_id, - &mmap_event->build_id_size); - mmap_event->build_id_fault = err == -EFAULT ; - } -perf_iterate_sb (perf_event_mmap_output, mmap_event,NULL );
Signature · 1 line
2.31.1
PATCH 3/10 perf: Add new read_format bit to read build id faults
In reply to Jiri OlsaIt's now possible to retrieve build id faults stats by read syscall on events with perf_event_attr::build_id set.
Adding PERF_FORMAT_BUILD_ID_FAULTS read_format bit to get the value of build_id_faults through the read syscall.
--- include/uapi/linux/perf_event.h | 17 ++++++++++------- kernel/events/core.c | 21 ++++++++++++++++++--- 2 files changed, 28 insertions(+), 10 deletions(-)
include/uapi/linux/perf_event.h +10 -7
@@ -296,16 +296,18 @@ enum {* * struct read_format { * { u64 value; -* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED -* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING -* { u64 id; } && PERF_FORMAT_ID +* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED +* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING +* { u64 id; } && PERF_FORMAT_ID +* { u64 build_id_faults; } && PERF_FORMAT_BUILD_ID_FAULTS * } && !PERF_FORMAT_GROUP * * { u64 nr; -* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED -* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING +* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED +* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING * { u64 value; -* { u64 id; } && PERF_FORMAT_ID +* { u64 id; } && PERF_FORMAT_ID +* { u64 build_id_faults; } && PERF_FORMAT_BUILD_ID_FAULTS * } cntr[nr]; * } && PERF_FORMAT_GROUP * };
@@ -315,8 +317,9 @@ enum perf_event_read_format {PERF_FORMAT_TOTAL_TIME_RUNNING =1U <<1 ,PERF_FORMAT_ID =1U <<2 ,PERF_FORMAT_GROUP =1U <<3 , +PERF_FORMAT_BUILD_ID_FAULTS =1U <<4 , - PERF_FORMAT_MAX = 1U << 4, /* non-ABI */ + PERF_FORMAT_MAX = 1U << 5, /* non-ABI */ };#define PERF_ATTR_SIZE_VER0 64/* sizeof first published struct */
kernel/events/core.c +18 -3
@@ -1840,6 +1840,9 @@ static void __perf_event_read_size(struct perf_event *event, int nr_siblings)if (event->attr.read_format &PERF_FORMAT_ID ) entry +=sizeof (u64); +if (event->attr.read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + entry +=sizeof (u64); +if (event->attr.read_format &PERF_FORMAT_GROUP ) { nr += nr_siblings; size +=sizeof (u64);
@@ -5247,11 +5250,15 @@ static int __perf_read_group_add(struct perf_event *leader, values[n++] +=perf_event_count (leader);if (read_format &PERF_FORMAT_ID ) values[n++] =primary_event_id (leader); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + values[n++] =local64_read (&leader->build_id_faults);for_each_sibling_event (sub, leader) { values[n++] +=perf_event_count (sub);if (read_format &PERF_FORMAT_ID ) values[n++] =primary_event_id (sub); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + values[n++] =local64_read (&sub->build_id_faults); }raw_spin_unlock_irqrestore (&ctx->lock, flags);
@@ -5308,7 +5315,7 @@ static int perf_read_one(struct perf_event *event, u64 read_format, char __user *buf) { u64 enabled, running; -u64 values[4 ]; +u64 values[5 ];int n =0 ; values[n++] =__perf_event_read_value (event, &enabled, &running);
@@ -5318,6 +5325,8 @@ static int perf_read_one(struct perf_event *event, values[n++] = running;if (read_format &PERF_FORMAT_ID ) values[n++] =primary_event_id (event); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + values[n++] =local64_read (&event->build_id_faults);if (copy_to_user (buf, values, n *sizeof (u64)))return -EFAULT ;
@@ -6820,7 +6829,7 @@ static void perf_output_read_one(struct perf_output_handle *handle, u64 enabled, u64 running) {u64 read_format = event->attr.read_format; -u64 values[4 ]; +u64 values[5 ];int n =0 ; values[n++] =perf_event_count (event);
@@ -6834,6 +6843,8 @@ static void perf_output_read_one(struct perf_output_handle *handle, }if (read_format &PERF_FORMAT_ID ) values[n++] =primary_event_id (event); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + values[n++] =local64_read (&event->build_id_faults);__output_copy (handle, values, n *sizeof (u64)); }
@@ -6844,7 +6855,7 @@ static void perf_output_read_group(struct perf_output_handle *handle, {struct perf_event *leader = event->group_leader, *sub;u64 read_format = event->attr.read_format; -u64 values[5 ]; +u64 values[6 ];int n =0 ; values[n++] =1 + leader->nr_siblings;
@@ -6862,6 +6873,8 @@ static void perf_output_read_group(struct perf_output_handle *handle, values[n++] =perf_event_count (leader);if (read_format &PERF_FORMAT_ID ) values[n++] =primary_event_id (leader); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + values[n++] =local64_read (&leader->build_id_faults);__output_copy (handle, values, n *sizeof (u64));
@@ -6875,6 +6888,8 @@ static void perf_output_read_group(struct perf_output_handle *handle, values[n++] =perf_event_count (sub);if (read_format &PERF_FORMAT_ID ) values[n++] =primary_event_id (sub); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + values[n++] =local64_read (&sub->build_id_faults);__output_copy (handle, values, n *sizeof (u64)); }
Signature · 1 line
2.31.1
PATCH 4/10 perf: Add new read_format bit to read lost events
In reply to Jiri OlsaIt's now possible to retrieve lost stats by read
syscall on events.
Adding PERF_FORMAT_LOST read_format bit to get the value of lost events through the read syscall.
--- include/linux/perf_event.h | 1 + include/uapi/linux/perf_event.h | 5 ++++- kernel/events/core.c | 21 ++++++++++++++++++--- kernel/events/ring_buffer.c | 3 +++ 4 files changed, 26 insertions(+), 4 deletions(-)
include/linux/perf_event.h +1 -0
@@ -670,6 +670,7 @@ struct perf_event {local64_t count;atomic64_t child_count;local64_t build_id_faults; +local64_t lost;/* * These are the total time in nanoseconds that the event
include/uapi/linux/perf_event.h +4 -1
@@ -300,6 +300,7 @@ enum {* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING * { u64 id; } && PERF_FORMAT_ID * { u64 build_id_faults; } && PERF_FORMAT_BUILD_ID_FAULTS +* { u64 lost; } && PERF_FORMAT_LOST * } && !PERF_FORMAT_GROUP * * { u64 nr;
@@ -308,6 +309,7 @@ enum {* { u64 value; * { u64 id; } && PERF_FORMAT_ID * { u64 build_id_faults; } && PERF_FORMAT_BUILD_ID_FAULTS +* { u64 lost; } && PERF_FORMAT_LOST * } cntr[nr]; * } && PERF_FORMAT_GROUP * };
@@ -318,8 +320,9 @@ enum perf_event_read_format {PERF_FORMAT_ID =1U <<2 ,PERF_FORMAT_GROUP =1U <<3 ,PERF_FORMAT_BUILD_ID_FAULTS =1U <<4 , +PERF_FORMAT_LOST =1U <<5 , - PERF_FORMAT_MAX = 1U << 5, /* non-ABI */ + PERF_FORMAT_MAX = 1U << 6, /* non-ABI */ };#define PERF_ATTR_SIZE_VER0 64/* sizeof first published struct */
kernel/events/core.c +18 -3
@@ -1843,6 +1843,9 @@ static void __perf_event_read_size(struct perf_event *event, int nr_siblings)if (event->attr.read_format &PERF_FORMAT_BUILD_ID_FAULTS ) entry +=sizeof (u64); +if (event->attr.read_format &PERF_FORMAT_LOST ) + entry +=sizeof (u64); +if (event->attr.read_format &PERF_FORMAT_GROUP ) { nr += nr_siblings; size +=sizeof (u64);
@@ -5252,6 +5255,8 @@ static int __perf_read_group_add(struct perf_event *leader, values[n++] =primary_event_id (leader);if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) values[n++] =local64_read (&leader->build_id_faults); +if (read_format &PERF_FORMAT_LOST ) + values[n++] =local64_read (&leader->lost);for_each_sibling_event (sub, leader) { values[n++] +=perf_event_count (sub);
@@ -5259,6 +5264,8 @@ static int __perf_read_group_add(struct perf_event *leader, values[n++] =primary_event_id (sub);if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) values[n++] =local64_read (&sub->build_id_faults); +if (read_format &PERF_FORMAT_LOST ) + values[n++] =local64_read (&sub->lost); }raw_spin_unlock_irqrestore (&ctx->lock, flags);
@@ -5315,7 +5322,7 @@ static int perf_read_one(struct perf_event *event, u64 read_format, char __user *buf) { u64 enabled, running; -u64 values[5 ]; +u64 values[6 ];int n =0 ; values[n++] =__perf_event_read_value (event, &enabled, &running);
@@ -5327,6 +5334,8 @@ static int perf_read_one(struct perf_event *event, values[n++] =primary_event_id (event);if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) values[n++] =local64_read (&event->build_id_faults); +if (read_format &PERF_FORMAT_LOST ) + values[n++] =local64_read (&event->lost);if (copy_to_user (buf, values, n *sizeof (u64)))return -EFAULT ;
@@ -6829,7 +6838,7 @@ static void perf_output_read_one(struct perf_output_handle *handle, u64 enabled, u64 running) {u64 read_format = event->attr.read_format; -u64 values[5 ]; +u64 values[6 ];int n =0 ; values[n++] =perf_event_count (event);
@@ -6845,6 +6854,8 @@ static void perf_output_read_one(struct perf_output_handle *handle, values[n++] =primary_event_id (event);if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) values[n++] =local64_read (&event->build_id_faults); +if (read_format &PERF_FORMAT_LOST ) + values[n++] =local64_read (&event->lost);__output_copy (handle, values, n *sizeof (u64)); }
@@ -6855,7 +6866,7 @@ static void perf_output_read_group(struct perf_output_handle *handle, {struct perf_event *leader = event->group_leader, *sub;u64 read_format = event->attr.read_format; -u64 values[6 ]; +u64 values[7 ];int n =0 ; values[n++] =1 + leader->nr_siblings;
@@ -6875,6 +6886,8 @@ static void perf_output_read_group(struct perf_output_handle *handle, values[n++] =primary_event_id (leader);if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) values[n++] =local64_read (&leader->build_id_faults); +if (read_format &PERF_FORMAT_LOST ) + values[n++] =local64_read (&leader->lost);__output_copy (handle, values, n *sizeof (u64));
@@ -6890,6 +6903,8 @@ static void perf_output_read_group(struct perf_output_handle *handle, values[n++] =primary_event_id (sub);if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) values[n++] =local64_read (&sub->build_id_faults); +if (read_format &PERF_FORMAT_LOST ) + values[n++] =local64_read (&sub->lost);__output_copy (handle, values, n *sizeof (u64)); }
kernel/events/ring_buffer.c +3 -0
@@ -248,6 +248,9 @@ __perf_output_begin(struct perf_output_handle *handle,perf_event_header__init_id (&lost_event.header, data, event);perf_output_put (handle, lost_event);perf_event__output_id_sample (event, handle, data); + +/* Keep track of lost events in event for PERF_FORMAT_LOST */ +local64_add (lost_event.lost, &event->lost); }return 0 ;
Signature · 1 line
2.31.1
PATCH 5/10 tools: Sync perf_event.h uapi
In reply to Jiri OlsaUpdate tools uapi headers with latest changes.
--- tools/include/uapi/linux/perf_event.h | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)
tools/include/uapi/linux/perf_event.h +13 -7
@@ -296,16 +296,20 @@ enum {* * struct read_format { * { u64 value; -* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED -* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING -* { u64 id; } && PERF_FORMAT_ID +* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED +* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING +* { u64 id; } && PERF_FORMAT_ID +* { u64 build_id_faults; } && PERF_FORMAT_BUILD_ID_FAULTS +* { u64 lost; } && PERF_FORMAT_LOST * } && !PERF_FORMAT_GROUP * * { u64 nr; -* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED -* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING +* { u64 time_enabled; } && PERF_FORMAT_TOTAL_TIME_ENABLED +* { u64 time_running; } && PERF_FORMAT_TOTAL_TIME_RUNNING * { u64 value; -* { u64 id; } && PERF_FORMAT_ID +* { u64 id; } && PERF_FORMAT_ID +* { u64 build_id_faults; } && PERF_FORMAT_BUILD_ID_FAULTS +* { u64 lost; } && PERF_FORMAT_LOST * } cntr[nr]; * } && PERF_FORMAT_GROUP * };
@@ -315,8 +319,10 @@ enum perf_event_read_format {PERF_FORMAT_TOTAL_TIME_RUNNING =1U <<1 ,PERF_FORMAT_ID =1U <<2 ,PERF_FORMAT_GROUP =1U <<3 , +PERF_FORMAT_BUILD_ID_FAULTS =1U <<4 , +PERF_FORMAT_LOST =1U <<5 , - PERF_FORMAT_MAX = 1U << 4, /* non-ABI */ + PERF_FORMAT_MAX = 1U << 6, /* non-ABI */ };#define PERF_ATTR_SIZE_VER0 64/* sizeof first published struct */
Signature · 1 line
2.31.1
PATCH 6/10 libperf: Do not allow PERF_FORMAT_GROUP in perf_evsel__read
In reply to Jiri OlsaThe struct perf_counts_values can hold only first three read_format bits. We can't allow PERF_FORMAT_GROUP, otherwise we get segfault.
--- tools/lib/perf/evsel.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/lib/perf/evsel.c +4 -0
@@ -260,8 +260,12 @@ int perf_evsel__read_size(struct perf_evsel *evsel)int perf_evsel__read (struct perf_evsel *evsel,int cpu,int thread,struct perf_counts_values *count) { +u64 read_format = evsel->attr.read_format;size_t size =perf_evsel__read_size (evsel); +if (read_format &PERF_FORMAT_GROUP ) +return -EINVAL ; +memset (count,0 ,sizeof (*count));if (FD (evsel, cpu, thread) <0 )
Signature · 1 line
2.31.1
PATCH 7/10 perf record: Add support to read build id fails
In reply to Jiri OlsaAdding support to read build id fails and lost events count and display it at the end of the record session, when recording with --buildid-mmap.
# perf record ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]
--- tools/lib/perf/evsel.c | 6 ++ tools/lib/perf/include/perf/evsel.h | 11 ++- tools/perf/builtin-record.c | 90 +++++++++++++++++++++++ tools/perf/util/evsel.c | 12 +++ tools/perf/util/perf_event_attr_fprintf.c | 3 +- 5 files changed, 120 insertions(+), 2 deletions(-)
tools/lib/perf/evsel.c +6 -0
@@ -248,6 +248,12 @@ int perf_evsel__read_size(struct perf_evsel *evsel)if (read_format &PERF_FORMAT_ID ) entry +=sizeof (u64); +if (read_format &PERF_FORMAT_BUILD_ID_FAULTS ) + entry +=sizeof (u64); + +if (read_format &PERF_FORMAT_LOST ) + entry +=sizeof (u64); +if (read_format &PERF_FORMAT_GROUP ) { nr = evsel->nr_members; size +=sizeof (u64);
tools/lib/perf/include/perf/evsel.h +10 -1
@@ -12,12 +12,21 @@ struct perf_thread_map;struct perf_counts_values {union { +/* Struct for specific perf interfaces. */ struct {uint64_t val;uint64_t ena;uint64_t run; }; -uint64_t values[3 ]; +/* +* Values to store all non-group data: +* PERF_FORMAT_TOTAL_TIME_ENABLED +* PERF_FORMAT_TOTAL_TIME_RUNNING +* PERF_FORMAT_ID +* PERF_FORMAT_BUILD_ID_FAULTS +* PERF_FORMAT_LOST +*/ +uint64_t values[6 ]; }; };
tools/perf/builtin-record.c +90 -0
108 lines not on this page · raw message
tools/perf/util/evsel.c +12 -0
@@ -1239,6 +1239,18 @@ void evsel__config(struct evsel *evsel, struct record_opts *opts,PERF_FORMAT_TOTAL_TIME_RUNNING ; } +/* +* We skip post processing for build_id, so we need +* to read stats via read syscall: +* - faults for mmap events +* - lost for each event +*/ +if (attr->build_id) + attr->read_format |=PERF_FORMAT_BUILD_ID_FAULTS ; + +if (opts->build_id) + attr->read_format |=PERF_FORMAT_LOST ; +/* * XXX see the function comment above *
tools/perf/util/perf_event_attr_fprintf.c +2 -1
@@ -64,7 +64,8 @@ static void __p_read_format(char *buf, size_t size, u64 value)#define bit_name (n) { PERF_FORMAT_##n, #n }struct bit_names bits[] = {bit_name (TOTAL_TIME_ENABLED ),bit_name (TOTAL_TIME_RUNNING ), -bit_name (ID ),bit_name (GROUP ), +bit_name (ID ),bit_name (GROUP ),bit_name (BUILD_ID_FAULTS ), +bit_name (LOST ), { .name =NULL , } };#undef bit_name
Signature · 1 line
2.31.1
PATCH 8/10 perf record: Add new HEADER_BUILD_ID_MMAP feature
In reply to Jiri OlsaAdding new HEADER_BUILD_ID_MMAP feature to store faulst/lost/fixed counts for --buildid-mmap setup. We skip post processing for build_id, so we need to store session stats.
The feature data has following format:
struct {
u32 version;
u64 faults;
u64 lost;
u64 fixed;
};The version is set to 1.
The faults has the value of faulted build id retrievals for the session. The lost has the value of faulted lost events for the session. The fixed has the value of fixed build ids by post-processing.
The perf report --header-only display for when fixes is 0:
# build id mmap stats: FAULTS 4, LOST 0, NOT FIXED
If fixed is defined:
# build id mmap stats: FAULTS 4, LOST 0, FIXED(4)
--- .../Documentation/perf.data-file-format.txt | 19 +++++ tools/perf/builtin-record.c | 7 ++ tools/perf/util/env.h | 6 ++ tools/perf/util/header.c | 80 +++++++++++++++++++ tools/perf/util/header.h | 1 + 5 files changed, 113 insertions(+)
tools/perf/Documentation/perf.data-file-format.txt +19 -0
@@ -438,6 +438,25 @@ struct { other bits are reserved and should ignored for now HEADER_FEAT_BITS = 256, + HEADER_BUILD_ID_MMAP = 32, + + It contains stats values for session with --buildid-mmap option. + +struct { + u32 version; + u64 faults; + u64 lost; + u64 fixed; +}; + + The version is set to 1. + The faults has the value of faulted build id retrievals for the session. + The lost has the value of faulted lost events for the session. + The fixed has the value of fixed build ids by post-processing. + + other bits are reserved and should ignored for now + HEADER_FEAT_BITS = 256, + Attributes This is an array of perf_event_attrs, each attr_size bytes long, which defines
tools/perf/builtin-record.c +7 -0
@@ -1221,6 +1221,9 @@ static void record__init_features(struct record *rec)if (!rec->opts.use_clockid)perf_header__clear_feat (&session->header,HEADER_CLOCK_DATA ); +if (!rec->buildid_mmap) +perf_header__clear_feat (&session->header,HEADER_BUILD_ID_MMAP ); +perf_header__clear_feat (&session->header,HEADER_DIR_FORMAT );if (!record__comp_enabled (rec))perf_header__clear_feat (&session->header,HEADER_COMPRESSED );
@@ -1296,6 +1299,7 @@ evlist__read_session_stats(struct evlist *evlist, struct session_stats *st)static void read_session_stats (struct record *rec) { +struct perf_session *session = rec->session;struct session_stats st;if (evlist__read_session_stats (rec->evlist, &st))
@@ -1310,6 +1314,9 @@ static void read_session_stats(struct record *rec)fprintf (stderr,"[ perf record: Lost %lu chunks]\n" , st.lost); } + + session->header.env.build_id_mmap.faults = st.build_id_faults; + session->header.env.build_id_mmap.lost = st.lost; } static void
tools/perf/util/env.h +6 -0
@@ -128,6 +128,12 @@ struct perf_env {*/ bool enabled; } clock; + +struct { +u64 faults; +u64 lost; +u64 fixed; + } build_id_mmap; };enum perf_compress_type {
tools/perf/util/header.c +80 -0
104 lines not on this page · raw message
tools/perf/util/header.h +1 -0
@@ -47,6 +47,7 @@ enum {HEADER_CLOCK_DATA ,HEADER_HYBRID_TOPOLOGY ,HEADER_HYBRID_CPU_PMU_CAPS , +HEADER_BUILD_ID_MMAP ,HEADER_LAST_FEATURE , HEADER_FEAT_BITS = 256, };
Signature · 1 line
2.31.1
PATCH 9/10 perf report: Display build id fails stats
In reply to Jiri OlsaAdding support to display build id fails in --stats option:
# perf report --stat
Aggregated stats:
TOTAL events: 104
COMM events: 2 ( 1.9%)
....
BUILD_ID fails: 4 (14.3%)
This stat is displayed only for session recorded with --buildid-mmap and contains HEADER_BUILD_ID_MMAP header feature.
We process all MMAP2 events and in case it does not contain build id and it should - it's regular file, we count the BUILD_ID fail and display it.
--- tools/perf/builtin-report.c | 35 +++++++++++++++++++++++++++++++++++ tools/perf/util/map.h | 15 +++++++++++++++ 2 files changed, 50 insertions(+)
tools/perf/builtin-report.c +35 -0
@@ -13,6 +13,7 @@#include "util/annotate.h" #include "util/color.h" #include "util/dso.h" +#include "util/vdso.h" #include <linux/list.h> #include <linux/rbtree.h> #include <linux/err.h>
@@ -100,6 +101,8 @@ struct report {u64 nr_entries;u64 queue_size;u64 total_cycles; +u64 buildid_fails; +u64 buildid_total;int socket_filter;DECLARE_BITMAP (cpu_bitmap,MAX_NR_CPUS );struct branch_type_stat brtype_stat;
@@ -729,10 +732,36 @@ static int count_sample_event(struct perf_tool *tool __maybe_unused,return 0 ; } +static int count_buildid_fails (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample__maybe_unused , +struct machine *machine__maybe_unused ) +{ + struct report *rep = container_of(tool, struct report, tool); +struct perf_record_mmap2 *mmap2 = &event->mmap2; + +/* No build id should be generated */ +if (!is_buildid_memory (mmap2->filename)) +return 0 ; + + rep->buildid_total++; + +/* The build id should be generated, but wasn't - fault. */ +if (!(mmap2->header.misc &PERF_RECORD_MISC_MMAP_BUILD_ID )) + rep->buildid_fails++; + +return 0 ; +} +static void stats_setup (struct report *rep) {memset (&rep->tool,0 ,sizeof (rep->tool)); rep->tool.sample = count_sample_event; + +if (perf_header__has_feat (&rep->session->header, +HEADER_BUILD_ID_MMAP )) + rep->tool.mmap2 = count_buildid_fails; + rep->tool.no_warn = true; }
@@ -742,6 +771,12 @@ static int stats_print(struct report *rep)perf_session__fprintf_nr_events (session, stdout, rep->skip_empty);evlist__fprintf_nr_events (session->evlist, stdout, rep->skip_empty); + +if (rep->buildid_fails) { +fprintf (stdout,"%23s: %10" PRIu64" (%4.1f%%)\n" ,"BUILD_ID fails" , + rep->buildid_fails, +100.0 * rep->buildid_fails / rep->buildid_total); + }return 0 ; }
tools/perf/util/map.h +15 -0
@@ -10,6 +10,7 @@#include <string.h> #include <stdbool.h> #include <linux/types.h> +#include "vdso.h" struct dso ;struct maps ;
@@ -186,4 +187,18 @@ static inline int is_no_dso_memory(const char *filename) !strncmp (filename,"/SYSV" ,5 ) || !strcmp(filename, "[heap]"); } + +static inline int is_vsyscall_memory (const char *filename) +{ +return !strcmp (filename,"[vsyscall]" ); +} + +static inline int is_buildid_memory (const char *filename) +{ +return !is_anon_memory (filename) && + !is_vdso_map (filename) && + !is_no_dso_memory (filename) && + !is_vsyscall_memory (filename); +} +#endif /* __PERF_MAP_H */
Signature · 1 line
2.31.1
PATCH 10/10 perf inject: Add --buildid-mmap2 option to fix failed build ids
In reply to Jiri OlsaAdding --buildid-mmap2 option that tried to fix failed build ids in mmap2 events.
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]
Check if there's only build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXED
There is, check the stats:
# perf report --stat
Aggregated stats:
TOTAL events: 104
....
BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.data
And verify:
# perf report -i perf-fixed.data --stats
Aggregated stats:
TOTAL events: 104
....
Good, let's see how many we fixed:
# perf report --header-only -i perf-fixed.data ... # build id mmap stats: FAULTS 4, LOST 0, FIXED(4)
--- tools/perf/Documentation/perf-inject.txt | 3 ++ tools/perf/builtin-inject.c | 45 ++++++++++++++++++++++-- 2 files changed, 46 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-inject.txt +3 -0
@@ -30,6 +30,9 @@ OPTIONS --buildid-all: Inject build-ids of all DSOs into the output stream +--buildid-mmap2: + Resolve failed buildids in MMAP2 events. + -v:: --verbose:: Be more verbose.
tools/perf/builtin-inject.c +43 -2
94 lines not on this page · raw message
Signature · 1 line
2.31.1
Arnaldo Carvalho de Melo
maintainerEm Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --stat
Quoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.data
Can we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
- Arnaldo
Quoted 63 lines
And verify:
# perf report -i perf-fixed.data --statsAggregated stats:
TOTAL events: 104
....Good, let's see how many we fixed:
# perf report --header-only -i perf-fixed.data ... # build id mmap stats: FAULTS 4, LOST 0, FIXED(4)I don't have a good way to test it, just by artificially adding the faults in kernel code, but Ian and Namhyung might have setup that could generate that.. would be great to have a perf test for this.
Also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/buildid_statsthoughts?
thanks,
jirka---
Jiri Olsa (10):
perf: Track build id faults for mmap2 event
perf: Movebuild_id_parseto check only regular files
perf: Add newread_formatbit to read build id faults
perf: Add newread_formatbit to read lost events
tools: Syncperf_event.huapi
libperf: Do not allowPERF_FORMAT_GROUPinperf_evsel__readperf record: Add support to read build id fails
perf record: Add newHEADER_BUILD_ID_MMAPfeature perf report: Display build id fails stats
perf inject: Add --buildid-mmap2 option to fix failed build idsinclude/linux/perf_event.h | 2 ++ include/uapi/linux/perf_event.h | 20 +++++++++++++------- kernel/events/core.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ kernel/events/ring_buffer.c | 3 +++ tools/include/uapi/linux/perf_event.h | 20 +++++++++++++------- tools/lib/perf/evsel.c | 10 ++++++++++ tools/lib/perf/include/perf/evsel.h | 11 ++++++++++- tools/perf/Documentation/perf-inject.txt | 3 +++ tools/perf/Documentation/perf.data-file-format.txt | 19 +++++++++++++++++++ tools/perf/builtin-inject.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- tools/perf/builtin-record.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin-report.c | 35 +++++++++++++++++++++++++++++++++++ tools/perf/util/env.h | 6 ++++++ tools/perf/util/evsel.c | 12 ++++++++++++ tools/perf/util/header.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 1 + tools/perf/util/map.h | 15 +++++++++++++++ tools/perf/util/perf_event_attr_fprintf.c | 3 ++- 18 files changed, 407 insertions(+), 24 deletions(-)
Signature · 1 line
- Arnaldo
Ian Rogers
reviewerQuoted 49 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
Quoted 3 lines
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
Firstly thanks for the patches! To Arnaldo's sideband idea, I wonder if we have a thread doing sideband buildid generation whether the same thread or threads could also be doing the synthesis job. Perhaps such work could be postponed until when the session closes, like with tail synthesis. It's a particular shame with tail synthesis that we synthesize mmap events for processes with no samples.
Thanks,
Ian
Quoted 69 lines
- Arnaldo
Quoted 54 lines
And verify:
# perf report -i perf-fixed.data --statsAggregated stats:
TOTAL events: 104
....Good, let's see how many we fixed:
# perf report --header-only -i perf-fixed.data ... # build id mmap stats: FAULTS 4, LOST 0, FIXED(4)I don't have a good way to test it, just by artificially adding the faults in kernel code, but Ian and Namhyung might have setup that could generate that.. would be great to have a perf test for this.
Also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/buildid_statsthoughts?
thanks,
jirka---
Jiri Olsa (10):
perf: Track build id faults for mmap2 event
perf: Movebuild_id_parseto check only regular files
perf: Add newread_formatbit to read build id faults
perf: Add newread_formatbit to read lost events
tools: Syncperf_event.huapi
libperf: Do not allowPERF_FORMAT_GROUPinperf_evsel__readperf record: Add support to read build id fails
perf record: Add newHEADER_BUILD_ID_MMAPfeature perf report: Display build id fails stats
perf inject: Add --buildid-mmap2 option to fix failed build idsinclude/linux/perf_event.h | 2 ++ include/uapi/linux/perf_event.h | 20 +++++++++++++------- kernel/events/core.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ kernel/events/ring_buffer.c | 3 +++ tools/include/uapi/linux/perf_event.h | 20 +++++++++++++------- tools/lib/perf/evsel.c | 10 ++++++++++ tools/lib/perf/include/perf/evsel.h | 11 ++++++++++- tools/perf/Documentation/perf-inject.txt | 3 +++ tools/perf/Documentation/perf.data-file-format.txt | 19 +++++++++++++++++++ tools/perf/builtin-inject.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- tools/perf/builtin-record.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Quoted 9 lines
tools/perf/builtin-report.c | 35 +++++++++++++++++++++++++++++++++++ tools/perf/util/env.h | 6 ++++++ tools/perf/util/evsel.c | 12 ++++++++++++ tools/perf/util/header.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 1 + tools/perf/util/map.h | 15 +++++++++++++++ tools/perf/util/perf_event_attr_fprintf.c | 3 ++- 18 files changed, 407 insertions(+), 24 deletions(-)--
- Arnaldo
Arnaldo Carvalho de Melo
maintainerEm Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 51 lines
Quoted 49 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
Quoted 8 lines
Quoted 3 lines
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
Firstly thanks for the patches! To Arnaldo's sideband idea, I wonder if we have a thread doing sideband buildid generation whether the same thread or threads could also be doing the synthesis job. Perhaps such work could be postponed until when the session closes, like with tail
I didn't suggest synthesizing the failed build-ids in the sideband thread, just receiving the MMAP2 records for the build-ids that faulted.
It may be interesting to do it right away, to avoid building up a potentially large number of entries to do at the end, but if this is something uncommon, with just a few entries, then leaving it for after the workload finishes may be a good idea.
Or perhaps this needs to be a knob, since for long running sessions such as with 'perf daemon' the "workload" may never end, so we better flush these things as the files where we'll get it from may go away.
Quoted 2 lines
synthesis. It's a particular shame with tail synthesis that we synthesize mmap events for processes with no samples.
Sure, but it is also very costly to process a potentially large perf.data file for looking at what MMAPs have samples. That is the raison d'être for PERF_RECORD_MMAP2 to carry build-ids :-)
I.e. there are pros and cons for tail synthesis, for looking at all samples to generate only build-ids for MMAPs with hits, for synthesizing it in the sideband thread immediately, for leaving this generation to be done at the end by traversing the list of MMAP records without build-ids, etc.
- Arnaldo
Quoted 72 lines
Thanks,
IanQuoted 69 lines
- Arnaldo
Quoted 54 lines
And verify:
# perf report -i perf-fixed.data --statsAggregated stats:
TOTAL events: 104
....Good, let's see how many we fixed:
# perf report --header-only -i perf-fixed.data ... # build id mmap stats: FAULTS 4, LOST 0, FIXED(4)I don't have a good way to test it, just by artificially adding the faults in kernel code, but Ian and Namhyung might have setup that could generate that.. would be great to have a perf test for this.
Also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/buildid_statsthoughts?
thanks,
jirka---
Jiri Olsa (10):
perf: Track build id faults for mmap2 event
perf: Movebuild_id_parseto check only regular files
perf: Add newread_formatbit to read build id faults
perf: Add newread_formatbit to read lost events
tools: Syncperf_event.huapi
libperf: Do not allowPERF_FORMAT_GROUPinperf_evsel__readperf record: Add support to read build id fails
perf record: Add newHEADER_BUILD_ID_MMAPfeature perf report: Display build id fails stats
perf inject: Add --buildid-mmap2 option to fix failed build idsinclude/linux/perf_event.h | 2 ++ include/uapi/linux/perf_event.h | 20 +++++++++++++------- kernel/events/core.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ kernel/events/ring_buffer.c | 3 +++ tools/include/uapi/linux/perf_event.h | 20 +++++++++++++------- tools/lib/perf/evsel.c | 10 ++++++++++ tools/lib/perf/include/perf/evsel.h | 11 ++++++++++- tools/perf/Documentation/perf-inject.txt | 3 +++ tools/perf/Documentation/perf.data-file-format.txt | 19 +++++++++++++++++++ tools/perf/builtin-inject.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- tools/perf/builtin-record.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Quoted 9 lines
tools/perf/builtin-report.c | 35 +++++++++++++++++++++++++++++++++++ tools/perf/util/env.h | 6 ++++++ tools/perf/util/evsel.c | 12 ++++++++++++ tools/perf/util/header.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 1 + tools/perf/util/map.h | 15 +++++++++++++++ tools/perf/util/perf_event_attr_fprintf.c | 3 ++- 18 files changed, 407 insertions(+), 24 deletions(-)--
- Arnaldo
Signature · 1 line
- Arnaldo
Quoted 55 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 54 lines
Quoted 52 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
right, that might be good to decide first.. because as I said, I never hit faulted build id, so it probably needs the special setup you guys are using.. could you try on your setup and check how many faulted build ids you see?
thanks,
jirka
Namhyung Kim
maintainerHi Jiri,
Thanks for your work!
Quoted 54 lines · Jiri Olsa
Quoted 52 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 51 lines
Quoted 49 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
right, that might be good to decide first.. because as I said, I never hit faulted build id, so it probably needs the special setup you guys are using.. could you try on your setup and check how many faulted build ids you see?
Did you check data mmaps? It might be easy to get faults from data files and we don't know if it's an ELF or not before reading the ELF header in the first page.
I'm not sure if we can limit it to exec mappings, there might be data-only DSOs and we may want to symbolize them too.
Thanks,
Namhyung
Namhyung Kim
maintainerQuoted 28 lines · Jiri Olsa
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
Quoted 8 lines
# perf report --statAggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
Depending on the failure, it might not need to be fixed. Say, one process mmapped a file A and succeeded. And then another process mmaped the same file A, but it failed to get a build-id (mmap itself was ok). And vice versa is fine too.
So even if it sees failures, we didn't lose anything.
Quoted 2 lines
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataNot sure this is really needed since perf inject -j can add BUILD_ID records without fixing MMAP2.
Quoted 64 lines
And verify:
# perf report -i perf-fixed.data --statsAggregated stats:
TOTAL events: 104
....Good, let's see how many we fixed:
# perf report --header-only -i perf-fixed.data ... # build id mmap stats: FAULTS 4, LOST 0, FIXED(4)I don't have a good way to test it, just by artificially adding the faults in kernel code, but Ian and Namhyung might have setup that could generate that.. would be great to have a perf test for this.
Also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/buildid_statsthoughts?
thanks,
jirka---
Jiri Olsa (10):
perf: Track build id faults for mmap2 event
perf: Movebuild_id_parseto check only regular files
perf: Add newread_formatbit to read build id faults
perf: Add newread_formatbit to read lost events
tools: Syncperf_event.huapi
libperf: Do not allowPERF_FORMAT_GROUPinperf_evsel__readperf record: Add support to read build id fails
perf record: Add newHEADER_BUILD_ID_MMAPfeature perf report: Display build id fails stats
perf inject: Add --buildid-mmap2 option to fix failed build idsinclude/linux/perf_event.h | 2 ++ include/uapi/linux/perf_event.h | 20 +++++++++++++------- kernel/events/core.c | 49 +++++++++++++++++++++++++++++++++++++++++++------ kernel/events/ring_buffer.c | 3 +++ tools/include/uapi/linux/perf_event.h | 20 +++++++++++++------- tools/lib/perf/evsel.c | 10 ++++++++++ tools/lib/perf/include/perf/evsel.h | 11 ++++++++++- tools/perf/Documentation/perf-inject.txt | 3 +++ tools/perf/Documentation/perf.data-file-format.txt | 19 +++++++++++++++++++ tools/perf/builtin-inject.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- tools/perf/builtin-record.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin-report.c | 35 +++++++++++++++++++++++++++++++++++ tools/perf/util/env.h | 6 ++++++ tools/perf/util/evsel.c | 12 ++++++++++++ tools/perf/util/header.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 1 + tools/perf/util/map.h | 15 +++++++++++++++ tools/perf/util/perf_event_attr_fprintf.c | 3 ++- 18 files changed, 407 insertions(+), 24 deletions(-)
Quoted 63 lines · Namhyung Kim
Hi Jiri,
Thanks for your work!
Quoted 58 lines · Jiri Olsa
Quoted 55 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 54 lines
Quoted 52 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
Quoted 8 lines
Quoted 4 lines
right, that might be good to decide first.. because as I said, I never hit faulted build id, so it probably needs the special setup you guys are using.. could you try on your setup and check how many faulted build ids you see?
Did you check data mmaps? It might be easy to get faults from data files and we don't know if it's an ELF or not before reading the ELF header in the first page.
Hi. Long ago, I have noticed samples pointing to purely data files, such as if the program execution was sampled just in the middle of them. However, these files couldn't certainly contain any executable code... It was quite hard to reproduce this.
Maybe what Namhyung says might have been a culprit for it? Just an idea...
Michael
Quoted 8 lines
I'm not sure if we can limit it to exec mappings, there might be data-only DSOs and we may want to symbolize them too.
Thanks,
Namhyung
Quoted 63 lines · Namhyung Kim
Hi Jiri,
Thanks for your work!
Quoted 58 lines · Jiri Olsa
Quoted 55 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 54 lines
Quoted 52 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
Quoted 8 lines
Quoted 4 lines
right, that might be good to decide first.. because as I said, I never hit faulted build id, so it probably needs the special setup you guys are using.. could you try on your setup and check how many faulted build ids you see?
Did you check data mmaps? It might be easy to get faults from data files and we don't know if it's an ELF or not before reading the ELF header in the first page.
well, AFAICS the mmap event is sent right after the elf file is loaded, so it does not have a chance to be swapped out
Quoted 3 lines
I'm not sure if we can limit it to exec mappings, there might be data-only DSOs and we may want to symbolize them too.
hum, I haven't checked the data-only DSO, which we'd load manually, not the loader.. will check
thanks,
jirka
Quoted 73 lines · Michael Petlan
Quoted 63 lines · Namhyung Kim
Hi Jiri,
Thanks for your work!
Quoted 58 lines · Jiri Olsa
Quoted 55 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 54 lines
Quoted 52 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
Quoted 8 lines
Quoted 4 lines
right, that might be good to decide first.. because as I said, I never hit faulted build id, so it probably needs the special setup you guys are using.. could you try on your setup and check how many faulted build ids you see?
Did you check data mmaps? It might be easy to get faults from data files and we don't know if it's an ELF or not before reading the ELF header in the first page.
Quoted 7 lines
Hi. Long ago, I have noticed samples pointing to purely data files, such as if the program execution was sampled just in the middle of them. However, these files couldn't certainly contain any executable code... It was quite hard to reproduce this.
Maybe what Namhyung says might have been a culprit for it? Just an idea...
yea, that sounds bad.. I guess you can no longer reproduce?
thanks,
jirka
Namhyung Kim
maintainerQuoted 70 lines · Jiri Olsa
Quoted 68 lines · Namhyung Kim
Hi Jiri,
Thanks for your work!
Quoted 62 lines · Jiri Olsa
Quoted 55 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 10:47:54AM -0700, Ian Rogers escreveu:
Quoted 54 lines
Quoted 52 lines · Arnaldo Carvalho de Melo
Em Tue, Jun 22, 2021 at 05:39:08PM +0200, Jiri Olsa escreveu:
Quoted 29 lines
hi,
this *RFC* patchset adds support to detect faults during mmap2's build id parsing and a way to fix such maps in generated perf.data.It adds support to record build id faults count for session and store it in perf.data and perf inject support to find these maps and reads build ids for them in user space.
It's probably best explained by the workflow:
Record data with --buildid-mmap option:
# perf record --buildid-mmap ... ... [ perf record: Woken up 1 times to write data ] [ perf record: Failed to parse 4 build ids] [ perf record: Captured and wrote 0.008 MB perf.data ]Check if there's any build id fault reported:
# perf report --header-only ... # build id mmap stats: FAULTS 4, LOST 0, NOT FIXEDThere is, check the stats:
# perf report --statQuoted 8 lines
Aggregated stats: TOTAL events: 104 .... BUILD_ID fails: 4 (14.3%)Yep, let's fix it:
# perf inject --buildid-mmap2 -i perf.data -o perf-fixed.dataCan we make it possible to automate this with --fixup-buildids or a perfconfig 'record' knob?
This would entail requesting that build-ids that _fail_ be sent to the side-band thread we have in 'perf record', this way we wouldn't have to traverse the whole perf.data file, be it with 'perf-record' at the end of a session with faulty build ids, or in a similar fashion using 'perf inject' as you suggest.
I even think that we can have all these modes and let the user to decide how important is this for them and how convenient they want the whole process to be.
right, that might be good to decide first.. because as I said, I never hit faulted build id, so it probably needs the special setup you guys are using.. could you try on your setup and check how many faulted build ids you see?
Quoted 6 lines
Quoted 3 lines
Did you check data mmaps? It might be easy to get faults from data files and we don't know if it's an ELF or not before reading the ELF header in the first page.
well, AFAICS the mmap event is sent right after the elf file is loaded, so it does not have a chance to be swapped out
I'm talking about the normal data files when we use perf record -d. Those mmap files might not have page 0 in memory. I'm afraid it's reported as a build-id fault.
Thanks,
Namhyung