perf,tools: add time out to force stop proc map processing
PATCH v3 1/2 perf,tools: add time out to force stop proc map processing
From: Kan Liang <[email protected]>
System wide sampling like 'perf top' or 'perf record -a' read all threads /proc/xxx/maps before sampling. If there are any threads which generating a keeping growing huge maps, perf will do infinite loop during synthesizing. Nothing will be sampled.
This patch fixes this issue by adding per-thread timeout to force stop this kind of endless proc map processing. PERF_RECORD_MISC_MMAP_TIME_OUT is introduced to indicate that the mmap record are truncated by time out. User will get warning notification when truncated mmap records are detected.
---
Changes since V1
- Add warning message for time out.
Changes since V2
- Handle time out warning in perf_session__warn_about_errors
- Configurable time out (patch 2/2)
include/uapi/linux/perf_event.h | 4 ++++ tools/perf/util/event.c | 18 ++++++++++++++++++ tools/perf/util/event.h | 1 + tools/perf/util/session.c | 11 +++++++++++ 4 files changed, 34 insertions(+)
include/uapi/linux/perf_event.h +4 -0
@@ -566,6 +566,10 @@ struct perf_event_mmap_page {#define PERF_RECORD_MISC_GUEST_USER (5 << 0)/* +* Indicates that /proc/PID/maps parsing are truncated by time out. +*/ +#define PERF_RECORD_MISC_MMAP_TIME_OUT (1 << 12) +/* * PERF_RECORD_MISC_MMAP_DATA and PERF_RECORD_MISC_COMM_EXEC are used on * different events so can reuse the same bit position. */
tools/perf/util/event.c +18 -0
@@ -213,6 +213,8 @@ static int perf_event__synthesize_fork(struct perf_tool *tool,return 0 ; } +#define MMAP_TIMEOUT (50 * 1000000ULL) + int perf_event__synthesize_mmap_events(struct perf_tool *tool, union perf_event *event, pid_t pid, pid_t tgid,
@@ -222,6 +224,8 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool, {char filename[PATH_MAX ];FILE *fp; +unsigned long long t; +bool timeout = false;int rc =0 ;if (machine__is_default_guest (machine))
@@ -240,6 +244,7 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool, } event->header.type =PERF_RECORD_MMAP2 ; + t =rdclock ();while (1 ) {char bf[BUFSIZ ];
@@ -253,6 +258,12 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,if (fgets (bf,sizeof (bf), fp) ==NULL )break ; +if ((rdclock () - t) >MMAP_TIMEOUT ) { +pr_warning ("Reading %s time out.\n" , filename); + timeout = true; + goto out; + } +/* ensure null termination since stack will be reused. */ strcpy (execname,"" );
@@ -301,6 +312,10 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool, event->header.misc |=PERF_RECORD_MISC_MMAP_DATA ; } +out: +if (timeout) + event->header.misc |=PERF_RECORD_MISC_MMAP_TIME_OUT ; +if (!strcmp (execname,"" ))strcpy (execname, anonstr);
@@ -319,6 +334,9 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool, rc =-1 ;break ; } + +if (timeout) +break ; }fclose (fp);
tools/perf/util/event.h +1 -0
@@ -265,6 +265,7 @@ struct events_stats {u32 nr_unknown_id;u32 nr_unprocessable_samples;u32 nr_auxtrace_errors[PERF_AUXTRACE_ERROR_MAX ]; +u32 nr_proc_map_timeout; };struct attr_event {
tools/perf/util/session.c +11 -0
@@ -1041,6 +1041,8 @@ static int machines__deliver_event(struct machines *machines,case PERF_RECORD_MMAP :return tool->mmap (tool, event, sample, machine);case PERF_RECORD_MMAP2 : +if (event->header.misc &PERF_RECORD_MISC_MMAP_TIME_OUT ) + ++evlist->stats.nr_proc_map_timeout;return tool->mmap2 (tool, event, sample, machine);case PERF_RECORD_COMM :return tool->comm (tool, event, sample, machine);
@@ -1337,6 +1339,15 @@ static void perf_session__warn_about_errors(const struct perf_session *session)ui__warning ("%u out of order events recorded.\n" , oe->nr_unordered_events);events_stats__auxtrace_error_warn (stats); + +if (stats->nr_proc_map_timeout !=0 ) { +ui__warning ("%d map information files for pre-existing threads were\n" +"not processed, if there are samples for addresses they\n" +"will not be resolved, you may find out which are these\n" +"threads by running with -v and redirecting the output\n" +"to a file.\n" , + stats->nr_proc_map_timeout); + } }volatile int session_done;
Signature · 1 line
1.8.3.1
PATCH v3 2/2 perf,tools: configurable per thread proc map processing time out
From: Kan Liang <[email protected]>
The time out to limit the individual proc map processing was hard code to 50ms. This patch introduce a new option --proc-map-timeout to make the time limit configurable.
--- tools/perf/Documentation/perf-kvm.txt | 6 ++++++ tools/perf/Documentation/perf-record.txt | 5 +++++ tools/perf/Documentation/perf-top.txt | 6 ++++++ tools/perf/Documentation/perf-trace.txt | 5 +++++ tools/perf/builtin-kvm.c | 5 ++++- tools/perf/builtin-record.c | 6 +++++- tools/perf/builtin-top.c | 5 ++++- tools/perf/builtin-trace.c | 6 +++++- tools/perf/perf.h | 1 + tools/perf/tests/code-reading.c | 2 +- tools/perf/tests/dwarf-unwind.c | 2 +- tools/perf/tests/mmap-thread-lookup.c | 4 ++-- tools/perf/util/event.c | 30 ++++++++++++++++++------------ tools/perf/util/event.h | 9 ++++++--- tools/perf/util/machine.c | 7 ++++--- tools/perf/util/machine.h | 9 ++++++--- 16 files changed, 79 insertions(+), 29 deletions(-)
tools/perf/Documentation/perf-kvm.txt +6 -0
@@ -151,6 +151,12 @@ STAT LIVE OPTIONS Show events other than HLT (x86 only) or Wait state (s390 only) that take longer than duration usecs. +--proc-map-timeout:: + When processing pre-existing threads /proc/XXX/mmap, it may take + a long time, because the file may be huge. A time out is needed + in such cases. + This option sets the time out limit. The default value is 50 ms. + SEE ALSO -------- linkperf:perf-top[1], linkperf:perf-record[1], linkperf:perf-report[1],
tools/perf/Documentation/perf-record.txt +5 -0
@@ -267,6 +267,11 @@ AUX area tracing event. Optionally the number of bytes to capture per snapshot can be specified. In Snapshot Mode, trace data is captured only when signal SIGUSR2 is received. +--proc-map-timeout:: +When processing pre-existing threads /proc/XXX/mmap, it may take a long time, +because the file may be huge. A time out is needed in such cases. +This option sets the time out limit. The default value is 50 ms. + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/Documentation/perf-top.txt +6 -0
@@ -201,6 +201,12 @@ Default is to monitor all CPUS. Force each column width to the provided list, for large terminal readability. 0 means no limit (default behavior). +--proc-map-timeout:: + When processing pre-existing threads /proc/XXX/mmap, it may take + a long time, because the file may be huge. A time out is needed + in such cases. + This option sets the time out limit. The default value is 50 ms. + INTERACTIVE PROMPTING KEYS --------------------------
tools/perf/Documentation/perf-trace.txt +5 -0
@@ -121,6 +121,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs. --event:: Trace other events, see 'perf list' for a complete list. +--proc-map-timeout:: + When processing pre-existing threads /proc/XXX/mmap, it may take a long time, + because the file may be huge. A time out is needed in such cases. + This option sets the time out limit. The default value is 50 ms. + PAGEFAULTS ----------
tools/perf/builtin-kvm.c +4 -1
@@ -1311,6 +1311,8 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,"show events other than" " HLT (x86 only) or Wait state (s390 only)" " that take longer than duration usecs"), +OPT_UINTEGER (0 ,"proc-map-timeout" , &kvm->opts.mmap_timeout, +"per thread proc mmap processing timeout in ms" ),OPT_END () }; const char * const live_usage[] = {
@@ -1338,6 +1340,7 @@ static int kvm_events_live(struct perf_kvm_stat *kvm, kvm->opts.target.uses_mmap = false; kvm->opts.target.uid_str =NULL ; kvm->opts.target.uid =UINT_MAX ; + kvm->opts.mmap_timeout =50 ;symbol__init (NULL );disable_buildid_cache ();
@@ -1393,7 +1396,7 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,perf_session__set_id_hdr_size (kvm->session);ordered_events__set_copy_on_queue (&kvm->session->ordered_events, true);machine__synthesize_threads (&kvm->session->machines.host, &kvm->opts.target, - kvm->evlist->threads, false); + kvm->evlist->threads, false, kvm->opts.mmap_timeout); err =kvm_live_open_events (kvm);if (err) goto out;
tools/perf/builtin-record.c +5 -1
@@ -598,7 +598,8 @@ static int __cmd_record(struct record *rec, int argc, const char **argv) } err =__machine__synthesize_threads (machine, tool, &opts->target, rec->evlist->threads, - process_synthesized_event, opts->sample_address); + process_synthesized_event, opts->sample_address, + opts->mmap_timeout);if (err !=0 ) goto out_child;
@@ -959,6 +960,7 @@ static struct record record = { .uses_mmap = true, .default_per_cpu = true, }, + .mmap_timeout = 50, }, .tool = { .sample = process_sample_event,
@@ -1067,6 +1069,8 @@ struct option __record_options[] = { parse_clockid),OPT_STRING_OPTARG ('S' ,"snapshot" , &record.opts.auxtrace_snapshot_opts,"opts" ,"AUX area tracing Snapshot Mode" ,"" ), +OPT_UINTEGER (0 ,"proc-map-timeout" , &record.opts.mmap_timeout, +"per thread proc mmap processing timeout in ms" ), OPT_END() };
tools/perf/builtin-top.c +4 -1
@@ -951,7 +951,7 @@ static int __cmd_top(struct perf_top *top) goto out_delete;machine__synthesize_threads (&top->session->machines.host, &opts->target, - top->evlist->threads, false); + top->evlist->threads, false, opts->mmap_timeout); ret =perf_top__start_counters (top);if (ret) goto out_delete;
@@ -1061,6 +1061,7 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused) .target = { .uses_mmap = true, }, + .mmap_timeout = 50, }, .max_stack = PERF_MAX_STACK_DEPTH, .sym_pcnt_filter = 5,
@@ -1160,6 +1161,8 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)OPT_STRING ('w' ,"column-widths" , &symbol_conf.col_width_list_str,"width[,width...]" ,"don't try to adjust column width, use these fixed values" ), +OPT_UINTEGER (0 ,"proc-map-timeout" , &opts->mmap_timeout, +"per thread proc mmap processing timeout in ms" ),OPT_END () }; const char * const top_usage[] = {
tools/perf/builtin-trace.c +5 -1
@@ -1460,7 +1460,8 @@ static int trace__symbols_init(struct trace *trace, struct perf_evlist *evlist)return -ENOMEM ; err =__machine__synthesize_threads (trace->host, &trace->tool, &trace->opts.target, - evlist->threads, trace__tool_process, false); + evlist->threads, trace__tool_process, false, + trace->opts.mmap_timeout);if (err)symbol__exit ();
@@ -2689,6 +2690,7 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused) .user_interval = ULLONG_MAX, .no_buffering = true, .mmap_pages = UINT_MAX, + .mmap_timeout = 50, }, .output = stdout, .show_comm = true,
@@ -2738,6 +2740,8 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused) "Trace pagefaults", parse_pagefaults, "maj"),OPT_BOOLEAN (0 ,"syscalls" , &trace.trace_syscalls,"Trace syscalls" ),OPT_BOOLEAN ('f' ,"force" , &trace.force,"don't complain, do it" ), +OPT_UINTEGER (0 ,"proc-map-timeout" , &trace.opts.mmap_timeout, +"per thread proc mmap processing timeout in ms" ), OPT_END() };const char *const trace_subcommands[] = {"record" ,NULL };
tools/perf/perf.h +1 -0
@@ -69,6 +69,7 @@ struct record_opts {unsigned initial_delay;bool use_clockid;clockid_t clockid; +unsigned int mmap_timeout; };struct option ;
tools/perf/tests/code-reading.c +1 -1
@@ -451,7 +451,7 @@ static int do_test_code_reading(bool try_kcore) } ret =perf_event__synthesize_thread_map (NULL , threads, - perf_event__process, machine, false); + perf_event__process, machine, false,50 );if (ret <0 ) {pr_debug ("perf_event__synthesize_thread_map failed\n" ); goto out_err;
tools/perf/tests/dwarf-unwind.c +1 -1
@@ -28,7 +28,7 @@ static int init_live_machine(struct machine *machine)pid_t pid =getpid ();return perf_event__synthesize_mmap_events (NULL , &event, pid, pid, - mmap_handler, machine, true); + mmap_handler, machine, true,50 ); }#define MAX_STACK 8
tools/perf/tests/mmap-thread-lookup.c +2 -2
@@ -129,7 +129,7 @@ static int synth_all(struct machine *machine) {return perf_event__synthesize_threads (NULL , perf_event__process, - machine,0 ); + machine,0 ,50 ); }static int synth_process (struct machine *machine)
@@ -141,7 +141,7 @@ static int synth_process(struct machine *machine) err =perf_event__synthesize_thread_map (NULL , map, perf_event__process, - machine,0 ); + machine,0 ,50 );thread_map__delete (map);return err;
tools/perf/util/event.c +18 -12
@@ -213,14 +213,13 @@ static int perf_event__synthesize_fork(struct perf_tool *tool,return 0 ; } -#define MMAP_TIMEOUT (50 * 1000000ULL) -int perf_event__synthesize_mmap_events (struct perf_tool *tool,union perf_event *event,pid_t pid,pid_t tgid,perf_event__handler_t process,struct machine *machine, -bool mmap_data) +bool mmap_data, +unsigned int mmap_timeout) {char filename[PATH_MAX ];FILE *fp;
@@ -258,7 +257,7 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,if (fgets (bf,sizeof (bf), fp) ==NULL )break ; -if ((rdclock () - t) >MMAP_TIMEOUT ) { +if ((rdclock () - t) > (mmap_timeout *1000000ULL )) {pr_warning ("Reading %s time out.\n" , filename); timeout = true; goto out;
@@ -404,7 +403,9 @@ static int __event__synthesize_thread(union perf_event *comm_event, pid_t pid, int full, perf_event__handler_t process, struct perf_tool *tool, - struct machine *machine, bool mmap_data) + struct machine *machine, + bool mmap_data, + unsigned int mmap_timeout) { char filename[PATH_MAX];DIR *tasks;
@@ -421,7 +422,8 @@ static int __event__synthesize_thread(union perf_event *comm_event,return -1 ;return perf_event__synthesize_mmap_events (tool, mmap_event, pid, tgid, - process, machine, mmap_data); + process, machine, mmap_data, + mmap_timeout); }if (machine__is_default_guest (machine))
@@ -462,7 +464,7 @@ static int __event__synthesize_thread(union perf_event *comm_event,if (_pid == pid) {/* process the parent's maps too */ rc =perf_event__synthesize_mmap_events (tool, mmap_event, pid, tgid, - process, machine, mmap_data); + process, machine, mmap_data, mmap_timeout);if (rc)break ; }
@@ -476,7 +478,8 @@ int perf_event__synthesize_thread_map(struct perf_tool *tool,struct thread_map *threads, perf_event__handler_t process, struct machine *machine, - bool mmap_data) + bool mmap_data, + unsigned int mmap_timeout) { union perf_event *comm_event, *mmap_event, *fork_event;int err =-1 , thread, j;
@@ -499,7 +502,7 @@ int perf_event__synthesize_thread_map(struct perf_tool *tool, fork_event, threads->map[thread], 0, process, tool, machine, - mmap_data)) { + mmap_data, mmap_timeout)) { err =-1 ;break ; }
@@ -525,7 +528,7 @@ int perf_event__synthesize_thread_map(struct perf_tool *tool, fork_event, comm_event->comm.pid, 0, process, tool, machine, - mmap_data)) { + mmap_data, mmap_timeout)) { err =-1 ;break ; }
@@ -542,7 +545,9 @@ out:int perf_event__synthesize_threads (struct perf_tool *tool,perf_event__handler_t process, -struct machine *machine,bool mmap_data) +struct machine *machine, +bool mmap_data, +unsigned int mmap_timeout) {DIR *proc;char proc_path[PATH_MAX ];
@@ -582,7 +587,8 @@ int perf_event__synthesize_threads(struct perf_tool *tool,* one thread couldn't be synthesized. */ __event__synthesize_thread (comm_event, mmap_event, fork_event, pid, -1 , process, tool, machine, mmap_data); +1 , process, tool, machine, mmap_data, + mmap_timeout); } err =0 ;
tools/perf/util/event.h +6 -3
@@ -384,10 +384,12 @@ typedef int (*perf_event__handler_t)(struct perf_tool *tool,int perf_event__synthesize_thread_map (struct perf_tool *tool,struct thread_map *threads,perf_event__handler_t process, -struct machine *machine,bool mmap_data); +struct machine *machine,bool mmap_data, +unsigned int mmap_timeout);int perf_event__synthesize_threads (struct perf_tool *tool,perf_event__handler_t process, -struct machine *machine,bool mmap_data); +struct machine *machine,bool mmap_data, +unsigned int mmap_timeout);int perf_event__synthesize_kernel_mmap (struct perf_tool *tool,perf_event__handler_t process,struct machine *machine);
@@ -469,7 +471,8 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool, pid_t pid, pid_t tgid, perf_event__handler_t process, struct machine *machine, - bool mmap_data); + bool mmap_data, + unsigned int mmap_timeout);size_t perf_event__fprintf_comm (union perf_event *event,FILE *fp);size_t perf_event__fprintf_mmap (union perf_event *event,FILE *fp);
tools/perf/util/machine.c +4 -3
@@ -1892,12 +1892,13 @@ int machine__for_each_thread(struct machine *machine,int __machine__synthesize_threads (struct machine *machine,struct perf_tool *tool,struct target *target,struct thread_map *threads, -perf_event__handler_t process,bool data_mmap) +perf_event__handler_t process,bool data_mmap, +unsigned int mmap_timeout) {if (target__has_task (target)) -return perf_event__synthesize_thread_map (tool, threads, process, machine, data_mmap); +return perf_event__synthesize_thread_map (tool, threads, process, machine, data_mmap, mmap_timeout);else if (target__has_cpu (target)) -return perf_event__synthesize_threads (tool, process, machine, data_mmap); +return perf_event__synthesize_threads (tool, process, machine, data_mmap, mmap_timeout);/* command specified */ return 0 ; }
tools/perf/util/machine.h +6 -3
@@ -219,13 +219,16 @@ int machine__for_each_thread(struct machine *machine,int __machine__synthesize_threads (struct machine *machine,struct perf_tool *tool,struct target *target,struct thread_map *threads, -perf_event__handler_t process,bool data_mmap); +perf_event__handler_t process,bool data_mmap, +unsigned int mmap_timeout);static inline int machine__synthesize_threads (struct machine *machine,struct target *target, -struct thread_map *threads,bool data_mmap) +struct thread_map *threads,bool data_mmap, +unsigned int mmap_timeout) {return __machine__synthesize_threads (machine,NULL , target, threads, - perf_event__process, data_mmap); + perf_event__process, data_mmap, + mmap_timeout); }pid_t machine__get_current_tid (struct machine *machine,int cpu);
Signature · 1 line
1.8.3.1
tools/perf/util/event.c static int perf_event__synthesize_fork(struct perf_tool *tool, · line 217
Quoted 10 lines · On 6/17/15 1:56 AM, [email protected] wrote:
tools/perf/util/event.c +2
-0@@ -213,6 +213,8 @@ static int perf_event__synthesize_fork(struct perf_tool *tool,return 0 ; } +#define MMAP_TIMEOUT (50 * 1000000ULL) +
How did you determine 50msec is a good time? This seems really low to me considering the range of platforms supported by perf and various run time conditions. The default needs to work right on all platforms.
Why not have the default be infinity and users who need the feature use the option provided in patch 2?
David
Quoted 12 lines · On 6/17/15 1:56 AM, [email protected] wrote:
tools/perf/util/event.c static int perf_event__synthesize_fork(struct
Quoted 5 lines
tools/perf/util/event.c index renamed from tools/perf/util/event.c +0
-0793b150..ac6cf2a 100644
tools/perf/util/event.c +0
-0@@ -213,6 +213,8 @@ static int perf_event__synthesize_fork(struct
perf_tool*tool,Quoted 3 lines
tools/perf/util/event.c +0
-0return 0 ; }
Quoted 7 lines
tools/perf/util/event.c static int perf_event__synthesize_fork(struct perf_tool *tool, · line 217
Quoted 2 lines
tools/perf/util/event.c +2
-0+#define MMAP_TIMEOUT (50 * 1000000ULL) +How did you determine 50msec is a good time? This seems really low to me considering the range of platforms supported by perf and various run time conditions. The default needs to work right on all platforms.
I only test 50ms on my platforms. For all platforms, any recommendation?
Quoted 2 lines
Why not have the default be infinity and users who need the feature use the option provided in patch 2?
If we set the default to infinity, the user will get nothing when perf top runs in our case. There will be no warning. How do they know that they should change the time out limit?
So I still think we need a finite default value. The problem is what's the number.
Thanks,
Kan
Arnaldo Carvalho de Melo
maintainerEm Wed, Jun 17, 2015 at 10:05:07AM -0600, David Ahern escreveu:
Quoted 9 lines · On 6/17/15 1:56 AM, [email protected] wrote:
Quoted 7 lines
tools/perf/util/event.c +0
-0@@ -213,6 +213,8 @@ static int perf_event__synthesize_fork(struct perf_tool *tool,return 0 ; }
Quoted 8 lines
tools/perf/util/event.c static int perf_event__synthesize_fork(struct perf_tool *tool, · line 216
Quoted 1 line
tools/perf/util/event.c +1
-0+#define MMAP_TIMEOUT (50 * 1000000ULL)How did you determine 50msec is a good time? This seems really low to me considering the range of platforms supported by perf and various run time conditions. The default needs to work right on all platforms.
Why not have the default be infinity and users who need the feature use the option provided in patch 2?
Don't think that is a reasonable approach, how would be the workflow, something like:
1) Fire up 'perf top'
2) Wait a long time.... maybe something is wrong, I guess I better exit, press Q, no response, ok, I think I'll kill this bugger
3) Read docs, etc, ah! That is it! --proc-map-timeout! How couldn't I think of that?
8-)
I think limiting this to say, half a second is ok, and the message about the truncation happening surely suggests using --proc-map-timeout, right?
Also please rename this MMAP_TIMEOUT define, this is not about a timeout for a mmap operation, it is a timeout for parsing the proc mmap info, i.e. something like:
#define PROC_MAP_PARSE_TIMEOUT (50 * 1000000ULL)
Clarifies this and doesn't pollutes ctags like tools.
Thanks
- Arnaldo