Skip to contentSkip to conversation

perf record: Allow poll timeout to be specified

PATCH In review David Ahern 4 participants 10 messages Mar 24, 2015 – Mar 25, 2015 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf record: Allow poll timeout to be specified

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Cc Ingo Molnar <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Adrian Hunter <[email protected]>
---
 tools/perf/Documentation/perf-record.txt | 6 ++++++
 tools/perf/builtin-record.c              | 5 ++++-
 tools/perf/perf.h                        | 1 +
 3 files changed, 11 insertions(+), 1 deletion(-)
tools/perf/Documentation/perf-record.txt +6 -0
index 355c4f5569b5..7010c363fdd1 100644
@@ -250,6 +250,12 @@ is off by default.
 --running-time::
 Record running and enabled time for read events (:S)
 
+--poll=::
+Polling interval in msec. Defaults to infinite which means record relies on
+watermarks to wakeup and read events from each mmap. Setting poll helps smooth
+the event collection across mmaps and the subsequent processing of the data
+file. For example perf-top uses a 100 msec polling interval.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +4 -1
index 5a2ff510b75b..091868288d29 100644
@@ -485,7 +485,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		if (hits == rec->samples) {
 			if (done || draining)
 				break;
-			err = perf_evlist__poll(rec->evlist, -1);
+			err = perf_evlist__poll(rec->evlist, opts->poll_timeout);
 			/*
 			 * Propagate error, only if there's any. Ignore positive
 			 * number of returned events and interrupt error.
@@ -734,6 +734,7 @@ static struct record record = {
 		.user_freq	     = UINT_MAX,
 		.user_interval	     = ULLONG_MAX,
 		.freq		     = 4000,
+		.poll_timeout	     = -1,
 		.target		     = {
 			.uses_mmap   = true,
 			.default_per_cpu = true,
@@ -841,6 +842,8 @@ struct option __record_options[] = {
 		    "Sample machine registers on interrupt"),
 	OPT_BOOLEAN(0, "running-time", &record.opts.running_time,
 		    "Record running/enabled time of read (:S) events"),
+	OPT_INTEGER(0, "poll", &record.opts.poll_timeout,
+		  "poll interval in ms (defaults to infinite)"),
 	OPT_END()
 };
 
tools/perf/perf.h +1 -0
index 1caa70a4a9e1..ee847c8af668 100644
@@ -62,6 +62,7 @@ struct record_opts {
 	u64	     user_interval;
 	bool	     sample_transaction;
 	unsigned     initial_delay;
+	int	     poll_timeout;
 };
 
 struct option;
Signature · 1 line
2.2.1
#1 lore raw
Quoted 11 lines · David Ahern

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

Quoted 4 lines

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Please tune the default value (perhaps influenced by N_PROC?) so that users will get sane behavior without having to specify this option!

Thanks,

Ingo

#2 lore raw
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · David Ahern

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

Quoted 7 lines
Quoted 4 lines

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Please tune the default value (perhaps influenced by N_PROC?) so that users will get sane behavior without having to specify this option!

I knew you were going to say that! ;-)

It's really a function of events coming in not cpus. The number of CPUs just compounds the problem.

I thought about making perf-record use a 100msec timeout like perf-top, but that can lead to unnecessary FINISHED_ROUND events in the file and unnecessary noise/overhead in the record side. On the other hand looking at scheduler tracepoints, kvm tracepoints, etc -- those can flood in to the point that even 100msec is too long.

#3 lore raw
In reply to Ingo Molnar

Em Tue, Mar 24, 2015 at 05:12:11PM +0100, Ingo Molnar escreveu:

Quoted 12 lines · David Ahern
Quoted 11 lines

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

Quoted 7 lines
Quoted 4 lines

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Please tune the default value (perhaps influenced by N_PROC?) so that users will get sane behavior without having to specify this option!

Isn't this a followup patch? I.e. changing the default from infinity to some sane value?

Applying it now.

- Arnaldo

#4 lore raw
Quoted 20 lines · Arnaldo Carvalho de Melo

Em Tue, Mar 24, 2015 at 05:12:11PM +0100, Ingo Molnar escreveu:

Quoted 19 lines · David Ahern
Quoted 17 lines

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Quoted 5 lines
Quoted 3 lines

Please tune the default value (perhaps influenced by N_PROC?) so that users will get sane behavior without having to specify this option!

Isn't this a followup patch? [...]

Will a followup patch be written?

Thanks,

Ingo

#5 lore raw

Em Wed, Mar 25, 2015 at 10:11:47AM +0100, Ingo Molnar escreveu:

Quoted 21 lines · Arnaldo Carvalho de Melo
Quoted 19 lines

Em Tue, Mar 24, 2015 at 05:12:11PM +0100, Ingo Molnar escreveu:

Quoted 18 lines · David Ahern
Quoted 17 lines

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Please tune the default value (perhaps influenced by N_PROC?) so that users will get sane behavior without having to specify this option!

Isn't this a followup patch? [...]

Will a followup patch be written?

Hope so :-)

If David doesn't come up with something I'll probably will, as making 'trace' use the ordered_samples, like 'perf top' does (initially with some arbitrary reasonable poll timeout value), is a low hanging fruit to get those multi-CPU tracepoints sorted until I get something better in place...

But what I said is independent of if a followup patch would come or not, right now we don't have that possibility, with his patch, we do.

Turning it from not possible to possible looks an improvement before we get it done automatically, and even by then allowing someone to tweak that value may be useful, no?

- Arnaldo

#6 lore raw
Quoted 5 lines · Arnaldo Carvalho de Melo

If David doesn't come up with something I'll probably will, as making 'trace' use the ordered_samples, like 'perf top' does (initially with some arbitrary reasonable poll timeout value), is a low hanging fruit to get those multi-CPU tracepoints sorted until I get something better in place...

I have thought about it. It needs to be an adaptive algorithm:
1. start at 100 msec.
2. Read the maps. How much data are there (not events, but data size)? 3. Adjust poll timeout up or down with some heuristic -- maybe something similar to the algorithm perf-top uses for removing entries from the histograms.

That said, I still thinking giving the user control is not a crazy idea.

David

#7 lore raw
Quoted 7 lines · David Ahern
Quoted 5 lines · Arnaldo Carvalho de Melo

If David doesn't come up with something I'll probably will, as making 'trace' use the ordered_samples, like 'perf top' does (initially with some arbitrary reasonable poll timeout value), is a low hanging fruit to get those multi-CPU tracepoints sorted until I get something better in place...

Quoted 8 lines

I have thought about it. It needs to be an adaptive algorithm:
1. start at 100 msec.
2. Read the maps. How much data are there (not events, but data size)? 3. Adjust poll timeout up or down with some heuristic -- maybe something similar to the algorithm perf-top uses for removing entries from the histograms.

That said, I still thinking giving the user control is not a crazy idea.

The user has absolutely zero control over the rate of data and about what the ideal refresh rate would be! So this really should be solved in software and automatically.

'try to tweak this parameter up and down until the tool is usable' is a really, really poor user interface concept ...

Thanks,

Ingo

Acked-by on v1 1/1
#8 lore raw
In reply to David Ahern
Quoted 9 lines · David Ahern

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

so you have pipe to perf script, right?

Quoted 5 lines

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

there's also the '--no-buffering' option that sets:

                attr->watermark = 0;
                attr->wakeup_events = 1;

but that's just the other edge, which is not what you'd want

I think it's good to have user side configurable as well

thanks,
jirka

#9 lore raw
Quoted 11 lines · Jiri Olsa
Quoted 9 lines · David Ahern

Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.

Quoted 1 line

so you have pipe to perf script, right?

$ perf record ....
$ perf script ...
<wait an eternity>
data
Quoted 7 lines
Quoted 6 lines

This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.

Quoted 6 lines

there's also the '--no-buffering' option that sets:

                 attr->watermark = 0;
                 attr->wakeup_events = 1;

but that's just the other edge, which is not what you'd want

right, that is the other extreme. record would never go to sleep.

Quoted 8 lines

I think it's good to have user side configurable as well

thanks,
jirka