Perf/stat: Solve problems with repeat and interval
Subsystems (2)
cover Perf/stat: Solve problems with repeat and interval
There are some problems in perf stat when using a combination of repeat and interval options. This series tries to fix them.
Srikar Dronamraju (2):
perf/stat: Reset previous counts on repeat with interval perf/stat: Fix a segmentation fault when using repeat forever
tools/perf/builtin-stat.c | 5 ++++- tools/perf/util/stat.c | 17 +++++++++++++++++ tools/perf/util/stat.h | 1 + 3 files changed, 22 insertions(+), 1 deletion(-)
Signature · 1 line
2.18.1
PATCH 1/2 perf/stat: Reset previous counts on repeat with interval
When using perf stat with repeat and interval option, perf stat shows wrong values for events. The wrong values will be shown for the first interval on the second and subsequent repetitions.
Without fix:
perf stat -r 3 -I 2000 -e faults -e sched:sched_switch -a sleep 5
2.000282489 53 faults
2.000282489 513 sched:sched_switch
4.005478208 3,721 faults
4.005478208 2,666 sched:sched_switch
5.025470933 395 faults
5.025470933 1,307 sched:sched_switch
2.009602825 1,84,46,74,40,73,70,95,47,520 faults <------
2.009602825 1,84,46,74,40,73,70,95,49,568 sched:sched_switch <------
4.019612206 4,730 faults
4.019612206 2,746 sched:sched_switch
5.039615484 3,953 faults
5.039615484 1,496 sched:sched_switch
2.000274620 1,84,46,74,40,73,70,95,47,520 faults <------
2.000274620 1,84,46,74,40,73,70,95,47,520 sched:sched_switch <------
4.000480342 4,282 faults
4.000480342 2,303 sched:sched_switch
5.000916811 1,322 faults
5.000916811 1,064 sched:sched_switchprev_raw_counts is allocated when using intervals. This is used when calculating the difference in the counts of events when using interval. The current counts are stored in prev_raw_counts to calculate the differences in the next iteration. On the first interval of the second and subsequent repetitions, prev_raw_counts would be the values stored in the last interval of the previous repetitions, while the current counts will only be for the first interval of the current repetition. Hence there is a possibility of events showing up as big number.
Fix this by resetting prev_raw_counts whenever perf stat repeats the command.
With fix:
perf stat -r 3 -I 2000 -e faults -e sched:sched_switch -a sleep 5
2.019349347 2,597 faults
2.019349347 2,753 sched:sched_switch
4.019577372 3,098 faults
4.019577372 2,532 sched:sched_switch
5.019415481 1,879 faults
5.019415481 1,356 sched:sched_switch
2.000178813 8,468 faults
2.000178813 2,254 sched:sched_switch
4.000404621 7,440 faults
4.000404621 1,266 sched:sched_switch
5.040196079 2,458 faults
5.040196079 556 sched:sched_switch
2.000191939 6,870 faults
2.000191939 1,170 sched:sched_switch
4.000414103 541 faults
4.000414103 902 sched:sched_switch
5.000809863 450 faults
5.000809863 364 sched:sched_switch--- tools/perf/builtin-stat.c | 3 +++ tools/perf/util/stat.c | 17 +++++++++++++++++ tools/perf/util/stat.h | 1 + 3 files changed, 21 insertions(+)
tools/perf/builtin-stat.c +3 -0
@@ -1961,6 +1961,9 @@ int cmd_stat(int argc, const char **argv)fprintf (output,"[ perf stat: executing run #%d ... ]\n" , run_idx +1 ); +if (run_idx !=0 ) +perf_evlist__reset_prev_raw_counts (evsel_list); + status =run_perf_stat (argc, argv, run_idx);if (forever && status !=-1 ) {print_counters (NULL , argc, argv);
tools/perf/util/stat.c +17 -0
@@ -155,6 +155,15 @@ static void perf_evsel__free_prev_raw_counts(struct perf_evsel *evsel) evsel->prev_raw_counts =NULL ; } +static void perf_evsel__reset_prev_raw_counts (struct perf_evsel *evsel) +{ +if (evsel->prev_raw_counts) { + evsel->prev_raw_counts->aggr.val =0 ; + evsel->prev_raw_counts->aggr.ena =0 ; + evsel->prev_raw_counts->aggr.run =0 ; + } +} +static int perf_evsel__alloc_stats (struct perf_evsel *evsel,bool alloc_raw) {int ncpus =perf_evsel__nr_cpus (evsel);
@@ -205,6 +214,14 @@ void perf_evlist__reset_stats(struct perf_evlist *evlist) } } +void perf_evlist__reset_prev_raw_counts (struct perf_evlist *evlist) +{ +struct perf_evsel *evsel; + +evlist__for_each_entry (evlist, evsel) +perf_evsel__reset_prev_raw_counts (evsel); +} +static void zero_per_pkg (struct perf_evsel *counter) {if (counter->per_pkg_mask)
tools/perf/util/stat.h +1 -0
@@ -194,6 +194,7 @@ void perf_stat__collect_metric_expr(struct perf_evlist *);int perf_evlist__alloc_stats (struct perf_evlist *evlist,bool alloc_raw);void perf_evlist__free_stats (struct perf_evlist *evlist);void perf_evlist__reset_stats (struct perf_evlist *evlist); +void perf_evlist__reset_prev_raw_counts (struct perf_evlist *evlist);int perf_stat_process_counter (struct perf_stat_config *config,struct perf_evsel *counter);
Signature · 1 line
2.18.1
[tip: perf/urgent] perf stat: Reset previous counts on repeat with interval
The following commit has been merged into the perf/urgent branch of tip:
The following commit has been merged into the perf/urgent branch of tip:
Commit-ID: b63fd11cced17fcb8e133def29001b0f6aaa5e06 Gitweb: https://git.kernel.org/tip/b63fd11cced17fcb8e133def29001b0f6aaa5e06 Author: Srikar Dronamraju <[email protected]> AuthorDate: Wed, 04 Sep 2019 15:17:37 +05:30 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitterDate: Fri, 20 Sep 2019 10:28:26 -03:00
perf stat: Reset previous counts on repeat with interval
When using 'perf stat' with repeat and interval option, it shows wrong values for events.
The wrong values will be shown for the first interval on the second and subsequent repetitions.
Without the fix:
# perf stat -r 3 -I 2000 -e faults -e sched:sched_switch -a sleep 5
2.000282489 53 faults
2.000282489 513 sched:sched_switch
4.005478208 3,721 faults
4.005478208 2,666 sched:sched_switch
5.025470933 395 faults
5.025470933 1,307 sched:sched_switch
2.009602825 1,84,46,74,40,73,70,95,47,520 faults <------
2.009602825 1,84,46,74,40,73,70,95,49,568 sched:sched_switch <------
4.019612206 4,730 faults
4.019612206 2,746 sched:sched_switch
5.039615484 3,953 faults
5.039615484 1,496 sched:sched_switch
2.000274620 1,84,46,74,40,73,70,95,47,520 faults <------
2.000274620 1,84,46,74,40,73,70,95,47,520 sched:sched_switch <------
4.000480342 4,282 faults
4.000480342 2,303 sched:sched_switch
5.000916811 1,322 faults
5.000916811 1,064 sched:sched_switch#
prev_raw_counts is allocated when using intervals. This is used when calculating the difference in the counts of events when using interval.
The current counts are stored in prev_raw_counts to calculate the differences in the next iteration.
On the first interval of the second and subsequent repetitions, prev_raw_counts would be the values stored in the last interval of the previous repetitions, while the current counts will only be for the first interval of the current repetition.
Hence there is a possibility of events showing up as big number.
Fix this by resetting prev_raw_counts whenever perf stat repeats the command.
With the fix:
# perf stat -r 3 -I 2000 -e faults -e sched:sched_switch -a sleep 5
2.019349347 2,597 faults
2.019349347 2,753 sched:sched_switch
4.019577372 3,098 faults
4.019577372 2,532 sched:sched_switch
5.019415481 1,879 faults
5.019415481 1,356 sched:sched_switch
2.000178813 8,468 faults
2.000178813 2,254 sched:sched_switch
4.000404621 7,440 faults
4.000404621 1,266 sched:sched_switch
5.040196079 2,458 faults
5.040196079 556 sched:sched_switch
2.000191939 6,870 faults
2.000191939 1,170 sched:sched_switch
4.000414103 541 faults
4.000414103 902 sched:sched_switch
5.000809863 450 faults
5.000809863 364 sched:sched_switch#
Committer notes:
This was broken since the cset introducing the --interval feature, i.e. --repeat + --interval wasn't tested at that point, add the Fixes tag so that automatic scripts can pick this up.
--- tools/perf/builtin-stat.c | 3 +++ tools/perf/util/stat.c | 17 +++++++++++++++++ tools/perf/util/stat.h | 1 + 3 files changed, 21 insertions(+)
tools/perf/builtin-stat.c +3 -0
@@ -1952,6 +1952,9 @@ int cmd_stat(int argc, const char **argv)fprintf (output,"[ perf stat: executing run #%d ... ]\n" , run_idx +1 ); +if (run_idx !=0 ) +perf_evlist__reset_prev_raw_counts (evsel_list); + status =run_perf_stat (argc, argv, run_idx);if (forever && status !=-1 ) {print_counters (NULL , argc, argv);
tools/perf/util/stat.c +17 -0
@@ -162,6 +162,15 @@ static void perf_evsel__free_prev_raw_counts(struct evsel *evsel) evsel->prev_raw_counts =NULL ; } +static void perf_evsel__reset_prev_raw_counts (struct evsel *evsel) +{ +if (evsel->prev_raw_counts) { + evsel->prev_raw_counts->aggr.val =0 ; + evsel->prev_raw_counts->aggr.ena =0 ; + evsel->prev_raw_counts->aggr.run =0 ; + } +} +static int perf_evsel__alloc_stats (struct evsel *evsel,bool alloc_raw) {int ncpus =perf_evsel__nr_cpus (evsel);
@@ -212,6 +221,14 @@ void perf_evlist__reset_stats(struct evlist *evlist) } } +void perf_evlist__reset_prev_raw_counts (struct evlist *evlist) +{ +struct evsel *evsel; + +evlist__for_each_entry (evlist, evsel) +perf_evsel__reset_prev_raw_counts (evsel); +} +static void zero_per_pkg (struct evsel *counter) {if (counter->per_pkg_mask)
tools/perf/util/stat.h +1 -0
@@ -193,6 +193,7 @@ void perf_stat__collect_metric_expr(struct evlist *);int perf_evlist__alloc_stats (struct evlist *evlist,bool alloc_raw);void perf_evlist__free_stats (struct evlist *evlist);void perf_evlist__reset_stats (struct evlist *evlist); +void perf_evlist__reset_prev_raw_counts (struct evlist *evlist);int perf_stat_process_counter (struct perf_stat_config *config,struct evsel *counter);
PATCH 2/2 perf/stat: Fix a segmentation fault when using repeat forever
In reply to Srikar DronamrajuObserve a segmentation fault when perf stat is asked to repeat forever with the interval option.
Without fix:
perf stat -r 0 -I 5000 -e cycles -a sleep 10
# time counts unit events
5.000211692 3,13,89,82,34,157 cycles
10.000380119 1,53,98,52,22,294 cycles
10.040467280 17,16,79,265 cyclesSegmentation fault
This problem was only observed when we use forever option aka -r 0 and works with limited repeats. Calling print_counter with ts being set to NULL, is not a correct option when interval is set. Hence avoid print_counter(NULL,..) if interval is set.
With fix:
perf stat -r 0 -I 5000 -e cycles -a sleep 10
# time counts unit events
5.019866622 3,15,14,43,08,697 cycles
10.039865756 3,15,16,31,95,261 cycles
10.059950628 1,26,05,47,158 cycles
5.009902655 3,14,52,62,33,932 cycles
10.019880228 3,14,52,22,89,154 cycles
10.030543876 66,90,18,333 cycles
5.009848281 3,14,51,98,25,437 cycles
10.029854402 3,15,14,93,04,918 cycles
5.009834177 3,14,51,95,92,316 cycles--- tools/perf/builtin-stat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/builtin-stat.c +1 -1
@@ -1965,7 +1965,7 @@ int cmd_stat(int argc, const char **argv)perf_evlist__reset_prev_raw_counts (evsel_list); status =run_perf_stat (argc, argv, run_idx); -if (forever && status !=-1 ) { +if (forever && status !=-1 && !interval) {print_counters (NULL , argc, argv);perf_stat__reset_stats (); }
Signature · 1 line
2.18.1
[tip: perf/urgent] perf stat: Fix a segmentation fault when using repeat forever
The following commit has been merged into the perf/urgent branch of tip:
The following commit has been merged into the perf/urgent branch of tip:
Commit-ID: 443f2d5ba13d65ccfd879460f77941875159d154 Gitweb: https://git.kernel.org/tip/443f2d5ba13d65ccfd879460f77941875159d154 Author: Srikar Dronamraju <[email protected]> AuthorDate: Wed, 04 Sep 2019 15:17:38 +05:30 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitterDate: Fri, 20 Sep 2019 10:28:26 -03:00
perf stat: Fix a segmentation fault when using repeat forever
Observe a segmentation fault when 'perf stat' is asked to repeat forever with the interval option.
Without fix:
# perf stat -r 0 -I 5000 -e cycles -a sleep 10
# time counts unit events
5.000211692 3,13,89,82,34,157 cycles
10.000380119 1,53,98,52,22,294 cycles
10.040467280 17,16,79,265 cycles
Segmentation faultThis problem was only observed when we use forever option aka -r 0 and works with limited repeats. Calling print_counter with ts being set to NULL, is not a correct option when interval is set. Hence avoid print_counter(NULL,..) if interval is set.
With fix:
# perf stat -r 0 -I 5000 -e cycles -a sleep 10
# time counts unit events
5.019866622 3,15,14,43,08,697 cycles
10.039865756 3,15,16,31,95,261 cycles
10.059950628 1,26,05,47,158 cycles
5.009902655 3,14,52,62,33,932 cycles
10.019880228 3,14,52,22,89,154 cycles
10.030543876 66,90,18,333 cycles
5.009848281 3,14,51,98,25,437 cycles
10.029854402 3,15,14,93,04,918 cycles
5.009834177 3,14,51,95,92,316 cyclesCommitter notes:
Did the 'git bisect' to find the cset introducing the problem to add the Fixes tag below, and at that time the problem reproduced as:
(gdb) run stat -r0 -I500 sleep 1
<SNIP>
Program received signal SIGSEGV, Segmentation fault.
print_interval (prefix=prefix@entry=0x7fffffffc8d0 "", ts=ts@entry=0x0) at builtin-stat.c:866
866 sprintf(prefix, "%6lu.%09lu%s", ts->tv_sec, ts->tv_nsec, csv_sep);
(gdb) bt
#0 print_interval (prefix=prefix@entry=0x7fffffffc8d0 "", ts=ts@entry=0x0) at builtin-stat.c:866 #1 0x000000000041860a in print_counters (ts=ts@entry=0x0, argc=argc@entry=2, argv=argv@entry=0x7fffffffd640) at builtin-stat.c:938 #2 0x0000000000419a7f in cmd_stat (argc=2, argv=0x7fffffffd640, prefix=<optimized out>) at builtin-stat.c:1411 #3 0x000000000045c65a in run_builtin (p=p@entry=0x6291b8 <commands+216>, argc=argc@entry=5, argv=argv@entry=0x7fffffffd640) at perf.c:370 #4 0x000000000045c893 in handle_internal_command (argc=5, argv=0x7fffffffd640) at perf.c:429 #5 0x000000000045c8f1 in run_argv (argcp=argcp@entry=0x7fffffffd4ac, argv=argv@entry=0x7fffffffd4a0) at perf.c:473 #6 0x000000000045cac9 in main (argc=<optimized out>, argv=<optimized out>) at perf.c:588
(gdb)
Mostly the same as just before this patch:
Program received signal SIGSEGV, Segmentation fault.
0x00000000005874a7 in print_interval (config=0xa1f2a0 <stat_config>, evlist=0xbc9b90, prefix=0x7fffffffd1c0 "`", ts=0x0) at util/stat-display.c:964
964 sprintf(prefix, "%6lu.%09lu%s", ts->tv_sec, ts->tv_nsec, config->csv_sep);
(gdb) bt
#0 0x00000000005874a7 in print_interval (config=0xa1f2a0 <stat_config>, evlist=0xbc9b90, prefix=0x7fffffffd1c0 "`", ts=0x0) at util/stat-display.c:964 #1 0x0000000000588047 in perf_evlist__print_counters (evlist=0xbc9b90, config=0xa1f2a0 <stat_config>, _target=0xa1f0c0 <target>, ts=0x0, argc=2, argv=0x7fffffffd670)
at util/stat-display.c:1172 #2 0x000000000045390f in print_counters (ts=0x0, argc=2, argv=0x7fffffffd670) at builtin-stat.c:656 #3 0x0000000000456bb5 in cmd_stat (argc=2, argv=0x7fffffffd670) at builtin-stat.c:1960 #4 0x00000000004dd2e0 in run_builtin (p=0xa30e00 <commands+288>, argc=5, argv=0x7fffffffd670) at perf.c:310 #5 0x00000000004dd54d in handle_internal_command (argc=5, argv=0x7fffffffd670) at perf.c:362 #6 0x00000000004dd694 in run_argv (argcp=0x7fffffffd4cc, argv=0x7fffffffd4c0) at perf.c:406 #7 0x00000000004dda11 in main (argc=5, argv=0x7fffffffd670) at perf.c:531 (gdb)
--- tools/perf/builtin-stat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/builtin-stat.c +1 -1
@@ -1956,7 +1956,7 @@ int cmd_stat(int argc, const char **argv)perf_evlist__reset_prev_raw_counts (evsel_list); status =run_perf_stat (argc, argv, run_idx); -if (forever && status !=-1 ) { +if (forever && status !=-1 && !interval) {print_counters (NULL , argc, argv);perf_stat__reset_stats (); }
Quoted 15 lines · Srikar Dronamraju
There are some problems in perf stat when using a combination of repeat and interval options. This series tries to fix them.
Srikar Dronamraju (2):
perf/stat: Reset previous counts on repeat with interval perf/stat: Fix a segmentation fault when using repeat forevertools/perf/builtin-stat.c | 5 ++++- tools/perf/util/stat.c | 17 +++++++++++++++++ tools/perf/util/stat.h | 1 + 3 files changed, 22 insertions(+), 1 deletion(-)Signature · 1 line
2.18.1
thanks,
jirka
Quoted 2 lines · Srikar Dronamraju
There are some problems in perf stat when using a combination of repeat and interval options. This series tries to fix them.
For the series:
Em Wed, Sep 11, 2019 at 09:44:20AM +0530, Ravi Bangoria escreveu:
Quoted 9 lines · Srikar Dronamraju
Quoted 2 lines
There are some problems in perf stat when using a combination of repeat and interval options. This series tries to fix them.
For the series:
Tested-by Ravi Bangoria <[email protected]>
I figured out where was that these features were broken so as to add Fixes tags for both patches, please consider doing that next time, this is important for getting this to be noticed by [email protected], added CC to them as well.
- Arnaldo