Skip to contentSkip to conversation

perf stat: Support overall statistics for interval mode

PATCH Superseded Jin Yao 3 participants 12 messages May 14, 2020 – May 19, 2020 perf events

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Jin Yao

author
#0 lore raw

v5 cover perf stat: Support overall statistics for interval mode

Currently perf-stat supports to print counts at regular interval (-I), but it's not very easy for user to get the overall statistics.

With this patchset, it supports to report the summary at the end of interval output.

For example,

 root@kbl-ppc:~# perf stat -e cycles -I1000 --interval-count 2
 #           time             counts unit events
      1.000412064          2,281,114      cycles
      2.001383658          2,547,880      cycles

Performance counter stats for 'system wide':

          4,828,994      cycles

        2.002860349 seconds time elapsed
 root@kbl-ppc:~# perf stat -e cycles,instructions -I1000 --interval-count 2
 #           time             counts unit events
      1.000389902          1,536,093      cycles
      1.000389902            420,226      instructions              #    0.27  insn per cycle
      2.001433453          2,213,952      cycles
      2.001433453            735,465      instructions              #    0.33  insn per cycle

Performance counter stats for 'system wide':

          3,750,045      cycles
          1,155,691      instructions              #    0.31  insn per cycle

        2.003023361 seconds time elapsed
 root@kbl-ppc:~# perf stat -M CPI,IPC -I1000 --interval-count 2
 #           time             counts unit events
      1.000435121            905,303      inst_retired.any          #      2.9 CPI
      1.000435121          2,663,333      cycles
      1.000435121            914,702      inst_retired.any          #      0.3 IPC
      1.000435121          2,676,559      cpu_clk_unhalted.thread
      2.001615941          1,951,092      inst_retired.any          #      1.8 CPI
      2.001615941          3,551,357      cycles
      2.001615941          1,950,837      inst_retired.any          #      0.5 IPC
      2.001615941          3,551,044      cpu_clk_unhalted.thread

Performance counter stats for 'system wide':

          2,856,395      inst_retired.any          #      2.2 CPI
          6,214,690      cycles
          2,865,539      inst_retired.any          #      0.5 IPC
          6,227,603      cpu_clk_unhalted.thread
        2.003403078 seconds time elapsed

v5:
---
1. Create new patch "perf stat: Save aggr value to first member of prev_raw_counts".

2. Call perf_evlist__save_aggr_prev_raw_counts to save aggr value to first member of prev_raw_counts for AGGR_GLOBAL. Then next, perf_stat_process_counter can create aggr values from per cpu values.

Following patches are impacted in v5:
perf stat: Copy counts from prev_raw_counts to evsel->counts perf stat: Save aggr value to first member of prev_raw_counts perf stat: Report summary for interval mode

v4:
---
1. Create runtime_stat_reset.

2. Zero the aggr in perf_counts__reset and use it to reset prev_raw_counts.

3. Move affinity setup and read_counter_cpu to a new function read_affinity_counters. It's only called when stat_config.summary is not set.

v3:
---
1. 'perf stat: Fix wrong per-thread runtime stat for interval mode' is a new patch which fixes an existing issue found in test.

2. We use the prev_raw_counts for summary counts. Drop the summary_counts in v2.

3. Fix some issues.

v2:
---
Rebase to perf/core branch

Jin Yao (5):
perf stat: Fix wrong per-thread runtime stat for interval mode perf counts: Reset prev_raw_counts counts
perf stat: Copy counts from prev_raw_counts to evsel->counts perf stat: Save aggr value to first member of prev_raw_counts perf stat: Report summary for interval mode

 tools/perf/builtin-stat.c | 101 ++++++++++++++++++++++++++------------
 tools/perf/util/counts.c  |   4 +-
 tools/perf/util/counts.h  |   1 +
 tools/perf/util/stat.c    |  43 +++++++++++++---
 tools/perf/util/stat.h    |   3 ++
 5 files changed, 113 insertions(+), 39 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v5 1/5 perf stat: Fix wrong per-thread runtime stat for interval mode

root@kbl-ppc:~# perf stat --per-thread -e cycles,instructions -I1000 --interval-count 2
     1.004171683             perf-3696              8,747,311      cycles
        ...
     1.004171683             perf-3696                691,730      instructions              #    0.08  insn per cycle
        ...
     2.006490373             perf-3696              1,749,936      cycles
        ...
     2.006490373             perf-3696              1,484,582      instructions              #    0.28  insn per cycle
        ...

Let's see interval 2.006490373

perf-3696              1,749,936      cycles
perf-3696              1,484,582      instructions              #    0.28  insn per cycle

insn per cycle = 1,484,582 / 1,749,936 = 0.85.
But now it's 0.28, that's not correct.

stat_config.stats[] records the per-thread runtime stat. But for interval mode, it should be reset for each interval.

So now, with this patch,

root@kbl-ppc:~# perf stat --per-thread -e cycles,instructions -I1000 --interval-count 2
     1.005818121             perf-8633              9,898,045      cycles
        ...
     1.005818121             perf-8633                693,298      instructions              #    0.07  insn per cycle
        ...
     2.007863743             perf-8633              1,551,619      cycles
        ...
     2.007863743             perf-8633              1,317,514      instructions              #    0.85  insn per cycle
        ...

Let's check interval 2.007863743.

insn per cycle = 1,317,514 / 1,551,619 = 0.85. It's correct.

This patch creates runtime_stat_reset, places it next to untime_stat_new/runtime_stat_delete and moves all runtime_stat functions before process_interval.

v4:
---
Create runtime_stat_reset.

Fixes commit 14e72a21c783 ("perf stat: Update or print per-thread stats")
---
 tools/perf/builtin-stat.c | 70 +++++++++++++++++++++++----------------
 1 file changed, 41 insertions(+), 29 deletions(-)
tools/perf/builtin-stat.c +41 -29
index e0c1ad23c768..f3b3a59ac7d2 100644
@@ -351,6 +351,46 @@ static void read_counters(struct timespec *rs)
 	}
 }
 
+static int runtime_stat_new(struct perf_stat_config *config, int nthreads)
+{
+	int i;
+
+	config->stats = calloc(nthreads, sizeof(struct runtime_stat));
+	if (!config->stats)
+		return -1;
+
+	config->stats_num = nthreads;
+
+	for (i = 0; i < nthreads; i++)
+		runtime_stat__init(&config->stats[i]);
+
+	return 0;
+}
+
+static void runtime_stat_delete(struct perf_stat_config *config)
+{
+	int i;
+
+	if (!config->stats)
+		return;
+
+	for (i = 0; i < config->stats_num; i++)
+		runtime_stat__exit(&config->stats[i]);
+
+	zfree(&config->stats);
+}
+
+static void runtime_stat_reset(struct perf_stat_config *config)
+{
+	int i;
+
+	if (!config->stats)
+		return;
+
+	for (i = 0; i < config->stats_num; i++)
+		perf_stat__reset_shadow_per_stat(&config->stats[i]);
+}
+
 static void process_interval(void)
 {
 	struct timespec ts, rs;
@@ -359,6 +399,7 @@ static void process_interval(void)
 	diff_timespec(&rs, &ts, &ref_time);
 
 	perf_stat__reset_shadow_per_stat(&rt_stat);
+	runtime_stat_reset(&stat_config);
 	read_counters(&rs);
 
 	if (STAT_RECORD) {
@@ -1737,35 +1778,6 @@ int process_cpu_map_event(struct perf_session *session,
 	return set_maps(st);
 }
 
-static int runtime_stat_new(struct perf_stat_config *config, int nthreads)
-{
-	int i;
-
-	config->stats = calloc(nthreads, sizeof(struct runtime_stat));
-	if (!config->stats)
-		return -1;
-
-	config->stats_num = nthreads;
-
-	for (i = 0; i < nthreads; i++)
-		runtime_stat__init(&config->stats[i]);
-
-	return 0;
-}
-
-static void runtime_stat_delete(struct perf_stat_config *config)
-{
-	int i;
-
-	if (!config->stats)
-		return;
-
-	for (i = 0; i < config->stats_num; i++)
-		runtime_stat__exit(&config->stats[i]);
-
-	zfree(&config->stats);
-}
-
 static const char * const stat_report_usage[] = {
 	"perf stat report [<options>]",
 	NULL,
Signature · 1 line
2.17.1
author
#2 lore raw

PATCH v5 2/5 perf counts: Reset prev_raw_counts counts

In reply to Jin Yao

When we want to reset the evsel->prev_raw_counts, zeroing the aggr is not enough, we need to reset the perf_counts too.

The perf_counts__reset zeros the perf_counts, and it should zero the aggr too. This patch changes perf_counts__reset to non-static, and calls it in evsel__reset_prev_raw_counts to reset the prev_raw_counts.

v4:
---
Zeroing the aggr in perf_counts__reset and use it to reset prev_raw_counts.

---
 tools/perf/util/counts.c | 4 +++-
 tools/perf/util/counts.h | 1 +
 tools/perf/util/stat.c   | 7 ++-----
 3 files changed, 6 insertions(+), 6 deletions(-)
tools/perf/util/counts.c +3 -1
index 615c9f3e95cb..582f3aeaf5e4 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <errno.h>
 #include <stdlib.h>
+#include <string.h>
 #include "evsel.h"
 #include "counts.h"
 #include <linux/zalloc.h>
@@ -42,10 +43,11 @@ void perf_counts__delete(struct perf_counts *counts)
 	}
 }
 
-static void perf_counts__reset(struct perf_counts *counts)
+void perf_counts__reset(struct perf_counts *counts)
 {
 	xyarray__reset(counts->loaded);
 	xyarray__reset(counts->values);
+	memset(&counts->aggr, 0, sizeof(struct perf_counts_values));
 }
 
 void evsel__reset_counts(struct evsel *evsel)
tools/perf/util/counts.h +1 -0
index 8f556c6d98fa..7ff36bf6d644 100644
@@ -37,6 +37,7 @@ perf_counts__set_loaded(struct perf_counts *counts, int cpu, int thread, bool lo
 
 struct perf_counts *perf_counts__new(int ncpus, int nthreads);
 void perf_counts__delete(struct perf_counts *counts);
+void perf_counts__reset(struct perf_counts *counts);
 
 void evsel__reset_counts(struct evsel *evsel);
 int evsel__alloc_counts(struct evsel *evsel, int ncpus, int nthreads);
tools/perf/util/stat.c +2 -5
index f4a44df9b221..e397815f0dfb 100644
@@ -163,11 +163,8 @@ static void evsel__free_prev_raw_counts(struct evsel *evsel)
 
 static void evsel__reset_prev_raw_counts(struct evsel *evsel)
 {
-	if (evsel->prev_raw_counts) {
-		evsel->prev_raw_counts->aggr.val = 0;
-		evsel->prev_raw_counts->aggr.ena = 0;
-		evsel->prev_raw_counts->aggr.run = 0;
-       }
+	if (evsel->prev_raw_counts)
+		perf_counts__reset(evsel->prev_raw_counts);
 }
 
 static int evsel__alloc_stats(struct evsel *evsel, bool alloc_raw)
Signature · 1 line
2.17.1
author
#3 lore raw

PATCH v5 3/5 perf stat: Copy counts from prev_raw_counts to evsel->counts

In reply to Jin Yao

It would be useful to support the overall statistics for perf-stat interval mode. For example, report the summary at the end of "perf-stat -I" output.

But since perf-stat can support many aggregation modes, such as --per-thread, --per-socket, -M and etc, we need a solution which doesn't bring much complexity.

The idea is to use 'evsel->prev_raw_counts' which is updated in each interval and it's saved with the latest counts. Before reporting the summary, we copy the counts from evsel->prev_raw_counts to evsel->counts, and next we just follow non-interval processing.

v5:
---
Don't save the previous aggr value to the member of [cpu0,thread0] in perf_counts. Originally that was a trick because the perf_stat_process_counter would create aggr values from per cpu values. But we don't need to do that all the time. We will handle it in next patch.

v4:
---
Change the commit message.
No functional change.

---
 tools/perf/util/stat.c | 24 ++++++++++++++++++++++++
 tools/perf/util/stat.h |  1 +
 2 files changed, 25 insertions(+)
tools/perf/util/stat.c +24 -0
index e397815f0dfb..aadc723ce871 100644
@@ -225,6 +225,30 @@ void perf_evlist__reset_prev_raw_counts(struct evlist *evlist)
 		evsel__reset_prev_raw_counts(evsel);
 }
 
+static void perf_evsel__copy_prev_raw_counts(struct evsel *evsel)
+{
+	int ncpus = evsel__nr_cpus(evsel);
+	int nthreads = perf_thread_map__nr(evsel->core.threads);
+
+	for (int thread = 0; thread < nthreads; thread++) {
+		for (int cpu = 0; cpu < ncpus; cpu++) {
+			*perf_counts(evsel->counts, cpu, thread) =
+				*perf_counts(evsel->prev_raw_counts, cpu,
+					     thread);
+		}
+	}
+
+	evsel->counts->aggr = evsel->prev_raw_counts->aggr;
+}
+
+void perf_evlist__copy_prev_raw_counts(struct evlist *evlist)
+{
+	struct evsel *evsel;
+
+	evlist__for_each_entry(evlist, evsel)
+		perf_evsel__copy_prev_raw_counts(evsel);
+}
+
 static void zero_per_pkg(struct evsel *counter)
 {
 	if (counter->per_pkg_mask)
tools/perf/util/stat.h +1 -0
index b4fdfaa7f2c0..62cf72c71869 100644
@@ -198,6 +198,7 @@ int perf_evlist__alloc_stats(struct evlist *evlist, bool alloc_raw);
 void perf_evlist__free_stats(struct evlist *evlist);
 void perf_evlist__reset_stats(struct evlist *evlist);
 void perf_evlist__reset_prev_raw_counts(struct evlist *evlist);
+void perf_evlist__copy_prev_raw_counts(struct evlist *evlist);
 
 int perf_stat_process_counter(struct perf_stat_config *config,
 			      struct evsel *counter);
Signature · 1 line
2.17.1
author
#4 lore raw

PATCH v5 4/5 perf stat: Save aggr value to first member of prev_raw_counts

In reply to Jin Yao

To collect the overall statistics for interval mode, we copy the counts from evsel->prev_raw_counts to evsel->counts.

For AGGR_GLOBAL mode, because the perf_stat_process_counter creates aggr values from per cpu values, but the per cpu values are 0, so the calculated aggr values will be always 0.

This patch uses a trick that saves the previous aggr value to the first member of perf_counts, then aggr calculation in process_counter_values can work correctly for AGGR_GLOBAL.

---
 tools/perf/util/stat.c | 10 ++++++++++
 tools/perf/util/stat.h |  1 +
 2 files changed, 11 insertions(+)
tools/perf/util/stat.c +10 -0
index aadc723ce871..fbabdd5b9b62 100644
@@ -249,6 +249,16 @@ void perf_evlist__copy_prev_raw_counts(struct evlist *evlist)
 		perf_evsel__copy_prev_raw_counts(evsel);
 }
 
+void perf_evlist__save_aggr_prev_raw_counts(struct evlist *evlist)
+{
+	struct evsel *evsel;
+
+	evlist__for_each_entry(evlist, evsel) {
+		*perf_counts(evsel->prev_raw_counts, 0, 0) =
+			evsel->prev_raw_counts->aggr;
+	}
+}
+
 static void zero_per_pkg(struct evsel *counter)
 {
 	if (counter->per_pkg_mask)
tools/perf/util/stat.h +1 -0
index 62cf72c71869..18ead55756cc 100644
@@ -199,6 +199,7 @@ void perf_evlist__free_stats(struct evlist *evlist);
 void perf_evlist__reset_stats(struct evlist *evlist);
 void perf_evlist__reset_prev_raw_counts(struct evlist *evlist);
 void perf_evlist__copy_prev_raw_counts(struct evlist *evlist);
+void perf_evlist__save_aggr_prev_raw_counts(struct evlist *evlist);
 
 int perf_stat_process_counter(struct perf_stat_config *config,
 			      struct evsel *counter);
Signature · 1 line
2.17.1
author
#7 lore raw

PATCH v5 5/5 perf stat: Report summary for interval mode

In reply to Jin Yao

Currently perf-stat supports to print counts at regular interval (-I), but it's not very easy for user to get the overall statistics.

The patch uses 'evsel->prev_raw_counts' to get counts for summary. Copy the counts to 'evsel->counts' after printing the interval results. Next, we just follow the non-interval processing.

Let's see some examples,

 root@kbl-ppc:~# perf stat -e cycles -I1000 --interval-count 2
 #           time             counts unit events
      1.000412064          2,281,114      cycles
      2.001383658          2,547,880      cycles

Performance counter stats for 'system wide':

          4,828,994      cycles

        2.002860349 seconds time elapsed
 root@kbl-ppc:~# perf stat -e cycles,instructions -I1000 --interval-count 2
 #           time             counts unit events
      1.000389902          1,536,093      cycles
      1.000389902            420,226      instructions              #    0.27  insn per cycle
      2.001433453          2,213,952      cycles
      2.001433453            735,465      instructions              #    0.33  insn per cycle

Performance counter stats for 'system wide':

          3,750,045      cycles
          1,155,691      instructions              #    0.31  insn per cycle

        2.003023361 seconds time elapsed
 root@kbl-ppc:~# perf stat -M CPI,IPC -I1000 --interval-count 2
 #           time             counts unit events
      1.000435121            905,303      inst_retired.any          #      2.9 CPI
      1.000435121          2,663,333      cycles
      1.000435121            914,702      inst_retired.any          #      0.3 IPC
      1.000435121          2,676,559      cpu_clk_unhalted.thread
      2.001615941          1,951,092      inst_retired.any          #      1.8 CPI
      2.001615941          3,551,357      cycles
      2.001615941          1,950,837      inst_retired.any          #      0.5 IPC
      2.001615941          3,551,044      cpu_clk_unhalted.thread

Performance counter stats for 'system wide':

          2,856,395      inst_retired.any          #      2.2 CPI
          6,214,690      cycles
          2,865,539      inst_retired.any          #      0.5 IPC
          6,227,603      cpu_clk_unhalted.thread
        2.003403078 seconds time elapsed

v5:
---
Call perf_evlist__save_aggr_prev_raw_counts to save aggr value to first member of prev_raw_counts for AGGR_GLOBAL. Then next, perf_stat_process_counter can create aggr values from per cpu values.

v4:
---
Move affinity setup and read_counter_cpu to a new function read_affinity_counters. It's only called when stat_config.summary is not set.

v3:
---
Use evsel->prev_raw_counts for summary counts

v2:
---
Rebase to perf/core branch

---
 tools/perf/builtin-stat.c | 31 ++++++++++++++++++++++++++++---
 tools/perf/util/stat.c    |  2 +-
 tools/perf/util/stat.h    |  1 +
 3 files changed, 30 insertions(+), 4 deletions(-)
tools/perf/builtin-stat.c +28 -3
index f3b3a59ac7d2..24deed746325 100644
@@ -314,14 +314,14 @@ static int read_counter_cpu(struct evsel *counter, struct timespec *rs, int cpu)
 	return 0;
 }
 
-static void read_counters(struct timespec *rs)
+static int read_affinity_counters(struct timespec *rs)
 {
 	struct evsel *counter;
 	struct affinity affinity;
 	int i, ncpus, cpu;
 
 	if (affinity__setup(&affinity) < 0)
-		return;
+		return -1;
 
 	ncpus = perf_cpu_map__nr(evsel_list->core.all_cpus);
 	if (!target__has_cpu(&target) || target__has_per_thread(&target))
@@ -341,6 +341,15 @@ static void read_counters(struct timespec *rs)
 		}
 	}
 	affinity__cleanup(&affinity);
+	return 0;
+}
+
+static void read_counters(struct timespec *rs)
+{
+	struct evsel *counter;
+
+	if (!stat_config.summary && (read_affinity_counters(rs) < 0))
+		return;
 
 	evlist__for_each_entry(evsel_list, counter) {
 		if (counter->err)
@@ -394,6 +403,7 @@ static void runtime_stat_reset(struct perf_stat_config *config)
 static void process_interval(void)
 {
 	struct timespec ts, rs;
+	struct stats walltime_nsecs_stats_bak;
 
 	clock_gettime(CLOCK_MONOTONIC, &ts);
 	diff_timespec(&rs, &ts, &ref_time);
@@ -407,9 +417,11 @@ static void process_interval(void)
 			pr_err("failed to write stat round event\n");
 	}
 
+	walltime_nsecs_stats_bak = walltime_nsecs_stats;
 	init_stats(&walltime_nsecs_stats);
 	update_stats(&walltime_nsecs_stats, stat_config.interval * 1000000);
 	print_counters(&rs, 0, NULL);
+	walltime_nsecs_stats = walltime_nsecs_stats_bak;
 }
 
 static void enable_counters(void)
@@ -765,6 +777,19 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 
 	update_stats(&walltime_nsecs_stats, t1 - t0);
 
+	if (interval) {
+		stat_config.interval = 0;
+		stat_config.summary = true;
+
+		if (stat_config.aggr_mode == AGGR_GLOBAL)
+			perf_evlist__save_aggr_prev_raw_counts(evsel_list);
+
+		perf_evlist__copy_prev_raw_counts(evsel_list);
+		perf_evlist__reset_prev_raw_counts(evsel_list);
+		runtime_stat_reset(&stat_config);
+		perf_stat__reset_shadow_per_stat(&rt_stat);
+	}
+
 	/*
 	 * Closing a group leader splits the group, and as we only disable
 	 * group leaders, results in remaining events becoming enabled. To
@@ -2159,7 +2184,7 @@ int cmd_stat(int argc, const char **argv)
 		}
 	}
 
-	if (!forever && status != -1 && !interval)
+	if (!forever && status != -1 && (!interval || stat_config.summary))
 		print_counters(NULL, argc, argv);
 
 	if (STAT_RECORD) {
tools/perf/util/stat.c +1 -1
index fbabdd5b9b62..481543c422a7 100644
@@ -398,7 +398,7 @@ int perf_stat_process_counter(struct perf_stat_config *config,
 	 * interval mode, otherwise overall avg running
 	 * averages will be shown for each interval.
 	 */
-	if (config->interval) {
+	if (config->interval || config->summary) {
 		for (i = 0; i < 3; i++)
 			init_stats(&ps->res_stats[i]);
 	}
tools/perf/util/stat.h +1 -0
index 18ead55756cc..a5604a20bdca 100644
@@ -110,6 +110,7 @@ struct perf_stat_config {
 	bool			 all_kernel;
 	bool			 all_user;
 	bool			 percore_show_thread;
+	bool			 summary;
 	FILE			*output;
 	unsigned int		 interval;
 	unsigned int		 timeout;
Signature · 1 line
2.17.1
#10 lore raw
In reply to Jin Yao
Quoted 44 lines · Jin Yao

Currently perf-stat supports to print counts at regular interval (-I), but it's not very easy for user to get the overall statistics.

With this patchset, it supports to report the summary at the end of interval output.

For example,

 root@kbl-ppc:~# perf stat -e cycles -I1000 --interval-count 2
 #           time             counts unit events
      1.000412064          2,281,114      cycles
      2.001383658          2,547,880      cycles

Performance counter stats for 'system wide':

          4,828,994      cycles

        2.002860349 seconds time elapsed
 root@kbl-ppc:~# perf stat -e cycles,instructions -I1000 --interval-count 2
 #           time             counts unit events
      1.000389902          1,536,093      cycles
      1.000389902            420,226      instructions              #    0.27  insn per cycle
      2.001433453          2,213,952      cycles
      2.001433453            735,465      instructions              #    0.33  insn per cycle

Performance counter stats for 'system wide':

          3,750,045      cycles
          1,155,691      instructions              #    0.31  insn per cycle

        2.003023361 seconds time elapsed
 root@kbl-ppc:~# perf stat -M CPI,IPC -I1000 --interval-count 2
 #           time             counts unit events
      1.000435121            905,303      inst_retired.any          #      2.9 CPI
      1.000435121          2,663,333      cycles
      1.000435121            914,702      inst_retired.any          #      0.3 IPC
      1.000435121          2,676,559      cpu_clk_unhalted.thread
      2.001615941          1,951,092      inst_retired.any          #      1.8 CPI
      2.001615941          3,551,357      cycles
      2.001615941          1,950,837      inst_retired.any          #      0.5 IPC
      2.001615941          3,551,044      cpu_clk_unhalted.thread
Quoted 8 lines

Performance counter stats for 'system wide':

          2,856,395      inst_retired.any          #      2.2 CPI
          6,214,690      cycles
          2,865,539      inst_retired.any          #      0.5 IPC
          6,227,603      cpu_clk_unhalted.thread
        2.003403078 seconds time elapsed

Hi Jin,
Reporting the summary will be great for understanding overall stats. So, Before the
patch where we are reseting rt_stat before read_counters to make sure, whatever printing in final aggregate is as per counts on that interval,

we used to update stats->means and other info as described in

RFC: https://lkml.org/lkml/2020/3/24/158

Now, stats->means is same as counts which we are using in generic_metric function. Is this expected behavior? I am not sure, if data like stats->means and all suppose to update per interval or we are using it somewhere else.

So, As we call update_stats for each event and for each interval, can we somehow use that
to print overall stats maybe by adding some var in struct stats to keep count of total counts for that event. Please let me know if my understanding is fine.

Thanks,
Kajol Jain

Quoted 54 lines

v5:
---
1. Create new patch "perf stat: Save aggr value to first member of prev_raw_counts".

2. Call perf_evlist__save_aggr_prev_raw_counts to save aggr value to first member of prev_raw_counts for AGGR_GLOBAL. Then next, perf_stat_process_counter can create aggr values from per cpu values.

Following patches are impacted in v5:
perf stat: Copy counts from prev_raw_counts to evsel->counts perf stat: Save aggr value to first member of prev_raw_counts perf stat: Report summary for interval mode

v4:
---
1. Create runtime_stat_reset.

2. Zero the aggr in perf_counts__reset and use it to reset prev_raw_counts.

3. Move affinity setup and read_counter_cpu to a new function read_affinity_counters. It's only called when stat_config.summary is not set.

v3:
---
1. 'perf stat: Fix wrong per-thread runtime stat for interval mode' is a new patch which fixes an existing issue found in test.

2. We use the prev_raw_counts for summary counts. Drop the summary_counts in v2.

3. Fix some issues.

v2:
---
Rebase to perf/core branch

Jin Yao (5):
perf stat: Fix wrong per-thread runtime stat for interval mode perf counts: Reset prev_raw_counts counts
perf stat: Copy counts from prev_raw_counts to evsel->counts perf stat: Save aggr value to first member of prev_raw_counts perf stat: Report summary for interval mode

 tools/perf/builtin-stat.c | 101 ++++++++++++++++++++++++++------------
 tools/perf/util/counts.c  |   4 +-
 tools/perf/util/counts.h  |   1 +
 tools/perf/util/stat.c    |  43 +++++++++++++---
 tools/perf/util/stat.h    |   3 ++
 5 files changed, 113 insertions(+), 39 deletions(-)
author
#11 lore raw

Hi Kajoljain,

Quoted 54 lines · kajoljain
Quoted 51 lines · Jin Yao

Currently perf-stat supports to print counts at regular interval (-I), but it's not very easy for user to get the overall statistics.

With this patchset, it supports to report the summary at the end of interval output.

For example,

  root@kbl-ppc:~# perf stat -e cycles -I1000 --interval-count 2
  #           time             counts unit events
       1.000412064          2,281,114      cycles
       2.001383658          2,547,880      cycles
   Performance counter stats for 'system wide':

           4,828,994      cycles

         2.002860349 seconds time elapsed
  root@kbl-ppc:~# perf stat -e cycles,instructions -I1000 --interval-count 2
  #           time             counts unit events
       1.000389902          1,536,093      cycles
       1.000389902            420,226      instructions              #    0.27  insn per cycle
       2.001433453          2,213,952      cycles
       2.001433453            735,465      instructions              #    0.33  insn per cycle

Performance counter stats for 'system wide':

           3,750,045      cycles
           1,155,691      instructions              #    0.31  insn per cycle

         2.003023361 seconds time elapsed
  root@kbl-ppc:~# perf stat -M CPI,IPC -I1000 --interval-count 2
  #           time             counts unit events
       1.000435121            905,303      inst_retired.any          #      2.9 CPI
       1.000435121          2,663,333      cycles
       1.000435121            914,702      inst_retired.any          #      0.3 IPC
       1.000435121          2,676,559      cpu_clk_unhalted.thread
       2.001615941          1,951,092      inst_retired.any          #      1.8 CPI
       2.001615941          3,551,357      cycles
       2.001615941          1,950,837      inst_retired.any          #      0.5 IPC
       2.001615941          3,551,044      cpu_clk_unhalted.thread
   Performance counter stats for 'system wide':

           2,856,395      inst_retired.any          #      2.2 CPI
           6,214,690      cycles
           2,865,539      inst_retired.any          #      0.5 IPC
           6,227,603      cpu_clk_unhalted.thread
Quoted 7 lines
Quoted 1 line
         2.003403078 seconds time elapsed

Hi Jin,
Reporting the summary will be great for understanding overall stats. So, Before the
patch where we are reseting rt_stat before read_counters to make sure, whatever printing in final aggregate is as per counts on that interval,

Yes, I had similar thoughts, so I posted following patch.

https://lore.kernel.org/lkml/[email protected]/

Quoted 4 lines

we used to update stats->means and other info as described in

RFC: https://lkml.org/lkml/2020/3/24/158

I've checked your patch but sorry I'm also not very sure if it's the expected behavior.

Quoted 3 lines

Now, stats->means is same as counts which we are using in generic_metric function. Is this expected behavior? I am not sure, if data like stats->means and all suppose to update per interval or we are using it somewhere else.

I just think it's easy to understand, that is the metric calculated by the counts per interval.

Quoted 4 lines

So, As we call update_stats for each event and for each interval, can we somehow use that
to print overall stats maybe by adding some var in struct stats to keep count of total counts for that event. Please let me know if my understanding is fine.

Adding var in 'struct stats' looks not enough (or more complicated), because perf-stat also needs to report some counts according to different aggregation modes (not only the metric). I just think copying total counts to current counts is a easy way because we can reuse most of existing non-interval processing code.

Thanks
Jin Yao

Quoted 59 lines

Thanks,
Kajol Jain

Quoted 54 lines

v5:
---
1. Create new patch "perf stat: Save aggr value to first member of prev_raw_counts".

2. Call perf_evlist__save_aggr_prev_raw_counts to save aggr value to first member of prev_raw_counts for AGGR_GLOBAL. Then next, perf_stat_process_counter can create aggr values from per cpu values.

Following patches are impacted in v5:
perf stat: Copy counts from prev_raw_counts to evsel->counts perf stat: Save aggr value to first member of prev_raw_counts perf stat: Report summary for interval mode

v4:
---
1. Create runtime_stat_reset.

2. Zero the aggr in perf_counts__reset and use it to reset prev_raw_counts.

3. Move affinity setup and read_counter_cpu to a new function read_affinity_counters. It's only called when stat_config.summary is not set.

v3:
---
1. 'perf stat: Fix wrong per-thread runtime stat for interval mode' is a new patch which fixes an existing issue found in test.

2. We use the prev_raw_counts for summary counts. Drop the summary_counts in v2.

3. Fix some issues.

v2:
---
Rebase to perf/core branch

Jin Yao (5):
perf stat: Fix wrong per-thread runtime stat for interval mode perf counts: Reset prev_raw_counts counts
perf stat: Copy counts from prev_raw_counts to evsel->counts perf stat: Save aggr value to first member of prev_raw_counts perf stat: Report summary for interval mode

  tools/perf/builtin-stat.c | 101 ++++++++++++++++++++++++++------------
  tools/perf/util/counts.c  |   4 +-
  tools/perf/util/counts.h  |   1 +
  tools/perf/util/stat.c    |  43 +++++++++++++---
  tools/perf/util/stat.h    |   3 ++
  5 files changed, 113 insertions(+), 39 deletions(-)
#8 lore raw

Re: [PATCH v5 5/5] perf stat: Report summary for interval mode

In reply to Jin Yao

On Thu, May 14, 2020 at 01:36:38PM +0800, Jin Yao wrote:

SNIP

Quoted 12 lines
tools/perf/builtin-stat.c +1 -0
 	evlist__for_each_entry(evsel_list, counter) {
 		if (counter->err)
@@ -394,6 +403,7 @@ static void runtime_stat_reset(struct perf_stat_config *config)
 static void process_interval(void)
 {
 	struct timespec ts, rs;
+	struct stats walltime_nsecs_stats_bak;
 
 	clock_gettime(CLOCK_MONOTONIC, &ts);
 	diff_timespec(&rs, &ts, &ref_time);

tools/perf/builtin-stat.c static void process_interval(void) · line 424

Quoted 8 lines
tools/perf/builtin-stat.c +2 -0
@@ -407,9 +417,11 @@ static void process_interval(void)
 			pr_err("failed to write stat round event\n");
 	}
 
+	walltime_nsecs_stats_bak = walltime_nsecs_stats;
 	init_stats(&walltime_nsecs_stats);
 	update_stats(&walltime_nsecs_stats, stat_config.interval * 1000000);
 	print_counters(&rs, 0, NULL);
+	walltime_nsecs_stats = walltime_nsecs_stats_bak;

could we instead of above initialize walltime_nsecs_stats in the condition below, like:

	init_stats(&walltime_nsecs_stats);
	update_stats(&walltime_nsecs_stats, stat_config.interval * 1000000);

jirka

Quoted 20 lines
tools/perf/builtin-stat.c +13 -0
 }
 
 static void enable_counters(void)
@@ -765,6 +777,19 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 
 	update_stats(&walltime_nsecs_stats, t1 - t0);
 
+	if (interval) {
+		stat_config.interval = 0;
+		stat_config.summary = true;
+
+		if (stat_config.aggr_mode == AGGR_GLOBAL)
+			perf_evlist__save_aggr_prev_raw_counts(evsel_list);
+
+		perf_evlist__copy_prev_raw_counts(evsel_list);
+		perf_evlist__reset_prev_raw_counts(evsel_list);
+		runtime_stat_reset(&stat_config);
+		perf_stat__reset_shadow_per_stat(&rt_stat);
+	}
+

SNIP

#5 lore raw

Re: [PATCH v5 4/5] perf stat: Save aggr value to first member of prev_raw_counts

In reply to Jin Yao
Quoted 22 lines · Jin Yao

To collect the overall statistics for interval mode, we copy the counts from evsel->prev_raw_counts to evsel->counts.

For AGGR_GLOBAL mode, because the perf_stat_process_counter creates aggr values from per cpu values, but the per cpu values are 0, so the calculated aggr values will be always 0.

This patch uses a trick that saves the previous aggr value to the first member of perf_counts, then aggr calculation in process_counter_values can work correctly for AGGR_GLOBAL.

---
 tools/perf/util/stat.c | 10 ++++++++++
 tools/perf/util/stat.h |  1 +
 2 files changed, 11 insertions(+)
tools/perf/util/stat.c +0 -0
index aadc723ce871..fbabdd5b9b62 100644

tools/perf/util/stat.c void perf_evlist__copy_prev_raw_counts(struct evlist *evlist) · line 251

Quoted 3 lines
tools/perf/util/stat.c +0 -0
@@ -249,6 +249,16 @@ void perf_evlist__copy_prev_raw_counts(struct evlist *evlist)
 		perf_evsel__copy_prev_raw_counts(evsel);
 }
 

much better, please put some comments in here explaning wha is this for, because it's not obvious ;-)

thanks,
jirka

Quoted 28 lines
tools/perf/util/stat.c +10 -0
+void perf_evlist__save_aggr_prev_raw_counts(struct evlist *evlist)
+{
+	struct evsel *evsel;
+
+	evlist__for_each_entry(evlist, evsel) {
+		*perf_counts(evsel->prev_raw_counts, 0, 0) =
+			evsel->prev_raw_counts->aggr;
+	}
+}
+
 static void zero_per_pkg(struct evsel *counter)
 {
 	if (counter->per_pkg_mask)
tools/perf/util/stat.h +1 -0
index 62cf72c71869..18ead55756cc 100644
@@ -199,6 +199,7 @@ void perf_evlist__free_stats(struct evlist *evlist);
 void perf_evlist__reset_stats(struct evlist *evlist);
 void perf_evlist__reset_prev_raw_counts(struct evlist *evlist);
 void perf_evlist__copy_prev_raw_counts(struct evlist *evlist);
+void perf_evlist__save_aggr_prev_raw_counts(struct evlist *evlist);
 
 int perf_stat_process_counter(struct perf_stat_config *config,
 			      struct evsel *counter);
Signature · 1 line
2.17.1
author
#6 lore raw

Re: [PATCH v5 4/5] perf stat: Save aggr value to first member of prev_raw_counts

Quoted 30 lines · Jiri Olsa
Quoted 22 lines · Jin Yao

To collect the overall statistics for interval mode, we copy the counts from evsel->prev_raw_counts to evsel->counts.

For AGGR_GLOBAL mode, because the perf_stat_process_counter creates aggr values from per cpu values, but the per cpu values are 0, so the calculated aggr values will be always 0.

This patch uses a trick that saves the previous aggr value to the first member of perf_counts, then aggr calculation in process_counter_values can work correctly for AGGR_GLOBAL.

---
  tools/perf/util/stat.c | 10 ++++++++++
  tools/perf/util/stat.h |  1 +
  2 files changed, 11 insertions(+)
tools/perf/util/stat.c +0 -0
index aadc723ce871..fbabdd5b9b62 100644

tools/perf/util/stat.c void perf_evlist__copy_prev_raw_counts(struct evlist *evlist) · line 251

Quoted 3 lines
tools/perf/util/stat.c +0 -0
@@ -249,6 +249,16 @@ void perf_evlist__copy_prev_raw_counts(struct evlist *evlist)
  		perf_evsel__copy_prev_raw_counts(evsel);
  }
 

much better, please put some comments in here explaning wha is this for, because it's not obvious ;-)

Quoted 3 lines

thanks,
jirka

Thanks, I will put some comments in v6.

Thanks
Jin Yao

Quoted 29 lines
Quoted 28 lines
tools/perf/util/stat.c +10 -0
+void perf_evlist__save_aggr_prev_raw_counts(struct evlist *evlist)
+{
+	struct evsel *evsel;
+
+	evlist__for_each_entry(evlist, evsel) {
+		*perf_counts(evsel->prev_raw_counts, 0, 0) =
+			evsel->prev_raw_counts->aggr;
+	}
+}
+
  static void zero_per_pkg(struct evsel *counter)
  {
  	if (counter->per_pkg_mask)
tools/perf/util/stat.h +1 -0
index 62cf72c71869..18ead55756cc 100644
@@ -199,6 +199,7 @@ void perf_evlist__free_stats(struct evlist *evlist);
  void perf_evlist__reset_stats(struct evlist *evlist);
  void perf_evlist__reset_prev_raw_counts(struct evlist *evlist);
  void perf_evlist__copy_prev_raw_counts(struct evlist *evlist);
+void perf_evlist__save_aggr_prev_raw_counts(struct evlist *evlist);
 
  int perf_stat_process_counter(struct perf_stat_config *config,
  			      struct evsel *counter);
Signature · 1 line
2.17.1
author
#9 lore raw

Re: [PATCH v5 5/5] perf stat: Report summary for interval mode

In reply to Jiri Olsa

Hi Jiri,

Quoted 28 lines · Jiri Olsa

On Thu, May 14, 2020 at 01:36:38PM +0800, Jin Yao wrote:

SNIP

Quoted 12 lines
  
  	evlist__for_each_entry(evsel_list, counter) {
  		if (counter->err)
tools/perf/builtin-stat.c +1 -0
@@ -394,6 +403,7 @@ static void runtime_stat_reset(struct perf_stat_config *config)
  static void process_interval(void)
  {
  	struct timespec ts, rs;
+	struct stats walltime_nsecs_stats_bak;
 
  	clock_gettime(CLOCK_MONOTONIC, &ts);
  	diff_timespec(&rs, &ts, &ref_time);

tools/perf/builtin-stat.c static void process_interval(void) · line 424

Quoted 8 lines
tools/perf/builtin-stat.c +2 -0
@@ -407,9 +417,11 @@ static void process_interval(void)
  			pr_err("failed to write stat round event\n");
  	}
 
+	walltime_nsecs_stats_bak = walltime_nsecs_stats;
  	init_stats(&walltime_nsecs_stats);
  	update_stats(&walltime_nsecs_stats, stat_config.interval * 1000000);
  	print_counters(&rs, 0, NULL);
+	walltime_nsecs_stats = walltime_nsecs_stats_bak;

could we instead of above initialize walltime_nsecs_stats in the condition below, like:

Quoted 5 lines
	init_stats(&walltime_nsecs_stats);
	update_stats(&walltime_nsecs_stats, stat_config.interval * 1000000);

jirka

Yes, I think that's OK and better. My fix is:

@@ -775,11 +772,11 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)

        if (stat_config.walltime_run_table)
                stat_config.walltime_run[run_idx] = t1 - t0;

- update_stats(&walltime_nsecs_stats, t1 - t0);
-

        if (interval) {
                stat_config.interval = 0;
                stat_config.summary = true;

+ init_stats(&walltime_nsecs_stats);
+ update_stats(&walltime_nsecs_stats, t1 - t0);

                if (stat_config.aggr_mode == AGGR_GLOBAL)
                        perf_evlist__save_aggr_prev_raw_counts(evsel_list);
@@ -788,7 +785,8 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
                perf_evlist__reset_prev_raw_counts(evsel_list);
                runtime_stat_reset(&stat_config);
                perf_stat__reset_shadow_per_stat(&rt_stat);

- }
+ } else
+ update_stats(&walltime_nsecs_stats, t1 - t0);

/*
* Closing a group leader splits the group, and as we only disable

Thanks
Jin Yao

Quoted 23 lines
Quoted 20 lines
tools/perf/builtin-stat.c +13 -0
  }
 
  static void enable_counters(void)
@@ -765,6 +777,19 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 
  	update_stats(&walltime_nsecs_stats, t1 - t0);
 
+	if (interval) {
+		stat_config.interval = 0;
+		stat_config.summary = true;
+
+		if (stat_config.aggr_mode == AGGR_GLOBAL)
+			perf_evlist__save_aggr_prev_raw_counts(evsel_list);
+
+		perf_evlist__copy_prev_raw_counts(evsel_list);
+		perf_evlist__reset_prev_raw_counts(evsel_list);
+		runtime_stat_reset(&stat_config);
+		perf_stat__reset_shadow_per_stat(&rt_stat);
+	}
+

SNIP