perf, tools: Refactor and support interval and CSV metrics

From: Andi Kleen <[email protected]>

Now support CSV output for metrics. With the new output callbacks this is relatively straight forward by creating new callbacks.

This allows to easily plot metrics from CSV files.

The new line callback needs to know the number of fields to skip them correctly

Example output before:

% perf stat -x, true
0.200687,,task-clock,200687,100.00
0,,context-switches,200687,100.00
0,,cpu-migrations,200687,100.00
40,,page-faults,200687,100.00
730871,,cycles,203601,100.00
551056,,stalled-cycles-frontend,203601,100.00
<not supported>,,stalled-cycles-backend,0,100.00
385523,,instructions,203601,100.00
78028,,branches,203601,100.00
3946,,branch-misses,203601,100.00

After:

% perf stat -x, true .502457,,task-clock,502457,100.00,0.485,CPUs utilized 0,,context-switches,502457,100.00,0.000,K/sec 0,,cpu-migrations,502457,100.00,0.000,K/sec 45,,page-faults,502457,100.00,0.090,M/sec 644692,,cycles,509102,100.00,1.283,GHz 423470,,stalled-cycles-frontend,509102,100.00,65.69,frontend cycles idle <not supported>,,stalled-cycles-backend,0,100.00,,,, 492701,,instructions,509102,100.00,0.76,insn per cycle ,,,,,0.86,stalled cycles per insn 97767,,branches,509102,100.00,194.578,M/sec 4788,,branch-misses,509102,100.00,4.90,of all branches

or easier readable

perf stat -x, -o x.csv true
[ak@tassilo hle]$ column -s, -t x.csv

0.490635                                 task-clock               490635  100.00  0.489    CPUs utilized
0                                        context-switches         490635  100.00  0.000    K/sec
0                                        cpu-migrations           490635  100.00  0.000    K/sec
45                                       page-faults              490635  100.00  0.092    M/sec
629080                                   cycles                   497698  100.00  1.282    GHz
409498                                   stalled-cycles-frontend  497698  100.00  65.09    frontend cycles idle
<not supported>                          stalled-cycles-backend   0       100.00
491424                                   instructions             497698  100.00  0.78     insn per cycle
                                                                                  0.83     stalled cycles per insn
97278                                    branches                 497698  100.00  198.270  M/sec
4569                                     branch-misses            497698  100.00  4.70     of all branches

Two new fields are added: metric value and metric name.

From: Andi Kleen <[email protected]>

Enable metrics printing in --per-core / --per-socket mode. We need to save the shadow metrics in a unique place. Always use the first CPU in the aggregation. Then use the same CPU to retrieve the shadow value later.

Example output:

% perf stat --per-core -a ./BC1s

Performance counter stats for 'system wide':

S0-C0           2        2966.020381      task-clock (msec)         #    2.004 CPUs utilized            (100.00%)
S0-C0           2                 49      context-switches          #    0.017 K/sec                    (100.00%)
S0-C0           2                  4      cpu-migrations            #    0.001 K/sec                    (100.00%)
S0-C0           2                467      page-faults               #    0.157 K/sec
S0-C0           2      4,599,061,773      cycles                    #    1.551 GHz                      (100.00%)
S0-C0           2      9,755,886,883      instructions              #    2.12  insn per cycle           (100.00%)
S0-C0           2      1,906,272,125      branches                  #  642.704 M/sec                    (100.00%)
S0-C0           2         81,180,867      branch-misses             #    4.26% of all branches
S0-C1           2        2965.995373      task-clock (msec)         #    2.003 CPUs utilized            (100.00%)
S0-C1           2                 62      context-switches          #    0.021 K/sec                    (100.00%)
S0-C1           2                  8      cpu-migrations            #    0.003 K/sec                    (100.00%)
S0-C1           2                281      page-faults               #    0.095 K/sec
S0-C1           2          6,347,290      cycles                    #    0.002 GHz                      (100.00%)
S0-C1           2          4,654,156      instructions              #    0.73  insn per cycle           (100.00%)
S0-C1           2            947,121      branches                  #    0.319 M/sec                    (100.00%)
S0-C1           2             37,322      branch-misses             #    3.94% of all branches

       1.480409747 seconds time elapsed

v2: Rebase to older patches
v3: Document shadow cpus. Fix aggr_get_id argument. Fix -A shadows (Jiri)

Signed-off-by Andi Kleen <[email protected]>

---
 tools/perf/builtin-stat.c     | 61 +++++++++++++++++++++++++++++++++++++------
 tools/perf/util/stat-shadow.c |  7 +++++
 2 files changed, 60 insertions(+), 8 deletions(-)

tools/perf/builtin-stat.c +2 -0

index 2ffb822..c79e571 100644
@@ -740,6 +740,8 @@ struct outstate {
 	bool newline;
 	const char *prefix;
 	int  nfields;
+	int  id, nr;
+	struct perf_evsel *evsel;
 };
 
 #define METRIC_LEN  35

Quoted 52 lines · Andi Kleen

From: Andi Kleen <[email protected]>

Enable metrics printing in --per-core / --per-socket mode. We need to save the shadow metrics in a unique place. Always use the first CPU in the aggregation. Then use the same CPU to retrieve the shadow value later.

Example output:

% perf stat --per-core -a ./BC1s

Performance counter stats for 'system wide':

S0-C0           2        2966.020381      task-clock (msec)         #    2.004 CPUs utilized            (100.00%)
S0-C0           2                 49      context-switches          #    0.017 K/sec                    (100.00%)
S0-C0           2                  4      cpu-migrations            #    0.001 K/sec                    (100.00%)
S0-C0           2                467      page-faults               #    0.157 K/sec
S0-C0           2      4,599,061,773      cycles                    #    1.551 GHz                      (100.00%)
S0-C0           2      9,755,886,883      instructions              #    2.12  insn per cycle           (100.00%)
S0-C0           2      1,906,272,125      branches                  #  642.704 M/sec                    (100.00%)
S0-C0           2         81,180,867      branch-misses             #    4.26% of all branches
S0-C1           2        2965.995373      task-clock (msec)         #    2.003 CPUs utilized            (100.00%)
S0-C1           2                 62      context-switches          #    0.021 K/sec                    (100.00%)
S0-C1           2                  8      cpu-migrations            #    0.003 K/sec                    (100.00%)
S0-C1           2                281      page-faults               #    0.095 K/sec
S0-C1           2          6,347,290      cycles                    #    0.002 GHz                      (100.00%)
S0-C1           2          4,654,156      instructions              #    0.73  insn per cycle           (100.00%)
S0-C1           2            947,121      branches                  #    0.319 M/sec                    (100.00%)
S0-C1           2             37,322      branch-misses             #    3.94% of all branches

       1.480409747 seconds time elapsed

v2: Rebase to older patches
v3: Document shadow cpus. Fix aggr_get_id argument. Fix -A shadows (Jiri)

Signed-off-by Andi Kleen <[email protected]>

---
 tools/perf/builtin-stat.c     | 61 +++++++++++++++++++++++++++++++++++++------
 tools/perf/util/stat-shadow.c |  7 +++++
 2 files changed, 60 insertions(+), 8 deletions(-)

tools/perf/builtin-stat.c +2 -0

index 2ffb822..c79e571 100644
@@ -740,6 +740,8 @@ struct outstate {
 	bool newline;
 	const char *prefix;
 	int  nfields;
+	int  id, nr;
+	struct perf_evsel *evsel;
 };
 

tools/perf/builtin-stat.c static void do_new_line_std(struct outstate *os) · line 762

Quoted 8 lines

tools/perf/builtin-stat.c +1 -0

 #define METRIC_LEN  35

@@ -755,12 +757,9 @@ static void do_new_line_std(struct outstate *os)
 {
 	fputc('\n', os->fh);
 	fputs(os->prefix, os->fh);
+	aggr_printout(os->evsel, os->id, os->nr);
 	if (stat_config.aggr_mode == AGGR_NONE)
 		fprintf(os->fh, "        ");

any reason for this extra space for AGGR_NONE?

From: Andi Kleen <[email protected]>

With all the recently added fields in the perf stat CSV output we should finally document them in the man page. Do this here.

v2: Fix fields in documentation (Jiri)
v3: fix order of fields again (Jiri)
v4: Change order again.

Signed-off-by Andi Kleen <[email protected]>

---
 tools/perf/Documentation/perf-stat.txt | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

tools/perf/Documentation/perf-stat.txt +0 -0

index 52ef7a9..3ae7907 100644
@@ -211,6 +211,27 @@ $ perf stat -- make -j
 
  Wall-clock time elapsed:   719.554352 msecs
 

tools/perf/Documentation/perf-stat.txt +11 -0

+	- counter value
+	- unit of the counter value or empty
+	- event name
+	- run time of counter
+	- percentage of measurement time the counter was running
+	- optional variance if multiple values are collected with -r
+	- optional metric value
+	- optional unit of metric
+
+Additional metrics may be printed with all earlier fields being empty.
+
 SEE ALSO
 --------
 linkperf:perf-top[1], linkperf:perf-list[1]

Signature · 1 line

2.5.0

From: Andi Kleen <[email protected]>

Add a new mode to only print metrics. Sometimes we don't care about the raw values, just want the computed metrics. This allows more compact printing, so with -I each sample is only a single line. This also allows easier plotting and processing with other tools.

The main target is with using --topdown, but it also works with -T and standard perf stat. A few metrics are not supported.

To avoiding having to hardcode all the metrics in the code it uses a two pass approach: first compute dummy metrics and only print the headers in the print_metric callback. Then use the callback to print the actual values.

There are some additional changes
in the stat printout code to handle all metrics being on a single line.

One issue is that the column code doesn't know in advance what events are not supported by the CPU, and it would be hard to find out as this could change based on dynamic conditions. That causes empty columns in some cases.

The output can be fairly wide, often you may need more than 80 columns.

Example:

% perf stat -a -I 1000 --metric-only
1.001452803 frontend cycles idle insn per cycle stalled cycles per insn branch-misses of all branches

     1.001452803  158.91%               0.66                2.39                    2.92%
     2.002192321  180.63%               0.76                2.08                    2.96%
     3.003088282  150.59%               0.62                2.57                    2.84%
     4.004369835  196.20%               0.98                1.62                    3.79%
     5.005227314  231.98%               0.84                1.90                    4.71%

Quoted 3 lines

% perf stat -a -I 1000 --metric-only
1.001452803 frontend cycles idle insn per cycle stalled cycles per insn branch-misses of all branches

where does 'frontend' collumn come from?
can't see it in the code or get it in output..

From: Andi Kleen <[email protected]>

Add a new mode to only print metrics. Sometimes we don't care about the raw values, just want the computed metrics. This allows more compact printing, so with -I each sample is only a single line. This also allows easier plotting and processing with other tools.

The main target is with using --topdown, but it also works with -T and standard perf stat. A few metrics are not supported.

To avoiding having to hardcode all the metrics in the code it uses a two pass approach: first compute dummy metrics and only print the headers in the print_metric callback. Then use the callback to print the actual values.

There are some additional changes
in the stat printout code to handle all metrics being on a single line.

One issue is that the column code doesn't know in advance what events are not supported by the CPU, and it would be hard to find out as this could change based on dynamic conditions. That causes empty columns in some cases.

The output can be fairly wide, often you may need more than 80 columns.

Example:

% perf stat -a -I 1000 --metric-only

     1.001452803 frontend cycles idle insn per cycle       stalled cycles per insn branch-misses of all branches
     1.001452803  158.91%               0.66                2.39                    2.92%
     2.002192321  180.63%               0.76                2.08                    2.96%
     3.003088282  150.59%               0.62                2.57                    2.84%
     4.004369835  196.20%               0.98                1.62                    3.79%
     5.005227314  231.98%               0.84                1.90                    4.71%

Quoted 7 lines

% perf stat -a -I 1000 --metric-only

     1.001452803 frontend cycles idle insn per cycle       stalled cycles per insn branch-misses of all branches
     1.001452803  158.91%               0.66                2.39                    2.92%
     2.002192321  180.63%               0.76                2.08                    2.96%
     3.003088282  150.59%               0.62                2.57                    2.84%
     4.004369835  196.20%               0.98                1.62                    3.79%
     5.005227314  231.98%               0.84                1.90                    4.71%

could you please align values with the headers

Quoted 27 lines · Andi Kleen

From: Andi Kleen <[email protected]>

Add a new mode to only print metrics. Sometimes we don't care about the raw values, just want the computed metrics. This allows more compact printing, so with -I each sample is only a single line. This also allows easier plotting and processing with other tools.

The main target is with using --topdown, but it also works with -T and standard perf stat. A few metrics are not supported.

To avoiding having to hardcode all the metrics in the code it uses a two pass approach: first compute dummy metrics and only print the headers in the print_metric callback. Then use the callback to print the actual values.

There are some additional changes
in the stat printout code to handle all metrics being on a single line.

One issue is that the column code doesn't know in advance what events are not supported by the CPU, and it would be hard to find out as this could change based on dynamic conditions. That causes empty columns in some cases.

The output can be fairly wide, often you may need more than 80 columns.

Example:

Quoted 7 lines

% perf stat -a -I 1000 --metric-only

     1.001452803 frontend cycles idle insn per cycle       stalled cycles per insn branch-misses of all branches
     1.001452803  158.91%               0.66                2.39                    2.92%
     2.002192321  180.63%               0.76                2.08                    2.96%
     3.003088282  150.59%               0.62                2.57                    2.84%
     4.004369835  196.20%               0.98                1.62                    3.79%
     5.005227314  231.98%               0.84                1.90                    4.71%

could you please align values with the headers

also I have same issue with the stalled cycles
being printed as empty in here:

[jolsa@krava perf]$ sudo ./perf stat -a -I 1000 --metric-only

     1.000583845 insn per cycle       stalled cycles per insn branch-misses of all branches 
     1.000583845    0.45                                         4.55%                      
     2.001055181    0.86                                         4.06%                      
     3.001262956    0.63                                         3.59%                      

Quoted 36 lines · Jiri Olsa

Quoted 34 lines · Andi Kleen

From: Andi Kleen <[email protected]>

Add a new mode to only print metrics. Sometimes we don't care about the raw values, just want the computed metrics. This allows more compact printing, so with -I each sample is only a single line. This also allows easier plotting and processing with other tools.

The main target is with using --topdown, but it also works with -T and standard perf stat. A few metrics are not supported.

To avoiding having to hardcode all the metrics in the code it uses a two pass approach: first compute dummy metrics and only print the headers in the print_metric callback. Then use the callback to print the actual values.

There are some additional changes
in the stat printout code to handle all metrics being on a single line.

One issue is that the column code doesn't know in advance what events are not supported by the CPU, and it would be hard to find out as this could change based on dynamic conditions. That causes empty columns in some cases.

The output can be fairly wide, often you may need more than 80 columns.

Example:

% perf stat -a -I 1000 --metric-only

     1.001452803 frontend cycles idle insn per cycle       stalled cycles per insn branch-misses of all branches
     1.001452803  158.91%               0.66                2.39                    2.92%
     2.002192321  180.63%               0.76                2.08                    2.96%
     3.003088282  150.59%               0.62                2.57                    2.84%
     4.004369835  196.20%               0.98                1.62                    3.79%
     5.005227314  231.98%               0.84                1.90                    4.71%

Quoted 1 line

could you please align values with the headers

you addressed this in new version

Quoted 5 lines

also I have same issue with the stalled cycles
being printed as empty in here:

Quoted 5 lines

[jolsa@krava perf]$ sudo ./perf stat -a -I 1000 --metric-only

     1.000583845 insn per cycle       stalled cycles per insn branch-misses of all branches 
     1.000583845    0.45                                         4.55%                      
     2.001055181    0.86                                         4.06%                      
     3.001262956    0.63                                         3.59%                      

but not this one..

jirka

Quoted 37 lines · Jiri Olsa

Quoted 36 lines · Jiri Olsa

Quoted 34 lines · Andi Kleen

From: Andi Kleen <[email protected]>

Add a new mode to only print metrics. Sometimes we don't care about the raw values, just want the computed metrics. This allows more compact printing, so with -I each sample is only a single line. This also allows easier plotting and processing with other tools.

The main target is with using --topdown, but it also works with -T and standard perf stat. A few metrics are not supported.

To avoiding having to hardcode all the metrics in the code it uses a two pass approach: first compute dummy metrics and only print the headers in the print_metric callback. Then use the callback to print the actual values.

There are some additional changes
in the stat printout code to handle all metrics being on a single line.

One issue is that the column code doesn't know in advance what events are not supported by the CPU, and it would be hard to find out as this could change based on dynamic conditions. That causes empty columns in some cases.

The output can be fairly wide, often you may need more than 80 columns.

Example:

% perf stat -a -I 1000 --metric-only

     1.001452803 frontend cycles idle insn per cycle       stalled cycles per insn branch-misses of all branches
     1.001452803  158.91%               0.66                2.39                    2.92%
     2.002192321  180.63%               0.76                2.08                    2.96%
     3.003088282  150.59%               0.62                2.57                    2.84%
     4.004369835  196.20%               0.98                1.62                    3.79%
     5.005227314  231.98%               0.84                1.90                    4.71%

Quoted 3 lines

Quoted 1 line

could you please align values with the headers

you addressed this in new version

It's not changed. The values are aligned, but not left aligned.

Left aligned numbers would be confusing if you have multiple orders of magnitudes.

And it uses 7 wide fields always like the rest of perf stat

Quoted 5 lines

Quoted 5 lines

also I have same issue with the stalled cycles
being printed as empty in here:

Quoted 7 lines

Quoted 5 lines

[jolsa@krava perf]$ sudo ./perf stat -a -I 1000 --metric-only

     1.000583845 insn per cycle       stalled cycles per insn branch-misses of all branches 
     1.000583845    0.45                                         4.55%                      
     2.001055181    0.86                                         4.06%                      
     3.001262956    0.63                                         3.59%                      

but not this one..

I think that is fixed by one of the later patches. At least it works for me.

% perf stat -a -I 1000 --metric-only

     1.003282611 insn per cycle       branch-misses of all branches 
     1.003282611    0.10               22.74%                      
     2.005159802    0.10               23.13%                     

-Andi

From: Andi Kleen <[email protected]>

Add an extra check for frontend stalled in the metrics. This avoids an extra column for the --metric-only case when the CPU does not support frontend stalled.

Signed-off-by Andi Kleen <[email protected]>

---
 tools/perf/util/stat-shadow.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

tools/perf/util/stat-shadow.c +2 -0

index 5e2d2e3..7f87ed7 100644
@@ -2,6 +2,7 @@
 #include "evsel.h"
 #include "stat.h"
 #include "color.h"
+#include "pmu.h"
 
 enum {
 	CTX_BIT_USER	= 1 << 0,

@@ -35,6 +36,7 @@ static struct stats runtime_dtlb_cache_stats[NUM_CTX][MAX_NR_CPUS];
 static struct stats runtime_cycles_in_tx_stats[NUM_CTX][MAX_NR_CPUS];
 static struct stats runtime_transaction_stats[NUM_CTX][MAX_NR_CPUS];
 static struct stats runtime_elision_stats[NUM_CTX][MAX_NR_CPUS];
+static bool have_frontend_stalled;
 
 struct stats walltime_nsecs_stats;
 

Fixed last feedback.

[v5: Fix mainly bisect problems. No regressions introduced by one patch and fixed again later. Some minor fixes in addition]
[v6: Fix running/noise printing patch.]
[v7: Reorder and merge two patches to avoid a bisect hole where unsupported was printed as 0]
[v8: Minor fixes for review feedback. See changelog in patches.] [v9: Fix newline bug. Add support for -A for --metric-only]
[v10: Remove extra "noise" printing (Jiri)
Fix fields in documentation (Jiri)]

[v11: Fix manpage again. Avoid extra metric output in CSV mode.] [v12: Move CSV metrics fields to after running/enabled/variance.
Minor fixes.]
[v13: Address review comments. Now probe for stalled events
in advance to avoid empty columns or lines. Fix -A shadowing. Various minor changes. Drop merged patches.]
[v14: Fix empty lines with CSV metrics. Avoid one more empty column
in metric-only.]