Skip to content

RFC Share events between metrics

Ian Rogers bpf linux-perf-users lkml netdev 4 versions – Applied

Status from this message: Thanks, applied.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 posting 2/27 patches 9 repliesIn review
  2. v3 RFC Latest14 patches 10 repliesApplied
  3. v2 RFC posting 1/223 patches 7 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v1 RFC7 patches8 repliesSuperseded

Patches in v3 Applied

#Subject and filesSizeRepliesApplied
1/14perf parse-events: expand add PMU error/verbose messagestools/perf/arch/x86/util/intel-pt.c tools/perf/tests/pmu.c tools/perf/util/parse-events.c +2 more+72 -281 reply
2/14perf test: improve pmu event metric testingtools/perf/tests/builtin-test.c tools/perf/tests/expr.c tools/perf/tests/pmu-events.c +1 more+160 -60 replies
3/14lib/bpf hashmap: increase portabilitytools/lib/bpf/hashmap.h+1 -20 replies
4/14libbpf: Fix memory leak and possible double-free in hashmap__cleartools/lib/bpf/hashmap.c+7 -00 replies
5/14perf expr: fix memory leaks in bisontools/perf/util/expr.y+3 -00 replies
6/14perf evsel: fix 2 memory leakstools/perf/util/evsel.c+2 -02 replies
7/14perf expr: migrate expr ids table to libbpf's hashmaptools/perf/tests/expr.c tools/perf/tests/pmu-events.c tools/perf/util/expr.c +4 more+193 -1810 replies
8/14perf metricgroup: change evlist_used to a bitmaptools/perf/util/metricgroup.c+8 -100 replies
9/14perf metricgroup: free metric_events on errortools/perf/util/metricgroup.c+3 -01 reply
10/14perf metricgroup: always place duration_time lasttools/perf/util/metricgroup.c+9 -90 replies
11/14perf metricgroup: delay events string creationtools/perf/util/metricgroup.c+25 -130 replies
12/14perf metricgroup: order event groups by sizetools/perf/util/metricgroup.c+15 -14 replies
13/14perf metricgroup: remove duped metric group eventstools/perf/util/metricgroup.c+60 -361 reply
14/14perf metricgroup: add options to not group or mergetools/perf/Documentation/perf-stat.txt tools/perf/builtin-stat.c tools/perf/util/metricgroup.c +2 more+71 -180 replies

Reviews trailers per patch

2 of 14 patches have a Reviewed-by or Acked-by.

PatchAndi Kleen
1/14R-b
2/14
3/14
4/14
5/14
6/14R-b
7/14
8/14
9/14
10/14
11/14
12/14
13/14
14/14

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [RFC PATCH v3 00/14] Share events between metrics

Metric groups contain metrics. Metrics create groups of events to ideally be scheduled together. Often metrics refer to the same events, for example, a cache hit and cache miss rate. Using separate event groups means these metrics are multiplexed at different times and the counts don't sum to 100%. More multiplexing also decreases the accuracy of the measurement.

This change orders metrics from groups or the command line, so that the ones with the most events are set up first. Later metrics see if groups already provide their events, and reuse them if possible. Unnecessary events and groups are eliminated.

The option --metric-no-group is added so that metrics aren't placed in groups. This affects multiplexing and may increase sharing.

The option --metric-mo-merge is added and with this option the existing grouping behavior is preserved.

RFC because:
- without this change events within a metric may get scheduled together, after they may appear as part of a larger group and be multiplexed at different times, lowering accuracy - however, less multiplexing may compensate for this.
- libbpf's hashmap is used, however, libbpf is an optional requirement for building perf.
- other things I'm not thinking of.

Thanks!

Example on Sandybridge:

$ perf stat -a --metric-no-merge -M TopDownL1_SMT sleep 1

Performance counter stats for 'system wide':

          14931177      cpu_clk_unhalted.one_thread_active #     0.47 Backend_Bound_SMT        (12.45%)
          32314653      int_misc.recovery_cycles_any                                     (16.23%)
         555020905      uops_issued.any                                               (18.85%)
        1038651176      idq_uops_not_delivered.core                                     (24.95%)
          43003170      cpu_clk_unhalted.ref_xclk                                     (25.20%)
        1154926272      cpu_clk_unhalted.thread                                       (31.50%)
         656873544      uops_retired.retire_slots                                     (31.11%)
          16491988      cpu_clk_unhalted.one_thread_active #     0.06 Bad_Speculation_SMT      (31.10%)
          32064061      int_misc.recovery_cycles_any                                     (31.04%)
         648394934      uops_issued.any                                               (31.14%)
          42107506      cpu_clk_unhalted.ref_xclk                                     (24.94%)
        1124565282      cpu_clk_unhalted.thread                                       (31.14%)
         523430886      uops_retired.retire_slots                                     (31.05%)
          12328380      cpu_clk_unhalted.one_thread_active #     0.35 Frontend_Bound_SMT       (10.08%)
          42651836      cpu_clk_unhalted.ref_xclk                                     (10.08%)
        1006287722      idq_uops_not_delivered.core                                     (10.08%)
        1130593027      cpu_clk_unhalted.thread                                       (10.08%)
          14209258      cpu_clk_unhalted.one_thread_active #     0.18 Retiring_SMT             (6.39%)
          41904474      cpu_clk_unhalted.ref_xclk                                     (6.39%)
         522251584      uops_retired.retire_slots                                     (6.39%)
        1111257754      cpu_clk_unhalted.thread                                       (6.39%)
          12930094      cpu_clk_unhalted.one_thread_active # 2865823806.05 SLOTS_SMT           (11.06%)
          40975376      cpu_clk_unhalted.ref_xclk                                     (11.06%)
        1089204936      cpu_clk_unhalted.thread                                       (11.06%)
       1.002165509 seconds time elapsed
$ perf stat -a -M TopDownL1_SMT sleep 1

Performance counter stats for 'system wide':

11893411 cpu_clk_unhalted.one_thread_active # 2715516883.49 SLOTS_SMT
# 0.19 Retiring_SMT
# 0.33 Frontend_Bound_SMT
# 0.04 Bad_Speculation_SMT
# 0.44 Backend_Bound_SMT (71.46%)
28458253 int_misc.recovery_cycles_any (71.44%)
562710994 uops_issued.any (71.42%)
907105260 idq_uops_not_delivered.core (57.12%)
39797715 cpu_clk_unhalted.ref_xclk (57.12%)
1045357060 cpu_clk_unhalted.thread (71.41%)
504809283 uops_retired.retire_slots (71.44%)

1.001939294 seconds time elapsed

Note that without merging the metrics sum to 1.06, but with merging the sum is 1.

Example on Cascadelake:

$ perf stat -a --metric-no-merge -M TopDownL1_SMT sleep 1

Performance counter stats for 'system wide':

          13678949      cpu_clk_unhalted.one_thread_active #     0.59 Backend_Bound_SMT        (13.35%)
         121286613      int_misc.recovery_cycles_any                                     (18.58%)
        4041490966      uops_issued.any                                               (18.81%)
        2665605457      idq_uops_not_delivered.core                                     (24.81%)
         111757608      cpu_clk_unhalted.ref_xclk                                     (25.03%)
        7579026491      cpu_clk_unhalted.thread                                       (31.27%)
        3848429110      uops_retired.retire_slots                                     (31.23%)
          15554046      cpu_clk_unhalted.one_thread_active #     0.02 Bad_Speculation_SMT      (31.19%)
         119582342      int_misc.recovery_cycles_any                                     (31.16%)
        3813943706      uops_issued.any                                               (31.14%)
         113151605      cpu_clk_unhalted.ref_xclk                                     (24.89%)
        7621196102      cpu_clk_unhalted.thread                                       (31.12%)
        3735690253      uops_retired.retire_slots                                     (31.12%)
          13727352      cpu_clk_unhalted.one_thread_active #     0.16 Frontend_Bound_SMT       (12.50%)
         115441454      cpu_clk_unhalted.ref_xclk                                     (12.50%)
        2824946246      idq_uops_not_delivered.core                                     (12.50%)
        7817227775      cpu_clk_unhalted.thread                                       (12.50%)
          13267908      cpu_clk_unhalted.one_thread_active #     0.21 Retiring_SMT             (6.31%)
         114015605      cpu_clk_unhalted.ref_xclk                                     (6.31%)
        3722498773      uops_retired.retire_slots                                     (6.31%)
        7771438396      cpu_clk_unhalted.thread                                       (6.31%)
          14948307      cpu_clk_unhalted.one_thread_active # 18085611559.36 SLOTS_SMT          (6.30%)
         115632797      cpu_clk_unhalted.ref_xclk                                     (6.30%)
        8007628156      cpu_clk_unhalted.thread                                       (6.30%)
       1.006256703 seconds time elapsed
$ perf stat -a -M TopDownL1_SMT sleep 1

Performance counter stats for 'system wide':

35999534 cpu_clk_unhalted.one_thread_active # 25969550384.66 SLOTS_SMT
# 0.40 Retiring_SMT
# 0.14 Frontend_Bound_SMT
# 0.02 Bad_Speculation_SMT
# 0.44 Backend_Bound_SMT (71.35%)
133499018 int_misc.recovery_cycles_any (71.36%)
10736468874 uops_issued.any (71.40%)
3518076530 idq_uops_not_delivered.core (57.24%)
78296616 cpu_clk_unhalted.ref_xclk (57.25%)
8894997400 cpu_clk_unhalted.thread (71.50%)
10409738753 uops_retired.retire_slots (71.40%)

1.011611791 seconds time elapsed

Note that without merging the metrics sum to 0.98, but with merging the sum is 1.

v3. is a rebase with following the merging of patches in v2. It also adds the metric-no-group and metric-no-merge flags.
v2. is the entire patch set based on acme's perf/core tree and includes a cherry-picks. Patch 13 was sent for review to the bpf maintainers here:
https://lore.kernel.org/lkml/[email protected]/
v1. was based on the perf metrics fixes and test sent here:
https://lore.kernel.org/lkml/[email protected]/

Andrii Nakryiko (1):
libbpf: Fix memory leak and possible double-free in hashmap__clear

Ian Rogers (13):
perf parse-events: expand add PMU error/verbose messages perf test: improve pmu event metric testing lib/bpf hashmap: increase portability
perf expr: fix memory leaks in bison
perf evsel: fix 2 memory leaks
perf expr: migrate expr ids table to libbpf's hashmap perf metricgroup: change evlist_used to a bitmap perf metricgroup: free metric_events on error perf metricgroup: always place duration_time last perf metricgroup: delay events string creation perf metricgroup: order event groups by size perf metricgroup: remove duped metric group events perf metricgroup: add options to not group or merge

 tools/lib/bpf/hashmap.c                |   7 +
 tools/lib/bpf/hashmap.h                |   3 +-
 tools/perf/Documentation/perf-stat.txt |  19 ++
 tools/perf/arch/x86/util/intel-pt.c    |  32 +--
 tools/perf/builtin-stat.c              |  11 +-
 tools/perf/tests/builtin-test.c        |   5 +
 tools/perf/tests/expr.c                |  41 ++--
 tools/perf/tests/pmu-events.c          | 159 +++++++++++++-
 tools/perf/tests/pmu.c                 |   4 +-
 tools/perf/tests/tests.h               |   2 +
 tools/perf/util/evsel.c                |   2 +
 tools/perf/util/expr.c                 | 129 +++++++-----
 tools/perf/util/expr.h                 |  22 +-
 tools/perf/util/expr.y                 |  25 +--
 tools/perf/util/metricgroup.c          | 277 ++++++++++++++++---------
 tools/perf/util/metricgroup.h          |   6 +-
 tools/perf/util/parse-events.c         |  29 ++-
 tools/perf/util/pmu.c                  |  33 +--
 tools/perf/util/pmu.h                  |   2 +-
 tools/perf/util/stat-shadow.c          |  49 +++--
 tools/perf/util/stat.h                 |   2 +
 21 files changed, 592 insertions(+), 267 deletions(-)
Signature · 1 line
2.26.2.645.ge9eca65c58-goog