Skip to content

AMD and Intel metric generation with Python

Ian Rogers linux-perf-users lkml 4 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v11 Latest35 patches 0 repliesNew
  2. v1035 patches 4 repliesSuperseded
  3. v948 patches 7 repliesApplied
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v852 patches12 repliesSuperseded

Patches in v11 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/35perf jevents: Build support for generating metrics from pythontools/perf/.gitignore tools/perf/Makefile.perf tools/perf/pmu-events/Build +3 more+184 -10 replies
2/35perf jevents: Add load event json to verify and allow fallbackstools/perf/pmu-events/Build tools/perf/pmu-events/amd_metrics.py tools/perf/pmu-events/arm64_metrics.py +2 more+101 -150 replies
3/35perf jevents: Add RAPL event metric for AMD zen modelstools/perf/pmu-events/amd_metrics.py+28 -30 replies
4/35perf jevents: Add idle metric for AMD zen modelstools/perf/pmu-events/amd_metrics.py+14 -20 replies
5/35perf jevents: Add upc metric for uops per cycle for AMDtools/perf/pmu-events/amd_metrics.py+19 -30 replies
6/35perf jevents: Add br metric group for branch statistics on AMDtools/perf/pmu-events/amd_metrics.py+105 -00 replies
7/35perf jevents: Add itlb metric group for AMDtools/perf/pmu-events/amd_metrics.py+49 -00 replies
8/35perf jevents: Add dtlb metric group for AMDtools/perf/pmu-events/amd_metrics.py+111 -00 replies
9/35perf jevents: Add uncore l3 metric group for AMDtools/perf/pmu-events/amd_metrics.py+19 -00 replies
10/35perf jevents: Add load store breakdown metrics ldst for AMDtools/perf/pmu-events/amd_metrics.py+75 -00 replies
11/35perf jevents: Add context switch metrics for AMDtools/perf/pmu-events/amd_metrics.py+33 -00 replies
12/35perf jevents: Add RAPL metrics for all Intel modelstools/perf/pmu-events/intel_metrics.py+41 -30 replies
13/35perf jevents: Add idle metric for Intel modelstools/perf/pmu-events/intel_metrics.py+14 -20 replies
14/35perf jevents: Add CheckPmu to see if a PMU is in loaded json eventstools/perf/pmu-events/metric.py+8 -00 replies
15/35perf jevents: Add smi metric group for Intel modelstools/perf/pmu-events/intel_metrics.py+22 -20 replies
16/35perf jevents: Mark metrics with experimental events as experimentaltools/perf/pmu-events/metric.py+44 -00 replies
17/35perf jevents: Add tsx metric group for Intel modelstools/perf/pmu-events/intel_metrics.py+50 -00 replies
18/35perf jevents: Add br metric group for branch statistics on Inteltools/perf/pmu-events/intel_metrics.py+138 -00 replies
19/35perf jevents: Add software prefetch (swpf) metric group for Inteltools/perf/pmu-events/intel_metrics.py+66 -00 replies
20/35perf jevents: Add ports metric group giving utilization on Inteltools/perf/pmu-events/intel_metrics.py+33 -20 replies
21/35perf jevents: Add L2 metrics for Inteltools/perf/pmu-events/intel_metrics.py+170 -00 replies
22/35perf jevents: Add load store breakdown metrics ldst for Inteltools/perf/pmu-events/intel_metrics.py+86 -10 replies
23/35perf jevents: Add ILP metrics for Inteltools/perf/pmu-events/intel_metrics.py+40 -00 replies
24/35perf jevents: Add context switch metrics for Inteltools/perf/pmu-events/intel_metrics.py+58 -00 replies
25/35perf jevents: Add FPU metrics for Inteltools/perf/pmu-events/intel_metrics.py+97 -00 replies
26/35perf jevents: Add Miss Level Parallelism (MLP) metric for Inteltools/perf/pmu-events/intel_metrics.py+15 -00 replies
27/35perf jevents: Add mem_bw metric for Inteltools/perf/pmu-events/intel_metrics.py+62 -00 replies
28/35perf jevents: Add local/remote "mem" breakdown metrics for Inteltools/perf/pmu-events/intel_metrics.py+31 -00 replies
29/35perf jevents: Add dir breakdown metrics for Inteltools/perf/pmu-events/intel_metrics.py+36 -00 replies
30/35perf jevents: Add C-State metrics from the PCU PMU for Inteltools/perf/pmu-events/intel_metrics.py+30 -00 replies
31/35perf jevents: Add local/remote miss latency metrics for Inteltools/perf/pmu-events/intel_metrics.py+67 -30 replies
32/35perf jevents: Add upi_bw metric for Inteltools/perf/pmu-events/intel_metrics.py+22 -00 replies
33/35perf jevents: Add mesh bandwidth saturation metric for Inteltools/perf/pmu-events/intel_metrics.py+17 -00 replies
34/35perf jevents: Add cycles breakdown metric for arm64/AMD/Inteltools/perf/pmu-events/Build tools/perf/pmu-events/amd_metrics.py tools/perf/pmu-events/arm64_metrics.py +2 more+28 -20 replies
35/35perf jevents: Validate that all names given an Eventtools/perf/pmu-events/metric.py+36 -00 replies

Reviews trailers per patch

9 of 35 patches have a Reviewed-by or Acked-by.

PatchThomas FalconSandipan Das
1/35T-b
2/35T-b
3/35R-b
4/35R-b
5/35R-b
6/35R-b
7/35R-b
8/35R-b
9/35R-b
10/35R-b
11/35R-b
12/35T-b
13/35T-b
14/35T-b
15/35T-b
16/35T-b
17/35T-b
18/35T-b
19/35T-b
20/35T-b
21/35T-b
22/35T-b
23/35T-b
24/35T-b
25/35T-b
26/35T-b
27/35T-b
28/35T-b
29/35T-b
30/35T-b
31/35T-b
32/35T-b
33/35T-b
34/35T-b
35/35T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v11 thread

Cover letter [PATCH v11 00/35] AMD and Intel metric generation with Python

Metrics in the perf tool come in via json. Json doesn't allow comments, line breaks, etc. making it an inconvenient way to write metrics. Further, it is useful to detect when writing a metric that the event specified is supported within the event json for a model. From the metric python code Event(s) are used, with fallback events provided, if no event is found then an exception is thrown and that can either indicate a failure or an unsupported model. To avoid confusion all the metrics and their metricgroups are prefixed with 'lpm_', where LPM is an abbreviation of Linux Perf Metric. While extra characters aren't ideal, this separates the metrics from other vendor provided metrics.

* The first 2 patches introduce infrastructure for the addition of metrics written in python for Arm64, AMD Zen and Intel CPUs.

* The next 9 patches generate additional metrics for AMD zen. Rapl and Idle metrics aren't specific to AMD but are placed here for ease and convenience. Uncore L3 metrics are added along with the majority of core metrics.

* The next 22 patches add additional metrics for Intel. Rapl and Idle metrics aren't specific to Intel but are placed here for ease and convenience. Smi and tsx metrics are added so they can be dropped from the per model json files. There are four uncore sets of metrics and eleven core metrics. Add a CheckPmu function to metric to simplify detecting the presence of hybrid PMUs in events. Metrics with experimental events are flagged as experimental in their description.

* The next patch adds a cycles metrics based on perf event modifiers for AMD, Intel and Arm64.

* The final patch validates that all events provided to an Event object exist in a json file somewhere. This is to avoid mistakes like unfortunate typos.

This series has benefitted from the input of Leo Yan
<[email protected]>, Sandipan Das <[email protected]>, Thomas Falcon <[email protected]> and Perry Taylor <[email protected]>.

v11. Rebase and add fix so that cmn isn't considered an ARM model.

v10. Drop already merged non-vendor patches (Namhyung). Drop "Add
collection of topdown like metrics for arm64" as requested by James Clark. Update AMD metrics for changes to AMD Zen6 event names from the series:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

v9. Drop (for now) 4 AMD sets of metrics for additional follow up. Add
reviewed-by tags from Sandipan Das (AMD) and tested-by tags from Thomas Falcon (Intel).
https://lore.kernel.org/lkml/[email protected]/

v8. Combine the previous 4 series for clarity. Rebase on top of the
more recent legacy metric and event changes. Make the python more pep8 and pylint compliant.
https://lore.kernel.org/lkml/[email protected]/

Foundations:
v6. Fix issue with '\-' escape not being '\\-' (reported-by Sandipan
Das <[email protected]>) which didn't alter the generated json.
https://lore.kernel.org/lkml/[email protected]/

v5. Rebase on top of legacy hardware/cache changes that now generate
events using python:
https://lore.kernel.org/lkml/[email protected]/
the v5 series is:
https://lore.kernel.org/lkml/[email protected]/

v4. Rebase and small Build/Makefile tweak

    https://lore.kernel.org/lkml/[email protected]/

v3. Some code tidying, make the input directory a command line
argument, but no other functional or output changes.
https://lore.kernel.org/lkml/[email protected]/

v2. Fixes two type issues in the python code but no functional or

    output changes.
    https://lore.kernel.org/lkml/[email protected]/

v1. https://lore.kernel.org/lkml/[email protected]/

AMD:
v6. Fix issue with '\-' escape not being '\\-' (reported-by Sandipan
Das <[email protected]>) which didn't alter the generated json.
https://lore.kernel.org/lkml/[email protected]/

v5. Rebase. Add uop cache hit/miss rates patch. Prefix all metric
names with lpm_ (short for Linux Perf Metric) so that python generated metrics are clearly namespaced.
https://lore.kernel.org/lkml/[email protected]/

v4. Rebase.

    https://lore.kernel.org/lkml/[email protected]/

v3. Some minor code cleanup changes.

    https://lore.kernel.org/lkml/[email protected]/

v2. Drop the cycles breakdown in favor of having it as a common
metric, suggested by Kan Liang <[email protected]>.
https://lore.kernel.org/lkml/[email protected]/

v1. https://lore.kernel.org/lkml/[email protected]/

Intel:
v6. Fix issue with '\-' escape not being '\\-' (reported-by Sandipan
Das <[email protected]>) which didn't alter the generated json.
https://lore.kernel.org/lkml/[email protected]/

v5. Rebase. Fix description for smi metric (Kan). Prefix all metric
names with lpm_ (short for Linux Perf Metric) so that python generated metrics are clearly namespaced. Kan requested a namespace in his review:
https://lore.kernel.org/lkml/[email protected]/
The v5 series is:
https://lore.kernel.org/lkml/[email protected]/

v4. Experimental metric descriptions. Add mesh bandwidth metric. Rebase.

    https://lore.kernel.org/lkml/[email protected]/

v3. Swap tsx and CheckPMU patches that were in the wrong order. Some
minor code cleanup changes. Drop reference to merged fix for umasks/occ_sel in PCU events and for cstate metrics.
https://lore.kernel.org/lkml/[email protected]/

v2. Drop the cycles breakdown in favor of having it as a common
metric, spelling and other improvements suggested by Kan Liang <[email protected]>.
https://lore.kernel.org/lkml/[email protected]/

v1. https://lore.kernel.org/lkml/[email protected]/

ARM:

v7. Switch a use of cycles to cpu-cycles due to ARM having too many

    cycles events.
    https://lore.kernel.org/lkml/[email protected]/

v6. Fix issue with '\-' escape not being '\\-' (reported-by Sandipan
Das <[email protected]>) which didn't alter the generated json.
https://lore.kernel.org/lkml/[email protected]/

v5. Rebase. Address review comments from Leo Yan
<[email protected]>. Prefix all metric names with lpm_ (short for Linux Perf Metric) so that python generated metrics are clearly namespaced. Use cpu-cycles rather than cycles legacy event for cycles metrics to avoid confusion with ARM PMUs. Add patch that checks events to ensure all possible event names are present in at least one json file.
https://lore.kernel.org/lkml/[email protected]/

v4. Tweak to build dependencies and rebase.

    https://lore.kernel.org/lkml/[email protected]/

v3. Some minor code cleanup changes.

    https://lore.kernel.org/lkml/[email protected]/

v2. The cycles metrics are now made common and shared with AMD and
Intel, suggested by Kan Liang <[email protected]>. This assumes these patches come after the AMD and Intel sets.
https://lore.kernel.org/lkml/[email protected]/

v1. https://lore.kernel.org/lkml/[email protected]/

Ian Rogers (35):
perf jevents: Build support for generating metrics from python perf jevents: Add load event json to verify and allow fallbacks perf jevents: Add RAPL event metric for AMD zen models perf jevents: Add idle metric for AMD zen models
perf jevents: Add upc metric for uops per cycle for AMD perf jevents: Add br metric group for branch statistics on AMD perf jevents: Add itlb metric group for AMD
perf jevents: Add dtlb metric group for AMD
perf jevents: Add uncore l3 metric group for AMD
perf jevents: Add load store breakdown metrics ldst for AMD perf jevents: Add context switch metrics for AMD
perf jevents: Add RAPL metrics for all Intel models
perf jevents: Add idle metric for Intel models
perf jevents: Add CheckPmu to see if a PMU is in loaded json events perf jevents: Add smi metric group for Intel models
perf jevents: Mark metrics with experimental events as experimental perf jevents: Add tsx metric group for Intel models
perf jevents: Add br metric group for branch statistics on Intel perf jevents: Add software prefetch (swpf) metric group for Intel perf jevents: Add ports metric group giving utilization on Intel perf jevents: Add L2 metrics for Intel
perf jevents: Add load store breakdown metrics ldst for Intel perf jevents: Add ILP metrics for Intel
perf jevents: Add context switch metrics for Intel
perf jevents: Add FPU metrics for Intel
perf jevents: Add Miss Level Parallelism (MLP) metric for Intel perf jevents: Add mem_bw metric for Intel
perf jevents: Add local/remote "mem" breakdown metrics for Intel perf jevents: Add dir breakdown metrics for Intel
perf jevents: Add C-State metrics from the PCU PMU for Intel perf jevents: Add local/remote miss latency metrics for Intel perf jevents: Add upi_bw metric for Intel
perf jevents: Add mesh bandwidth saturation metric for Intel perf jevents: Add cycles breakdown metric for arm64/AMD/Intel perf jevents: Validate that all names given an Event

 tools/perf/.gitignore                   |    5 +
 tools/perf/Makefile.perf                |    2 +
 tools/perf/pmu-events/Build             |   51 +-
 tools/perf/pmu-events/amd_metrics.py    |  492 ++++++++++
 tools/perf/pmu-events/arm64_metrics.py  |   49 +
 tools/perf/pmu-events/common_metrics.py |   19 +
 tools/perf/pmu-events/intel_metrics.py  | 1129 +++++++++++++++++++++++
 tools/perf/pmu-events/metric.py         |  171 +++-
 8 files changed, 1914 insertions(+), 4 deletions(-)
 create mode 100755 tools/perf/pmu-events/amd_metrics.py
 create mode 100755 tools/perf/pmu-events/arm64_metrics.py
 create mode 100644 tools/perf/pmu-events/common_metrics.py
 create mode 100755 tools/perf/pmu-events/intel_metrics.py
Signature · 1 line
2.52.0.457.g6b5491de43-goog