perf stat: Fix bperf cgroup counters (v2)
Subsystems (2)
Namhyung Kim
maintainercover perf stat: Fix bperf cgroup counters (v2)
Hello,
This fixes random failures in the perf stat cgroup BPF counters (bperf). I've also added a new test to ensure it working properly.
v2 changes)
* fix a segfault with uncore event
You can get it from 'perf/bperf-cgrp-fix-v2' branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks,
Namhyung
Namhyung Kim (4):
perf stat: Fix BPF program section name
perf stat: Fix cpu map index in bperf cgroup code
perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters perf test: Add a new test for perf stat cgroup BPF counter
.../tests/shell/stat_bpf_counters_cgrp.sh | 83 +++++++++++++++++++ tools/perf/util/bpf_counter_cgroup.c | 10 +-- tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +- 3 files changed, 89 insertions(+), 6 deletions(-) create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
base-commit: 62e64c9d2fd12839c02f1b3e8b873e7cb34e8720
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Namhyung Kim
maintainerPATCH 1/4 perf stat: Fix BPF program section name
It seems the recent libbpf got more strict about the section name. I'm seeing a failure like this:
$ sudo ./perf stat -a --bpf-counters --for-each-cgroup ^. sleep 1 libbpf: prog 'on_cgrp_switch': missing BPF prog type, check ELF section name 'perf_events' libbpf: prog 'on_cgrp_switch': failed to load: -22 libbpf: failed to load object 'bperf_cgroup_bpf' libbpf: failed to load BPF skeleton 'bperf_cgroup_bpf': -22 Failed to load cgroup skeleton
The section name should be 'perf_event' (without the trailing 's'). Although it's related to the libbpf change, it'd be better fix the section name in the first place.
--- tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c +1 -1
@@ -176,7 +176,7 @@ static int bperf_cgroup_count(void) }// This will be attached to cgroup-switches event for each cpu -SEC("perf_events") +SEC("perf_event")int BPF_PROG (on_cgrp_switch ) {return bperf_cgroup_count ();
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Namhyung Kim
maintainerPATCH 2/4 perf stat: Fix cpu map index in bperf cgroup code
In reply to Namhyung KimThe previous cpu map introduced a bug in the bperf cgroup counter. This results in a failure when user gives a partial cpu map starting from non-zero.
$ sudo ./perf stat -C 1-2 --bpf-counters --for-each-cgroup ^. sleep 1
libbpf: prog 'on_cgrp_switch': failed to create BPF link for perf_event FD 0:
-9 (Bad file descriptor)
Failed to attach cgroup programTo get the FD of an evsel, it should use a map index not the CPU number.
--- tools/perf/util/bpf_counter_cgroup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/bpf_counter_cgroup.c +2 -2
@@ -95,7 +95,7 @@ static int bperf_load_program(struct evlist *evlist)perf_cpu_map__for_each_cpu (cpu, i, evlist->core.all_cpus) { link =bpf_program__attach_perf_event (skel->progs.on_cgrp_switch, -FD (cgrp_switch, cpu.cpu)); +FD (cgrp_switch, i));if (IS_ERR (link)) {pr_err ("Failed to attach cgroup program\n" ); err =PTR_ERR (link);
@@ -123,7 +123,7 @@ static int bperf_load_program(struct evlist *evlist) map_fd =bpf_map__fd (skel->maps.events);perf_cpu_map__for_each_cpu (cpu, j, evlist->core.all_cpus) { -int fd =FD (evsel, cpu.cpu); +int fd =FD (evsel, j);__u32 idx = evsel->core.idx * total_cpus + cpu.cpu; err = bpf_map_update_elem(map_fd, &idx, &fd,
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Namhyung Kim
maintainerPATCH 3/4 perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters
In reply to Namhyung KimIf it mixes core and uncore events, each evsel would have different cpu map. But it assumed they are same with evlist's all_cpus and accessed by the same index. This resulted in a crash like below.
$ perf stat -a --bpf-counters --for-each_cgroup ^. -e cycles,imc/cas_count_read/ sleep 1 Segmentation fault
While it's not recommended to use uncore events for cgroup aggregation, it should not crash.
--- tools/perf/util/bpf_counter_cgroup.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/bpf_counter_cgroup.c +3 -3
@@ -115,14 +115,14 @@ static int bperf_load_program(struct evlist *evlist) evsel->cgrp =NULL ;/* open single copy of the events w/o cgroup */ - err =evsel__open_per_cpu (evsel, evlist->core.all_cpus,-1 ); + err =evsel__open_per_cpu (evsel, evsel->core.cpus,-1 );if (err) {pr_err ("Failed to open first cgroup events\n" ); goto out; } map_fd =bpf_map__fd (skel->maps.events); -perf_cpu_map__for_each_cpu (cpu, j, evlist->core.all_cpus) { +perf_cpu_map__for_each_cpu (cpu, j, evsel->core.cpus) {int fd =FD (evsel, j);__u32 idx = evsel->core.idx * total_cpus + cpu.cpu;
@@ -269,7 +269,7 @@ static int bperf_cgrp__read(struct evsel *evsel) goto out; } -perf_cpu_map__for_each_cpu (cpu, i, evlist->core.all_cpus) { +perf_cpu_map__for_each_cpu (cpu, i, evsel->core.cpus) { counts =perf_counts (evsel->counts, i,0 ); counts->val = values[cpu.cpu].counter; counts->ena = values[cpu.cpu].enabled;
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Namhyung Kim
maintainerPATCH 4/4 perf test: Add a new test for perf stat cgroup BPF counter
In reply to Namhyung Kim$ sudo ./perf test -v each-cgroup 96: perf stat --bpf-counters --for-each-cgroup test : --- start --- test child forked, pid 79600 test child finished with 0 ---- end ---- perf stat --bpf-counters --for-each-cgroup test: Ok
--- .../tests/shell/stat_bpf_counters_cgrp.sh | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
tools/perf/tests/shell/stat_bpf_counters_cgrp.sh new file +83 -0
@@ -0,0 +1,83 @@ +#!/bin/sh +# perf stat --bpf-counters --for-each-cgroup test +# SPDX-License-Identifier: GPL-2.0 + +set -e + +test_cgroups= +if [ "$1" = "-v" ]; then + verbose="1" +fi + +# skip if --bpf-counters --for-each-cgroup is not supported +check_bpf_counter() +{ + if ! perf stat -a --bpf-counters --for-each-cgroup / true > /dev/null 2>&1; then + if [ "${verbose}" = "1" ]; then + echo "Skipping: --bpf-counters --for-each-cgroup not supported" + perf --no-pager stat -a --bpf-counters --for-each-cgroup / true || true + fi + exit 2 + fi +} + +# find two cgroups to measure +find_cgroups() +{ + # try usual systemd slices first + if [ -d /sys/fs/cgroup/system.slice -a -d /sys/fs/cgroup/user.slice ]; then + test_cgroups="system.slice,user.slice" + return + fi + + # try root and self cgroups + local self_cgrp=$(grep perf_event /proc/self/cgroup | cut -d: -f3) + if [ -z ${self_cgrp} ]; then + # cgroup v2 doesn't specify perf_event + self_cgrp=$(grep ^0: /proc/self/cgroup | cut -d: -f3) + fi + + if [ -z ${self_cgrp} ]; then + test_cgroups="/" + else + test_cgroups="/,${self_cgrp}" + fi +} + +# As cgroup events are cpu-wide, we cannot simply compare the result. +# Just check if it runs without failure and has non-zero results. +check_system_wide_counted() +{ + local output + + output=$(perf stat -a --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, sleep 1 2>&1) + if echo ${output} | grep -q -F "<not "; then + echo "Some system-wide events are not counted" + if [ "${verbose}" = "1" ]; then + echo ${output} + fi + exit 1 + fi +} + +check_cpu_list_counted() +{ + local output + + output=$(perf stat -C 1 --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, taskset -c 1 sleep 1 2>&1) + if echo ${output} | grep -q -F "<not "; then + echo "Some CPU events are not counted" + if [ "${verbose}" = "1" ]; then + echo ${output} + fi + exit 1 + fi +} + +check_bpf_counter +find_cgroups + +check_system_wide_counted +check_cpu_list_counted + +exit 0
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Em Fri, Sep 16, 2022 at 11:41:28AM -0700, Namhyung Kim escreveu:
Quoted 7 lines
Hello,
This fixes random failures in the perf stat cgroup BPF counters (bperf). I've also added a new test to ensure it working properly.
v2 changes)
* fix a segfault with uncore event
Thanks, applied.
- Arnaldo
Quoted 24 lines
You can get it from 'perf/bperf-cgrp-fix-v2' branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThanks,
NamhyungNamhyung Kim (4):
perf stat: Fix BPF program section name
perf stat: Fix cpu map index in bperf cgroup code
perf stat: Useevsel->core.cpusto iterate cpus in BPF cgroup counters perf test: Add a new test for perf stat cgroup BPF counter.../tests/shell/stat_bpf_counters_cgrp.sh | 83 +++++++++++++++++++ tools/perf/util/bpf_counter_cgroup.c | 10 +-- tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +- 3 files changed, 89 insertions(+), 6 deletions(-) create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.shbase-commit:
62e64c9d2fd12839c02f1b3e8b873e7cb34e8720Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Signature · 1 line
- Arnaldo