Skip to contentSkip to conversation

perf stat: Fix bperf cgroup counters (v2)

PATCH Applied Namhyung Kim 2 participants 6 messages Sep 16, 2022 – Sep 20, 2022

Subsystems (2)
lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

cover perf stat: Fix bperf cgroup counters (v2)

Hello,

This fixes random failures in the perf stat cgroup BPF counters (bperf). I've also added a new test to ensure it working properly.

v2 changes)
* fix a segfault with uncore event

You can get it from 'perf/bperf-cgrp-fix-v2' branch in

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (4):
perf stat: Fix BPF program section name
perf stat: Fix cpu map index in bperf cgroup code
perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters perf test: Add a new test for perf stat cgroup BPF counter

 .../tests/shell/stat_bpf_counters_cgrp.sh     | 83 +++++++++++++++++++
 tools/perf/util/bpf_counter_cgroup.c          | 10 +--
 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c   |  2 +-
 3 files changed, 89 insertions(+), 6 deletions(-)
 create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh

base-commit: 62e64c9d2fd12839c02f1b3e8b873e7cb34e8720

Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
author maintainer
#1 lore raw

PATCH 1/4 perf stat: Fix BPF program section name

It seems the recent libbpf got more strict about the section name. I'm seeing a failure like this:

  $ sudo ./perf stat -a --bpf-counters --for-each-cgroup ^. sleep 1
  libbpf: prog 'on_cgrp_switch': missing BPF prog type, check ELF section name 'perf_events'
  libbpf: prog 'on_cgrp_switch': failed to load: -22
  libbpf: failed to load object 'bperf_cgroup_bpf'
  libbpf: failed to load BPF skeleton 'bperf_cgroup_bpf': -22
  Failed to load cgroup skeleton

The section name should be 'perf_event' (without the trailing 's'). Although it's related to the libbpf change, it'd be better fix the section name in the first place.

Fixes 944138f048f7 ("perf stat: Enable BPF counter with --for-each-cgroup")
---
 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c +1 -1
index 292c430768b5..c72f8ad96f75 100644
@@ -176,7 +176,7 @@ static int bperf_cgroup_count(void)
 }
 
 // This will be attached to cgroup-switches event for each cpu
-SEC("perf_events")
+SEC("perf_event")
 int BPF_PROG(on_cgrp_switch)
 {
 	return bperf_cgroup_count();
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
author maintainer
#2 lore raw

PATCH 2/4 perf stat: Fix cpu map index in bperf cgroup code

In reply to Namhyung Kim

The previous cpu map introduced a bug in the bperf cgroup counter. This results in a failure when user gives a partial cpu map starting from non-zero.

  $ sudo ./perf stat -C 1-2 --bpf-counters --for-each-cgroup ^. sleep 1
  libbpf: prog 'on_cgrp_switch': failed to create BPF link for perf_event FD 0:
                                 -9 (Bad file descriptor)
  Failed to attach cgroup program

To get the FD of an evsel, it should use a map index not the CPU number.

Fixes 0255571a1605 ("perf cpumap: Switch to using perf_cpu_map API")
---
 tools/perf/util/bpf_counter_cgroup.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/bpf_counter_cgroup.c +2 -2
index 63b9db657442..97c69a249c6e 100644
@@ -95,7 +95,7 @@ static int bperf_load_program(struct evlist *evlist)
 
 	perf_cpu_map__for_each_cpu(cpu, i, evlist->core.all_cpus) {
 		link = bpf_program__attach_perf_event(skel->progs.on_cgrp_switch,
-						      FD(cgrp_switch, cpu.cpu));
+						      FD(cgrp_switch, i));
 		if (IS_ERR(link)) {
 			pr_err("Failed to attach cgroup program\n");
 			err = PTR_ERR(link);
@@ -123,7 +123,7 @@ static int bperf_load_program(struct evlist *evlist)
 
 			map_fd = bpf_map__fd(skel->maps.events);
 			perf_cpu_map__for_each_cpu(cpu, j, evlist->core.all_cpus) {
-				int fd = FD(evsel, cpu.cpu);
+				int fd = FD(evsel, j);
 				__u32 idx = evsel->core.idx * total_cpus + cpu.cpu;
 
 				err = bpf_map_update_elem(map_fd, &idx, &fd,
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
author maintainer
#3 lore raw

PATCH 3/4 perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters

In reply to Namhyung Kim

If it mixes core and uncore events, each evsel would have different cpu map. But it assumed they are same with evlist's all_cpus and accessed by the same index. This resulted in a crash like below.

  $ perf stat -a --bpf-counters --for-each_cgroup ^. -e cycles,imc/cas_count_read/ sleep 1
  Segmentation fault

While it's not recommended to use uncore events for cgroup aggregation, it should not crash.

---
 tools/perf/util/bpf_counter_cgroup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/bpf_counter_cgroup.c +3 -3
index 97c69a249c6e..3c2df7522f6f 100644
@@ -115,14 +115,14 @@ static int bperf_load_program(struct evlist *evlist)
 			evsel->cgrp = NULL;
 
 			/* open single copy of the events w/o cgroup */
-			err = evsel__open_per_cpu(evsel, evlist->core.all_cpus, -1);
+			err = evsel__open_per_cpu(evsel, evsel->core.cpus, -1);
 			if (err) {
 				pr_err("Failed to open first cgroup events\n");
 				goto out;
 			}
 
 			map_fd = bpf_map__fd(skel->maps.events);
-			perf_cpu_map__for_each_cpu(cpu, j, evlist->core.all_cpus) {
+			perf_cpu_map__for_each_cpu(cpu, j, evsel->core.cpus) {
 				int fd = FD(evsel, j);
 				__u32 idx = evsel->core.idx * total_cpus + cpu.cpu;
 
@@ -269,7 +269,7 @@ static int bperf_cgrp__read(struct evsel *evsel)
 			goto out;
 		}
 
-		perf_cpu_map__for_each_cpu(cpu, i, evlist->core.all_cpus) {
+		perf_cpu_map__for_each_cpu(cpu, i, evsel->core.cpus) {
 			counts = perf_counts(evsel->counts, i, 0);
 			counts->val = values[cpu.cpu].counter;
 			counts->ena = values[cpu.cpu].enabled;
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
author maintainer
#4 lore raw

PATCH 4/4 perf test: Add a new test for perf stat cgroup BPF counter

In reply to Namhyung Kim
  $ sudo ./perf test -v each-cgroup
   96: perf stat --bpf-counters --for-each-cgroup test                 :
  --- start ---
  test child forked, pid 79600
  test child finished with 0
  ---- end ----
  perf stat --bpf-counters --for-each-cgroup test: Ok
---
 .../tests/shell/stat_bpf_counters_cgrp.sh     | 83 +++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
tools/perf/tests/shell/stat_bpf_counters_cgrp.sh new file +83 -0
new file mode 100755
index 000000000000..d724855d097c
@@ -0,0 +1,83 @@
+#!/bin/sh
+# perf stat --bpf-counters --for-each-cgroup test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+test_cgroups=
+if [ "$1" = "-v" ]; then
+	verbose="1"
+fi
+
+# skip if --bpf-counters --for-each-cgroup is not supported
+check_bpf_counter()
+{
+	if ! perf stat -a --bpf-counters --for-each-cgroup / true > /dev/null 2>&1; then
+		if [ "${verbose}" = "1" ]; then
+			echo "Skipping: --bpf-counters --for-each-cgroup not supported"
+			perf --no-pager stat -a --bpf-counters --for-each-cgroup / true || true
+		fi
+		exit 2
+	fi
+}
+
+# find two cgroups to measure
+find_cgroups()
+{
+	# try usual systemd slices first
+	if [ -d /sys/fs/cgroup/system.slice -a -d /sys/fs/cgroup/user.slice ]; then
+		test_cgroups="system.slice,user.slice"
+		return
+	fi
+
+	# try root and self cgroups
+	local self_cgrp=$(grep perf_event /proc/self/cgroup | cut -d: -f3)
+	if [ -z ${self_cgrp} ]; then
+		# cgroup v2 doesn't specify perf_event
+		self_cgrp=$(grep ^0: /proc/self/cgroup | cut -d: -f3)
+	fi
+
+	if [ -z ${self_cgrp} ]; then
+		test_cgroups="/"
+	else
+		test_cgroups="/,${self_cgrp}"
+	fi
+}
+
+# As cgroup events are cpu-wide, we cannot simply compare the result.
+# Just check if it runs without failure and has non-zero results.
+check_system_wide_counted()
+{
+	local output
+
+	output=$(perf stat -a --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, sleep 1  2>&1)
+	if echo ${output} | grep -q -F "<not "; then
+		echo "Some system-wide events are not counted"
+		if [ "${verbose}" = "1" ]; then
+			echo ${output}
+		fi
+		exit 1
+	fi
+}
+
+check_cpu_list_counted()
+{
+	local output
+
+	output=$(perf stat -C 1 --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, taskset -c 1 sleep 1  2>&1)
+	if echo ${output} | grep -q -F "<not "; then
+		echo "Some CPU events are not counted"
+		if [ "${verbose}" = "1" ]; then
+			echo ${output}
+		fi
+		exit 1
+	fi
+}
+
+check_bpf_counter
+find_cgroups
+
+check_system_wide_counted
+check_cpu_list_counted
+
+exit 0
Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
maintainer Applied
#5 lore raw
In reply to Namhyung Kim

Em Fri, Sep 16, 2022 at 11:41:28AM -0700, Namhyung Kim escreveu:

Quoted 7 lines

Hello,

This fixes random failures in the perf stat cgroup BPF counters (bperf). I've also added a new test to ensure it working properly.

v2 changes)
* fix a segfault with uncore event

Thanks, applied.

- Arnaldo

Quoted 24 lines

You can get it from 'perf/bperf-cgrp-fix-v2' branch in

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (4):
perf stat: Fix BPF program section name
perf stat: Fix cpu map index in bperf cgroup code
perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters perf test: Add a new test for perf stat cgroup BPF counter

 .../tests/shell/stat_bpf_counters_cgrp.sh     | 83 +++++++++++++++++++
 tools/perf/util/bpf_counter_cgroup.c          | 10 +--
 tools/perf/util/bpf_skel/bperf_cgroup.bpf.c   |  2 +-
 3 files changed, 89 insertions(+), 6 deletions(-)
 create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh

base-commit: 62e64c9d2fd12839c02f1b3e8b873e7cb34e8720

Signature · 1 line
2.37.3.968.ga6b4b080e4-goog
Signature · 1 line
- Arnaldo