Skip to contentSkip to conversation

perf stat: Fix aggregation of cgroup events

PATCH Applied Namhyung Kim 4 participants 6 messages Oct 2 – Nov 6 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

PATCH 1/1 perf stat: Fix aggregation of cgroup events

I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.

  $ sudo perf stat -a --bpf-counters --per-socket -e cycles \
    --for-each-cgroup /user.slice,/system.slice  sleep 1

Performance counter stats for 'system wide':

  S0       12      <not counted>      cpu_atom/cycles/                 user.slice
  S0       16      <not counted>      cpu_core/cycles/                 user.slice
  S0       12      <not counted>      cpu_atom/cycles/                 system.slice
  S0       16      <not counted>      cpu_core/cycles/                 system.slice
         1.002798147 seconds time elapsed

That's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.

After the fix, the same command produces:

Performance counter stats for 'system wide':

  S0       12          5,094,112      cpu_atom/cycles/                 user.slice
  S0       16         28,075,944      cpu_core/cycles/                 user.slice
  S0       12          1,516,568      cpu_atom/cycles/                 system.slice
  S0       16          5,569,575      cpu_core/cycles/                 system.slice
         1.003231856 seconds time elapsed
---
 tools/perf/util/stat.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/perf/util/stat.c +4 -0
index 25f31a17436828aa..6da3dbbde0e2ad8d 100644
@@ -381,6 +381,10 @@ static bool evsel__count_has_error(struct evsel *evsel,
 	if (config->aggr_mode == AGGR_GLOBAL)
 		return false;
 
+	/* cgroup events may not be scheduled on some CPUs */
+	if (evsel->cgrp)
+		return false;
+
 	/* it's considered ok when it actually ran */
 	if (count->ena != 0 && count->run != 0)
 		return false;
Signature · 1 line
2.55.0
#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 14 lines

I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.

  $ sudo perf stat -a --bpf-counters --per-socket -e cycles \
    --for-each-cgroup /user.slice,/system.slice  sleep 1

Performance counter stats for 'system wide':

  S0       12      <not counted>      cpu_atom/cycles/                 user.slice
  S0       16      <not counted>      cpu_core/cycles/                 user.slice
  S0       12      <not counted>      cpu_atom/cycles/                 system.slice
  S0       16      <not counted>      cpu_core/cycles/                 system.slice
Quoted 7 lines
         1.002798147 seconds time elapsed

[ ... ]

         1.003231856 seconds time elapsed

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Tested-by on v1 1/1
#2 lore raw
In reply to Namhyung Kim
Quoted 33 lines · Namhyung Kim

I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.

  $ sudo perf stat -a --bpf-counters --per-socket -e cycles \
    --for-each-cgroup /user.slice,/system.slice  sleep 1

Performance counter stats for 'system wide':

  S0       12      <not counted>      cpu_atom/cycles/                 user.slice
  S0       16      <not counted>      cpu_core/cycles/                 user.slice
  S0       12      <not counted>      cpu_atom/cycles/                 system.slice
  S0       16      <not counted>      cpu_core/cycles/                 system.slice
         1.002798147 seconds time elapsed

That's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.

After the fix, the same command produces:

Performance counter stats for 'system wide':

  S0       12          5,094,112      cpu_atom/cycles/                 user.slice
  S0       16         28,075,944      cpu_core/cycles/                 user.slice
  S0       12          1,516,568      cpu_atom/cycles/                 system.slice
  S0       16          5,569,575      cpu_core/cycles/                 system.slice
Quoted 4 lines
         1.003231856 seconds time elapsed

Thanks for the fix!

-CT

Quoted 22 lines
---
 tools/perf/util/stat.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/perf/util/stat.c +4 -1
index 25f31a17436828aa..6da3dbbde0e2ad8d 100644
@@ -381,6 +381,10 @@ static bool evsel__count_has_error(struct evsel *evsel,
        if (config->aggr_mode == AGGR_GLOBAL)
                return false;
 
+       /* cgroup events may not be scheduled on some CPUs */
+       if (evsel->cgrp)
+               return false;
+
        /* it's considered ok when it actually ran */
        if (count->ena != 0 && count->run != 0)
                return false;
--

2.55.0

author maintainer
#3 lore raw
Quoted 34 lines · Chun-Tse Shao
Quoted 33 lines · Namhyung Kim

I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.

  $ sudo perf stat -a --bpf-counters --per-socket -e cycles \
    --for-each-cgroup /user.slice,/system.slice  sleep 1

Performance counter stats for 'system wide':

  S0       12      <not counted>      cpu_atom/cycles/                 user.slice
  S0       16      <not counted>      cpu_core/cycles/                 user.slice
  S0       12      <not counted>      cpu_atom/cycles/                 system.slice
  S0       16      <not counted>      cpu_core/cycles/                 system.slice
         1.002798147 seconds time elapsed

That's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.

After the fix, the same command produces:

Performance counter stats for 'system wide':

  S0       12          5,094,112      cpu_atom/cycles/                 user.slice
  S0       16         28,075,944      cpu_core/cycles/                 user.slice
  S0       12          1,516,568      cpu_atom/cycles/                 system.slice
  S0       16          5,569,575      cpu_core/cycles/                 system.slice
Quoted 6 lines
Quoted 4 lines
         1.003231856 seconds time elapsed

Thanks for the test!

Arnaldo, can you please take this?

Thanks,
Namhyung

maintainer Applied
#4 lore raw
Quoted 37 lines · Namhyung Kim
Quoted 36 lines · Chun-Tse Shao
Quoted 35 lines · Namhyung Kim

I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.

  $ sudo perf stat -a --bpf-counters --per-socket -e cycles \
    --for-each-cgroup /user.slice,/system.slice  sleep 1

Performance counter stats for 'system wide':

  S0       12      <not counted>      cpu_atom/cycles/                 user.slice
  S0       16      <not counted>      cpu_core/cycles/                 user.slice
  S0       12      <not counted>      cpu_atom/cycles/                 system.slice
  S0       16      <not counted>      cpu_core/cycles/                 system.slice
         1.002798147 seconds time elapsed

That's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.

After the fix, the same command produces:

Performance counter stats for 'system wide':

  S0       12          5,094,112      cpu_atom/cycles/                 user.slice
  S0       16         28,075,944      cpu_core/cycles/                 user.slice
  S0       12          1,516,568      cpu_atom/cycles/                 system.slice
  S0       16          5,569,575      cpu_core/cycles/                 system.slice
         1.003231856 seconds time elapsed
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Thanks for the test!

Arnaldo, can you please take this?

applied,

- Arnaldo

Quoted 2 lines

Thanks,
Namhyung

author maintainer Applied
#5 lore raw
In reply to Namhyung Kim
Quoted 8 lines · Namhyung Kim

I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.

  $ sudo perf stat -a --bpf-counters --per-socket -e cycles \
    --for-each-cgroup /user.slice,/system.slice  sleep 1

[...]

Applied to perf-tools-next, thanks!

Best regards
Namhyung