perf stat: Fix aggregation of cgroup events
Namhyung Kim
maintainerPATCH 1/1 perf stat: Fix aggregation of cgroup events
I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.
$ sudo perf stat -a --bpf-counters --per-socket -e cycles \
--for-each-cgroup /user.slice,/system.slice sleep 1Performance counter stats for 'system wide':
S0 12 <not counted> cpu_atom/cycles/ user.slice S0 16 <not counted> cpu_core/cycles/ user.slice S0 12 <not counted> cpu_atom/cycles/ system.slice S0 16 <not counted> cpu_core/cycles/ system.slice
1.002798147 seconds time elapsed
That's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.
After the fix, the same command produces:
Performance counter stats for 'system wide':
S0 12 5,094,112 cpu_atom/cycles/ user.slice S0 16 28,075,944 cpu_core/cycles/ user.slice S0 12 1,516,568 cpu_atom/cycles/ system.slice S0 16 5,569,575 cpu_core/cycles/ system.slice
1.003231856 seconds time elapsed
--- tools/perf/util/stat.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/perf/util/stat.c +4 -0
@@ -381,6 +381,10 @@ static bool evsel__count_has_error(struct evsel *evsel,if (config->aggr_mode ==AGGR_GLOBAL )return false; +/* cgroup events may not be scheduled on some CPUs */ +if (evsel->cgrp) +return false; +/* it's considered ok when it actually ran */ if (count->ena !=0 && count->run !=0 )return false;
Signature · 1 line
2.55.0
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 14 lines
I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.
$ sudo perf stat -a --bpf-counters --per-socket -e cycles \ --for-each-cgroup /user.slice,/system.slice sleep 1Performance counter stats for 'system wide':
S0 12 <not counted> cpu_atom/cycles/ user.slice S0 16 <not counted> cpu_core/cycles/ user.slice S0 12 <not counted> cpu_atom/cycles/ system.slice S0 16 <not counted> cpu_core/cycles/ system.slice
Quoted 7 lines
1.002798147 seconds time elapsed[ ... ]
1.003231856 seconds time elapsedReported-by Chun-Tse Shao <[email protected]>Signed-off-by Namhyung Kim <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 33 lines · Namhyung Kim
I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.
$ sudo perf stat -a --bpf-counters --per-socket -e cycles \ --for-each-cgroup /user.slice,/system.slice sleep 1Performance counter stats for 'system wide':
S0 12 <not counted> cpu_atom/cycles/ user.slice S0 16 <not counted> cpu_core/cycles/ user.slice S0 12 <not counted> cpu_atom/cycles/ system.slice S0 16 <not counted> cpu_core/cycles/ system.slice1.002798147 seconds time elapsedThat's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.
After the fix, the same command produces:
Performance counter stats for 'system wide':
S0 12 5,094,112 cpu_atom/cycles/ user.slice S0 16 28,075,944 cpu_core/cycles/ user.slice S0 12 1,516,568 cpu_atom/cycles/ system.slice S0 16 5,569,575 cpu_core/cycles/ system.slice
Quoted 4 lines
1.003231856 seconds time elapsedReported-by Chun-Tse Shao <[email protected]>Signed-off-by Namhyung Kim <[email protected]>
Thanks for the fix!
-CT
Quoted 22 lines
--- tools/perf/util/stat.c | 4 ++++ 1 file changed, 4 insertions(+)tools/perf/util/stat.c +4
-1@@ -381,6 +381,10 @@ static bool evsel__count_has_error(struct evsel *evsel,if (config->aggr_mode ==AGGR_GLOBAL )return false; +/* cgroup events may not be scheduled on some CPUs */ +if (evsel->cgrp) +return false; +/* it's considered ok when it actually ran */ if (count->ena !=0 && count->run !=0 )return false; --2.55.0
Namhyung Kim
maintainerQuoted 34 lines · Chun-Tse Shao
Quoted 33 lines · Namhyung Kim
I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.
$ sudo perf stat -a --bpf-counters --per-socket -e cycles \ --for-each-cgroup /user.slice,/system.slice sleep 1Performance counter stats for 'system wide':
S0 12 <not counted> cpu_atom/cycles/ user.slice S0 16 <not counted> cpu_core/cycles/ user.slice S0 12 <not counted> cpu_atom/cycles/ system.slice S0 16 <not counted> cpu_core/cycles/ system.slice1.002798147 seconds time elapsedThat's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.
After the fix, the same command produces:
Performance counter stats for 'system wide':
S0 12 5,094,112 cpu_atom/cycles/ user.slice S0 16 28,075,944 cpu_core/cycles/ user.slice S0 12 1,516,568 cpu_atom/cycles/ system.slice S0 16 5,569,575 cpu_core/cycles/ system.slice
Quoted 6 lines
Quoted 4 lines
1.003231856 seconds time elapsedReported-by Chun-Tse Shao <[email protected]>Signed-off-by Namhyung Kim <[email protected]>Tested-by Chun-Tse Shao <[email protected]>
Thanks for the test!
Arnaldo, can you please take this?
Thanks,
Namhyung
Quoted 37 lines · Namhyung Kim
Quoted 36 lines · Chun-Tse Shao
Quoted 35 lines · Namhyung Kim
I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.
$ sudo perf stat -a --bpf-counters --per-socket -e cycles \ --for-each-cgroup /user.slice,/system.slice sleep 1Performance counter stats for 'system wide':
S0 12 <not counted> cpu_atom/cycles/ user.slice S0 16 <not counted> cpu_core/cycles/ user.slice S0 12 <not counted> cpu_atom/cycles/ system.slice S0 16 <not counted> cpu_core/cycles/ system.slice1.002798147 seconds time elapsedThat's because there's a logic to make the whole event failed if result from any CPU looks bad when aggregation is enabled. Normally it considers bad when an event has no enabled and running time. But it's possible for a cgroup event to have no chance to run on some CPU during the window and then it will have 0 enabled and running time. Let's not treat them as errors.
After the fix, the same command produces:
Performance counter stats for 'system wide':
S0 12 5,094,112 cpu_atom/cycles/ user.slice S0 16 28,075,944 cpu_core/cycles/ user.slice S0 12 1,516,568 cpu_atom/cycles/ system.slice S0 16 5,569,575 cpu_core/cycles/ system.slice1.003231856 seconds time elapsed
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Reported-by Chun-Tse Shao <[email protected]>Signed-off-by Namhyung Kim <[email protected]>Tested-by Chun-Tse Shao <[email protected]>Thanks for the test!
Arnaldo, can you please take this?
applied,
- Arnaldo
Quoted 2 lines
Thanks,
Namhyung
Quoted 8 lines · Namhyung Kim
I got a report that perf stat with BPF and cgroup is broken with aggregation like per-socket or node. On my machine, running the following command shows the problem.
$ sudo perf stat -a --bpf-counters --per-socket -e cycles \ --for-each-cgroup /user.slice,/system.slice sleep 1[...]
Applied to perf-tools-next, thanks!
Best regards
Namhyung