perf test: Directory file descriptor leak
Ian Rogers
reviewerPATCH 1/4 perf test: Directory file descriptor leak
Add missed close when iterating over the script directories.
--- tools/perf/tests/tests-scripts.c | 1 + 1 file changed, 1 insertion(+)
tools/perf/tests/tests-scripts.c +1 -0
@@ -260,6 +260,7 @@ static void append_scripts_in_dir(int dir_fd,continue ;/* Skip scripts that have a separate driver. */ fd =openat (dir_fd, ent->d_name,O_PATH );append_scripts_in_dir (fd, result, result_sz); +close (fd); }for (i =0 ; i < n_dirs; i++)/* Clean up */ zfree (&entlist[i]);
Signature · 1 line
2.50.0.rc1.591.g9c95f17f64-goog
Ian Rogers
reviewerPATCH 2/4 perf evsel: Missed close when probing hybrid core PMUs
Add missing close to avoid leaking perf events. In past perfs this mattered little as the function was just used by perf list. As the function is now used to detect hybrid PMUs leaking the perf event is somewhat more painful.
--- tools/perf/util/print-events.c | 1 + 1 file changed, 1 insertion(+)
tools/perf/util/print-events.c +1 -0
@@ -268,6 +268,7 @@ bool is_event_supported(u8 type, u64 config) ret =evsel__open (evsel,NULL , tmap) >=0 ; } +evsel__close (evsel);evsel__delete (evsel); }
Signature · 1 line
2.50.0.rc1.591.g9c95f17f64-goog
Ian Rogers
reviewerRe: [PATCH v1 2/4] perf evsel: Missed close when probing hybrid core PMUs
Quoted 5 lines · Ian Rogers
Add missing close to avoid leaking perf events. In past perfs this mattered little as the function was just used by perf list. As the function is now used to detect hybrid PMUs leaking the perf event is somewhat more painful.
Given this leads to leaking perf events on hybrid systems it would be nice to land this 1 liner in the next v6.15 rc pull request fixes if possible.
Thanks,
Ian
Quoted 21 lines
Fixes b41f1cec91c3 ("perf list: Skip unsupported events")Signed-off-by Ian Rogers <[email protected]>--- tools/perf/util/print-events.c | 1 + 1 file changed, 1 insertion(+)tools/perf/util/print-events.c +1
-1@@ -268,6 +268,7 @@ bool is_event_supported(u8 type, u64 config) ret =evsel__open (evsel,NULL , tmap) >=0 ; } +evsel__close (evsel);evsel__delete (evsel); } --2.50.0.rc1.591.g9c95f17f64-goog
Arnaldo Carvalho de Melo
maintainerRe: [PATCH v1 2/4] perf evsel: Missed close when probing hybrid core PMUs
Quoted 9 lines · Ian Rogers
Quoted 4 lines · Ian Rogers
Add missing close to avoid leaking perf events. In past perfs this mattered little as the function was just used by perf list. As the function is now used to detect hybrid PMUs leaking the perf event is somewhat more painful.
Given this leads to leaking perf events on hybrid systems it would be nice to land this 1 liner in the next v6.15 rc pull request fixes if possible.
Sure.
Just one nit: while looking at:
void evsel__close(struct evsel *evsel)
{
if (evsel__is_retire_lat(evsel))
evsel__tpebs_close(evsel);
perf_evsel__close(&evsel->core);
perf_evsel__free_id(&evsel->core);
}And then:
void evsel__delete(struct evsel *evsel)
{
if (!evsel)
return;
evsel__exit(evsel);
free(evsel);
}and:
void evsel__exit(struct evsel *evsel)
{
assert(list_empty(&evsel->core.node));
assert(evsel->evlist == NULL);
if (evsel__is_retire_lat(evsel))
evsel__tpebs_close(evsel);<SNIP>
I think that tpebs_close could be done just at evsel__close(), no?
The way it works I think there is no problem with calling it now twice, but it fits better in evsel__close().
- Arnaldo
Ian Rogers
reviewerRe: [PATCH v1 2/4] perf evsel: Missed close when probing hybrid core PMUs
Quoted 43 lines · Arnaldo Carvalho de Melo
Quoted 9 lines · Ian Rogers
Quoted 4 lines · Ian Rogers
Add missing close to avoid leaking perf events. In past perfs this mattered little as the function was just used by perf list. As the function is now used to detect hybrid PMUs leaking the perf event is somewhat more painful.
Given this leads to leaking perf events on hybrid systems it would be nice to land this 1 liner in the next v6.15 rc pull request fixes if possible.
Sure.
Just one nit: while looking at:
void
evsel__close(struct evsel *evsel){ if (evsel__is_retire_lat(evsel)) evsel__tpebs_close(evsel); perf_evsel__close(&evsel->core); perf_evsel__free_id(&evsel->core); }And then:
void
evsel__delete(struct evsel *evsel){ if (!evsel) return; evsel__exit(evsel); free(evsel); }and:
void
evsel__exit(struct evsel *evsel){ assert(list_empty(&evsel->core.node)); assert(evsel->evlist == NULL); if (evsel__is_retire_lat(evsel)) evsel__tpebs_close(evsel);
Quoted 6 lines
<SNIP>
I think that
tpebs_closecould be done just atevsel__close(), no?The way it works I think there is no problem with calling it now twice, but it fits better in
evsel__close().
Agreed. It'd be nice if we could assert that it was closed given this has been a problem. Retirement latency events are pretty unusual so I'm not overly worried about needing this :-)
Thanks,
Ian
Ian Rogers
reviewerPATCH 3/4 perf trace: Add missed freeing of ordered events and thread
In reply to Ian RogersCaught by leak sanitizer running "perf trace BTF general tests".
--- tools/perf/builtin-trace.c | 2 ++ 1 file changed, 2 insertions(+)
tools/perf/builtin-trace.c +2 -0
@@ -5359,6 +5359,7 @@ static int trace__config(const char *var, const char *value, void *arg)static void trace__exit (struct trace *trace) { +thread__zput (trace->current);strlist__delete (trace->ev_qualifier);zfree (&trace->ev_qualifier_ids.entries);if (trace->syscalls.table) {
@@ -5369,6 +5370,7 @@ static void trace__exit(struct trace *trace)zfree (&trace->perfconfig_events);evlist__delete (trace->evlist); trace->evlist =NULL ; +ordered_events__free (&trace->oe.data);#ifdef HAVE_LIBBPF_SUPPORT btf__free (trace->btf); trace->btf =NULL ;
Signature · 1 line
2.50.0.rc1.591.g9c95f17f64-goog
Re: [PATCH v1 3/4] perf trace: Add missed freeing of ordered events and thread
Hello Ian,
before:
~~~
$ sudo valgrind --leak-check=full /tmp/perf/perf trace --sort-events -e openat* --max-events=1 -- echo 1 ==248676== Memcheck, a memory error detector ==248676== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al. ==248676== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info ==248676== Command: /tmp/perf/perf trace --sort-events -e openat* --max-events=1 -- echo 1 ... ==248676== 3,842 (256 direct, 3,586 indirect) bytes in 1 blocks are definitely lost in loss record 2,773 of 2,808 ==248676== at 0x484FBE3: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==248676== by 0x3634E1: thread__new (in /tmp/perf/perf) ==248676== by 0x363BBE: threads__findnew (in /tmp/perf/perf) ==248676== by 0x34DAB6: machine__findnew_thread (in /tmp/perf/perf) ==248676== by 0x34E655: machine__process_comm_event (in /tmp/perf/perf) ==248676== by 0x2071B1: __trace__deliver_event.isra.0 (in /tmp/perf/perf) ==248676== by 0x2072EA: ordered_events__deliver_event (in /tmp/perf/perf) ==248676== by 0x3608D2: __ordered_events__flush (in /tmp/perf/perf) ==248676== by 0x20CB19: trace__run (in /tmp/perf/perf) ==248676== by 0x21183D: cmd_trace (in /tmp/perf/perf) ==248676== by 0x215E6F: run_builtin (in /tmp/perf/perf) ==248676== by 0x21618A: handle_internal_command (in /tmp/perf/perf) ... ==248676== 4,992 bytes in 16 blocks are definitely lost in loss record 2,776 of 2,808 ==248676== at 0x484FE10: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==248676== by 0x57CCBCD: __vasprintf_internal (vasprintf.c:123) ==248676== by 0x5878152: __asprintf_chk (asprintf_chk.c:34) ==248676== by 0x31ED18: parse_events_formats_error_string (in /tmp/perf/perf) ==248676== by 0x36DCA7: perf_pmu__config_terms (in /tmp/perf/perf) ==248676== by 0x370CD8: perf_pmu__name_from_config (in /tmp/perf/perf) ==248676== by 0x43216A: arch_is_topdown_metrics (in /tmp/perf/perf) ==248676== by 0x432AD7: arch_evsel__must_be_in_group (in /tmp/perf/perf) ==248676== by 0x31C657: __parse_events (in /tmp/perf/perf) ==248676== by 0x31CD1E: parse_event (in /tmp/perf/perf) ==248676== by 0x3A2D76: perf_can_comm_exec (in /tmp/perf/perf) ==248676== by 0x3A374F: evlist__config (in /tmp/perf/perf) ... ==248676== ==248676== LEAK SUMMARY: ==248676== definitely lost: 5,248 bytes in 17 blocks ==248676== indirectly lost: 3,586 bytes in 12 blocks ==248676== possibly lost: 311,943 bytes in 5,292 blocks ==248676== still reachable: 740,220 bytes in 21,186 blocks ==248676== suppressed: 0 bytes in 0 blocks ==248676== Reachable blocks (those to which a pointer was found) are not shown. ==248676== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==248676== ==248676== For lists of detected and suppressed errors, rerun with: -s ==248676== ERROR SUMMARY: 56 errors from 56 contexts (suppressed: 0 from 0) ~~~
after:
~~~
$ sudo valgrind --leak-check=full /tmp/perf/perf trace --sort-events -e openat* --max-events=1 -- echo 1 ==245044== Memcheck, a memory error detector ==245044== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al. ==245044== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info ==245044== Command: /tmp/perf/perf trace --sort-events -e openat* --max-events=1 -- echo 1 ==245044== --245044-- WARNING: unhandled eBPF command 36 --245044-- WARNING: unhandled eBPF command 28 --245044-- WARNING: unhandled eBPF command 28 --245044-- WARNING: unhandled eBPF command 28 --245044-- WARNING: unhandled eBPF command 28 --245044-- WARNING: unhandled eBPF command 28 libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV libbpf: failed to read kernel BTF from '/sys/kernel/btf/vmlinux': -ENODEV ... ==245044== 4,992 bytes in 16 blocks are definitely lost in loss record 2,763 of 2,794 ==245044== at 0x484FE10: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==245044== by 0x57CCBCD: __vasprintf_internal (vasprintf.c:123) ==245044== by 0x5878152: __asprintf_chk (asprintf_chk.c:34) ==245044== by 0x31ED98: parse_events_formats_error_string (in /tmp/perf/perf) ==245044== by 0x36DD27: perf_pmu__config_terms (in /tmp/perf/perf) ==245044== by 0x370D58: perf_pmu__name_from_config (in /tmp/perf/perf) ==245044== by 0x4321EA: arch_is_topdown_metrics (in /tmp/perf/perf) ==245044== by 0x432B57: arch_evsel__must_be_in_group (in /tmp/perf/perf) ==245044== by 0x31C6D7: __parse_events (in /tmp/perf/perf) ==245044== by 0x31CD9E: parse_event (in /tmp/perf/perf) ==245044== by 0x3A2DF6: perf_can_comm_exec (in /tmp/perf/perf) ==245044== by 0x3A37CF: evlist__config (in /tmp/perf/perf) ... ==245044== LEAK SUMMARY: ==245044== definitely lost: 4,992 bytes in 16 blocks ==245044== indirectly lost: 0 bytes in 0 blocks ==245044== possibly lost: 245,988 bytes in 5,288 blocks ==245044== still reachable: 740,211 bytes in 21,185 blocks ==245044== suppressed: 0 bytes in 0 blocks ==245044== Reachable blocks (those to which a pointer was found) are not shown. ==245044== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==245044== ==245044== For lists of detected and suppressed errors, rerun with: -s ==245044== ERROR SUMMARY: 51 errors from 51 contexts (suppressed: 0 from 0) ~~~
TL;DR: (definitely lost: 5,248 bytes in 17 blocks, 3,586 bytes in 12 blocks) -> (definitely lost: 4,992 bytes in 16 blocks, indirectly
lost: 0 bytes in 0 blocks), the leak of thread__new() is fixed.
Thank you so much for fixing this.
Quoted 4 lines · Ian Rogers
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>
Thanks,
Howard
Arnaldo Carvalho de Melo
maintainerRe: [PATCH v1 3/4] perf trace: Add missed freeing of ordered events and thread
Quoted 6 lines · Howard Chu
TL;DR: (definitely lost: 5,248 bytes in 17 blocks, 3,586 bytes in 12 blocks) -> (definitely lost: 4,992 bytes in 16 blocks, indirectly
lost: 0 bytes in 0 blocks), the leak ofthread__new()is fixed.Thank you so much for fixing this.
Quoted 7 lines
Quoted 4 lines · Ian Rogers
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>Acked-by Howard Chu <[email protected]>
Small enough, applied to perf-tools.
- Arnaldo
Re: [PATCH v1 3/4] perf trace: Add missed freeing of ordered events and thread
Quoted 9 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Howard Chu
TL;DR: (definitely lost: 5,248 bytes in 17 blocks, 3,586 bytes in 12 blocks) -> (definitely lost: 4,992 bytes in 16 blocks, indirectly
lost: 0 bytes in 0 blocks), the leak ofthread__new()is fixed.Thank you so much for fixing this.
Quoted 1 line · Ian Rogers
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>Acked-by Howard Chu <[email protected]>Small enough, applied to perf-tools.
root@number:~# perf trace -e *sleep ls anaconda-ks.cfg bin bla commands dtel firefly logind.conf perf-install.txt python perf: Segmentation fault Obtained 11 stack frames. perf() [0x5c595e] perf() [0x5c59f9] /lib64/libc.so.6(+0x19c30) [0x7fd43ce27c30] perf() [0x5dc497] perf() [0x492d54] perf() [0x49860e] perf() [0x49890e] perf() [0x413413] /lib64/libc.so.6(+0x35f5) [0x7fd43ce115f5] /lib64/libc.so.6(__libc_start_main+0x88) [0x7fd43ce116a8] perf() [0x413a45] Segmentation fault (core dumped) root@number:~#
⬢ [acme@toolbx perf-tools]$ git bisect good e340815ea559052d8d590a145bab7d7105608e7d is the first bad commit commit e340815ea559052d8d590a145bab7d7105608e7d
Author: Ian Rogers <[email protected]> Date: Fri Jun 13 17:41:06 2025 -0700
perf trace: Add missed freeing of ordered events and thread
Caught by leak sanitizer running "perf trace BTF general tests".
tools/perf/builtin-trace.c | 2 ++ 1 file changed, 2 insertions(+)
⬢ [acme@toolbx perf-tools]$
Removing it:
root@number:~# perf trace -e *sleep sleep 1
0.000 (1000.057 ms): sleep/2693449 clock_nanosleep(rqtp: { .tv_sec: 1, .tv_nsec: 0 }, rmtp: 0x7fffedc43ab0) = 0
root@number:~#- Arnaldo
Ian Rogers
reviewerRe: [PATCH v1 3/4] perf trace: Add missed freeing of ordered events and thread
Quoted 28 lines · Arnaldo Carvalho de Melo
Quoted 9 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Howard Chu
TL;DR: (definitely lost: 5,248 bytes in 17 blocks, 3,586 bytes in 12 blocks) -> (definitely lost: 4,992 bytes in 16 blocks, indirectly
lost: 0 bytes in 0 blocks), the leak ofthread__new()is fixed.Thank you so much for fixing this.
Quoted 1 line · Ian Rogers
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>Acked-by Howard Chu <[email protected]>Small enough, applied to perf-tools.
root@number:~# perf trace -e *sleep ls anaconda-ks.cfg bin bla commands dtel firefly logind.conf perf-install.txt python perf: Segmentation fault Obtained 11 stack frames. perf() [0x5c595e] perf() [0x5c59f9] /lib64/libc.so.6(+0x19c30) [0x7fd43ce27c30] perf() [0x5dc497] perf() [0x492d54]
Quoted 8 lines
perf() [0x49860e] perf() [0x49890e] perf() [0x413413] /lib64/libc.so.6(+0x35f5) [0x7fd43ce115f5] /lib64/libc.so.6(__libc_start_main+0x88) [0x7fd43ce116a8] perf() [0x413a45] Segmentation fault (core dumped) root@number:~#
Thanks, I'll take a look to see if I can spot what's broken. Seeing this stack trace makes me remember we haven't landed:
https://lore.kernel.org/lkml/[email protected]/
Ian
Quoted 38 lines
⬢ [acme@toolbx perf-tools]$ git bisect good
e340815ea559052d8d590a145bab7d7105608e7dis the first bad commit commite340815ea559052d8d590a145bab7d7105608e7dAuthor: Ian Rogers <[email protected]> Date: Fri Jun 13 17:41:06 2025 -0700perf trace: Add missed freeing of ordered events and thread
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>Acked-by Howard Chu <[email protected]>Cc Adrian Hunter <[email protected]>Cc Alexander Shishkin <[email protected]>Cc Andi Kleen <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jiapeng Chong <[email protected]>Cc Jiri Olsa <[email protected]>Cc Kan Liang <[email protected]>Cc Mark Rutland <[email protected]>Cc Michael Petlan <[email protected]>Cc Namhyung Kim <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Tiezhu Yang <[email protected]>Signed-off-by Arnaldo Carvalho de Melo <[email protected]>tools/perf/builtin-trace.c | 2 ++ 1 file changed, 2 insertions(+)⬢ [acme@toolbx perf-tools]$
Removing it:
root@number:~# perf trace -e *sleep sleep 1 0.000 (1000.057 ms): sleep/2693449 clock_nanosleep(rqtp: { .tv_sec: 1, .tv_nsec: 0 }, rmtp: 0x7fffedc43ab0) = 0 root@number:~#- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [PATCH v1 3/4] perf trace: Add missed freeing of ordered events and thread
Quoted 29 lines · Ian Rogers
Quoted 19 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
Quoted 6 lines · Ian Rogers
Quoted 3 lines
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>Acked-by Howard Chu <[email protected]>Small enough, applied to perf-tools.
root@number:~# perf trace -e *sleep ls anaconda-ks.cfg bin bla commands dtel firefly logind.conf perf-install.txt python perf: Segmentation fault Obtained 11 stack frames. perf() [0x5c595e] perf() [0x5c59f9] /lib64/libc.so.6(+0x19c30) [0x7fd43ce27c30] perf() [0x5dc497] perf() [0x492d54]Quoted 8 lines
perf() [0x49860e] perf() [0x49890e] perf() [0x413413] /lib64/libc.so.6(+0x35f5) [0x7fd43ce115f5] /lib64/libc.so.6(__libc_start_main+0x88) [0x7fd43ce116a8] perf() [0x413a45] Segmentation fault (core dumped) root@number:~#
Quoted 3 lines
Thanks, I'll take a look to see if I can spot what's broken. Seeing this stack trace makes me remember we haven't landed:
https://lore.kernel.org/lkml/[email protected]/
Yeah, I just pushed perf-tools to soak in linux-next/pending-fixes for a few days and will switch to processing patches for perf-tools-next, will try and pick that one, I also noticed that the backtrace wasn't symbolized, thus your patch wasn't there :-\
- Arnaldo
Ian Rogers
reviewerRe: [PATCH v1 3/4] perf trace: Add missed freeing of ordered events and thread
Quoted 31 lines · Arnaldo Carvalho de Melo
Quoted 28 lines · Ian Rogers
Quoted 27 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
Quoted 6 lines · Ian Rogers
Quoted 3 lines
Caught by leak sanitizer running "perf trace BTF general tests".
Signed-off-by Ian Rogers <[email protected]>Acked-by Howard Chu <[email protected]>Small enough, applied to perf-tools.
root@number:~# perf trace -e *sleep ls anaconda-ks.cfg bin bla commands dtel firefly logind.conf perf-install.txt python perf: Segmentation fault Obtained 11 stack frames. perf() [0x5c595e] perf() [0x5c59f9] /lib64/libc.so.6(+0x19c30) [0x7fd43ce27c30] perf() [0x5dc497] perf() [0x492d54] perf() [0x49860e] perf() [0x49890e] perf() [0x413413] /lib64/libc.so.6(+0x35f5) [0x7fd43ce115f5] /lib64/libc.so.6(__libc_start_main+0x88) [0x7fd43ce116a8] perf() [0x413a45] Segmentation fault (core dumped) root@number:~#
Quoted 8 lines
Quoted 3 lines
Thanks, I'll take a look to see if I can spot what's broken. Seeing this stack trace makes me remember we haven't landed:
https://lore.kernel.org/lkml/[email protected]/Yeah, I just pushed perf-tools to soak in linux-next/pending-fixes for a few days and will switch to processing patches for perf-tools-next, will try and pick that one, I also noticed that the backtrace wasn't symbolized, thus your patch wasn't there :-\
I sent out a fix for this issue in v2:
https://lore.kernel.org/lkml/[email protected]/
Patches 1 and 2 I see are in the v6.16 fixes PR:
https://lore.kernel.org/lkml/[email protected]/
Thanks,
Ian
Quoted 1 line
- Arnaldo
Ian Rogers
reviewerPATCH 4/4 libperf evsel: In exit add missed puts and assert close, etc. were called
In reply to Ian RogersA missed evsel__close before evsel__delete was the source of leaking perf events due to a hybrid test. Add asserts in debug builds so that this shouldn't happen in the future. Add puts missing on the cpu map and thread maps.
--- tools/lib/perf/evsel.c | 6 ++++++ 1 file changed, 6 insertions(+)
tools/lib/perf/evsel.c +6 -0
@@ -42,6 +42,12 @@ struct perf_evsel *perf_evsel__new(struct perf_event_attr *attr)void perf_evsel__delete (struct perf_evsel *evsel) { +assert (evsel->fd ==NULL );/* If not fds were not closed. */ +assert (evsel->mmap ==NULL );/* If not munmap wasn't called. */ +assert (evsel->sample_id ==NULL );/* If not free_id wasn't called. */ +perf_cpu_map__put (evsel->cpus); +perf_cpu_map__put (evsel->own_cpus); +perf_thread_map__put (evsel->threads);free (evsel); }
Signature · 1 line
2.50.0.rc1.591.g9c95f17f64-goog