Skip to contentSkip to conversation

perf test: Check test suite description properly

PATCH Applied Namhyung Kim 4 participants 15 messages Jul 1, 2025 – Dec 11, 2025 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

PATCH v2 1/3 perf test: Check test suite description properly

Currently perf test checks the given string with descriptions for both test suites and cases (subtests). But sometimes it's confusing since the subtests don't contain the important keyword.

I think it's better to check the suite level and run the whole suite together if it matches description in the suite.

Before:

  $ perf test hwmon
  (no output)

After:

  $ perf test hwmon
   10: Hwmon PMU                                                       :
   10.1: Basic parsing test                                            : Ok
   10.2: Parsing without PMU name                                      : Ok
   10.3: Parsing with PMU name                                         : Ok

And keep the existing behavior when it only matches test description only.

  $ perf test "Equal cpu map"
   39.5: Equal cpu map                                                 : Ok
---
 tools/perf/tests/builtin-test.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/tests/builtin-test.c +5 -2
index 80375ca39a37a256..846c9b3a732c9b3a 100644
@@ -539,6 +539,7 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
 
 		for (struct test_suite **t = suites; *t; t++, curr_suite++) {
 			int curr_test_case;
+			bool suite_matched = false;
 
 			if (!perf_test__matches(test_description(*t, -1), curr_suite, argc, argv)) {
 				/*
@@ -556,6 +557,8 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
 				}
 				if (skip)
 					continue;
+			} else {
+				suite_matched = true;
 			}
 
 			if (intlist__find(skiplist, curr_suite + 1)) {
@@ -567,10 +570,10 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
 
 			for (unsigned int run = 0; run < runs_per_test; run++) {
 				test_suite__for_each_test_case(*t, curr_test_case) {
-					if (!perf_test__matches(test_description(*t, curr_test_case),
+					if (!suite_matched &&
+					    !perf_test__matches(test_description(*t, curr_test_case),
 								curr_suite, argc, argv))
 						continue;
-
 					err = start_test(*t, curr_suite, curr_test_case,
 							 &child_tests[child_test_num++],
 							 width, pass);
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
author maintainer
#1 lore raw

PATCH v2 2/3 perf test: Add libsubcmd help tests

Add a set of tests for subcmd routines. Currently it fails the last one since there's a bug. It'll be fixed by the next commit.

  $ perf test subcmd
   69: libsubcmd help tests                                            :
   69.1: Load subcmd names                                             : Ok
   69.2: Uniquify subcmd names                                         : Ok
   69.3: Exclude duplicate subcmd names                                : FAILED!
---
 tools/perf/tests/Build          |   1 +
 tools/perf/tests/builtin-test.c |   1 +
 tools/perf/tests/subcmd-help.c  | 108 ++++++++++++++++++++++++++++++++
 tools/perf/tests/tests.h        |   2 +
 4 files changed, 112 insertions(+)
 create mode 100644 tools/perf/tests/subcmd-help.c
tools/perf/tests/Build +1 -0
index d6c35dd0de3bd6ce..3e8394be15aea4fb 100644
@@ -69,6 +69,7 @@ perf-test-y += symbols.o
 perf-test-y += util.o
 perf-test-y += hwmon_pmu.o
 perf-test-y += tool_pmu.o
+perf-test-y += subcmd-help.o
 
 ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc))
 perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
index 846c9b3a732c9b3a..e242d56523cee171 100644
@@ -139,6 +139,7 @@ static struct test_suite *generic_tests[] = {
 	&suite__event_groups,
 	&suite__symbols,
 	&suite__util,
+	&suite__subcmd_help,
 	NULL,
 };
 
tools/perf/tests/subcmd-help.c new file +108 -0

108 lines not on this page · raw message

tools/perf/tests/tests.h +2 -0
index 4c128a9594413b32..97e62db8764a0537 100644
@@ -3,6 +3,7 @@
 #define TESTS_H
 
 #include <stdbool.h>
+#include "util/debug.h"
 
 enum {
 	TEST_OK   =  0,
@@ -177,6 +178,7 @@ DECLARE_SUITE(sigtrap);
 DECLARE_SUITE(event_groups);
 DECLARE_SUITE(symbols);
 DECLARE_SUITE(util);
+DECLARE_SUITE(subcmd_help);
 
 /*
  * PowerPC and S390 do not support creation of instruction breakpoints using the
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
author maintainer
#2 lore raw

PATCH v2 3/3 perf tools: Fix use-after-free in help_unknown_cmd()

In reply to Namhyung Kim

Currently perf aborts when it finds an invalid command. I guess it depends on the environment as I have some custom commands in the path.

  $ perf bad-command
  perf: 'bad-command' is not a perf-command. See 'perf --help'.
  Aborted (core dumped)

It's because the exclude_cmds() in libsubcmd has a use-after-free when it removes some entries. After copying one to another entry, it keeps the pointer in the both position. And the next copy operation will free the later one but it's the same entry in the previous one.

For example, let's say cmds = { A, B, C, D, E } and excludes = { B, E }.

ci cj ei cmds-name excludes
-----------+--------------------

   0   0   0 |     A         B       :    cmp < 0, ci == cj
   1   1   0 |     B         B       :    cmp == 0
   2   1   1 |     C         E       :    cmp < 0, ci != cj

At this point, it frees cmds->names[1] and cmds->names[1] is assigned to cmds->names[2].

3 2 1 | D E : cmp < 0, ci != cj

Now it frees cmds->names[2] but it's the same as cmds->names[1]. So accessing cmds->names[1] will be invalid.

This makes the subcmd tests succeed.

  $ perf test subcmd
   69: libsubcmd help tests                                            :
   69.1: Load subcmd names                                             : Ok
   69.2: Uniquify subcmd names                                         : Ok
   69.3: Exclude duplicate subcmd names                                : Ok
Fixes 657a3efee43a ("libsubcmd: Avoid SEGV/use-after-free when commands aren't excluded")
---
 tools/lib/subcmd/help.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
tools/lib/subcmd/help.c +7 -5
index 8561b0f01a247690..9ef569492560efd7 100644
@@ -9,6 +9,7 @@
 #include <sys/stat.h>
 #include <unistd.h>
 #include <dirent.h>
+#include <assert.h>
 #include "subcmd-util.h"
 #include "help.h"
 #include "exec-cmd.h"
@@ -82,10 +83,11 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
 				ci++;
 				cj++;
 			} else {
-				zfree(&cmds->names[cj]);
-				cmds->names[cj++] = cmds->names[ci++];
+				cmds->names[cj++] = cmds->names[ci];
+				cmds->names[ci++] = NULL;
 			}
 		} else if (cmp == 0) {
+			zfree(&cmds->names[ci]);
 			ci++;
 			ei++;
 		} else if (cmp > 0) {
@@ -94,12 +96,12 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
 	}
 	if (ci != cj) {
 		while (ci < cmds->cnt) {
-			zfree(&cmds->names[cj]);
-			cmds->names[cj++] = cmds->names[ci++];
+			cmds->names[cj++] = cmds->names[ci];
+			cmds->names[ci++] = NULL;
 		}
 	}
 	for (ci = cj; ci < cmds->cnt; ci++)
-		zfree(&cmds->names[ci]);
+		assert(cmds->names[ci] == NULL);
 	cmds->cnt = cj;
 }
 
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
#3 lore raw

perf --help triggers an assertion

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL", 
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104, 
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>, 
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

reviewer
#4 lore raw

Re: perf --help triggers an assertion

Quoted 73 lines · Guilherme Amadio

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
Quoted 6 lines

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

Thanks Guilherme,

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

Thanks,
Ian

#5 lore raw

Re: perf --help triggers an assertion

Quoted 83 lines · Ian Rogers
Quoted 73 lines · Guilherme Amadio

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
Quoted 6 lines

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

Thanks Guilherme,

Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

I do not have any special setup on my machine (if you consider Gentoo not special, of course). I just did a git bisect and arrived at commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked above. I used

$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1

then called perf --help each time (but perf help record or equivalent also triggers the assertion). I'm using GCC 14 on a 3950X. The features I have enabled and installed dependencies are below. This patch has been backported to at least 6.12 stable, and I can reproduce this with v6.12.45.

gentoo linux $ tools/perf/perf version --build-options perf version 6.16.g45effee4b85a

                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ on  ]  # HAVE_BPF_SKEL
            debuginfod: [ on  ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ on  ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ on  ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ on  ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ on  ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ on  ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ on  ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ on  ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ on  ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ on  ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ on  ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on  ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ on  ]  # HAVE_ZSTD_SUPPORT

And here are the version of all dependencies I have installed:

* dependency graph for dev-util/perf-6.16-r1
`-- dev-util/perf-6.16-r1

   `--  dev-util/babeltrace-1.5.11
   `--  dev-libs/libbpf-1.6.2
   `--  dev-util/bpftool-7.6.0
   `--  dev-util/pahole-1.30
   `--  llvm-core/clang-18.1.8-r6
   `--  llvm-core/llvm-18.1.8-r6
   `--  llvm-core/clang-19.1.7
   `--  llvm-core/llvm-19.1.7
   `--  llvm-core/clang-20.1.8
   `--  llvm-core/llvm-20.1.8
   `--  llvm-core/clang-21.1.0
   `--  llvm-core/llvm-21.1.0
   `--  sys-libs/libcap-2.76
   `--  dev-libs/capstone-5.0.6
   `--  dev-libs/openssl-3.5.2
   `--  x11-libs/gtk+-2.24.33-r3
   `--  virtual/jre-21
   `--  dev-libs/libpfm-4.13.0
   `--  dev-libs/libtraceevent-1.8.4
   `--  dev-libs/libtracefs-1.8.2
   `--  app-arch/xz-utils-5.8.1-r1
   `--  sys-process/numactl-2.0.19
   `--  dev-lang/perl-5.42.0
   `--  dev-lang/python-0.3.13.7
   `--  dev-lang/python-0.3.14.0_rc2
   `--  dev-lang/python-3.11.13_p1
   `--  dev-lang/python-3.12.11_p1
   `--  dev-lang/python-3.13.9999
   `--  dev-lang/python-3.14.0_rc2
   `--  sys-libs/slang-2.3.3-r1
   `--  dev-debug/systemtap-5.3
   `--  dev-util/google-perftools-2.17.2
   `--  sys-libs/libunwind-1.8.2-r1
   `--  app-arch/zstd-1.5.7-r1
   `--  dev-libs/elfutils-0.193
   `--  sys-libs/binutils-libs-2.45-r1
   `--  sys-libs/zlib-1.3.1-r1
   `--  virtual/libcrypt-2-r1
   `--  sys-kernel/linux-headers-6.16
   `--  virtual/jdk-21
   `--  app-arch/tar-1.35
   `--  dev-python/setuptools-80.9.0-r1
   `--  app-alternatives/yacc-1-r2
   `--  app-alternatives/lex-0-r1
   `--  sys-apps/which-2.23
   `--  virtual/pkgconfig-3
   `--  app-text/asciidoc-10.2.1
   `--  app-text/sgml-common-0.6.3-r7
   `--  app-text/xmlto-0.0.28-r11
   `--  sys-process/time-1.9-r1

I hope that this helps with reproducing the problem, but if not, let me know which additional information you'd like to have and I will send it. My running kernel is 6.16.4, and glibc is version 2.42.

Best regards,
-Guilherme

reviewer
#6 lore raw

Re: perf --help triggers an assertion

Quoted 189 lines · Guilherme Amadio
Quoted 83 lines · Ian Rogers
Quoted 73 lines · Guilherme Amadio

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
Quoted 6 lines

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

Thanks Guilherme,

Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

I do not have any special setup on my machine (if you consider Gentoo not special, of course). I just did a git bisect and arrived at commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked above. I used

$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1

then called perf --help each time (but perf help record or equivalent also triggers the assertion). I'm using GCC 14 on a 3950X. The features I have enabled and installed dependencies are below. This patch has been backported to at least 6.12 stable, and I can reproduce this with v6.12.45.

gentoo linux $ tools/perf/perf version --build-options perf version 6.16.g45effee4b85a

                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ on  ]  # HAVE_BPF_SKEL
            debuginfod: [ on  ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ on  ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ on  ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ on  ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ on  ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ on  ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ on  ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ on  ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ on  ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ on  ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ on  ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on  ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ on  ]  # HAVE_ZSTD_SUPPORT

And here are the version of all dependencies I have installed:

* dependency graph for dev-util/perf-6.16-r1
`-- dev-util/perf-6.16-r1

   `--  dev-util/babeltrace-1.5.11
   `--  dev-libs/libbpf-1.6.2
   `--  dev-util/bpftool-7.6.0
   `--  dev-util/pahole-1.30
   `--  llvm-core/clang-18.1.8-r6
   `--  llvm-core/llvm-18.1.8-r6
   `--  llvm-core/clang-19.1.7
   `--  llvm-core/llvm-19.1.7
   `--  llvm-core/clang-20.1.8
   `--  llvm-core/llvm-20.1.8
   `--  llvm-core/clang-21.1.0
   `--  llvm-core/llvm-21.1.0
   `--  sys-libs/libcap-2.76
   `--  dev-libs/capstone-5.0.6
   `--  dev-libs/openssl-3.5.2
   `--  x11-libs/gtk+-2.24.33-r3
   `--  virtual/jre-21
   `--  dev-libs/libpfm-4.13.0
   `--  dev-libs/libtraceevent-1.8.4
   `--  dev-libs/libtracefs-1.8.2
   `--  app-arch/xz-utils-5.8.1-r1
   `--  sys-process/numactl-2.0.19
   `--  dev-lang/perl-5.42.0
   `--  dev-lang/python-0.3.13.7
   `--  dev-lang/python-0.3.14.0_rc2
   `--  dev-lang/python-3.11.13_p1
   `--  dev-lang/python-3.12.11_p1
   `--  dev-lang/python-3.13.9999
   `--  dev-lang/python-3.14.0_rc2
   `--  sys-libs/slang-2.3.3-r1
   `--  dev-debug/systemtap-5.3
   `--  dev-util/google-perftools-2.17.2
   `--  sys-libs/libunwind-1.8.2-r1
   `--  app-arch/zstd-1.5.7-r1
   `--  dev-libs/elfutils-0.193
   `--  sys-libs/binutils-libs-2.45-r1
   `--  sys-libs/zlib-1.3.1-r1
   `--  virtual/libcrypt-2-r1
   `--  sys-kernel/linux-headers-6.16
   `--  virtual/jdk-21
   `--  app-arch/tar-1.35
   `--  dev-python/setuptools-80.9.0-r1
   `--  app-alternatives/yacc-1-r2
   `--  app-alternatives/lex-0-r1
   `--  sys-apps/which-2.23
   `--  virtual/pkgconfig-3
   `--  app-text/asciidoc-10.2.1
   `--  app-text/sgml-common-0.6.3-r7
   `--  app-text/xmlto-0.0.28-r11
   `--  sys-process/time-1.9-r1
Quoted 3 lines

I hope that this helps with reproducing the problem, but if not, let me know which additional information you'd like to have and I will send it. My running kernel is 6.16.4, and glibc is version 2.42.

Thanks Guilherme,

There is also:
https://lore.kernel.org/lkml/CADHxFxSBKuKUWDvki9QDqc5jf1Y+k2EpvOnDsg8O09pZ10n2NA@mail.gmail.com/
which may give me a reproduction. The fix there doesn't look right. I'll try to find time to look into it again.

Thanks,
Ian

#7 lore raw

Re: perf --help triggers an assertion

Hi Ian,

Quoted 86 lines · Ian Rogers
Quoted 85 lines · Guilherme Amadio
Quoted 83 lines · Ian Rogers
Quoted 73 lines · Guilherme Amadio

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
Quoted 6 lines

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

Thanks Guilherme,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.

BTW, I've followed your advice and turned off BUILD_NONDISTRO:
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f

It looks like the logic to exclude commands sometimes doesn't work as expected, and you reach a line where there is an assertion for cmds->names[ci] == NULL, but it's not actually NULL. The difference with other setups is probably the installation of perf-read-vdso32 into a default path in our packaging. In gdb I saw that when running perf --help, I arrive at exclude_cmds() with the following:

Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
74 {
(gdb) p *cmds
$1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
(gdb) p *excludes
$2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
(gdb) p *cmds->names[0]
$3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
(gdb) p *excludes->names[0]
$4 = {len = 7, name = 0x555556890198 "archive"}

I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:

$ ls /usr/libexec/perf-core/
dlfilters  libperf-gtk.so  perf-archive  perf-iostat  scripts  tests

If I remove /usr/bin/perf-read-vdso32, then perf --help works again:

gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
gentoo ~ $ perf version
perf version 6.18
gentoo ~ $ perf --help | head -3

usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]

gentoo ~ $ touch ~/bin/perf-read-vdso32
gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
gentoo ~ $ perf --help
perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) perf --help

So, I think that if you don't have perf-read-vdso32 installed by default, you will likely be able to reproduce the problem like shown above. So, the assumption that cmds->names[0] == NULL is not correct, since we have the "read-vdso32" command at position 0, and the excludes list only contained "archive". The patch below is my attempt to fix the issue:

tools/lib/subcmd/help.c +3 -3
index ddaeb4eb3e24..0c4ba87422e5 100644
@@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
                        cmds->names[cj++] = cmds->names[ci];
                        cmds->names[ci++] = NULL;
                }
+               for (ci = cj; ci < cmds->cnt; ci++)
+                       assert(cmds->names[ci] == NULL);
+               cmds->cnt = cj;
        }
-       for (ci = cj; ci < cmds->cnt; ci++)
-               assert(cmds->names[ci] == NULL);
-       cmds->cnt = cj;
 }
 

So, if there are no actual command exclusions, at the end of the while loop we get out with ei > excludes->cnt, and ci == cj, like the example above, in which ci == cj == 0, and ei == 1. So the assumption that cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32" command at position 0, and setting cmds->cnt = cj; is also wrong, since cnt should be 1 and is set to 0. The patch moves the asserts and the update of cmds->cnt inside the if(ci != cj), which indicates something was actually excluded. If cmds was not modified, then the initial value of cmds->cnt was already correct. Please let me know if you'd like me to submit this properly as a patch, but since it's so simple, please feel free to just take this and apply yourself.

Best regards,
-Guilherme

reviewer
#8 lore raw

Re: perf --help triggers an assertion

Quoted 167 lines · Guilherme Amadio

Hi Ian,

Quoted 86 lines · Ian Rogers
Quoted 85 lines · Guilherme Amadio
Quoted 83 lines · Ian Rogers
Quoted 73 lines · Guilherme Amadio

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
Quoted 6 lines

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

Thanks Guilherme,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.

BTW, I've followed your advice and turned off BUILD_NONDISTRO:
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f

It looks like the logic to exclude commands sometimes doesn't work as expected, and you reach a line where there is an assertion for cmds->names[ci] == NULL, but it's not actually NULL. The difference with other setups is probably the installation of perf-read-vdso32 into a default path in our packaging. In gdb I saw that when running perf --help, I arrive at exclude_cmds() with the following:

Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
74 {
(gdb) p *cmds
$1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
(gdb) p *excludes
$2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
(gdb) p *cmds->names[0]
$3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
(gdb) p *excludes->names[0]
$4 = {len = 7, name = 0x555556890198 "archive"}

I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:

$ ls /usr/libexec/perf-core/
dlfilters  libperf-gtk.so  perf-archive  perf-iostat  scripts  tests

If I remove /usr/bin/perf-read-vdso32, then perf --help works again:

gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
gentoo ~ $ perf version
perf version 6.18
gentoo ~ $ perf --help | head -3

usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]

gentoo ~ $ touch ~/bin/perf-read-vdso32
gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
gentoo ~ $ perf --help
perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) perf --help

So, I think that if you don't have perf-read-vdso32 installed by default, you will likely be able to reproduce the problem like shown above. So, the assumption that cmds->names[0] == NULL is not correct, since we have the "read-vdso32" command at position 0, and the excludes list only contained "archive". The patch below is my attempt to fix the issue:

tools/lib/subcmd/help.c +3 -3
index ddaeb4eb3e24..0c4ba87422e5 100644
@@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
                        cmds->names[cj++] = cmds->names[ci];
                        cmds->names[ci++] = NULL;
                }
+               for (ci = cj; ci < cmds->cnt; ci++)
+                       assert(cmds->names[ci] == NULL);
+               cmds->cnt = cj;
        }
-       for (ci = cj; ci < cmds->cnt; ci++)
-               assert(cmds->names[ci] == NULL);
-       cmds->cnt = cj;
 }
 

So, if there are no actual command exclusions, at the end of the while loop we get out with ei > excludes->cnt, and ci == cj, like the example above, in which ci == cj == 0, and ei == 1. So the assumption that

Quoted 8 lines

cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32" command at position 0, and setting cmds->cnt = cj; is also wrong, since cnt should be 1 and is set to 0. The patch moves the asserts and the update of cmds->cnt inside the if(ci != cj), which indicates something was actually excluded. If cmds was not modified, then the initial value of cmds->cnt was already correct. Please let me know if you'd like me to submit this properly as a patch, but since it's so simple, please feel free to just take this and apply yourself.

Thanks Guilherme,

This code is the gift that keeps on giving in terms of crashes. We did have a recent and related patch sent to the list:
https://lore.kernel.org/lkml/[email protected]/
I need some time to get my head around the issue and the fixes. I wandered into this code originally due to address/leak sanitizer issues.

Thanks,
Ian

Quoted 2 lines

Best regards,
-Guilherme

#9 lore raw

Re: perf --help triggers an assertion

Quoted 179 lines · Ian Rogers
Quoted 167 lines · Guilherme Amadio

Hi Ian,

Quoted 86 lines · Ian Rogers
Quoted 85 lines · Guilherme Amadio
Quoted 83 lines · Ian Rogers
Quoted 73 lines · Guilherme Amadio

Hi Namhyung,

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

Cheers,
-Guilherme

The problem:

gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help

Some extra information:

gentoo perf $ ./perf version
perf version 6.17.rc5.gf777d1112ee5
gentoo perf $ ./perf test subcmd

 69: libsubcmd help tests                                            :
 69.1: Load subcmd names                                             : Ok
 69.2: Uniquify subcmd names                                         : Ok
 69.3: Exclude duplicate subcmd names                                : Ok

gentoo perf $ ./perf bad-command
perf: 'bad-command' is not a perf-command. See 'perf --help'. gentoo perf $ ./perf --help
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) ./perf --help
gentoo perf $ gdb run --args ./perf --help
GNU gdb (Gentoo 16.3 vanilla) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"... Reading symbols from ./perf...
(gdb) run
Starting program: /home/amadio/src/linux/tools/perf/perf --help [Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
perf: help.c:104: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff74c1656 in __pthread_kill_internal (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:89
#2  0x00007ffff74c166d in __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at pthread_kill.c:100
#3  0x00007ffff747509c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff747637e in __GI_abort () at abort.c:77
#5  0x00007ffff746e023 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x555555974d03 "cmds->names[ci] == NULL",
    file=file@entry=0x555555974cfc "help.c", line=line@entry=104, function=function@entry=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds")
    at assert.c:118
#6  0x00007ffff746e075 in __assert_fail (assertion=0x555555974d03 "cmds->names[ci] == NULL", file=0x555555974cfc "help.c", line=104,
    function=0x555555974dc8 <__PRETTY_FUNCTION__.0> "exclude_cmds") at assert.c:127
#7  0x0000555555693813 in exclude_cmds (cmds=0x55555614e5e0 <other_cmds>, excludes=0x55555614e5c0 <main_cmds>) at help.c:104
#8  0x0000555555693eb3 in load_command_list (prefix=0x555555954071 "perf-", main_cmds=0x55555614e5c0 <main_cmds>,
    other_cmds=0x55555614e5e0 <other_cmds>) at help.c:252
Quoted 6 lines

#9 0x00005555555e9987 in cmd_help (argc=1, argv=0x7fffffffd1a0) at builtin-help.c:458
#10 0x0000555555685d45 in run_builtin (p=0x555556130de0 <commands+192>, argc=1, argv=0x7fffffffd1a0) at perf.c:349 #11 0x0000555555685fe1 in handle_internal_command (argc=1, argv=0x7fffffffd1a0) at perf.c:401 #12 0x0000555555686142 in run_argv (argcp=0x7fffffffcfac, argv=0x7fffffffcfa0) at perf.c:445 #13 0x0000555555686493 in main (argc=1, argv=0x7fffffffd1a0) at perf.c:553
(gdb) quit

Thanks Guilherme,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

I had a look at this with gdb again while I was updating perf to 6.18 in Gentoo.

BTW, I've followed your advice and turned off BUILD_NONDISTRO:
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b92c5b8bd89c70c40a84f86013c4ecf321f3132f

It looks like the logic to exclude commands sometimes doesn't work as expected, and you reach a line where there is an assertion for cmds->names[ci] == NULL, but it's not actually NULL. The difference with other setups is probably the installation of perf-read-vdso32 into a default path in our packaging. In gdb I saw that when running perf --help, I arrive at exclude_cmds() with the following:

Breakpoint 1, exclude_cmds (cmds=cmds@entry=0x55555605bd90 <other_cmds>, excludes=excludes@entry=0x55555605bdb0 <main_cmds>) at help.c:74
74 {
(gdb) p *cmds
$1 = {alloc = 24, cnt = 1, names = 0x5555568fe180}
(gdb) p *excludes
$2 = {alloc = 24, cnt = 1, names = 0x5555568fe0c0}
(gdb) p *cmds->names[0]
$3 = {len = 11, name = 0x5555568db788 "read-vdso32"}
(gdb) p *excludes->names[0]
$4 = {len = 7, name = 0x555556890198 "archive"}

I have /usr/bin/perf and /usr/bin/perf-read-vdso32, as well as this:

$ ls /usr/libexec/perf-core/
dlfilters  libperf-gtk.so  perf-archive  perf-iostat  scripts  tests

If I remove /usr/bin/perf-read-vdso32, then perf --help works again:

gentoo ~ $ sudo rm /usr/bin/perf-read-vdso32
gentoo ~ $ perf version
perf version 6.18
gentoo ~ $ perf --help | head -3

usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]

gentoo ~ $ touch ~/bin/perf-read-vdso32
gentoo ~ $ chmod +x ~/bin/perf-read-vdso32
gentoo ~ $ perf --help
perf: help.c:107: exclude_cmds: Assertion `cmds->names[ci] == NULL' failed.
Aborted (core dumped) perf --help

So, I think that if you don't have perf-read-vdso32 installed by default, you will likely be able to reproduce the problem like shown above. So, the assumption that cmds->names[0] == NULL is not correct, since we have the "read-vdso32" command at position 0, and the excludes list only contained "archive". The patch below is my attempt to fix the issue:

tools/lib/subcmd/help.c +3 -3
index ddaeb4eb3e24..0c4ba87422e5 100644
@@ -102,10 +102,10 @@ void exclude_cmds(struct cmdnames *cmds, struct cmdnames *excludes)
                        cmds->names[cj++] = cmds->names[ci];
                        cmds->names[ci++] = NULL;
                }
+               for (ci = cj; ci < cmds->cnt; ci++)
+                       assert(cmds->names[ci] == NULL);
+               cmds->cnt = cj;
        }
-       for (ci = cj; ci < cmds->cnt; ci++)
-               assert(cmds->names[ci] == NULL);
-       cmds->cnt = cj;
 }
 

So, if there are no actual command exclusions, at the end of the while loop we get out with ei > excludes->cnt, and ci == cj, like the example above, in which ci == cj == 0, and ei == 1. So the assumption that

Quoted 8 lines

cmds->names[0] == NULL is wrong, since there's a valid "read-vdso32" command at position 0, and setting cmds->cnt = cj; is also wrong, since cnt should be 1 and is set to 0. The patch moves the asserts and the update of cmds->cnt inside the if(ci != cj), which indicates something was actually excluded. If cmds was not modified, then the initial value of cmds->cnt was already correct. Please let me know if you'd like me to submit this properly as a patch, but since it's so simple, please feel free to just take this and apply yourself.

Thanks Guilherme,

Quoted 6 lines

This code is the gift that keeps on giving in terms of crashes. We did have a recent and related patch sent to the list:
https://lore.kernel.org/lkml/[email protected]/
I need some time to get my head around the issue and the fixes. I wandered into this code originally due to address/leak sanitizer issues.

Thank you, Ian,

I've applied Sri's patch in Gentoo. It seems to be a better fix than what I proposed above, so please feel free to add my Tested-by: to Sri's patch.

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a77ba80e013b62ca44a3123167aa3ee376e7faeb

Best regards,
-Guilherme

#10 lore raw

Re: perf --help triggers an assertion

In reply to Guilherme Amadio
Quoted 8 lines · Guilherme Amadio
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Guilherme Amadio

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

<SNIP>

Quoted 8 lines
Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

Quoted 4 lines

I do not have any special setup on my machine (if you consider Gentoo not special, of course). I just did a git bisect and arrived at commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked above. I used>

⬢ [acme@toolbx perf-tools-next]$ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d
fatal: bad object 9401d1771dad99bfc795dd2ae0c292343fd1f78d
⬢ [acme@toolbx perf-tools-next]$

Looking for the patch title I got to this one:

commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae

Author: Namhyung Kim <[email protected]>
Date:   Tue Jul 1 13:10:27 2025 -0700

perf tools: Fix use-after-free in help_unknown_cmd()

?

- Arnaldo

Quoted 97 lines
$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1

then called perf --help each time (but perf help record or equivalent also triggers the assertion). I'm using GCC 14 on a 3950X. The features I have enabled and installed dependencies are below. This patch has been backported to at least 6.12 stable, and I can reproduce this with v6.12.45.

gentoo linux $ tools/perf/perf version --build-options perf version 6.16.g45effee4b85a

                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ on  ]  # HAVE_BPF_SKEL
            debuginfod: [ on  ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ on  ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ on  ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ on  ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ on  ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ on  ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ on  ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ on  ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ on  ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ on  ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ on  ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on  ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ on  ]  # HAVE_ZSTD_SUPPORT

And here are the version of all dependencies I have installed:

* dependency graph for dev-util/perf-6.16-r1
`-- dev-util/perf-6.16-r1

   `--  dev-util/babeltrace-1.5.11
   `--  dev-libs/libbpf-1.6.2
   `--  dev-util/bpftool-7.6.0
   `--  dev-util/pahole-1.30
   `--  llvm-core/clang-18.1.8-r6
   `--  llvm-core/llvm-18.1.8-r6
   `--  llvm-core/clang-19.1.7
   `--  llvm-core/llvm-19.1.7
   `--  llvm-core/clang-20.1.8
   `--  llvm-core/llvm-20.1.8
   `--  llvm-core/clang-21.1.0
   `--  llvm-core/llvm-21.1.0
   `--  sys-libs/libcap-2.76
   `--  dev-libs/capstone-5.0.6
   `--  dev-libs/openssl-3.5.2
   `--  x11-libs/gtk+-2.24.33-r3
   `--  virtual/jre-21
   `--  dev-libs/libpfm-4.13.0
   `--  dev-libs/libtraceevent-1.8.4
   `--  dev-libs/libtracefs-1.8.2
   `--  app-arch/xz-utils-5.8.1-r1
   `--  sys-process/numactl-2.0.19
   `--  dev-lang/perl-5.42.0
   `--  dev-lang/python-0.3.13.7
   `--  dev-lang/python-0.3.14.0_rc2
   `--  dev-lang/python-3.11.13_p1
   `--  dev-lang/python-3.12.11_p1
   `--  dev-lang/python-3.13.9999
   `--  dev-lang/python-3.14.0_rc2
   `--  sys-libs/slang-2.3.3-r1
   `--  dev-debug/systemtap-5.3
   `--  dev-util/google-perftools-2.17.2
   `--  sys-libs/libunwind-1.8.2-r1
   `--  app-arch/zstd-1.5.7-r1
   `--  dev-libs/elfutils-0.193
   `--  sys-libs/binutils-libs-2.45-r1
   `--  sys-libs/zlib-1.3.1-r1
   `--  virtual/libcrypt-2-r1
   `--  sys-kernel/linux-headers-6.16
   `--  virtual/jdk-21
   `--  app-arch/tar-1.35
   `--  dev-python/setuptools-80.9.0-r1
   `--  app-alternatives/yacc-1-r2
   `--  app-alternatives/lex-0-r1
   `--  sys-apps/which-2.23
   `--  virtual/pkgconfig-3
   `--  app-text/asciidoc-10.2.1
   `--  app-text/sgml-common-0.6.3-r7
   `--  app-text/xmlto-0.0.28-r11
   `--  sys-process/time-1.9-r1

I hope that this helps with reproducing the problem, but if not, let me know which additional information you'd like to have and I will send it. My running kernel is 6.16.4, and glibc is version 2.42.

Best regards,
-Guilherme

author maintainer
#11 lore raw

Re: perf --help triggers an assertion

Quoted 29 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Guilherme Amadio
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Guilherme Amadio

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

<SNIP>

Quoted 8 lines
Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

Quoted 4 lines

I do not have any special setup on my machine (if you consider Gentoo not special, of course). I just did a git bisect and arrived at commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked above. I used>

⬢ [acme@toolbx perf-tools-next]$ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d
fatal: bad object 9401d1771dad99bfc795dd2ae0c292343fd1f78d
⬢ [acme@toolbx perf-tools-next]$

Quoted 7 lines

Looking for the patch title I got to this one:

commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae

Author: Namhyung Kim <[email protected]>
Date:   Tue Jul 1 13:10:27 2025 -0700

perf tools: Fix use-after-free in help_unknown_cmd()

Sorry for the trouble. The hupu's patch should fix the problem.

Thanks,
Namhyung

#12 lore raw

Re: perf --help triggers an assertion

In reply to Arnaldo Carvalho de Melo
Quoted 31 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Guilherme Amadio
Quoted 7 lines · Ian Rogers
Quoted 6 lines · Guilherme Amadio

I was updating perf's package in Gentoo Linux and noticed some problems which were not there before. I tested with the version below and the problem still seems to be there. perf --help triggers an assertion (see below). Looking in the list, it seems related to the patch below:

https://lore.kernel.org/linux-perf-users/[email protected]/

<SNIP>

Quoted 8 lines
Quoted 7 lines

I tried to reproduce the same version with various options: DEBUG=1 -UNDEBUG in EXTRA_CFLAGS, -fsanitize=address. Being in various directories with "perf-" prefixed files. I'm afraid I wasn't able to reproduce. The assert is trying to avoid a memory leak, so non-critical, and I couldn't in a quick inspection eye-ball an issue. Without getting a reproduction I don't think I can make progress with the issue.

Quoted 4 lines

I do not have any special setup on my machine (if you consider Gentoo not special, of course). I just did a git bisect and arrived at commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d, which is the commit I linked above. I used>

⬢ [acme@toolbx perf-tools-next]$ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d
fatal: bad object 9401d1771dad99bfc795dd2ae0c292343fd1f78d
⬢ [acme@toolbx perf-tools-next]$

Looking for the patch title I got to this one:

Quoted 7 lines

commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae

Author: Namhyung Kim <[email protected]>
Date:   Tue Jul 1 13:10:27 2025 -0700

perf tools: Fix use-after-free in help_unknown_cmd()

?

Sorry, I used the tag I arrived at with git bisect, which was probably cherry-picked:

gentoo linux $ git remote -v
amadio [email protected]:amadio/linux (fetch)
amadio [email protected]:amadio/linux (push)
perf-tools git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools.git (fetch)
perf-tools git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools.git (push)
perf-tools-next git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git (fetch)
perf-tools-next git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git (push)
stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (fetch)
stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (push)
torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (fetch)
torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (push)
gentoo linux $ git show 9401d1771dad99bfc795dd2ae0c292343fd1f78d | head commit 9401d1771dad99bfc795dd2ae0c292343fd1f78d
Author: Namhyung Kim <[email protected]>
Date: Tue Jul 1 13:10:27 2025 -0700

perf tools: Fix use-after-free in help_unknown_cmd()

[ Upstream commit 1fdf938168c4d26fa279d4f204768690d1f9c4ae ]

Currently perf aborts when it finds an invalid command. I guess it depends on the environment as I have some custom commands in the path.
gentoo linux $ git tag --contains 9401d1771dad99bfc795dd2ae0c292343fd1f78d v6.16.1
v6.16.2
v6.16.3
v6.16.4
v6.16.5
v6.16.6
v6.16.7
gentoo linux $ git tag --contains 1fdf938168c4d26fa279d4f204768690d1f9c4ae perf-tools-fixes-for-v6.17-2025-08-27 perf-tools-fixes-for-v6.17-2025-09-05 perf-tools-for-v6.17-2025-08-01
v6.17-rc1
v6.17-rc2
v6.17-rc3
v6.17-rc4
v6.17-rc5

Cheers,
-Guilherme

Quoted 100 lines

- Arnaldo

Quoted 97 lines
$ make -C tools/perf clean && make -B -j16 -C tools/perf DEBUG=1 WERROR=0 NO_SHELLCHECK=1

then called perf --help each time (but perf help record or equivalent also triggers the assertion). I'm using GCC 14 on a 3950X. The features I have enabled and installed dependencies are below. This patch has been backported to at least 6.12 stable, and I can reproduce this with v6.12.45.

gentoo linux $ tools/perf/perf version --build-options perf version 6.16.g45effee4b85a

                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ on  ]  # HAVE_BPF_SKEL
            debuginfod: [ on  ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ on  ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ on  ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ on  ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ on  ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ on  ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ on  ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ on  ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ on  ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ on  ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ on  ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on  ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ on  ]  # HAVE_ZSTD_SUPPORT

And here are the version of all dependencies I have installed:

* dependency graph for dev-util/perf-6.16-r1
`-- dev-util/perf-6.16-r1

   `--  dev-util/babeltrace-1.5.11
   `--  dev-libs/libbpf-1.6.2
   `--  dev-util/bpftool-7.6.0
   `--  dev-util/pahole-1.30
   `--  llvm-core/clang-18.1.8-r6
   `--  llvm-core/llvm-18.1.8-r6
   `--  llvm-core/clang-19.1.7
   `--  llvm-core/llvm-19.1.7
   `--  llvm-core/clang-20.1.8
   `--  llvm-core/llvm-20.1.8
   `--  llvm-core/clang-21.1.0
   `--  llvm-core/llvm-21.1.0
   `--  sys-libs/libcap-2.76
   `--  dev-libs/capstone-5.0.6
   `--  dev-libs/openssl-3.5.2
   `--  x11-libs/gtk+-2.24.33-r3
   `--  virtual/jre-21
   `--  dev-libs/libpfm-4.13.0
   `--  dev-libs/libtraceevent-1.8.4
   `--  dev-libs/libtracefs-1.8.2
   `--  app-arch/xz-utils-5.8.1-r1
   `--  sys-process/numactl-2.0.19
   `--  dev-lang/perl-5.42.0
   `--  dev-lang/python-0.3.13.7
   `--  dev-lang/python-0.3.14.0_rc2
   `--  dev-lang/python-3.11.13_p1
   `--  dev-lang/python-3.12.11_p1
   `--  dev-lang/python-3.13.9999
   `--  dev-lang/python-3.14.0_rc2
   `--  sys-libs/slang-2.3.3-r1
   `--  dev-debug/systemtap-5.3
   `--  dev-util/google-perftools-2.17.2
   `--  sys-libs/libunwind-1.8.2-r1
   `--  app-arch/zstd-1.5.7-r1
   `--  dev-libs/elfutils-0.193
   `--  sys-libs/binutils-libs-2.45-r1
   `--  sys-libs/zlib-1.3.1-r1
   `--  virtual/libcrypt-2-r1
   `--  sys-kernel/linux-headers-6.16
   `--  virtual/jdk-21
   `--  app-arch/tar-1.35
   `--  dev-python/setuptools-80.9.0-r1
   `--  app-alternatives/yacc-1-r2
   `--  app-alternatives/lex-0-r1
   `--  sys-apps/which-2.23
   `--  virtual/pkgconfig-3
   `--  app-text/asciidoc-10.2.1
   `--  app-text/sgml-common-0.6.3-r7
   `--  app-text/xmlto-0.0.28-r11
   `--  sys-process/time-1.9-r1

I hope that this helps with reproducing the problem, but if not, let me know which additional information you'd like to have and I will send it. My running kernel is 6.16.4, and glibc is version 2.42.

Best regards,
-Guilherme

reviewer Reviewed-by on v2 1/3
#13 lore raw
In reply to Namhyung Kim
Quoted 19 lines · Namhyung Kim

Currently perf test checks the given string with descriptions for both test suites and cases (subtests). But sometimes it's confusing since the subtests don't contain the important keyword.

I think it's better to check the suite level and run the whole suite together if it matches description in the suite.

Before:

  $ perf test hwmon
  (no output)

After:

  $ perf test hwmon
   10: Hwmon PMU                                                       :
   10.1: Basic parsing test                                            : Ok
   10.2: Parsing without PMU name                                      : Ok
   10.3: Parsing with PMU name                                         : Ok
Quoted 6 lines

And keep the existing behavior when it only matches test description only.

  $ perf test "Equal cpu map"
   39.5: Equal cpu map                                                 : Ok

Thanks,
Ian

Quoted 41 lines
---
 tools/perf/tests/builtin-test.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/tests/builtin-test.c +5 -3
index 80375ca39a37a256..846c9b3a732c9b3a 100644
@@ -539,6 +539,7 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
 
                for (struct test_suite **t = suites; *t; t++, curr_suite++) {
                        int curr_test_case;
+                       bool suite_matched = false;
 
                        if (!perf_test__matches(test_description(*t, -1), curr_suite, argc, argv)) {
                                /*
@@ -556,6 +557,8 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
                                }
                                if (skip)
                                        continue;
+                       } else {
+                               suite_matched = true;
                        }
 
                        if (intlist__find(skiplist, curr_suite + 1)) {
@@ -567,10 +570,10 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[],
 
                        for (unsigned int run = 0; run < runs_per_test; run++) {
                                test_suite__for_each_test_case(*t, curr_test_case) {
-                                       if (!perf_test__matches(test_description(*t, curr_test_case),
+                                       if (!suite_matched &&
+                                           !perf_test__matches(test_description(*t, curr_test_case),
                                                                curr_suite, argc, argv))
                                                continue;
-
                                        err = start_test(*t, curr_suite, curr_test_case,
                                                         &child_tests[child_test_num++],
                                                         width, pass);
--

2.50.0.727.gbf7dc18ff4-goog

author maintainer Applied
#14 lore raw
In reply to Namhyung Kim
Quoted 8 lines · Namhyung Kim

Currently perf test checks the given string with descriptions for both test suites and cases (subtests). But sometimes it's confusing since the subtests don't contain the important keyword.

I think it's better to check the suite level and run the whole suite together if it matches description in the suite.

[...]

Applied to perf-tools-next, thanks!

Best regards,
Namhyung