"perf list" built from git repository Segmentation fault with 4.4.0 kernels

DISCUSSION William Cohen 2 participants 3 messages Dec 10, 2015 – Dec 11, 2015

Share & tools

William Cohen

author

2015-12-10 22:54 UTC #0 lore raw

I have noticed that with the 4.4.0 kernels "perf list" has been segfaulting on both aarch64 and x86_64. Below is the failure on fedora rawhide running on x86_64:

[wcohen@localhost ~]$ rpm -q perf
perf-4.4.0-0.rc4.git0.1.fc24.x86_64
[wcohen@localhost ~]$ gdb /bin/perf ./core.2110
GNU gdb (GDB) Fedora 7.10.50.20151113-33.fc24
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/perf...Reading symbols from /usr/lib/debug/usr/bin/perf.debug...done. done.
[New LWP 2110]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `perf list'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055b1fc97fdce in print_symbol_events (type=<optimized out>,

    syms=0x55b1fcd81580 <event_symbols_sw+160>, max=<optimized out>, 
    name_only=<optimized out>, event_glob=<optimized out>)
    at util/parse-events.c:1895

1895 if (!name_only && strlen(syms->alias))
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-19.fc24.x86_64 nss-softokn-freebl-3.21.0-3.fc24.x86_64 (gdb) where
#0 0x000055b1fc97fdce in print_symbol_events (type=<optimized out>,

    syms=0x55b1fcd81580 <event_symbols_sw+160>, max=<optimized out>, 
    name_only=<optimized out>, event_glob=<optimized out>)
    at util/parse-events.c:1895
#1  print_events (event_glob=event_glob@entry=0x0, name_only=false)
    at util/parse-events.c:1943
#2  0x000055b1fc90a63e in cmd_list (argc=0, argv=0x7ffde1523f80, 
    prefix=<optimized out>) at builtin-list.c:43
#3  0x000055b1fc95c413 in run_builtin (p=0x55b1fcd803d8 <commands+120>, 
    argc=1, argv=0x7ffde1523f80) at perf.c:386
#4  0x000055b1fc8fe308 in handle_internal_command (argv=0x7ffde1523f80, argc=1)
    at perf.c:447

#5 run_argv (argv=0x7ffde1523d10, argcp=0x7ffde1523d1c) at perf.c:491 #6 main (argc=1, argv=0x7ffde1523f80) at perf.c:608
(gdb)

I did a git bisect on the kernel source code to find out when "perf list" broke and the git bisect pointed to the following commit causing the problem:

# git bisect good
a43eec304259a6c637f4014a6d4767159b6a3aa3 is the first bad commit
commit a43eec304259a6c637f4014a6d4767159b6a3aa3
Author: Alexei Starovoitov <[email protected]>
Date:   Tue Oct 20 20:02:34 2015 -0700

bpf: introduce bpf_perf_event_output() helper

This helper is used to send raw data from eBPF program into special PERF_TYPE_SOFTWARE/PERF_COUNT_SW_BPF_OUTPUT perf_event. User space needs to perf_event_open() it (either for one or all cpus) and store FD into perf_event_array (similar to bpf_perf_event_read() helper) before eBPF program can send data into it.

Today the programs triggered by kprobe collect the data and either store it into the maps or print it via bpf_trace_printk() where latter is the debug facility and not suitable to stream the data. This new helper replaces such bpf_trace_printk() usage and allows programs to have dedicated channel into user space for post-processing of the raw data collected.

Signed-off-by Alexei Starovoitov <[email protected]>

Signed-off-by David S. Miller <[email protected]>

:040000 040000 322abd03170df11e43da73d3fa4f74f1826788e9 14f3ee14b7fb4b6c76daf18e026cf67acd3154fa M include
:040000 040000 f84baec2884fe97ec501ae8d45bbb8a7aad3a578 ea440aeb13f1dd25aa8f2e1dadfc32dbf4b3d2b3 M kernel

-Will

Arnaldo Carvalho de Melo

2015-12-11 13:10 UTC #1 lore raw

Em Thu, Dec 10, 2015 at 05:54:03PM -0500, William Cohen escreveu:

Quoted 3 lines

I have noticed that with the 4.4.0 kernels "perf list" has been segfaulting on both aarch64 and x86_64. Below is the failure on fedora rawhide running on x86_64:

Ok, this was fixed, but just on the perf/core branch:

"perf list: Add support for PERF_COUNT_SW_BPF_OUT"
https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/tools/perf/util/parse-events.c?h=perf/core&id=bae9cc41105b9edd74d68a9636be2ba240e74b9e

And then this one on top to avoid such things in the future:
"perf list: Robustify event printing routine"
https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/tools/perf/util/parse-events.c?h=perf/core&id=e37df6c76cb19971f1228bfaff504d8a3ea6f748

I'll cherry-pick those and push to Ingo.

Thanks for the report!

- Arnaldo

Quoted 82 lines

[wcohen@localhost ~]$ rpm -q perf
perf-4.4.0-0.rc4.git0.1.fc24.x86_64
[wcohen@localhost ~]$ gdb /bin/perf ./core.2110
GNU gdb (GDB) Fedora 7.10.50.20151113-33.fc24
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/perf...Reading symbols from /usr/lib/debug/usr/bin/perf.debug...done. done.
[New LWP 2110]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `perf list'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055b1fc97fdce in print_symbol_events (type=<optimized out>,
    syms=0x55b1fcd81580 <event_symbols_sw+160>, max=<optimized out>, 
    name_only=<optimized out>, event_glob=<optimized out>)
    at util/parse-events.c:1895
1895 if (!name_only && strlen(syms->alias))
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-19.fc24.x86_64 nss-softokn-freebl-3.21.0-3.fc24.x86_64 (gdb) where
#0 0x000055b1fc97fdce in print_symbol_events (type=<optimized out>,
    syms=0x55b1fcd81580 <event_symbols_sw+160>, max=<optimized out>, 
    name_only=<optimized out>, event_glob=<optimized out>)
    at util/parse-events.c:1895
#1  print_events (event_glob=event_glob@entry=0x0, name_only=false)
    at util/parse-events.c:1943
#2  0x000055b1fc90a63e in cmd_list (argc=0, argv=0x7ffde1523f80, 
    prefix=<optimized out>) at builtin-list.c:43
#3  0x000055b1fc95c413 in run_builtin (p=0x55b1fcd803d8 <commands+120>, 
    argc=1, argv=0x7ffde1523f80) at perf.c:386
#4  0x000055b1fc8fe308 in handle_internal_command (argv=0x7ffde1523f80, argc=1)
    at perf.c:447
#5 run_argv (argv=0x7ffde1523d10, argcp=0x7ffde1523d1c) at perf.c:491 #6 main (argc=1, argv=0x7ffde1523f80) at perf.c:608
(gdb)
I did a git bisect on the kernel source code to find out when "perf list" broke and the git bisect pointed to the following commit causing the problem:
# git bisect good
a43eec304259a6c637f4014a6d4767159b6a3aa3 is the first bad commit
commit a43eec304259a6c637f4014a6d4767159b6a3aa3
Author: Alexei Starovoitov <[email protected]>
Date:   Tue Oct 20 20:02:34 2015 -0700
bpf: introduce bpf_perf_event_output() helper
This helper is used to send raw data from eBPF program into special PERF_TYPE_SOFTWARE/PERF_COUNT_SW_BPF_OUTPUT perf_event. User space needs to perf_event_open() it (either for one or all cpus) and store FD into perf_event_array (similar to bpf_perf_event_read() helper) before eBPF program can send data into it.
Today the programs triggered by kprobe collect the data and either store it into the maps or print it via bpf_trace_printk() where latter is the debug facility and not suitable to stream the data. This new helper replaces such bpf_trace_printk() usage and allows programs to have dedicated channel into user space for post-processing of the raw data collected.
Signed-off-by Alexei Starovoitov <[email protected]>
Signed-off-by David S. Miller <[email protected]>
:040000 040000 322abd03170df11e43da73d3fa4f74f1826788e9 14f3ee14b7fb4b6c76daf18e026cf67acd3154fa M include
:040000 040000 f84baec2884fe97ec501ae8d45bbb8a7aad3a578 ea440aeb13f1dd25aa8f2e1dadfc32dbf4b3d2b3 M kernel
-Will
--
To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

William Cohen

author

2015-12-11 15:08 UTC #2 lore raw

Quoted 15 lines · Arnaldo Carvalho de Melo

Em Thu, Dec 10, 2015 at 05:54:03PM -0500, William Cohen escreveu:
Quoted 3 lines
I have noticed that with the 4.4.0 kernels "perf list" has been segfaulting on both aarch64 and x86_64. Below is the failure on fedora rawhide running on x86_64:
Ok, this was fixed, but just on the perf/core branch:
"perf list: Add support for PERF_COUNT_SW_BPF_OUT"
https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/tools/perf/util/parse-events.c?h=perf/core&id=bae9cc41105b9edd74d68a9636be2ba240e74b9e
And then this one on top to avoid such things in the future:
"perf list: Robustify event printing routine"
https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/tools/perf/util/parse-events.c?h=perf/core&id=e37df6c76cb19971f1228bfaff504d8a3ea6f748

Quoted 5 lines

I'll cherry-pick those and push to Ingo.
Thanks for the report!
- Arnaldo

Thanks Arnaldo. Glad to hear that this is already addressed and just needed to be pulled into the appropriate git repo.

-Will

Quoted 83 lines

Quoted 83 lines
[wcohen@localhost ~]$ rpm -q perf
perf-4.4.0-0.rc4.git0.1.fc24.x86_64
[wcohen@localhost ~]$ gdb /bin/perf ./core.2110
GNU gdb (GDB) Fedora 7.10.50.20151113-33.fc24
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/perf...Reading symbols from /usr/lib/debug/usr/bin/perf.debug...done. done.
[New LWP 2110]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `perf list'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055b1fc97fdce in print_symbol_events (type=<optimized out>,
    syms=0x55b1fcd81580 <event_symbols_sw+160>, max=<optimized out>, 
    name_only=<optimized out>, event_glob=<optimized out>)
    at util/parse-events.c:1895
1895 if (!name_only && strlen(syms->alias))
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-19.fc24.x86_64 nss-softokn-freebl-3.21.0-3.fc24.x86_64 (gdb) where
#0 0x000055b1fc97fdce in print_symbol_events (type=<optimized out>,
    syms=0x55b1fcd81580 <event_symbols_sw+160>, max=<optimized out>, 
    name_only=<optimized out>, event_glob=<optimized out>)
    at util/parse-events.c:1895
#1  print_events (event_glob=event_glob@entry=0x0, name_only=false)
    at util/parse-events.c:1943
#2  0x000055b1fc90a63e in cmd_list (argc=0, argv=0x7ffde1523f80, 
    prefix=<optimized out>) at builtin-list.c:43
#3  0x000055b1fc95c413 in run_builtin (p=0x55b1fcd803d8 <commands+120>, 
    argc=1, argv=0x7ffde1523f80) at perf.c:386
#4  0x000055b1fc8fe308 in handle_internal_command (argv=0x7ffde1523f80, argc=1)
    at perf.c:447
#5 run_argv (argv=0x7ffde1523d10, argcp=0x7ffde1523d1c) at perf.c:491 #6 main (argc=1, argv=0x7ffde1523f80) at perf.c:608
(gdb)
I did a git bisect on the kernel source code to find out when "perf list" broke and the git bisect pointed to the following commit causing the problem:
# git bisect good
a43eec304259a6c637f4014a6d4767159b6a3aa3 is the first bad commit
commit a43eec304259a6c637f4014a6d4767159b6a3aa3
Author: Alexei Starovoitov <[email protected]>
Date:   Tue Oct 20 20:02:34 2015 -0700
bpf: introduce bpf_perf_event_output() helper
This helper is used to send raw data from eBPF program into special PERF_TYPE_SOFTWARE/PERF_COUNT_SW_BPF_OUTPUT perf_event. User space needs to perf_event_open() it (either for one or all cpus) and store FD into perf_event_array (similar to bpf_perf_event_read() helper) before eBPF program can send data into it.
Today the programs triggered by kprobe collect the data and either store it into the maps or print it via bpf_trace_printk() where latter is the debug facility and not suitable to stream the data. This new helper replaces such bpf_trace_printk() usage and allows programs to have dedicated channel into user space for post-processing of the raw data collected.
Signed-off-by Alexei Starovoitov <[email protected]>
Signed-off-by David S. Miller <[email protected]>
:040000 040000 322abd03170df11e43da73d3fa4f74f1826788e9 14f3ee14b7fb4b6c76daf18e026cf67acd3154fa M include
:040000 040000 f84baec2884fe97ec501ae8d45bbb8a7aad3a578 ea440aeb13f1dd25aa8f2e1dadfc32dbf4b3d2b3 M kernel
-Will
--
To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html