Skip to contentSkip to conversation

Fedora 33 and perf probe failures

PATCH In review Thomas Richter 3 participants 11 messages Nov 18, 2020 – Nov 27, 2020 perf events

lore
Share & tools
lore
#0 lore raw

I debugged the failure of some perf test cases on Fedora 33. The following test cases fail on s390 and x86_64 using

[root@f33 yum.repos.d]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020

                                x86_64 x86_64 x86_64 GNU/Linux

[root@f33 yum.repos.d]#

[root@f33 yum.repos.d]# perf test -F 74 77 78
74: Use vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
77: Check open filename arg using perf trace + vfs_getname
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
78: Add vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
[root@f33 yum.repos.d]#

The root cause is the output of perf probe -L which returns:
[root@f33 yum.repos.d]# perf probe -L getname_flags Specified source line is not found.
Error: Failed to show lines.
[root@f33 yum.repos.d]#

whereas on my Fedora 32 installation this command shows [root@t35lp46 perf]# cat /etc/redhat-release Fedora release 32 (Thirty Two)
[root@t35lp46 perf]#

[root@t35lp46 perf]# ./perf probe -L getname_flags | tail -10

     68                         return ERR_PTR(-ENOENT);
                        }
                }

     72         result->uptr = filename;
     73         result->aname = NULL;
     74         audit_getname(result);
                return result;
}

[root@t35lp46 perf]#

Any ideas how to fix this? Is this failure related to a missing debuginfod setup, even when I have installed the debuginfo packages for the kernel:

[root@f33 ~]# perf probe -L getname_flags
Specified source line is not found.
Error: Failed to show lines.
[root@f33 ~]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@f33 ~]# rpm -qa | fgrep kernel | fgrep debuginfo kernel-debuginfo-common-x86_64-5.9.8-200.fc33.x86_64 kernel-debuginfo-5.9.8-200.fc33.x86_64 kernel-tools-debuginfo-5.9.7-200.fc33.x86_64
[root@f33 ~]#
~

Thanks a lot

Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
#1 lore raw

Hi Thomas,

Quoted 2 lines · On Wed, 18 Nov 2020 17:37:21 +0100 Thomas Richter <[email protected]> wrote:

I debugged the failure of some perf test cases on Fedora 33. The following test cases fail on s390 and x86_64 using

Thanks for the report!
Let me setup Fedora33.

Quoted 6 lines

[root@f33 yum.repos.d]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020

                                x86_64 x86_64 x86_64 GNU/Linux

[root@f33 yum.repos.d]#

Quoted 14 lines

[root@f33 yum.repos.d]# perf test -F 74 77 78
74: Use vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
77: Check open filename arg using perf trace + vfs_getname
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
78: Add vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
[root@f33 yum.repos.d]#

BTW, do we have vfs_getname in the kernel?

Quoted 5 lines

The root cause is the output of perf probe -L which returns:
[root@f33 yum.repos.d]# perf probe -L getname_flags Specified source line is not found.
Error: Failed to show lines.

Hmm, could you run it with -vvv option?

Quoted 18 lines

[root@f33 yum.repos.d]#

whereas on my Fedora 32 installation this command shows [root@t35lp46 perf]# cat /etc/redhat-release Fedora release 32 (Thirty Two)
[root@t35lp46 perf]#

[root@t35lp46 perf]# ./perf probe -L getname_flags | tail -10

     68                         return ERR_PTR(-ENOENT);
                        }
                }

     72         result->uptr = filename;
     73         result->aname = NULL;
     74         audit_getname(result);
                return result;
}
Quoted 4 lines

[root@t35lp46 perf]#

Any ideas how to fix this? Is this failure related to a missing debuginfod setup, even when I have installed the debuginfo packages for the kernel:

If you can find the vmlinux in debuginfo package, could you try eu-addr2line to find getname_flags?

eu-addr2line -e /somewhere/to/vmlinux getname_flags+0

Thank you,

Quoted 20 lines

[root@f33 ~]# perf probe -L getname_flags
Specified source line is not found.
Error: Failed to show lines.
[root@f33 ~]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@f33 ~]# rpm -qa | fgrep kernel | fgrep debuginfo kernel-debuginfo-common-x86_64-5.9.8-200.fc33.x86_64 kernel-debuginfo-5.9.8-200.fc33.x86_64 kernel-tools-debuginfo-5.9.7-200.fc33.x86_64
[root@f33 ~]#
~

Thanks a lot

Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Signature · 1 line
Masami Hiramatsu <[email protected]>
#2 lore raw
Quoted 32 lines · Masami Hiramatsu

Hi Thomas,

Quoted 2 lines · On Wed, 18 Nov 2020 17:37:21 +0100 Thomas Richter <[email protected]> wrote:

I debugged the failure of some perf test cases on Fedora 33. The following test cases fail on s390 and x86_64 using

Thanks for the report!
Let me setup Fedora33.

Quoted 6 lines

[root@f33 yum.repos.d]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020

                                x86_64 x86_64 x86_64 GNU/Linux

[root@f33 yum.repos.d]#

Quoted 14 lines

[root@f33 yum.repos.d]# perf test -F 74 77 78
74: Use vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
77: Check open filename arg using perf trace + vfs_getname
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
78: Add vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
[root@f33 yum.repos.d]#

Quoted 2 lines

BTW, do we have vfs_getname in the kernel?

No, that is the name of the probe when calling perf probe command, as in .../perf/tests/shell/lib/probe_vfs_getname.sh:

perf probe -q "vfs_getname=getname_flags:...." The probe is set on kernel function getname_flags() in fs/namei.c source file and named vfs_getname inside perf/tracing subsystem.

Quoted 7 lines
Quoted 5 lines

The root cause is the output of perf probe -L which returns:
[root@f33 yum.repos.d]# perf probe -L getname_flags Specified source line is not found.
Error: Failed to show lines.

Hmm, could you run it with -vvv option?

[root@f33 ~]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@f33 ~]# cat /etc/fedora-release
Fedora release 33 (Thirty Three)
[root@f33 ~]# perf kallsyms getname_flags
getname_flags: [kernel] [kernel.kallsyms] 0xffffffffa8325000-0xffffffffa8325060 (0xffffffffa8325000-0xffffffffa8325060) [root@f33 ~]# find / -name vmlinux
/home/richter/linux/arch/x86/boot/compressed/vmlinux
/home/richter/linux/vmlinux
/usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux
/sys/kernel/btf/vmlinux
[root@f33 ~]# nm -g /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux| fgrep getname_flags ffffffff81325000 T getname_flags
[root@f33 ~]#

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.
Error: Failed to show lines. Reason: No such file or directory (Code: -2)
[root@f33 ~]#

Quoted 21 lines
Quoted 21 lines

[root@f33 yum.repos.d]#

whereas on my Fedora 32 installation this command shows [root@t35lp46 perf]# cat /etc/redhat-release Fedora release 32 (Thirty Two)
[root@t35lp46 perf]#

[root@t35lp46 perf]# ./perf probe -L getname_flags | tail -10

     68                         return ERR_PTR(-ENOENT);
                        }
                }

     72         result->uptr = filename;
     73         result->aname = NULL;
     74         audit_getname(result);
                return result;
}

[root@t35lp46 perf]#

Quoted 7 lines
Quoted 2 lines

Any ideas how to fix this? Is this failure related to a missing debuginfod setup, even when I have installed the debuginfo packages for the kernel:

If you can find the vmlinux in debuginfo package, could you try eu-addr2line to find getname_flags?

eu-addr2line -e /somewhere/to/vmlinux getname_flags+0

[root@f33 ~]# eu-addr2line -e /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux getname_flags+0 fs/namei.c:129:1
[root@f33 ~]#

Quoted 3 lines

Thank you,

You are welcome, thanks for looking into this.

I have added my Fedora32 output for completeness:

[root@t35lp46 perf]# ./perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.10.0-rc4d-perf+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/5.10.0-rc4d-perf+/build/vmlinux
fname: fs/namei.c, lineno:128
New line range: 128 to 2147483647
Get 20989 lines from this CU
path: fs/namei.c
Search fs/namei.c from debuginfod -> -38
Failed to find fs/namei.c in debuginfod (8a690c7c154db1283cebb8953128584dd4f94b3d) <getname_flags@/root/linux/fs/namei.c:0>

      0  getname_flags(const char __user *filename, int flags, int *empty)
         {
      2         struct filename *result;
                char *kname;
                int len;

......

root@t35lp46 perf]# eu-addr2line -e ~/linux/vmlinux getname_flags+0 fs/namei.c:129:1
[root@t35lp46 perf]#

Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
#3 lore raw

Hi,

Quoted 33 lines · On Thu, 19 Nov 2020 09:16:57 +0100 Thomas Richter <[email protected]> wrote:
Quoted 32 lines · Masami Hiramatsu

Hi Thomas,

Quoted 2 lines · On Wed, 18 Nov 2020 17:37:21 +0100 Thomas Richter <[email protected]> wrote:

I debugged the failure of some perf test cases on Fedora 33. The following test cases fail on s390 and x86_64 using

Thanks for the report!
Let me setup Fedora33.

Quoted 20 lines

[root@f33 yum.repos.d]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020

                                x86_64 x86_64 x86_64 GNU/Linux

[root@f33 yum.repos.d]#

[root@f33 yum.repos.d]# perf test -F 74 77 78
74: Use vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
77: Check open filename arg using perf trace + vfs_getname
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
78: Add vfs_getname probe to get syscall args filenames
:Failed to write event: Invalid argument
Error: Failed to add events.
FAILED!
[root@f33 yum.repos.d]#

Quoted 7 lines
Quoted 2 lines

BTW, do we have vfs_getname in the kernel?

No, that is the name of the probe when calling perf probe command, as in .../perf/tests/shell/lib/probe_vfs_getname.sh:

perf probe -q "vfs_getname=getname_flags:...."

Ah, OK.

Quoted 3 lines

The probe is set on kernel function getname_flags() in fs/namei.c source file and named vfs_getname inside perf/tracing subsystem.

Hmm, this seems debuginfo is not correctly handled.

Quoted 22 lines
Quoted 6 lines
Quoted 4 lines

The root cause is the output of perf probe -L which returns:
[root@f33 yum.repos.d]# perf probe -L getname_flags Specified source line is not found.
Error: Failed to show lines.

Hmm, could you run it with -vvv option?

[root@f33 ~]# uname -a
Linux f33 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@f33 ~]# cat /etc/fedora-release
Fedora release 33 (Thirty Three)
[root@f33 ~]# perf kallsyms getname_flags
getname_flags: [kernel] [kernel.kallsyms] 0xffffffffa8325000-0xffffffffa8325060 (0xffffffffa8325000-0xffffffffa8325060) [root@f33 ~]# find / -name vmlinux
/home/richter/linux/arch/x86/boot/compressed/vmlinux
/home/richter/linux/vmlinux
/usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux
/sys/kernel/btf/vmlinux
[root@f33 ~]# nm -g /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux| fgrep getname_flags ffffffff81325000 T getname_flags
[root@f33 ~]#

Quoted 8 lines

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.

The fname seems wrong, there is a declaration of getname_flags, but not the definition. Hmm, maybe the dwarf format is different?

Quoted 27 lines

Error: Failed to show lines. Reason: No such file or directory (Code: -2)
[root@f33 ~]#

Quoted 24 lines
Quoted 23 lines

[root@f33 yum.repos.d]#

whereas on my Fedora 32 installation this command shows [root@t35lp46 perf]# cat /etc/redhat-release Fedora release 32 (Thirty Two)
[root@t35lp46 perf]#

[root@t35lp46 perf]# ./perf probe -L getname_flags | tail -10

     68                         return ERR_PTR(-ENOENT);
                        }
                }

     72         result->uptr = filename;
     73         result->aname = NULL;
     74         audit_getname(result);
                return result;
}

[root@t35lp46 perf]#

Any ideas how to fix this? Is this failure related to a missing debuginfod setup, even when I have installed the debuginfo packages for the kernel:

Quoted 8 lines
Quoted 4 lines

If you can find the vmlinux in debuginfo package, could you try eu-addr2line to find getname_flags?

eu-addr2line -e /somewhere/to/vmlinux getname_flags+0

[root@f33 ~]# eu-addr2line -e /usr/lib/debug/usr/lib/modules/5.9.8-200.fc33.x86_64/vmlinux getname_flags+0 fs/namei.c:129:1
[root@f33 ~]#

OK, so elfutils can decode it.
I'll try to decode dwarf, trace elfutils and find what happens.

Thank you,

Quoted 38 lines
Quoted 4 lines

Thank you,

You are welcome, thanks for looking into this.

I have added my Fedora32 output for completeness:

[root@t35lp46 perf]# ./perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.10.0-rc4d-perf+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/5.10.0-rc4d-perf+/build/vmlinux
fname: fs/namei.c, lineno:128
New line range: 128 to 2147483647
Get 20989 lines from this CU
path: fs/namei.c
Search fs/namei.c from debuginfod -> -38
Failed to find fs/namei.c in debuginfod (8a690c7c154db1283cebb8953128584dd4f94b3d) <getname_flags@/root/linux/fs/namei.c:0>

      0  getname_flags(const char __user *filename, int flags, int *empty)
         {
      2         struct filename *result;
                char *kname;
                int len;

......

root@t35lp46 perf]# eu-addr2line -e ~/linux/vmlinux getname_flags+0 fs/namei.c:129:1
[root@t35lp46 perf]#

Signature · 5 lines
Thomas Richter, Dept 3252, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Signature · 1 line
Masami Hiramatsu <[email protected]>
#4 lore raw
In reply to Thomas Richter

Hi Thomas,

I've setup fedora33 and found strange debuginfo.

Quoted 9 lines · On Thu, 19 Nov 2020 09:16:57 +0100 Thomas Richter <[email protected]> wrote:

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.
Error: Failed to show lines. Reason: No such file or directory (Code: -2)

Above error was reproduced, and I found that the perf-probe's routine failed to find the real function definition.

I decoded the vmlinux with eu-readelf -w, and I found there were 2 entries (DIEs) for the getname_flags.

(1)
[2dbe7c6] subprogram abbrev: 54

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) fs.h (8)
             decl_line            (data2) 2527
             decl_column          (data1) 25
             prototyped           (flag_present) yes
             type                 (ref4) [2dbe7e7]
             sibling              (ref4) [2dbe7e7]
 [2dbe7d7]      formal_parameter     abbrev: 4
               type                 (ref4) [2da9721]

[2dbe7dc] formal_parameter abbrev: 4
type (ref4) [2da97a1]
[2dbe7e1] formal_parameter abbrev: 4
type (ref4) [2dabbdd]

(2)
[2e0fde0] subprogram abbrev: 162

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) namei.c (1)
             decl_line            (data1) 128
             decl_column          (data1) 1
             prototyped           (flag_present) yes
             type                 (ref4) [2dfced6]
             inline               (data1) inlined (1)
             sibling              (ref4) [2e0fe49]
 [2e0fdf2]      formal_parameter     abbrev: 31
               name                 (strp) "filename"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 34
               type                 (ref4) [2de77f3]
 [2e0fdfe]      formal_parameter     abbrev: 31
               name                 (strp) "flags"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 48
               type                 (ref4) [2de7873]
 [2e0fe0a]      formal_parameter     abbrev: 31
               name                 (strp) "empty"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 60
               type                 (ref4) [2de9ce4]

The first one is not the actual function, but that is a DIE for the declaration. However, this DIE doesn't have DW_AT_declaration attribute, it should be there if the DIE represents the declaration.

In perf, die_is_func_def() checks that attribute, and if the DIE doen't have DW_AT_declaration, line_range_search_cb() think the DIE represents the function definition (function prototype). And search the instance to find the line range.

However, since the (1) has no DW_AT_declaration, perf probe was mislead that the DIE is the function definition, and search the function instance which links to the DIE. And it failed to find any instances because the DIE it found was just a prototype in the header file. (But it seems some call-site DIE refers the declaration DIE.)

Let me try to fix if we skip the first DIE.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#5 lore raw

Em Thu, Nov 26, 2020 at 10:08:48PM +0900, Masami Hiramatsu escreveu:

Quoted 70 lines

Hi Thomas,

I've setup fedora33 and found strange debuginfo.

Quoted 9 lines · On Thu, 19 Nov 2020 09:16:57 +0100 Thomas Richter <[email protected]> wrote:

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.
Error: Failed to show lines. Reason: No such file or directory (Code: -2)

Above error was reproduced, and I found that the perf-probe's routine failed to find the real function definition.

I decoded the vmlinux with eu-readelf -w, and I found there were 2 entries (DIEs) for the getname_flags.

(1)
[2dbe7c6] subprogram abbrev: 54

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) fs.h (8)
             decl_line            (data2) 2527
             decl_column          (data1) 25
             prototyped           (flag_present) yes
             type                 (ref4) [2dbe7e7]
             sibling              (ref4) [2dbe7e7]
 [2dbe7d7]      formal_parameter     abbrev: 4
               type                 (ref4) [2da9721]

[2dbe7dc] formal_parameter abbrev: 4
type (ref4) [2da97a1]
[2dbe7e1] formal_parameter abbrev: 4
type (ref4) [2dabbdd]

(2)
[2e0fde0] subprogram abbrev: 162

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) namei.c (1)
             decl_line            (data1) 128
             decl_column          (data1) 1
             prototyped           (flag_present) yes
             type                 (ref4) [2dfced6]
             inline               (data1) inlined (1)
             sibling              (ref4) [2e0fe49]
 [2e0fdf2]      formal_parameter     abbrev: 31
               name                 (strp) "filename"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 34
               type                 (ref4) [2de77f3]
 [2e0fdfe]      formal_parameter     abbrev: 31
               name                 (strp) "flags"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 48
               type                 (ref4) [2de7873]
 [2e0fe0a]      formal_parameter     abbrev: 31
               name                 (strp) "empty"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 60
               type                 (ref4) [2de9ce4]
Quoted 3 lines

The first one is not the actual function, but that is a DIE for the declaration. However, this DIE doesn't have DW_AT_declaration attribute, it should be there if the DIE represents the declaration.

Its a bug in some gdb versions, see:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97060

And here is the pahole cset dealing with that:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=5a22c2de79fb9edf2d36ff4e8d440a862fc99c2a

Quoted 17 lines

In perf, die_is_func_def() checks that attribute, and if the DIE doen't have DW_AT_declaration, line_range_search_cb() think the DIE represents the function definition (function prototype). And search the instance to find the line range.

However, since the (1) has no DW_AT_declaration, perf probe was mislead that the DIE is the function definition, and search the function instance which links to the DIE. And it failed to find any instances because the DIE it found was just a prototype in the header file. (But it seems some call-site DIE refers the declaration DIE.)

Let me try to fix if we skip the first DIE.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
Signature · 1 line
- Arnaldo
#6 lore raw
Quoted 71 lines · On Thu, 26 Nov 2020 14:28:49 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Thu, Nov 26, 2020 at 10:08:48PM +0900, Masami Hiramatsu escreveu:

Quoted 70 lines

Hi Thomas,

I've setup fedora33 and found strange debuginfo.

Quoted 9 lines · On Thu, 19 Nov 2020 09:16:57 +0100 Thomas Richter <[email protected]> wrote:

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.
Error: Failed to show lines. Reason: No such file or directory (Code: -2)

Above error was reproduced, and I found that the perf-probe's routine failed to find the real function definition.

I decoded the vmlinux with eu-readelf -w, and I found there were 2 entries (DIEs) for the getname_flags.

(1)
[2dbe7c6] subprogram abbrev: 54

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) fs.h (8)
             decl_line            (data2) 2527
             decl_column          (data1) 25
             prototyped           (flag_present) yes
             type                 (ref4) [2dbe7e7]
             sibling              (ref4) [2dbe7e7]
 [2dbe7d7]      formal_parameter     abbrev: 4
               type                 (ref4) [2da9721]

[2dbe7dc] formal_parameter abbrev: 4
type (ref4) [2da97a1]
[2dbe7e1] formal_parameter abbrev: 4
type (ref4) [2dabbdd]

(2)
[2e0fde0] subprogram abbrev: 162

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) namei.c (1)
             decl_line            (data1) 128
             decl_column          (data1) 1
             prototyped           (flag_present) yes
             type                 (ref4) [2dfced6]
             inline               (data1) inlined (1)
             sibling              (ref4) [2e0fe49]
 [2e0fdf2]      formal_parameter     abbrev: 31
               name                 (strp) "filename"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 34
               type                 (ref4) [2de77f3]
 [2e0fdfe]      formal_parameter     abbrev: 31
               name                 (strp) "flags"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 48
               type                 (ref4) [2de7873]
 [2e0fe0a]      formal_parameter     abbrev: 31
               name                 (strp) "empty"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 60
               type                 (ref4) [2de9ce4]
Quoted 7 lines
Quoted 3 lines

The first one is not the actual function, but that is a DIE for the declaration. However, this DIE doesn't have DW_AT_declaration attribute, it should be there if the DIE represents the declaration.

Its a bug in some gdb versions, see:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97060

Thanks, so there is a compiler bug.

Quoted 4 lines

And here is the pahole cset dealing with that:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=5a22c2de79fb9edf2d36ff4e8d440a862fc99c2a

Oh, that's nice!

Anyway, I found how I can handle it for --lines option. Let me fix it first.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#7 lore raw

Em Fri, Nov 27, 2020 at 09:10:04AM +0900, Masami Hiramatsu escreveu:

Quoted 74 lines
Quoted 71 lines · On Thu, 26 Nov 2020 14:28:49 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Thu, Nov 26, 2020 at 10:08:48PM +0900, Masami Hiramatsu escreveu:

Quoted 70 lines

Hi Thomas,

I've setup fedora33 and found strange debuginfo.

Quoted 9 lines · On Thu, 19 Nov 2020 09:16:57 +0100 Thomas Richter <[email protected]> wrote:

[root@f33 ~]# perf probe -vvv -L getname_flags
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux for symbols Open Debuginfo file: /lib/modules/5.9.8-200.fc33.x86_64/build/vmlinux
fname: ./include/linux/fs.h, lineno:2527
New line range: 2527 to 2147483647
path: (null)
Specified source line is not found.
Error: Failed to show lines. Reason: No such file or directory (Code: -2)

Above error was reproduced, and I found that the perf-probe's routine failed to find the real function definition.

I decoded the vmlinux with eu-readelf -w, and I found there were 2 entries (DIEs) for the getname_flags.

(1)
[2dbe7c6] subprogram abbrev: 54

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) fs.h (8)
             decl_line            (data2) 2527
             decl_column          (data1) 25
             prototyped           (flag_present) yes
             type                 (ref4) [2dbe7e7]
             sibling              (ref4) [2dbe7e7]
 [2dbe7d7]      formal_parameter     abbrev: 4
               type                 (ref4) [2da9721]

[2dbe7dc] formal_parameter abbrev: 4
type (ref4) [2da97a1]
[2dbe7e1] formal_parameter abbrev: 4
type (ref4) [2dabbdd]

(2)
[2e0fde0] subprogram abbrev: 162

             external             (flag_present) yes
             name                 (strp) "getname_flags"
             decl_file            (data1) namei.c (1)
             decl_line            (data1) 128
             decl_column          (data1) 1
             prototyped           (flag_present) yes
             type                 (ref4) [2dfced6]
             inline               (data1) inlined (1)
             sibling              (ref4) [2e0fe49]
 [2e0fdf2]      formal_parameter     abbrev: 31
               name                 (strp) "filename"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 34
               type                 (ref4) [2de77f3]
 [2e0fdfe]      formal_parameter     abbrev: 31
               name                 (strp) "flags"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 48
               type                 (ref4) [2de7873]
 [2e0fe0a]      formal_parameter     abbrev: 31
               name                 (strp) "empty"
               decl_file            (data1) namei.c (1)
               decl_line            (data1) 128
               decl_column          (data1) 60
               type                 (ref4) [2de9ce4]
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The first one is not the actual function, but that is a DIE for the declaration. However, this DIE doesn't have DW_AT_declaration attribute, it should be there if the DIE represents the declaration.

Its a bug in some gdb versions, see:

ARgh, s/gdb/gcc/ :-)

Quoted 19 lines
Quoted 1 line

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97060

Thanks, so there is a compiler bug.

Quoted 4 lines

And here is the pahole cset dealing with that:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=5a22c2de79fb9edf2d36ff4e8d440a862fc99c2a

Oh, that's nice!

Anyway, I found how I can handle it for --lines option. Let me fix it first.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
Signature · 1 line
- Arnaldo
#8 lore raw

PATCH 1/2 perf-probe: Fix to die_entrypc() returns error correctly

In reply to Arnaldo Carvalho de Melo

Fix die_entrypc() to return error correctly if the DIE has no DW_AT_ranges attribute. Since dwarf_ranges() will treat the case as an empty ranges and return 0, we have to check it by ourselves.

Fixes 91e2f539eeda ("perf probe: Fix to show function entry line as probe-able")
---
 tools/perf/util/dwarf-aux.c |    8 ++++++++
 1 file changed, 8 insertions(+)
tools/perf/util/dwarf-aux.c +8 -0
index aa898014ad12..03c1a39c312a 100644
@@ -373,6 +373,7 @@ bool die_is_func_def(Dwarf_Die *dw_die)
 int die_entrypc(Dwarf_Die *dw_die, Dwarf_Addr *addr)
 {
 	Dwarf_Addr base, end;
+	Dwarf_Attribute attr;
 
 	if (!addr)
 		return -EINVAL;
@@ -380,6 +381,13 @@ int die_entrypc(Dwarf_Die *dw_die, Dwarf_Addr *addr)
 	if (dwarf_entrypc(dw_die, addr) == 0)
 		return 0;
 
+	/*
+	 *  Since the dwarf_ranges() will return 0 if there is no
+	 * DW_AT_ranges attribute, we should check it first.
+	 */
+	if (!dwarf_attr(dw_die, DW_AT_ranges, &attr))
+		return -ENOENT;
+
 	return dwarf_ranges(dw_die, 0, &base, addr, &end) < 0 ? -ENOENT : 0;
 }
 
#9 lore raw

PATCH 2/2 perf-probe: Change function definition check due to broken dwarf

In reply to Arnaldo Carvalho de Melo

Since some gcc generates a broken DWARF which lacks DW_AT_declaration attribute from the subprogram DIE of function prototype.
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97060)

So, in addition to the DW_AT_declaration check, we also check the subprogram DIE has DW_AT_inline or actual entry pc.

---
 tools/perf/util/dwarf-aux.c    |   20 ++++++++++++++++++--
 tools/perf/util/probe-finder.c |    3 +--
 2 files changed, 19 insertions(+), 4 deletions(-)
tools/perf/util/dwarf-aux.c +18 -2
index 03c1a39c312a..7b2d471a6419 100644
@@ -356,9 +356,25 @@ bool die_is_signed_type(Dwarf_Die *tp_die)
 bool die_is_func_def(Dwarf_Die *dw_die)
 {
 	Dwarf_Attribute attr;
+	Dwarf_Addr addr = 0;
+
+	if (dwarf_tag(dw_die) != DW_TAG_subprogram)
+		return false;
+
+	if (dwarf_attr(dw_die, DW_AT_declaration, &attr))
+		return false;
 
-	return (dwarf_tag(dw_die) == DW_TAG_subprogram &&
-		dwarf_attr(dw_die, DW_AT_declaration, &attr) == NULL);
+	/*
+	 * DW_AT_declaration can be lost from function declaration
+	 * by gcc's bug #97060.
+	 * So we need to check this subprogram DIE has DW_AT_inline
+	 * or an entry address.
+	 */
+	if (!dwarf_attr(dw_die, DW_AT_inline, &attr) &&
+	    die_entrypc(dw_die, &addr) < 0)
+		return false;
+
+	return true;
 }
 
 /**
tools/perf/util/probe-finder.c +1 -2
index 2c4061035f77..76dd349aa48d 100644
@@ -1885,8 +1885,7 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)
 	if (lr->file && strtailcmp(lr->file, dwarf_decl_file(sp_die)))
 		return DWARF_CB_OK;
 
-	if (die_is_func_def(sp_die) &&
-	    die_match_name(sp_die, lr->function)) {
+	if (die_match_name(sp_die, lr->function) && die_is_func_def(sp_die)) {
 		lf->fname = dwarf_decl_file(sp_die);
 		dwarf_decl_line(sp_die, &lr->offset);
 		pr_debug("fname: %s, lineno:%d\n", lf->fname, lr->offset);
#10 lore raw

Re: [PATCH 2/2] perf-probe: Change function definition check due to broken dwarf

Em Fri, Nov 27, 2020 at 02:48:55PM +0900, Masami Hiramatsu escreveu:

Quoted 6 lines

Since some gcc generates a broken DWARF which lacks DW_AT_declaration attribute from the subprogram DIE of function prototype.
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97060)

So, in addition to the DW_AT_declaration check, we also check the subprogram DIE has DW_AT_inline or actual entry pc.

Thanks, applied and tested both patches on a fedora 33 system where previously those tests were failing:

Committer testing:

# cat /etc/fedora-release
Fedora release 33 (Thirty Three)
#

Before:

# perf test vfs_getname
78: Use vfs_getname probe to get syscall args filenames : FAILED!
79: Check open filename arg using perf trace + vfs_getname : FAILED!
81: Add vfs_getname probe to get syscall args filenames : FAILED!
#

After:

# perf test vfs_getname
78: Use vfs_getname probe to get syscall args filenames : Ok
79: Check open filename arg using perf trace + vfs_getname : Ok
81: Add vfs_getname probe to get syscall args filenames : Ok
#

- Arnaldo

Quoted 54 lines
---
 tools/perf/util/dwarf-aux.c    |   20 ++++++++++++++++++--
 tools/perf/util/probe-finder.c |    3 +--
 2 files changed, 19 insertions(+), 4 deletions(-)
tools/perf/util/dwarf-aux.c +18 -2
index 03c1a39c312a..7b2d471a6419 100644
@@ -356,9 +356,25 @@ bool die_is_signed_type(Dwarf_Die *tp_die)
 bool die_is_func_def(Dwarf_Die *dw_die)
 {
 	Dwarf_Attribute attr;
+	Dwarf_Addr addr = 0;
+
+	if (dwarf_tag(dw_die) != DW_TAG_subprogram)
+		return false;
+
+	if (dwarf_attr(dw_die, DW_AT_declaration, &attr))
+		return false;
 
-	return (dwarf_tag(dw_die) == DW_TAG_subprogram &&
-		dwarf_attr(dw_die, DW_AT_declaration, &attr) == NULL);
+	/*
+	 * DW_AT_declaration can be lost from function declaration
+	 * by gcc's bug #97060.
+	 * So we need to check this subprogram DIE has DW_AT_inline
+	 * or an entry address.
+	 */
+	if (!dwarf_attr(dw_die, DW_AT_inline, &attr) &&
+	    die_entrypc(dw_die, &addr) < 0)
+		return false;
+
+	return true;
 }
 
 /**
tools/perf/util/probe-finder.c +1 -2
index 2c4061035f77..76dd349aa48d 100644
@@ -1885,8 +1885,7 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)
 	if (lr->file && strtailcmp(lr->file, dwarf_decl_file(sp_die)))
 		return DWARF_CB_OK;
 
-	if (die_is_func_def(sp_die) &&
-	    die_match_name(sp_die, lr->function)) {
+	if (die_match_name(sp_die, lr->function) && die_is_func_def(sp_die)) {
 		lf->fname = dwarf_decl_file(sp_die);
 		dwarf_decl_line(sp_die, &lr->offset);
 		pr_debug("fname: %s, lineno:%d\n", lf->fname, lr->offset);
Signature · 1 line
- Arnaldo