Skip to contentSkip to conversation

perf probe segfaulting when asked for variable it doesn't find

BUG In review Arnaldo Carvalho de Melo 6 participants 19 messages May 28, 2014 – Jun 5, 2014 perf events

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer
#0 lore raw

Masami-san,

While trying:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Failed to find the location of result at this address.
Perhaps, it has been optimized out.
Failed to find 'result' in this function.
Error: Failed to add events. (-2)
[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Added new event:
Segmentation fault (core dumped)

I got segfaulted while in the past I would get the much nicer:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Failed to find the location of result at this address.
Perhaps, it has been optimized out.
Failed to find 'result' in this function.
Error: Failed to add events. (-2)
[root@zoo ~]#

The "Error:" part can and should be trimmed, just the two middle ones should be enough, but I'm digressing, I bisected it down to the cset at the bottom of this message.

In the past there was a 'result' variable at getname_flags, but now 'probe' isn't finding any, as it seems to have been optimized away, still haven't checked thoroughly:

[root@zoo ~]# perf probe -V getname_flags
Available variables at getname_flags

        @<getname_flags+0>
                char*   filename
                int     flags
                int*    empty
        @<getname+18>
                (No matched variables)
        @<user_path_create+37>
                (No matched variables)
        @<user_path_parent+37>
                (No matched variables)
        @<user_path_at_empty+50>
                (No matched variables)
        @<user_path_mountpoint_at+37>
                (No matched variables)
        @<sys_symlinkat+38>
                (No matched variables)
        @<sys_symlink+34>
                (No matched variables)

[root@zoo ~]

3d918a12a1b3088ac16ff37fa52760639d6e2403 is the first bad commit commit 3d918a12a1b3088ac16ff37fa52760639d6e2403

Author: Masami Hiramatsu <[email protected]>
Date:   Fri Oct 11 16:10:26 2013 +0900

perf probe: Find fentry mcount fuzzed parameter location

At this point, --fentry (mcount function entry) option for gcc fuzzes the debuginfo variable locations by skipping the mcount instruction offset (on x86, this is a 5 byte call instruction).

This makes variable searching fail at the entry of functions which are mcount'ed.

e.g.)
Available variables at vfs_read
@<vfs_read+0>
(No matched variables)

This patch adds additional location search at the function entry point to solve this issue, which tries to find the earliest address for the variable location.

Note that this only works with function parameters (formal parameters) because any local variables should not exist on the function entry address (those are not initialized yet).

With this patch, perf probe shows correct parameters if possible;
# perf probe --vars vfs_read
Available variables at vfs_read
@<vfs_read+0>
char* buf
loff_t* pos
size_t count
struct file* file

:040000 040000 4c416906285aa1488ed2badbaf3b3feee86f9578 978c6b47f442845e6e93a79a24aeb36bbca0b7da M tools
[acme@zoo linux]$

#1 lore raw
Quoted 13 lines · Arnaldo Carvalho de Melo

Masami-san,

While trying:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Failed to find the location of result at this address.
Perhaps, it has been optimized out.
Failed to find 'result' in this function.
Error: Failed to add events. (-2)
[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Added new event:
Segmentation fault (core dumped)

Quoted 8 lines

I got segfaulted while in the past I would get the much nicer:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Failed to find the location of result at this address.
Perhaps, it has been optimized out.
Failed to find 'result' in this function.
Error: Failed to add events. (-2)
[root@zoo ~]#

Oops, I got that. Bad SEGV was reproduced here too. And I've found an error-handling miss in the convert_variable(), I'll send a patch for fixing it.
Thank you very much for bisecting and finding my fault.

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#2 lore raw

PATCH 1/1 perf/probe: Fix a segfault if asked for variable it doesn't find

In reply to Arnaldo Carvalho de Melo

Fix a segfault bug by asking for variable it doesn't find. Since the convert_variable() didn't handle error code returned from convert_variable_location(), it just passed an incomplete variable field and then a segfault was occured when formatting the field.

This fixes that bug by handling success code correctly in convert_variable(). Other callers of convert_variable_location() are correctly checking the return code.

This bug was introduced by following commit. But another hidden erroneous error handling has been there previuosly (-ENOMEM case).

commit 3d918a12a1b3088ac16ff37fa52760639d6e2403

Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/probe-finder.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/probe-finder.c +2 -2
index 5627621..9d8eb26 100644
@@ -511,12 +511,12 @@ static int convert_variable(Dwarf_Die *vr_die, struct probe_finder *pf)
 
 	ret = convert_variable_location(vr_die, pf->addr, pf->fb_ops,
 					&pf->sp_die, pf->tvar);
-	if (ret == -ENOENT)
+	if (ret == -ENOENT || ret == -EINVAL)
 		pr_err("Failed to find the location of %s at this address.\n"
 		       " Perhaps, it has been optimized out.\n", pf->pvar->var);
 	else if (ret == -ENOTSUP)
 		pr_err("Sorry, we don't support this variable location yet.\n");
-	else if (pf->pvar->field) {
+	else if (ret == 0 && pf->pvar->field) {
 		ret = convert_variable_fields(vr_die, pf->pvar->var,
 					      pf->pvar->field, &pf->tvar->ref,
 					      &die_mem);
#3 lore raw

Re: [PATCH -tip/urgent ] [BUGFIX] perf/probe: Fix a segfault if asked for variable it doesn't find

Arnaldo, this fixes the SEGV bug which you reported.

But I've found that perf probe sometimes loses the location of variables with recent DWARF implementation. I need to check and fix that too.

Anyway, this patch should be applied for fixing critical bug.

Thank you,

Quoted 49 lines · Masami Hiramatsu

Fix a segfault bug by asking for variable it doesn't find. Since the convert_variable() didn't handle error code returned from convert_variable_location(), it just passed an incomplete variable field and then a segfault was occured when formatting the field.

This fixes that bug by handling success code correctly in convert_variable(). Other callers of convert_variable_location() are correctly checking the return code.

This bug was introduced by following commit. But another hidden erroneous error handling has been there previuosly (-ENOMEM case).

commit 3d918a12a1b3088ac16ff37fa52760639d6e2403

Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/probe-finder.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/probe-finder.c +2 -2
index 5627621..9d8eb26 100644
@@ -511,12 +511,12 @@ static int convert_variable(Dwarf_Die *vr_die, struct probe_finder *pf)
 
 	ret = convert_variable_location(vr_die, pf->addr, pf->fb_ops,
 					&pf->sp_die, pf->tvar);
-	if (ret == -ENOENT)
+	if (ret == -ENOENT || ret == -EINVAL)
 		pr_err("Failed to find the location of %s at this address.\n"
 		       " Perhaps, it has been optimized out.\n", pf->pvar->var);
 	else if (ret == -ENOTSUP)
 		pr_err("Sorry, we don't support this variable location yet.\n");
-	else if (pf->pvar->field) {
+	else if (ret == 0 && pf->pvar->field) {
 		ret = convert_variable_fields(vr_die, pf->pvar->var,
 					      pf->pvar->field, &pf->tvar->ref,
 					      &die_mem);
Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#4 lore raw

PATCH 1/1 perf/probe: Fix perf probe to find correct variable DIE

In reply to Masami Hiramatsu

Fix perf probe to find correct variable DIE which has location or external instance by tracking down the lexical blocks.

Current die_find_variable() expects that the all variable DIEs which has DW_TAG_variable have a location. However, since recent dwarf information may have declaration variable DIEs at the entry of function (subprogram), die_find_variable() returns it.

To solve this problem, it must track down the DIE tree to find a DIE which has an actual location or a reference for external instance.

e.g. finding a DIE which origin is <0xdc73>;

<1><11496>: Abbrev Number: 95 (DW_TAG_subprogram)

    <11497>   DW_AT_abstract_origin: <0xdc42>
    <1149b>   DW_AT_low_pc      : 0x1850
[...]
 <2><114cc>: Abbrev Number: 119 (DW_TAG_variable) <- this is a declaration
    <114cd>   DW_AT_abstract_origin: <0xdc73>
 <2><114d1>: Abbrev Number: 119 (DW_TAG_variable)
[...]
 <3><115a7>: Abbrev Number: 105 (DW_TAG_lexical_block)
    <115a8>   DW_AT_ranges      : 0xaa0
 <4><115ac>: Abbrev Number: 96 (DW_TAG_variable) <- this has a location
    <115ad>   DW_AT_abstract_origin: <0xdc73>
    <115b1>   DW_AT_location    : 0x486c        (location list)
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/dwarf-aux.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/util/dwarf-aux.c +5 -2
index 7defd77..cc66c40 100644
@@ -747,14 +747,17 @@ struct __find_variable_param {
 static int __die_find_variable_cb(Dwarf_Die *die_mem, void *data)
 {
 	struct __find_variable_param *fvp = data;
+	Dwarf_Attribute attr;
 	int tag;
 
 	tag = dwarf_tag(die_mem);
 	if ((tag == DW_TAG_formal_parameter ||
 	     tag == DW_TAG_variable) &&
-	    die_compare_name(die_mem, fvp->name))
+	    die_compare_name(die_mem, fvp->name) &&
+	/* Does the DIE have location information or external instance? */
+	    (dwarf_attr(die_mem, DW_AT_external, &attr) ||
+	     dwarf_attr(die_mem, DW_AT_location, &attr)))
 		return DIE_FIND_CB_END;
-
 	if (dwarf_haspc(die_mem, fvp->addr))
 		return DIE_FIND_CB_CONTINUE;
 	else
#5 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Hi Arnaldo,

Here is the patch which fixes perf probe to find variable location correctly, on the recent dwarf format. This is not related to the SEGV issue which I fixed in previous mail.

Thank you,

Quoted 67 lines · Masami Hiramatsu

Fix perf probe to find correct variable DIE which has location or external instance by tracking down the lexical blocks.

Current die_find_variable() expects that the all variable DIEs which has DW_TAG_variable have a location. However, since recent dwarf information may have declaration variable DIEs at the entry of function (subprogram), die_find_variable() returns it.

To solve this problem, it must track down the DIE tree to find a DIE which has an actual location or a reference for external instance.

e.g. finding a DIE which origin is <0xdc73>;

<1><11496>: Abbrev Number: 95 (DW_TAG_subprogram)

    <11497>   DW_AT_abstract_origin: <0xdc42>
    <1149b>   DW_AT_low_pc      : 0x1850
[...]
 <2><114cc>: Abbrev Number: 119 (DW_TAG_variable) <- this is a declaration
    <114cd>   DW_AT_abstract_origin: <0xdc73>
 <2><114d1>: Abbrev Number: 119 (DW_TAG_variable)
[...]
 <3><115a7>: Abbrev Number: 105 (DW_TAG_lexical_block)
    <115a8>   DW_AT_ranges      : 0xaa0
 <4><115ac>: Abbrev Number: 96 (DW_TAG_variable) <- this has a location
    <115ad>   DW_AT_abstract_origin: <0xdc73>
    <115b1>   DW_AT_location    : 0x486c        (location list)
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/dwarf-aux.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/util/dwarf-aux.c +5 -2
index 7defd77..cc66c40 100644
@@ -747,14 +747,17 @@ struct __find_variable_param {
 static int __die_find_variable_cb(Dwarf_Die *die_mem, void *data)
 {
 	struct __find_variable_param *fvp = data;
+	Dwarf_Attribute attr;
 	int tag;
 
 	tag = dwarf_tag(die_mem);
 	if ((tag == DW_TAG_formal_parameter ||
 	     tag == DW_TAG_variable) &&
-	    die_compare_name(die_mem, fvp->name))
+	    die_compare_name(die_mem, fvp->name) &&
+	/* Does the DIE have location information or external instance? */
+	    (dwarf_attr(die_mem, DW_AT_external, &attr) ||
+	     dwarf_attr(die_mem, DW_AT_location, &attr)))
 		return DIE_FIND_CB_END;
-
 	if (dwarf_haspc(die_mem, fvp->addr))
 		return DIE_FIND_CB_CONTINUE;
 	else
Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#6 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Quoted 7 lines · Masami Hiramatsu

Hi Arnaldo,

Here is the patch which fixes perf probe to find variable location correctly, on the recent dwarf format. This is not related to the SEGV issue which I fixed in previous mail.

Thank you,

hi,
I took both patches to my perf/core.. will check
if Ingo wants the segfault fix via perf/urgent

thanks,
jirka

#7 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Quoted 7 lines · Jiri Olsa
Quoted 6 lines · Masami Hiramatsu

Hi Arnaldo,

Here is the patch which fixes perf probe to find variable location correctly, on the recent dwarf format. This is not related to the SEGV issue which I fixed in previous mail.

Quoted 5 lines
Quoted 1 line

Thank you,

hi,
I took both patches to my perf/core.. will check
if Ingo wants the segfault fix via perf/urgent

Yeah, segfaults (or any other random/uncontrolled user space behavior) we generally want to fix via perf/urgent. (Please also put a Cc:
[email protected] tag into it, in case it misses v3.15.)

Thanks,

Ingo

author maintainer
#8 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

In reply to Masami Hiramatsu

Em Thu, May 29, 2014 at 09:39:14PM +0900, Masami Hiramatsu escreveu:

Quoted 5 lines

Hi Arnaldo,

Here is the patch which fixes perf probe to find variable location correctly, on the recent dwarf format. This is not related to the SEGV issue which I fixed in previous mail.

Thanks for the prompt reply! I'm testing them and will report results here!

- Arnaldo

author maintainer
#9 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

In reply to Masami Hiramatsu

Em Thu, May 29, 2014 at 09:39:14PM +0900, Masami Hiramatsu escreveu:

Quoted 5 lines

Hi Arnaldo,

Here is the patch which fixes perf probe to find variable location correctly, on the recent dwarf format. This is not related to the SEGV issue which I fixed in previous mail.

Hey, if I apply just this one I got my problem solved, i.e. no more segfaults and also I managed to add that problem, the variable is there, all works, why do I need the first one?

Without your two patches:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Added new event:
Segmentation fault (core dumped)
[root@zoo ~]#

With this ([BUGFIX] perf/probe: Fix perf probe to find correct variable DIE):

Trying to reference a bogus variable, one that is not available at this function:

Removed event: probe:vfs_getname
[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=BOGUS->name:string' Failed to find 'BOGUS' in this function.
Error: Failed to add events. (-2)
[root@zoo ~]#

Perfect!

Now trying to reference a bogus member name:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->BOGUS:string' result(type:filename) has no member BOGUS.
Failed to find 'result' in this function.
Error: Failed to add events. (-22)
[root@zoo ~]#

No segfault, albeit it produces a bogus error message, as it clearly _finds_ the 'result' variable, its just that it is of a struct type that _has_ no such 'BOGUS' _member_.

Also I suggest removing that last "Error:" line, what is its value for users or developers?

Now if I do it all correctly:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Added new event:
probe:vfs_getname (on getname_flags:65 with pathname=result->name:string)

You can now use it in all perf tools, such as:

perf record -e probe:vfs_getname -aR sleep 1

[root@zoo ~]#

Ok, it works again!

And if I try to use it:

[root@zoo ~]# perf record -e probe:vfs_getname -a sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.562 MB perf.data (~24570 samples) ] [root@zoo ~]# perf script | head -5

            perf   716 [000] 17842.271777: probe:vfs_getname: (ffffffff811c2e43) pathname="/home/acme/libexec/perf-core/sleep"
            perf   716 [000] 17842.271794: probe:vfs_getname: (ffffffff811c2e43) pathname="/usr/lib64/ccache/sleep"
            perf   716 [000] 17842.271800: probe:vfs_getname: (ffffffff811c2e43) pathname="/usr/local/sbin/sleep"
            perf   716 [000] 17842.271804: probe:vfs_getname: (ffffffff811c2e43) pathname="/usr/local/bin/sleep"
            perf   716 [000] 17842.271808: probe:vfs_getname: (ffffffff811c2e43) pathname="/sbin/sleep"

[root@zoo ~]#

Works as expected.

So no segfault anywhere, while if I apply just that one that fixes the segfault I get:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Failed to find the location of result at this address.
Perhaps, it has been optimized out.
Failed to find 'result' in this function.
Error: Failed to add events. (-22)
[root@zoo ~]#

It doesn't work, which is a regression, as applying just the first one allows us to retrieve that variable.

[root@zoo ~]# perf probe -V getname_flags:65
Available variables at getname_flags:65
@<getname_flags+227>
char* filename
int len
int* empty
long int max
struct filename* result
[root@zoo ~]#

I.e. applying just the one that fixes just the segfault:

perf probe: Fix a segfault if asked for variable it doesn't find

"Fixes" the segfault but introduces a regression.

Can we apply just this one:

perf probe: Fix to find correct variable DIE

Because even when we specify a "variable that it doesn't find", it works well.

Regards,

- Arnaldo

Quoted 78 lines

Thank you,

Quoted 67 lines · Masami Hiramatsu

Fix perf probe to find correct variable DIE which has location or external instance by tracking down the lexical blocks.

Current die_find_variable() expects that the all variable DIEs which has DW_TAG_variable have a location. However, since recent dwarf information may have declaration variable DIEs at the entry of function (subprogram), die_find_variable() returns it.

To solve this problem, it must track down the DIE tree to find a DIE which has an actual location or a reference for external instance.

e.g. finding a DIE which origin is <0xdc73>;

<1><11496>: Abbrev Number: 95 (DW_TAG_subprogram)

    <11497>   DW_AT_abstract_origin: <0xdc42>
    <1149b>   DW_AT_low_pc      : 0x1850
[...]
 <2><114cc>: Abbrev Number: 119 (DW_TAG_variable) <- this is a declaration
    <114cd>   DW_AT_abstract_origin: <0xdc73>
 <2><114d1>: Abbrev Number: 119 (DW_TAG_variable)
[...]
 <3><115a7>: Abbrev Number: 105 (DW_TAG_lexical_block)
    <115a8>   DW_AT_ranges      : 0xaa0
 <4><115ac>: Abbrev Number: 96 (DW_TAG_variable) <- this has a location
    <115ad>   DW_AT_abstract_origin: <0xdc73>
    <115b1>   DW_AT_location    : 0x486c        (location list)
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/dwarf-aux.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/util/dwarf-aux.c +5 -2
index 7defd77..cc66c40 100644
@@ -747,14 +747,17 @@ struct __find_variable_param {
 static int __die_find_variable_cb(Dwarf_Die *die_mem, void *data)
 {
 	struct __find_variable_param *fvp = data;
+	Dwarf_Attribute attr;
 	int tag;
 
 	tag = dwarf_tag(die_mem);
 	if ((tag == DW_TAG_formal_parameter ||
 	     tag == DW_TAG_variable) &&
-	    die_compare_name(die_mem, fvp->name))
+	    die_compare_name(die_mem, fvp->name) &&
+	/* Does the DIE have location information or external instance? */
+	    (dwarf_attr(die_mem, DW_AT_external, &attr) ||
+	     dwarf_attr(die_mem, DW_AT_location, &attr)))
 		return DIE_FIND_CB_END;
-
 	if (dwarf_haspc(die_mem, fvp->addr))
 		return DIE_FIND_CB_CONTINUE;
 	else
Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#10 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Quoted 11 lines · Arnaldo Carvalho de Melo

Now trying to reference a bogus member name:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->BOGUS:string' result(type:filename) has no member BOGUS.
Failed to find 'result' in this function.
Error: Failed to add events. (-22)
[root@zoo ~]#

No segfault, albeit it produces a bogus error message, as it clearly _finds_ the 'result' variable, its just that it is of a struct type that _has_ no such 'BOGUS' _member_.

Right, I'll fix that.

Quoted 3 lines

Also I suggest removing that last "Error:" line, what is its value for users or developers?

It tells them the perf probe finally failed because of an error (with error code). Indeed that message is bluntness. I'll update it to use strerr instead of giving mysterious error code.
FYI, if it gets some general errors, like -ENOMEM, will just indicate the last error message.

Thank you!

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#11 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Quoted 9 lines · Masami Hiramatsu
Quoted 8 lines · Arnaldo Carvalho de Melo

Now trying to reference a bogus member name:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->BOGUS:string' result(type:filename) has no member BOGUS.
Failed to find 'result' in this function.
Error: Failed to add events. (-22)
[root@zoo ~]#

Quoted 5 lines
Quoted 3 lines

No segfault, albeit it produces a bogus error message, as it clearly _finds_ the 'result' variable, its just that it is of a struct type that _has_ no such 'BOGUS' _member_.

Right, I'll fix that.

should I wait for new patchset, or should I take those 2?

thanks,
jirka

#12 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Quoted 10 lines · Jiri Olsa
Quoted 9 lines · Masami Hiramatsu
Quoted 8 lines · Arnaldo Carvalho de Melo

Now trying to reference a bogus member name:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->BOGUS:string' result(type:filename) has no member BOGUS.
Failed to find 'result' in this function.
Error: Failed to add events. (-22)
[root@zoo ~]#

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

No segfault, albeit it produces a bogus error message, as it clearly _finds_ the 'result' variable, its just that it is of a struct type that _has_ no such 'BOGUS' _member_.

Right, I'll fix that.

should I wait for new patchset, or should I take those 2?

No, those 2 patches are urgently required, because one fixes SEGV critical error, and the other fixes a wrong behavior (catch up to the latest dwarf), which is also degradation.
I think changing error messages is a kind of usability enhancement.

Thank you,

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#13 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

Quoted 15 lines · Masami Hiramatsu
Quoted 14 lines · Jiri Olsa
Quoted 13 lines · Masami Hiramatsu
Quoted 11 lines · Arnaldo Carvalho de Melo

Now trying to reference a bogus member name:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->BOGUS:string' result(type:filename) has no member BOGUS.
Failed to find 'result' in this function.
Error: Failed to add events. (-22)
[root@zoo ~]#

No segfault, albeit it produces a bogus error message, as it clearly _finds_ the 'result' variable, its just that it is of a struct type that _has_ no such 'BOGUS' _member_.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Right, I'll fix that.

should I wait for new patchset, or should I take those 2?

No, those 2 patches are urgently required, because one fixes SEGV critical error, and the other fixes a wrong behavior (catch up to the latest dwarf), which is also degradation.

ok, I'll queue both of them to perf/urgent

thanks,
jirka

author maintainer Acked-by Tested-by on v1 1/1
#14 lore raw

Re: [PATCH -tip ] [BUGFIX] perf/probe: Fix perf probe to find correct variable DIE

In reply to Masami Hiramatsu

Em Thu, May 29, 2014 at 09:19:30PM +0900, Masami Hiramatsu escreveu:

Quoted 11 lines

Fix perf probe to find correct variable DIE which has location or external instance by tracking down the lexical blocks.

Current die_find_variable() expects that the all variable DIEs which has DW_TAG_variable have a location. However, since recent dwarf information may have declaration variable DIEs at the entry of function (subprogram), die_find_variable() returns it.

To solve this problem, it must track down the DIE tree to find a DIE which has an actual location or a reference for external instance.

Before:

[root@zoo ~]# perf probe --del probe:vfs_getname
Removed event: probe:vfs_getname
[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Added new event:
Segmentation fault (core dumped)

After:

[root@zoo ~]# perf probe 'vfs_getname=getname_flags:65 pathname=result->name:string' Added new event:
probe:vfs_getname (on getname_flags:65 with pathname=result->name:string)

You can now use it in all perf tools, such as:

perf record -e probe:vfs_getname -aR sleep 1

[root@zoo ~]# perf record -e probe:vfs_getname -a sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.550 MB perf.data (~24022 samples) ] [root@zoo ~]# perf script | head -5

            perf 27758 [001] 12022.551584: probe:vfs_getname: (ffffffff811c2e43) pathname="/home/acme/libexec/perf-core/sleep"
            perf 27758 [001] 12022.551616: probe:vfs_getname: (ffffffff811c2e43) pathname="/usr/lib64/ccache/sleep"
            perf 27758 [001] 12022.551628: probe:vfs_getname: (ffffffff811c2e43) pathname="/usr/local/sbin/sleep"
            perf 27758 [001] 12022.551636: probe:vfs_getname: (ffffffff811c2e43) pathname="/usr/local/bin/sleep"
            perf 27758 [001] 12022.551644: probe:vfs_getname: (ffffffff811c2e43) pathname="/sbin/sleep"

[root@zoo ~]#

- Arnaldo

#15 lore raw

[tip:perf/urgent] perf probe: Fix perf probe to find correct variable DIE

In reply to Masami Hiramatsu
perf probe: Fix perf probe to find correct variable DIE
Commit-ID:  082f96a93eb5ba9bf771518a0dda590624568e8e
Gitweb:     http://git.kernel.org/tip/082f96a93eb5ba9bf771518a0dda590624568e8e
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Thu, 29 May 2014 21:19:30 +0900
Committer:  Jiri Olsa <[email protected]>
CommitDate: Wed, 4 Jun 2014 14:49:20 +0200

perf probe: Fix perf probe to find correct variable DIE

Fix perf probe to find correct variable DIE which has location or external instance by tracking down the lexical blocks.

Current die_find_variable() expects that the all variable DIEs which has DW_TAG_variable have a location. However, since recent dwarf information may have declaration variable DIEs at the entry of function (subprogram), die_find_variable() returns it.

To solve this problem, it must track down the DIE tree to find a DIE which has an actual location or a reference for external instance.

e.g. finding a DIE which origin is <0xdc73>;

<1><11496>: Abbrev Number: 95 (DW_TAG_subprogram)

    <11497>   DW_AT_abstract_origin: <0xdc42>
    <1149b>   DW_AT_low_pc      : 0x1850
[...]
 <2><114cc>: Abbrev Number: 119 (DW_TAG_variable) <- this is a declaration
    <114cd>   DW_AT_abstract_origin: <0xdc73>
 <2><114d1>: Abbrev Number: 119 (DW_TAG_variable)
[...]
 <3><115a7>: Abbrev Number: 105 (DW_TAG_lexical_block)
    <115a8>   DW_AT_ranges      : 0xaa0
 <4><115ac>: Abbrev Number: 96 (DW_TAG_variable) <- this has a location
    <115ad>   DW_AT_abstract_origin: <0xdc73>
    <115b1>   DW_AT_location    : 0x486c        (location list)
---
 tools/perf/util/dwarf-aux.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/util/dwarf-aux.c +5 -2
index 7defd77..cc66c40 100644
@@ -747,14 +747,17 @@ struct __find_variable_param {
 static int __die_find_variable_cb(Dwarf_Die *die_mem, void *data)
 {
 	struct __find_variable_param *fvp = data;
+	Dwarf_Attribute attr;
 	int tag;
 
 	tag = dwarf_tag(die_mem);
 	if ((tag == DW_TAG_formal_parameter ||
 	     tag == DW_TAG_variable) &&
-	    die_compare_name(die_mem, fvp->name))
+	    die_compare_name(die_mem, fvp->name) &&
+	/* Does the DIE have location information or external instance? */
+	    (dwarf_attr(die_mem, DW_AT_external, &attr) ||
+	     dwarf_attr(die_mem, DW_AT_location, &attr)))
 		return DIE_FIND_CB_END;
-
 	if (dwarf_haspc(die_mem, fvp->addr))
 		return DIE_FIND_CB_CONTINUE;
 	else
#16 lore raw

Re: [PATCH -tip/urgent ] [BUGFIX] perf/probe: Fix a segfault if asked for variable it doesn't find

In reply to Masami Hiramatsu

Hi Masami,

Quoted 6 lines · Masami Hiramatsu

Fix a segfault bug by asking for variable it doesn't find. Since the convert_variable() didn't handle error code returned from convert_variable_location(), it just passed an incomplete variable field and then a segfault was occured when formatting the field.

Quoted 6 lines

This fixes that bug by handling success code correctly in convert_variable(). Other callers of convert_variable_location() are correctly checking the return code.

This bug was introduced by following commit. But another hidden erroneous error handling has been there previuosly (-ENOMEM case).

And -ERANGE too.. :)

Thanks,
Namhyung

Quoted 33 lines

commit 3d918a12a1b3088ac16ff37fa52760639d6e2403

Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/probe-finder.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/probe-finder.c +2 -2
index 5627621..9d8eb26 100644
@@ -511,12 +511,12 @@ static int convert_variable(Dwarf_Die *vr_die, struct probe_finder *pf)
 
 	ret = convert_variable_location(vr_die, pf->addr, pf->fb_ops,
 					&pf->sp_die, pf->tvar);
-	if (ret == -ENOENT)
+	if (ret == -ENOENT || ret == -EINVAL)
 		pr_err("Failed to find the location of %s at this address.\n"
 		       " Perhaps, it has been optimized out.\n", pf->pvar->var);
 	else if (ret == -ENOTSUP)
 		pr_err("Sorry, we don't support this variable location yet.\n");
-	else if (pf->pvar->field) {
+	else if (ret == 0 && pf->pvar->field) {
 		ret = convert_variable_fields(vr_die, pf->pvar->var,
 					      pf->pvar->field, &pf->tvar->ref,
 					      &die_mem);
#17 lore raw

Re: [PATCH -tip/urgent ] [BUGFIX] perf/probe: Fix a segfault if asked for variable it doesn't find

Quoted 9 lines · Namhyung Kim

Hi Masami,

Quoted 6 lines · Masami Hiramatsu

Fix a segfault bug by asking for variable it doesn't find. Since the convert_variable() didn't handle error code returned from convert_variable_location(), it just passed an incomplete variable field and then a segfault was occured when formatting the field.

Quoted 8 lines
Quoted 6 lines

This fixes that bug by handling success code correctly in convert_variable(). Other callers of convert_variable_location() are correctly checking the return code.

This bug was introduced by following commit. But another hidden erroneous error handling has been there previuosly (-ENOMEM case).

And -ERANGE too.. :)

Right, Anyway, I've added ret==0 sanity check. That error should be handled too. :-)

Thank you,

Quoted 38 lines

Thanks,
Namhyung

Quoted 33 lines

commit 3d918a12a1b3088ac16ff37fa52760639d6e2403

Cc Peter Zijlstra <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Jiri Olsa <[email protected]>
---
 tools/perf/util/probe-finder.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/probe-finder.c +2 -2
index 5627621..9d8eb26 100644
@@ -511,12 +511,12 @@ static int convert_variable(Dwarf_Die *vr_die, struct probe_finder *pf)
 
 	ret = convert_variable_location(vr_die, pf->addr, pf->fb_ops,
 					&pf->sp_die, pf->tvar);
-	if (ret == -ENOENT)
+	if (ret == -ENOENT || ret == -EINVAL)
 		pr_err("Failed to find the location of %s at this address.\n"
 		       " Perhaps, it has been optimized out.\n", pf->pvar->var);
 	else if (ret == -ENOTSUP)
 		pr_err("Sorry, we don't support this variable location yet.\n");
-	else if (pf->pvar->field) {
+	else if (ret == 0 && pf->pvar->field) {
 		ret = convert_variable_fields(vr_die, pf->pvar->var,
 					      pf->pvar->field, &pf->tvar->ref,
 					      &die_mem);
Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#18 lore raw

[tip:perf/urgent] perf probe: Fix a segfault if asked for variable it doesn't find

In reply to Masami Hiramatsu
perf probe: Fix a segfault if asked for variable it doesn't find
Commit-ID:  0c188a07b6a399e3df66534c29fef0a2082aaf57
Gitweb:     http://git.kernel.org/tip/0c188a07b6a399e3df66534c29fef0a2082aaf57
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Thu, 29 May 2014 19:52:32 +0900
Committer:  Jiri Olsa <[email protected]>
CommitDate: Wed, 4 Jun 2014 14:48:03 +0200

perf probe: Fix a segfault if asked for variable it doesn't find

Fix a segfault bug by asking for variable it doesn't find. Since the convert_variable() didn't handle error code returned from convert_variable_location(), it just passed an incomplete variable field and then a segfault was occurred when formatting the field.

This fixes that bug by handling success code correctly in convert_variable(). Other callers of convert_variable_location() are correctly checking the return code.

This bug was introduced by following commit. But another hidden erroneous error handling has been there previously (-ENOMEM case).

commit 3d918a12a1b3088ac16ff37fa52760639d6e2403

---
 tools/perf/util/probe-finder.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/probe-finder.c +2 -2
index 5627621..9d8eb26 100644
@@ -511,12 +511,12 @@ static int convert_variable(Dwarf_Die *vr_die, struct probe_finder *pf)
 
 	ret = convert_variable_location(vr_die, pf->addr, pf->fb_ops,
 					&pf->sp_die, pf->tvar);
-	if (ret == -ENOENT)
+	if (ret == -ENOENT || ret == -EINVAL)
 		pr_err("Failed to find the location of %s at this address.\n"
 		       " Perhaps, it has been optimized out.\n", pf->pvar->var);
 	else if (ret == -ENOTSUP)
 		pr_err("Sorry, we don't support this variable location yet.\n");
-	else if (pf->pvar->field) {
+	else if (ret == 0 && pf->pvar->field) {
 		ret = convert_variable_fields(vr_die, pf->pvar->var,
 					      pf->pvar->field, &pf->tvar->ref,
 					      &die_mem);