perf-probe: bugfix and update
cover perf-probe: bugfix and update
Hi,
These patches are a bugfix and an update of the document about perf-probe.
- The first one fixes probing on the function which has been inlined by optimizer but is not marked as inlined in the debuginfo.
- The second one just updates perf-probe document.
Thank you,
---
Masami Hiramatsu (2):
[BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance [DOC] perf-probe: Update perf-probe document
tools/perf/Documentation/perf-probe.txt | 16 ++++++++++++++-- tools/perf/util/dwarf-aux.c | 15 +++++++++++++++ tools/perf/util/dwarf-aux.h | 3 +++ tools/perf/util/probe-finder.c | 12 ++++-------- 4 files changed, 36 insertions(+), 10 deletions(-)
--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
PATCH 1/2 perf-probe: Fix to handle optimized not-inlined but has no instance
Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.
Without this patch:
----
# perf probe -a __up
Failed to get entry address of __up.
Error: Failed to add events.
# perf probe -L __up
Specified source line is not found.
Error: Failed to show lines.
----With this patch:
----
# perf probe -a __up
Added new event:
probe:__up (on __up)You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
# perf probe -L __up
<__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
0 static noinline void __sched __up(struct semaphore *sem)
{
struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
struct semaphore_waite
4 list_del(&waiter->list);
5 waiter->up = true;
6 wake_up_process(waiter->task);
7 }
------- tools/perf/util/dwarf-aux.c | 15 +++++++++++++++ tools/perf/util/dwarf-aux.h | 3 +++ tools/perf/util/probe-finder.c | 12 ++++-------- 3 files changed, 22 insertions(+), 8 deletions(-)
tools/perf/util/dwarf-aux.c +15 -0
@@ -278,6 +278,21 @@ bool die_is_func_def(Dwarf_Die *dw_die) }/** +* die_is_func_instance - Ensure that this DIE is an instance of a subprogram +* @dw_die: a DIE +* +* Ensure that this DIE is an instance (which has an entry address). +* This returns true if @dw_die is a function instance. If not, you need to +* call die_walk_instances() to find actual instances. +**/ +bool die_is_func_instance (Dwarf_Die *dw_die) +{ +Dwarf_Addr tmp; + +/* Actually gcc optimizes non-inline as like as inlined */ +return !dwarf_func_inline (dw_die) &&dwarf_entrypc (dw_die, &tmp) ==0 ; +} +/** * die_get_data_member_location - Get the data-member offset * @mb_die: a DIE of a member of a data structure * @offs: The offset of the member in the data structure
tools/perf/util/dwarf-aux.h +3 -0
@@ -41,6 +41,9 @@ extern int cu_walk_functions_at(Dwarf_Die *cu_die, Dwarf_Addr addr,/* Ensure that this DIE is a subprogram and definition (not declaration) */ extern bool die_is_func_def (Dwarf_Die *dw_die); +/* Ensure that this DIE is an instance of a subprogram */ +extern bool die_is_func_instance (Dwarf_Die *dw_die); +/* Compare diename and tname */ extern bool die_compare_name (Dwarf_Die *dw_die,const char *tname);
tools/perf/util/probe-finder.c +4 -8
@@ -915,17 +915,13 @@ static int probe_point_search_cb(Dwarf_Die *sp_die, void *data)dwarf_decl_line (sp_die, &pf->lno); pf->lno += pp->line; param->retval =find_probe_point_by_line (pf); - }else if (!dwarf_func_inline (sp_die)) { + }else if (die_is_func_instance (sp_die)) { +/* Instances always have the entry address */ +dwarf_entrypc (sp_die, &pf->addr);/* Real function */ if (pp->lazy_line) param->retval =find_probe_point_lazy (sp_die, pf);else { -if (dwarf_entrypc (sp_die, &pf->addr) !=0 ) { -pr_warning ("Failed to get entry address of " -"%s.\n" ,dwarf_diename (sp_die)); - param->retval = -ENOENT ; -return DWARF_CB_ABORT ; - } pf->addr += pp->offset;/* TODO: Check the address in this function */ param->retval =call_probe_finder (sp_die, pf);
@@ -1536,7 +1532,7 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)pr_debug ("New line range: %d to %d\n" , lf->lno_s, lf->lno_e); lr->start = lf->lno_s; lr->end = lf->lno_e; -if (dwarf_func_inline (sp_die)) +if (!die_is_func_instance (sp_die)) param->retval =die_walk_instances (sp_die, line_range_inline_cb, lf);else
Arnaldo Carvalho de Melo
maintainerRe: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance
Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:
Quoted 25 lines
Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.
Without this patch:
----# perf probe -a __up Failed to get entry address of __up. Error: Failed to add events. # perf probe -L __up Specified source line is not found. Error: Failed to show lines. ----With this patch:
----# perf probe -a __up Added new event: probe:__up (on __up)You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
Quoted 11 lines
# perf probe -L __up
<__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
0 static noinline void __sched __up(struct semaphore *sem)
{
struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
struct semaphore_waite
4 list_del(&waiter->list);
5 waiter->up = true;
6 wake_up_process(waiter->task);
7 }
----Since __up here was built in some other way, I looked for another 'noinline' function to try, and it failed here:
[root@zoo ~]# perf probe -L vmalloc_fault | head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>
0 static noinline int vmalloc_fault(unsigned long address)
1 {
pgd_t *pgd, *pgd_ref;
pud_t *pud, *pud_ref; [root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
---->>>>>> APPLY THE PATCH, rebuild and try again:
[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -v vmalloc_fault probe-definition(0): vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/3.19.0-rc6+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/3.19.0-rc6+/build/vmlinux Try to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -w vmalloc_fault /proc/kallsyms ffffffff81046b40 t vmalloc_fault
[root@zoo ~]# grep -w _text /proc/kallsyms ffffffff81000000 T _text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Quoted 1 line
Quoted 1 line
Quoted 1 line
0xffffffff81046b40-0xffffffff81000000
289600L
Quoted 1 line
Quoted 1 line
Quoted 1 line
[root@zoo ~]# readelf -s /lib/modules/3.19.0-rc6+/build/vmlinux | egrep -w _text\|vmalloc_fault
3499: ffffffff81046b40 410 FUNC LOCAL DEFAULT 1 vmalloc_fault 48873: ffffffff81000000 0 NOTYPE GLOBAL DEFAULT 1 _text
[root@zoo ~]#
---------------------------------------------------------------------------
So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?
Looking for one, will post here after objdump -dS finishes...
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance
Em Fri, Jan 30, 2015 at 11:21:35AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 38 lines
Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:
Quoted 25 lines
Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.
Without this patch:
----# perf probe -a __up Failed to get entry address of __up. Error: Failed to add events. # perf probe -L __up Specified source line is not found. Error: Failed to show lines. ----With this patch:
----# perf probe -a __up Added new event: probe:__up (on __up)You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
Quoted 11 lines
# perf probe -L __up <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0> 0 static noinline void __sched __up(struct semaphore *sem) { struct semaphore_waiter *waiter = list_first_entry(&sem->wait_ struct semaphore_waite 4 list_del(&waiter->list); 5 waiter->up = true; 6 wake_up_process(waiter->task); 7 } ----
Quoted 1 line
Since
__uphere was built in some other way, I looked for another
Nah, I saw you trying to add "perf probe -a __up" and stopped there, hadn't seen that it fails _as well_ for just listing the source code...
Yes, I managed to reproduce the problem and test your fix, it works:
[root@zoo ~]# perf probe --del __up
Removed event: probe:__up
[root@zoo ~]# perf probe -L __up
Specified source line is not found.
Error: Failed to show lines.
---->>>>>> APPLY THE PATCH, rebuild and try again:
[root@zoo ~]#
[root@zoo ~]# perf probe -L __up
<__up@/home/git/linux/kernel/locking/semaphore.c:0>
0 static noinline void __sched __up(struct semaphore *sem)
{
struct semaphore_waiter *waiter = list_first_entry(&sem->wait_list,
struct semaphore_waiter, list);
4 list_del(&waiter->list);
5 waiter->up = true;
6 wake_up_process(waiter->task);
7 }
[root@zoo ~]#--------------------------------------------------------------
So its just this other case that fails, vmalloc_fault, can you check this one? Does it happen in your machine as well?
Applying your patch, thanks,
- Arnaldo
Quoted 59 lines
'noinline' function to try, and it failed here:
[root@zoo ~]# perf probe -L
vmalloc_fault| head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>0 static noinline int vmalloc_fault(unsigned long address) 1 { pgd_t *pgd, *pgd_ref; pud_t *pud, *pud_ref;[root@zoo ~]# perf probe
vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.---->>>>>> APPLY THE PATCH, rebuild and try again:
[root@zoo ~]# perf probe
vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -vvmalloc_faultprobe-definition(0):vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at thevmlinux_path(7 entries long)
Using/lib/modules/3.19.0-rc6+/build/vmlinuxfor symbols Open Debuginfo file:/lib/modules/3.19.0-rc6+/build/vmlinuxTry to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1probe_trace_events.
Opening/sys/kernel/debug/tracing/kprobe_eventswrite=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -wvmalloc_fault/proc/kallsymsffffffff81046b40tvmalloc_fault
[root@zoo ~]# grep -w_text/proc/kallsymsffffffff81000000T_text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Quoted 1 line
Quoted 1 line
Quoted 1 line
0xffffffff81046b40-0xffffffff81000000289600L
Quoted 1 line
Quoted 1 line
Quoted 1 line
[root@zoo ~]# readelf -s
/lib/modules/3.19.0-rc6+/build/vmlinux| egrep -w _text\|vmalloc_fault3499: ffffffff81046b40 410 FUNC LOCAL DEFAULT 1 vmalloc_fault 48873: ffffffff81000000 0 NOTYPE GLOBAL DEFAULT 1 _text[root@zoo ~]#
---------------------------------------------------------------------------
So perhaps I need a 'noinline' _and_ '
__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?Looking for one, will post here after objdump -dS finishes...
- Arnaldo
Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance
Quoted 39 lines · Arnaldo Carvalho de Melo
Em Fri, Jan 30, 2015 at 11:21:35AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 38 lines
Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:
Quoted 36 lines
Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.
Without this patch:
----# perf probe -a __up Failed to get entry address of __up. Error: Failed to add events. # perf probe -L __up Specified source line is not found. Error: Failed to show lines. ----With this patch:
----# perf probe -a __up Added new event: probe:__up (on __up)You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
# perf probe -L __up <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0> 0 static noinline void __sched __up(struct semaphore *sem) { struct semaphore_waiter *waiter = list_first_entry(&sem->wait_ struct semaphore_waite 4 list_del(&waiter->list); 5 waiter->up = true; 6 wake_up_process(waiter->task); 7 } ----
Quoted 6 lines
Quoted 1 line
Since
__uphere was built in some other way, I looked for anotherNah, I saw you trying to add "perf probe -a
__up" and stopped there, hadn't seen that it fails_aswell_for just listing the source code...Yes, I managed to reproduce the problem and test your fix, it works:
Thanks for testing!
Quoted 22 lines
[root@zoo ~]# perf probe --del
__up
Removed event: probe:__up
[root@zoo ~]# perf probe -L__up
Specified source line is not found.
Error: Failed to show lines.---->>>>>> APPLY THE PATCH, rebuild and try again:
[root@zoo ~]# [root@zoo ~]# perf probe -L __up <__up@/home/git/linux/kernel/locking/semaphore.c:0> 0 static noinline void __sched __up(struct semaphore *sem) { struct semaphore_waiter *waiter = list_first_entry(&sem->wait_list, struct semaphore_waiter, list); 4 list_del(&waiter->list); 5 waiter->up = true; 6 wake_up_process(waiter->task); 7 } [root@zoo ~]#
Quoted 4 lines
--------------------------------------------------------------
So its just this other case that fails,
vmalloc_fault, can you check this one? Does it happen in your machine as well?
OK, I'll try reproducing this and dig deeper.
Quoted 2 lines
Applying your patch, thanks,
Thank you!
Quoted 63 lines
- Arnaldo
Quoted 59 lines
'noinline' function to try, and it failed here:
[root@zoo ~]# perf probe -L
vmalloc_fault| head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>0 static noinline int vmalloc_fault(unsigned long address) 1 { pgd_t *pgd, *pgd_ref; pud_t *pud, *pud_ref;[root@zoo ~]# perf probe
vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.---->>>>>> APPLY THE PATCH, rebuild and try again:
[root@zoo ~]# perf probe
vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -vvmalloc_faultprobe-definition(0):vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at thevmlinux_path(7 entries long)
Using/lib/modules/3.19.0-rc6+/build/vmlinuxfor symbols Open Debuginfo file:/lib/modules/3.19.0-rc6+/build/vmlinuxTry to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1probe_trace_events.
Opening/sys/kernel/debug/tracing/kprobe_eventswrite=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -wvmalloc_fault/proc/kallsymsffffffff81046b40tvmalloc_fault
[root@zoo ~]# grep -w_text/proc/kallsymsffffffff81000000T_text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Quoted 1 line
Quoted 1 line
Quoted 1 line
0xffffffff81046b40-0xffffffff81000000289600L
Quoted 1 line
Quoted 1 line
Quoted 1 line
[root@zoo ~]# readelf -s
/lib/modules/3.19.0-rc6+/build/vmlinux| egrep -w _text\|vmalloc_fault3499: ffffffff81046b40 410 FUNC LOCAL DEFAULT 1 vmalloc_fault 48873: ffffffff81000000 0 NOTYPE GLOBAL DEFAULT 1 _text[root@zoo ~]#
---------------------------------------------------------------------------
So perhaps I need a 'noinline' _and_ '
__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?Looking for one, will post here after objdump -dS finishes...
- Arnaldo
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance
In reply to Arnaldo Carvalho de MeloQuoted 87 lines · Arnaldo Carvalho de Melo
Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:
Quoted 25 lines
Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.
Without this patch:
----# perf probe -a __up Failed to get entry address of __up. Error: Failed to add events. # perf probe -L __up Specified source line is not found. Error: Failed to show lines. ----With this patch:
----# perf probe -a __up Added new event: probe:__up (on __up)You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
Quoted 11 lines
# perf probe -L __up <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0> 0 static noinline void __sched __up(struct semaphore *sem) { struct semaphore_waiter *waiter = list_first_entry(&sem->wait_ struct semaphore_waite 4 list_del(&waiter->list); 5 waiter->up = true; 6 wake_up_process(waiter->task); 7 } ----Since
__uphere was built in some other way, I looked for another 'noinline' function to try, and it failed here:[root@zoo ~]# perf probe -L
vmalloc_fault| head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>0 static noinline int vmalloc_fault(unsigned long address) 1 { pgd_t *pgd, *pgd_ref; pud_t *pud, *pud_ref;[root@zoo ~]# perf probe
vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.---->>>>>> APPLY THE PATCH, rebuild and try again:
[root@zoo ~]# perf probe
vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -vvmalloc_faultprobe-definition(0):vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at thevmlinux_path(7 entries long)
Using/lib/modules/3.19.0-rc6+/build/vmlinuxfor symbols Open Debuginfo file:/lib/modules/3.19.0-rc6+/build/vmlinuxTry to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1probe_trace_events.
Opening/sys/kernel/debug/tracing/kprobe_eventswrite=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -wvmalloc_fault/proc/kallsymsffffffff81046b40tvmalloc_fault
[root@zoo ~]# grep -w_text/proc/kallsymsffffffff81000000T_text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Quoted 1 line
Quoted 1 line
Quoted 1 line
0xffffffff81046b40-0xffffffff81000000289600L
Quoted 1 line
Quoted 1 line
Quoted 1 line
[root@zoo ~]# readelf -s
/lib/modules/3.19.0-rc6+/build/vmlinux| egrep -w _text\|vmalloc_fault3499: ffffffff81046b40 410 FUNC LOCAL DEFAULT 1 vmalloc_fault 48873: ffffffff81000000 0 NOTYPE GLOBAL DEFAULT 1 _text
Quoted 7 lines
[root@zoo ~]#
---------------------------------------------------------------------------
So perhaps I need a 'noinline' _and_ '
__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?
Ok, actually vmalloc_fault is marked as a nokprobe symbol.
$ grep vmalloc_fault -r arch/x86/ arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address) arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault); arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address) arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault); arch/x86/mm/fault.c: if (vmalloc_fault(address) >= 0)
All the symbols which marked by NOKPROBE_SYMBOL macro can not be kprobed. And we now have <debugfs>/kprobes/blacklist special file to get all nokprobe symbols.
I'll make another perf-probe patch which looks the blacklist before trying define new events.
Thank you,
Quoted 10 lines
Looking for one, will post here after objdump -dS finishes...
- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Arnaldo Carvalho de Melo
maintainerRe: Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance
Em Sun, Feb 08, 2015 at 05:37:08PM +0900, Masami Hiramatsu escreveu:
Quoted 14 lines
Quoted 3 lines · Arnaldo Carvalho de Melo
So perhaps I need a 'noinline' _and_ '
__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?Ok, actually
vmalloc_faultis marked as a nokprobe symbol.$ grep vmalloc_fault -r arch/x86/ arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address) arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault); arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address) arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault); arch/x86/mm/fault.c: if (vmalloc_fault(address) >= 0)
Quoted 2 lines
All the symbols which marked by
NOKPROBE_SYMBOLmacro can not be kprobed. And we now have <debugfs>/kprobes/blacklist special file
Ok!
Quoted 4 lines
to get all nokprobe symbols.
I'll make another perf-probe patch which looks the blacklist before trying define new events.
Ok, that will help, improving the error messages is a big priority.
- Arnaldo
PATCH 1/1 perf-probe: Check kprobes blacklist when adding new events
Recent linux kernel provides a blacklist of the functions which can not be probed. perf probe can now check this blacklist before setting new events and indicate better error message for users.
Without this patch,
----
# perf probe --add vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
----
With this patch
----
# perf probe --add vmalloc_fault
Added new event:
Warning: Skipped probing on blacklisted function: vmalloc_fault
------- tools/perf/util/probe-event.c | 109 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+), 1 deletion(-)
tools/perf/util/probe-event.c +108 -1
@@ -1903,6 +1903,95 @@ static struct strlist *get_probe_trace_command_rawlist(int fd)return sl; } +struct kprobe_blacklist_node { +struct list_head list; +unsigned long start; +unsigned long end; +char *symbol; +}; + +static void kprobe_blacklist__delete (struct list_head *blacklist) +{ +struct kprobe_blacklist_node *node; + +while (!list_empty (blacklist)) { + node =list_first_entry (blacklist, + struct kprobe_blacklist_node, list); +list_del (&node->list); +free (node->symbol); +free (node); + } +} + +static int kprobe_blacklist__load (struct list_head *blacklist) +{ +struct kprobe_blacklist_node *node; +const char *__debugfs =debugfs_find_mountpoint (); +char buf[PATH_MAX ], *p; +FILE *fp; +int ret; + +if (__debugfs ==NULL ) +return -ENOTSUP ; + + ret =e_snprintf (buf,PATH_MAX ,"%s/kprobes/blacklist" , __debugfs); +if (ret <0 ) +return ret; + + fp =fopen (buf,"r" ); +if (!fp) +return -errno; + + ret =0 ; +while (fgets (buf,PATH_MAX , fp)) { + node =zalloc (sizeof (*node)); +if (!node) { + ret = -ENOMEM ; +break ; + } +INIT_LIST_HEAD (&node->list); +list_add_tail (&node->list, blacklist); +if (sscanf (buf,"0x%lx-0x%lx" , &node->start, &node->end) !=2 ) { + ret = -EINVAL ; +break ; + } + p =strchr (buf,'\t' ); +if (p) { + p++; +if (p[strlen (p) -1 ] =='\n' ) + p[strlen (p) -1 ] ='\0' ; + }else + p = (char *)"unknown" ; + node->symbol =strdup (p); +if (!node->symbol) { + ret = -ENOMEM ; +break ; + } +pr_debug2 ("Blacklist: 0x%lx-0x%lx, %s\n" , + node->start, node->end, node->symbol); + ret++; + } +if (ret <0 ) +kprobe_blacklist__delete (blacklist); +fclose (fp); + +return ret; +} + +static struct kprobe_blacklist_node * +kprobe_blacklist__find_by_address (struct list_head *blacklist, +unsigned long address) +{ +struct kprobe_blacklist_node *node; + +list_for_each_entry (node, blacklist, list) { +if (node->start <= address && address <= node->end) +return node; + } + +return NULL ; +} +/* Show an event */ static int show_perf_probe_event (struct perf_probe_event *pev,const char *module)
@@ -2117,6 +2206,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,char buf[64 ];const char *event, *group;struct strlist *namelist; +LIST_HEAD (blacklist); +struct kprobe_blacklist_node *node;if (pev->uprobes) fd =open_uprobe_events (true);
@@ -2134,11 +2225,25 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,pr_debug ("Failed to get current event list.\n" );return -EIO ; } +/* Get kprobe blacklist if exists */ +if (!pev->uprobes) { + ret =kprobe_blacklist__load (&blacklist); +if (ret <0 ) +pr_debug ("No kprobe blacklist support, ignored\n" ); + } ret =0 ;pr_info ("Added new event%s\n" , (ntevs >1 ) ?"s:" :":" );for (i =0 ; i < ntevs; i++) { tev = &tevs[i]; +/* Ensure that the address is NOT blacklisted */ + node =kprobe_blacklist__find_by_address (&blacklist, + tev->point.address); +if (node) { +pr_warning ("Warning: Skipped probing on blacklisted function: %s\n" , node->symbol); +continue ; + } +if (pev->event) event = pev->event;else
@@ -2189,13 +2294,15 @@ static int __add_probe_trace_events(struct perf_probe_event *pev, allow_suffix = true; } -if (ret >=0 ) { +/* Note that it is possible to skip all events because of blacklist */ +if (ret >=0 && tev->event) {/* Show how to use the event. */ pr_info ("\nYou can now use it in all perf tools, such as:\n\n" );pr_info ("\tperf record -e %s:%s -aR sleep 1\n\n" , tev->group, tev->event); } +kprobe_blacklist__delete (&blacklist);strlist__delete (namelist);close (fd);return ret;
Arnaldo Carvalho de Melo
maintainerRe: [PATCH perf/core ] perf-probe: Check kprobes blacklist when adding new events
Em Thu, Feb 19, 2015 at 11:31:13PM +0900, Masami Hiramatsu escreveu:
Quoted 18 lines
Recent linux kernel provides a blacklist of the functions which can not be probed. perf probe can now check this blacklist before setting new events and indicate better error message for users.
Without this patch,
----# perf probe --add vmalloc_fault Added new event: Failed to write event: Invalid argument Error: Failed to add events. ---- With this patch ---- # perf probe --add vmalloc_fault Added new event: Warning: Skipped probing on blacklisted function: vmalloc_fault ----
Quoted 2 lines
Signed-off-by Masami Hiramatsu <[email protected]>Reported-by Arnaldo Carvalho de Melo <[email protected]>
Ok, I really need to use a newer kernel on this RHEL7 box, a 4.0-rc one, as when trying this I saw no difference, but probably that is because it didn't find the kprobes blacklist, tried inserting the probe and then failed with the exact previous message...
A follow up patch could try to improve the situation by saying something like:
"Failed to insert the probe, maybe this is a blacklisted function and you're running an old kernel?
Please check the function definition, it should not be marked as, for instance:
NOKPROBE_SYMBOL(vmalloc_fault);"
Anyway, applying your patch, clearly an improvement when the blacklist is available.
Thanks,
- Arnaldo
tip-bot for Masami Hiramatsu
merge bot[tip:perf/core] perf probe: Check kprobes blacklist when adding new events
In reply to Masami Hiramatsuperf probe: Check kprobes blacklist when adding new events
Commit-ID: 9aaf5a5f479bd68699f2e6f6e5e5f1253377b6da Gitweb: http://git.kernel.org/tip/9aaf5a5f479bd68699f2e6f6e5e5f1253377b6da Author: Masami Hiramatsu <[email protected]> AuthorDate: Thu, 19 Feb 2015 23:31:13 +0900 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Thu, 26 Feb 2015 11:59:05 -0300
perf probe: Check kprobes blacklist when adding new events
Recent linux kernel provides a blacklist of the functions which can not be probed. perf probe can now check this blacklist before setting new events and indicate better error message for users.
Without this patch,
----
# perf probe --add vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
----
With this patch
----
# perf probe --add vmalloc_fault
Added new event:
Warning: Skipped probing on blacklisted function: vmalloc_fault
------- tools/perf/util/probe-event.c | 109 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 108 insertions(+), 1 deletion(-)
tools/perf/util/probe-event.c +108 -1
@@ -1903,6 +1903,95 @@ static struct strlist *get_probe_trace_command_rawlist(int fd)return sl; } +struct kprobe_blacklist_node { +struct list_head list; +unsigned long start; +unsigned long end; +char *symbol; +}; + +static void kprobe_blacklist__delete (struct list_head *blacklist) +{ +struct kprobe_blacklist_node *node; + +while (!list_empty (blacklist)) { + node =list_first_entry (blacklist, + struct kprobe_blacklist_node, list); +list_del (&node->list); +free (node->symbol); +free (node); + } +} + +static int kprobe_blacklist__load (struct list_head *blacklist) +{ +struct kprobe_blacklist_node *node; +const char *__debugfs =debugfs_find_mountpoint (); +char buf[PATH_MAX ], *p; +FILE *fp; +int ret; + +if (__debugfs ==NULL ) +return -ENOTSUP ; + + ret =e_snprintf (buf,PATH_MAX ,"%s/kprobes/blacklist" , __debugfs); +if (ret <0 ) +return ret; + + fp =fopen (buf,"r" ); +if (!fp) +return -errno; + + ret =0 ; +while (fgets (buf,PATH_MAX , fp)) { + node =zalloc (sizeof (*node)); +if (!node) { + ret = -ENOMEM ; +break ; + } +INIT_LIST_HEAD (&node->list); +list_add_tail (&node->list, blacklist); +if (sscanf (buf,"0x%lx-0x%lx" , &node->start, &node->end) !=2 ) { + ret = -EINVAL ; +break ; + } + p =strchr (buf,'\t' ); +if (p) { + p++; +if (p[strlen (p) -1 ] =='\n' ) + p[strlen (p) -1 ] ='\0' ; + }else + p = (char *)"unknown" ; + node->symbol =strdup (p); +if (!node->symbol) { + ret = -ENOMEM ; +break ; + } +pr_debug2 ("Blacklist: 0x%lx-0x%lx, %s\n" , + node->start, node->end, node->symbol); + ret++; + } +if (ret <0 ) +kprobe_blacklist__delete (blacklist); +fclose (fp); + +return ret; +} + +static struct kprobe_blacklist_node * +kprobe_blacklist__find_by_address (struct list_head *blacklist, +unsigned long address) +{ +struct kprobe_blacklist_node *node; + +list_for_each_entry (node, blacklist, list) { +if (node->start <= address && address <= node->end) +return node; + } + +return NULL ; +} +/* Show an event */ static int show_perf_probe_event (struct perf_probe_event *pev,const char *module)
@@ -2117,6 +2206,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,char buf[64 ];const char *event, *group;struct strlist *namelist; +LIST_HEAD (blacklist); +struct kprobe_blacklist_node *node;if (pev->uprobes) fd =open_uprobe_events (true);
@@ -2134,11 +2225,25 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,pr_debug ("Failed to get current event list.\n" );return -EIO ; } +/* Get kprobe blacklist if exists */ +if (!pev->uprobes) { + ret =kprobe_blacklist__load (&blacklist); +if (ret <0 ) +pr_debug ("No kprobe blacklist support, ignored\n" ); + } ret =0 ;pr_info ("Added new event%s\n" , (ntevs >1 ) ?"s:" :":" );for (i =0 ; i < ntevs; i++) { tev = &tevs[i]; +/* Ensure that the address is NOT blacklisted */ + node =kprobe_blacklist__find_by_address (&blacklist, + tev->point.address); +if (node) { +pr_warning ("Warning: Skipped probing on blacklisted function: %s\n" , node->symbol); +continue ; + } +if (pev->event) event = pev->event;else
@@ -2189,13 +2294,15 @@ static int __add_probe_trace_events(struct perf_probe_event *pev, allow_suffix = true; } -if (ret >=0 ) { +/* Note that it is possible to skip all events because of blacklist */ +if (ret >=0 && tev->event) {/* Show how to use the event. */ pr_info ("\nYou can now use it in all perf tools, such as:\n\n" );pr_info ("\tperf record -e %s:%s -aR sleep 1\n\n" , tev->group, tev->event); } +kprobe_blacklist__delete (&blacklist);strlist__delete (namelist);close (fd);return ret;
tip-bot for Masami Hiramatsu
merge bot[tip:perf/core] perf probe: Fix to handle optimized not-inlined functions
In reply to Masami Hiramatsuperf probe: Fix to handle optimized not-inlined functions
Commit-ID: e1ecbbc3fa834cc6b4b344edb1968e734d57189b Gitweb: http://git.kernel.org/tip/e1ecbbc3fa834cc6b4b344edb1968e734d57189b Author: Masami Hiramatsu <[email protected]> AuthorDate: Fri, 30 Jan 2015 18:37:44 +0900 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Fri, 6 Feb 2015 11:46:36 +0100
perf probe: Fix to handle optimized not-inlined functions
Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point.
To fix this problem, we need to find actual instance of the function.
Without this patch:
----
# perf probe -a __up
Failed to get entry address of __up.
Error: Failed to add events.
# perf probe -L __up
Specified source line is not found.
Error: Failed to show lines.
----With this patch:
----
# perf probe -a __up
Added new event:
probe:__up (on __up)You can now use it in all perf tools, such as:
perf record -e probe:__up -aR sleep 1
# perf probe -L __up
<__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
0 static noinline void __sched __up(struct semaphore *sem)
{
struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
struct semaphore_waite
4 list_del(&waiter->list);
5 waiter->up = true;
6 wake_up_process(waiter->task);
7 }
------- tools/perf/util/dwarf-aux.c | 15 +++++++++++++++ tools/perf/util/dwarf-aux.h | 3 +++ tools/perf/util/probe-finder.c | 12 ++++-------- 3 files changed, 22 insertions(+), 8 deletions(-)
tools/perf/util/dwarf-aux.c +15 -0
@@ -278,6 +278,21 @@ bool die_is_func_def(Dwarf_Die *dw_die) }/** +* die_is_func_instance - Ensure that this DIE is an instance of a subprogram +* @dw_die: a DIE +* +* Ensure that this DIE is an instance (which has an entry address). +* This returns true if @dw_die is a function instance. If not, you need to +* call die_walk_instances() to find actual instances. +**/ +bool die_is_func_instance (Dwarf_Die *dw_die) +{ +Dwarf_Addr tmp; + +/* Actually gcc optimizes non-inline as like as inlined */ +return !dwarf_func_inline (dw_die) &&dwarf_entrypc (dw_die, &tmp) ==0 ; +} +/** * die_get_data_member_location - Get the data-member offset * @mb_die: a DIE of a member of a data structure * @offs: The offset of the member in the data structure
tools/perf/util/dwarf-aux.h +3 -0
@@ -41,6 +41,9 @@ extern int cu_walk_functions_at(Dwarf_Die *cu_die, Dwarf_Addr addr,/* Ensure that this DIE is a subprogram and definition (not declaration) */ extern bool die_is_func_def (Dwarf_Die *dw_die); +/* Ensure that this DIE is an instance of a subprogram */ +extern bool die_is_func_instance (Dwarf_Die *dw_die); +/* Compare diename and tname */ extern bool die_compare_name (Dwarf_Die *dw_die,const char *tname);
tools/perf/util/probe-finder.c +4 -8
@@ -915,17 +915,13 @@ static int probe_point_search_cb(Dwarf_Die *sp_die, void *data)dwarf_decl_line (sp_die, &pf->lno); pf->lno += pp->line; param->retval =find_probe_point_by_line (pf); - }else if (!dwarf_func_inline (sp_die)) { + }else if (die_is_func_instance (sp_die)) { +/* Instances always have the entry address */ +dwarf_entrypc (sp_die, &pf->addr);/* Real function */ if (pp->lazy_line) param->retval =find_probe_point_lazy (sp_die, pf);else { -if (dwarf_entrypc (sp_die, &pf->addr) !=0 ) { -pr_warning ("Failed to get entry address of " -"%s.\n" ,dwarf_diename (sp_die)); - param->retval = -ENOENT ; -return DWARF_CB_ABORT ; - } pf->addr += pp->offset;/* TODO: Check the address in this function */ param->retval =call_probe_finder (sp_die, pf);
@@ -1536,7 +1532,7 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)pr_debug ("New line range: %d to %d\n" , lf->lno_s, lf->lno_e); lr->start = lf->lno_s; lr->end = lf->lno_e; -if (dwarf_func_inline (sp_die)) +if (!die_is_func_instance (sp_die)) param->retval =die_walk_instances (sp_die, line_range_inline_cb, lf);else
PATCH 2/2 perf-probe: Update perf-probe document
In reply to Masami HiramatsuUpdate Documentation/perf-probe.txt to add descriptions of some newer options.
--- tools/perf/Documentation/perf-probe.txt | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-probe.txt +14 -2
@@ -47,6 +47,12 @@ OPTIONS -v:: --verbose:: Be more verbose (show parsed arguments, etc). + Can not use with -q. + +-q:: +--quiet:: + Be quiet (do not show any messages including errors). + Can not use with -v. -a:: --add=::
@@ -96,7 +102,7 @@ OPTIONS Dry run. With this option, --add and --del doesn't execute actual adding and removal operations. ---max-probes:: +--max-probes=NUM:: Set the maximum number of probe points for an event. Default is 128. -x::
@@ -104,8 +110,13 @@ OPTIONS Specify path to the executable or shared library file for user space tracing. Can also be used with --funcs option. +--demangle:: + Demangle application symbols. --no-demangle is also available + for disabling demangling. + --demangle-kernel:: - Demangle kernel symbols. + Demangle kernel symbols. --no-demangle-kernel is also available + for disabling kernel demangling. In absence of -m/-x options, perf probe checks if the first argument after the options is an absolute path name. If its an absolute path, perf probe
@@ -137,6 +148,7 @@ Each probe argument follows below syntax. [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE] 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.) +'$vars' special argument is also available for NAME, it is expanded to the local variables which can access at given probe point. 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo. You can specify 'string' type only for the local variable or structure member which is an array of or a pointer to 'char' or 'unsigned char' type. On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid.
tip-bot for Masami Hiramatsu
merge bot[tip:perf/core] perf probe: Update man page
perf probe: Update man page
Commit-ID: 8b72805fd1dbfd697c5d4492d0cf1ebbd994950d Gitweb: http://git.kernel.org/tip/8b72805fd1dbfd697c5d4492d0cf1ebbd994950d Author: Masami Hiramatsu <[email protected]> AuthorDate: Fri, 30 Jan 2015 18:37:46 +0900 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Fri, 6 Feb 2015 11:46:36 +0100
perf probe: Update man page
Update Documentation/perf-probe.txt to add descriptions of some newer options.
--- tools/perf/Documentation/perf-probe.txt | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-probe.txt +14 -2
@@ -47,6 +47,12 @@ OPTIONS -v:: --verbose:: Be more verbose (show parsed arguments, etc). + Can not use with -q. + +-q:: +--quiet:: + Be quiet (do not show any messages including errors). + Can not use with -v. -a:: --add=::
@@ -96,7 +102,7 @@ OPTIONS Dry run. With this option, --add and --del doesn't execute actual adding and removal operations. ---max-probes:: +--max-probes=NUM:: Set the maximum number of probe points for an event. Default is 128. -x::
@@ -104,8 +110,13 @@ OPTIONS Specify path to the executable or shared library file for user space tracing. Can also be used with --funcs option. +--demangle:: + Demangle application symbols. --no-demangle is also available + for disabling demangling. + --demangle-kernel:: - Demangle kernel symbols. + Demangle kernel symbols. --no-demangle-kernel is also available + for disabling kernel demangling. In absence of -m/-x options, perf probe checks if the first argument after the options is an absolute path name. If its an absolute path, perf probe
@@ -137,6 +148,7 @@ Each probe argument follows below syntax. [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE] 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.) +'$vars' special argument is also available for NAME, it is expanded to the local variables which can access at given probe point. 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo. You can specify 'string' type only for the local variable or structure member which is an array of or a pointer to 'char' or 'unsigned char' type. On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid.