Skip to contentSkip to conversation

perf-probe: bugfix and update

PATCH In review Masami Hiramatsu 3 participants 13 messages Jan 30, 2015 – Feb 28, 2015 perf events

lore
Share & tools
lore
#0 lore raw

cover perf-probe: bugfix and update

Hi,

These patches are a bugfix and an update of the document about perf-probe.

- The first one fixes probing on the function which has been inlined by optimizer but is not marked as inlined in the debuginfo.

- The second one just updates perf-probe document.

Thank you,

---

Masami Hiramatsu (2):
[BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance [DOC] perf-probe: Update perf-probe document

 tools/perf/Documentation/perf-probe.txt |   16 ++++++++++++++--
 tools/perf/util/dwarf-aux.c             |   15 +++++++++++++++
 tools/perf/util/dwarf-aux.h             |    3 +++
 tools/perf/util/probe-finder.c          |   12 ++++--------
 4 files changed, 36 insertions(+), 10 deletions(-)

--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

#1 lore raw

PATCH 1/2 perf-probe: Fix to handle optimized not-inlined but has no instance

Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.

Without this patch:
----

  # perf probe -a __up
  Failed to get entry address of __up.
    Error: Failed to add events.
  # perf probe -L __up
  Specified source line is not found.
    Error: Failed to show lines.
  ----

With this patch:
----

  # perf probe -a __up
  Added new event:
    probe:__up           (on __up)

You can now use it in all perf tools, such as:

perf record -e probe:__up -aR sleep 1

  # perf probe -L __up
  <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
                                                          struct semaphore_waite
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  ----
---
 tools/perf/util/dwarf-aux.c    |   15 +++++++++++++++
 tools/perf/util/dwarf-aux.h    |    3 +++
 tools/perf/util/probe-finder.c |   12 ++++--------
 3 files changed, 22 insertions(+), 8 deletions(-)
tools/perf/util/dwarf-aux.c +15 -0
index cc66c40..780b2bc 100644
@@ -278,6 +278,21 @@ bool die_is_func_def(Dwarf_Die *dw_die)
 }
 
 /**
+ * die_is_func_instance - Ensure that this DIE is an instance of a subprogram
+ * @dw_die: a DIE
+ *
+ * Ensure that this DIE is an instance (which has an entry address).
+ * This returns true if @dw_die is a function instance. If not, you need to
+ * call die_walk_instances() to find actual instances.
+ **/
+bool die_is_func_instance(Dwarf_Die *dw_die)
+{
+	Dwarf_Addr tmp;
+
+	/* Actually gcc optimizes non-inline as like as inlined */
+	return !dwarf_func_inline(dw_die) && dwarf_entrypc(dw_die, &tmp) == 0;
+}
+/**
  * die_get_data_member_location - Get the data-member offset
  * @mb_die: a DIE of a member of a data structure
  * @offs: The offset of the member in the data structure
tools/perf/util/dwarf-aux.h +3 -0
index b4fe90c..af7dbcd 100644
@@ -41,6 +41,9 @@ extern int cu_walk_functions_at(Dwarf_Die *cu_die, Dwarf_Addr addr,
 /* Ensure that this DIE is a subprogram and definition (not declaration) */
 extern bool die_is_func_def(Dwarf_Die *dw_die);
 
+/* Ensure that this DIE is an instance of a subprogram */
+extern bool die_is_func_instance(Dwarf_Die *dw_die);
+
 /* Compare diename and tname */
 extern bool die_compare_name(Dwarf_Die *dw_die, const char *tname);
 
tools/perf/util/probe-finder.c +4 -8
index b5247d7..d141935 100644
@@ -915,17 +915,13 @@ static int probe_point_search_cb(Dwarf_Die *sp_die, void *data)
 		dwarf_decl_line(sp_die, &pf->lno);
 		pf->lno += pp->line;
 		param->retval = find_probe_point_by_line(pf);
-	} else if (!dwarf_func_inline(sp_die)) {
+	} else if (die_is_func_instance(sp_die)) {
+		/* Instances always have the entry address */
+		dwarf_entrypc(sp_die, &pf->addr);
 		/* Real function */
 		if (pp->lazy_line)
 			param->retval = find_probe_point_lazy(sp_die, pf);
 		else {
-			if (dwarf_entrypc(sp_die, &pf->addr) != 0) {
-				pr_warning("Failed to get entry address of "
-					   "%s.\n", dwarf_diename(sp_die));
-				param->retval = -ENOENT;
-				return DWARF_CB_ABORT;
-			}
 			pf->addr += pp->offset;
 			/* TODO: Check the address in this function */
 			param->retval = call_probe_finder(sp_die, pf);
@@ -1536,7 +1532,7 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)
 		pr_debug("New line range: %d to %d\n", lf->lno_s, lf->lno_e);
 		lr->start = lf->lno_s;
 		lr->end = lf->lno_e;
-		if (dwarf_func_inline(sp_die))
+		if (!die_is_func_instance(sp_die))
 			param->retval = die_walk_instances(sp_die,
 						line_range_inline_cb, lf);
 		else
#2 lore raw

Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance

Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:

Quoted 25 lines

Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.

Without this patch:
----

  # perf probe -a __up
  Failed to get entry address of __up.
    Error: Failed to add events.
  # perf probe -L __up
  Specified source line is not found.
    Error: Failed to show lines.
  ----

With this patch:
----

  # perf probe -a __up
  Added new event:
    probe:__up           (on __up)

You can now use it in all perf tools, such as:

perf record -e probe:__up -aR sleep 1

Quoted 11 lines
  # perf probe -L __up
  <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
                                                          struct semaphore_waite
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  ----

Since __up here was built in some other way, I looked for another 'noinline' function to try, and it failed here:

[root@zoo ~]# perf probe -L vmalloc_fault | head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>

        0  static noinline int vmalloc_fault(unsigned long address)
        1  {
           	pgd_t *pgd, *pgd_ref;
           	pud_t *pud, *pud_ref;

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.

---->>>>>> APPLY THE PATCH, rebuild and try again:

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -v vmalloc_fault probe-definition(0): vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/3.19.0-rc6+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/3.19.0-rc6+/build/vmlinux Try to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -w vmalloc_fault /proc/kallsyms ffffffff81046b40 t vmalloc_fault
[root@zoo ~]# grep -w _text /proc/kallsyms ffffffff81000000 T _text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

0xffffffff81046b40 - 0xffffffff81000000

289600L

Quoted 1 line
Quoted 1 line
Quoted 1 line

[root@zoo ~]# readelf -s /lib/modules/3.19.0-rc6+/build/vmlinux | egrep -w _text\|vmalloc_fault

    3499: ffffffff81046b40   410 FUNC    LOCAL  DEFAULT    1 vmalloc_fault
   48873: ffffffff81000000     0 NOTYPE  GLOBAL DEFAULT    1 _text

[root@zoo ~]#

---------------------------------------------------------------------------

So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?

Looking for one, will post here after objdump -dS finishes...

- Arnaldo

#3 lore raw

Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance

Em Fri, Jan 30, 2015 at 11:21:35AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 38 lines

Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:

Quoted 25 lines

Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.

Without this patch:
----

  # perf probe -a __up
  Failed to get entry address of __up.
    Error: Failed to add events.
  # perf probe -L __up
  Specified source line is not found.
    Error: Failed to show lines.
  ----

With this patch:
----

  # perf probe -a __up
  Added new event:
    probe:__up           (on __up)

You can now use it in all perf tools, such as:

perf record -e probe:__up -aR sleep 1

Quoted 11 lines
  # perf probe -L __up
  <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
                                                          struct semaphore_waite
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  ----
Quoted 1 line

Since __up here was built in some other way, I looked for another

Nah, I saw you trying to add "perf probe -a __up" and stopped there, hadn't seen that it fails _as well_ for just listing the source code...

Yes, I managed to reproduce the problem and test your fix, it works:

[root@zoo ~]# perf probe --del __up
Removed event: probe:__up
[root@zoo ~]# perf probe -L __up
Specified source line is not found.
Error: Failed to show lines.

---->>>>>> APPLY THE PATCH, rebuild and try again:

  [root@zoo ~]# 
  [root@zoo ~]# perf probe -L __up
  <__up@/home/git/linux/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_list,
                                                          struct semaphore_waiter, list);
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  [root@zoo ~]#

--------------------------------------------------------------

So its just this other case that fails, vmalloc_fault, can you check this one? Does it happen in your machine as well?

Applying your patch, thanks,

- Arnaldo

Quoted 59 lines

'noinline' function to try, and it failed here:

[root@zoo ~]# perf probe -L vmalloc_fault | head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>

        0  static noinline int vmalloc_fault(unsigned long address)
        1  {
           	pgd_t *pgd, *pgd_ref;
           	pud_t *pud, *pud_ref;

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.

---->>>>>> APPLY THE PATCH, rebuild and try again:

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -v vmalloc_fault probe-definition(0): vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/3.19.0-rc6+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/3.19.0-rc6+/build/vmlinux Try to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -w vmalloc_fault /proc/kallsyms ffffffff81046b40 t vmalloc_fault
[root@zoo ~]# grep -w _text /proc/kallsyms ffffffff81000000 T _text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

0xffffffff81046b40 - 0xffffffff81000000

289600L

Quoted 1 line
Quoted 1 line
Quoted 1 line

[root@zoo ~]# readelf -s /lib/modules/3.19.0-rc6+/build/vmlinux | egrep -w _text\|vmalloc_fault

    3499: ffffffff81046b40   410 FUNC    LOCAL  DEFAULT    1 vmalloc_fault
   48873: ffffffff81000000     0 NOTYPE  GLOBAL DEFAULT    1 _text

[root@zoo ~]#

---------------------------------------------------------------------------

So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?

Looking for one, will post here after objdump -dS finishes...

- Arnaldo

#4 lore raw

Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance

Quoted 39 lines · Arnaldo Carvalho de Melo

Em Fri, Jan 30, 2015 at 11:21:35AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 38 lines

Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:

Quoted 36 lines

Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.

Without this patch:
----

  # perf probe -a __up
  Failed to get entry address of __up.
    Error: Failed to add events.
  # perf probe -L __up
  Specified source line is not found.
    Error: Failed to show lines.
  ----

With this patch:
----

  # perf probe -a __up
  Added new event:
    probe:__up           (on __up)

You can now use it in all perf tools, such as:

perf record -e probe:__up -aR sleep 1

  # perf probe -L __up
  <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
                                                          struct semaphore_waite
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  ----
Quoted 6 lines
Quoted 1 line

Since __up here was built in some other way, I looked for another

Nah, I saw you trying to add "perf probe -a __up" and stopped there, hadn't seen that it fails _as well_ for just listing the source code...

Yes, I managed to reproduce the problem and test your fix, it works:

Thanks for testing!

Quoted 22 lines

[root@zoo ~]# perf probe --del __up
Removed event: probe:__up
[root@zoo ~]# perf probe -L __up
Specified source line is not found.
Error: Failed to show lines.

---->>>>>> APPLY THE PATCH, rebuild and try again:

  [root@zoo ~]# 
  [root@zoo ~]# perf probe -L __up
  <__up@/home/git/linux/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_list,
                                                          struct semaphore_waiter, list);
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  [root@zoo ~]#
Quoted 4 lines

--------------------------------------------------------------

So its just this other case that fails, vmalloc_fault, can you check this one? Does it happen in your machine as well?

OK, I'll try reproducing this and dig deeper.

Quoted 2 lines

Applying your patch, thanks,

Thank you!

Quoted 63 lines

- Arnaldo

Quoted 59 lines

'noinline' function to try, and it failed here:

[root@zoo ~]# perf probe -L vmalloc_fault | head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>

        0  static noinline int vmalloc_fault(unsigned long address)
        1  {
           	pgd_t *pgd, *pgd_ref;
           	pud_t *pud, *pud_ref;

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.

---->>>>>> APPLY THE PATCH, rebuild and try again:

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -v vmalloc_fault probe-definition(0): vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/3.19.0-rc6+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/3.19.0-rc6+/build/vmlinux Try to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -w vmalloc_fault /proc/kallsyms ffffffff81046b40 t vmalloc_fault
[root@zoo ~]# grep -w _text /proc/kallsyms ffffffff81000000 T _text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

0xffffffff81046b40 - 0xffffffff81000000

289600L

Quoted 1 line
Quoted 1 line
Quoted 1 line

[root@zoo ~]# readelf -s /lib/modules/3.19.0-rc6+/build/vmlinux | egrep -w _text\|vmalloc_fault

    3499: ffffffff81046b40   410 FUNC    LOCAL  DEFAULT    1 vmalloc_fault
   48873: ffffffff81000000     0 NOTYPE  GLOBAL DEFAULT    1 _text

[root@zoo ~]#

---------------------------------------------------------------------------

So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?

Looking for one, will post here after objdump -dS finishes...

- Arnaldo

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#5 lore raw

Re: Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance

In reply to Arnaldo Carvalho de Melo
Quoted 87 lines · Arnaldo Carvalho de Melo

Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:

Quoted 25 lines

Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point. To fix this problem, we need to find actual instance of the function.

Without this patch:
----

  # perf probe -a __up
  Failed to get entry address of __up.
    Error: Failed to add events.
  # perf probe -L __up
  Specified source line is not found.
    Error: Failed to show lines.
  ----

With this patch:
----

  # perf probe -a __up
  Added new event:
    probe:__up           (on __up)

You can now use it in all perf tools, such as:

perf record -e probe:__up -aR sleep 1

Quoted 11 lines
  # perf probe -L __up
  <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
                                                          struct semaphore_waite
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  ----

Since __up here was built in some other way, I looked for another 'noinline' function to try, and it failed here:

[root@zoo ~]# perf probe -L vmalloc_fault | head -5 <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>

        0  static noinline int vmalloc_fault(unsigned long address)
        1  {
           	pgd_t *pgd, *pgd_ref;
           	pud_t *pud, *pud_ref;

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.

---->>>>>> APPLY THE PATCH, rebuild and try again:

[root@zoo ~]# perf probe vmalloc_fault
Added new event:
Failed to write event: Invalid argument
Error: Failed to add events.
[root@zoo ~]# perf probe -v vmalloc_fault probe-definition(0): vmalloc_fault
symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/3.19.0-rc6+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/3.19.0-rc6+/build/vmlinux Try to find probe point from debuginfo.
Probe point found: vmalloc_fault+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Added new event:
Writing event: p:probe/vmalloc_fault _text+289600
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[root@zoo ~]# grep -w vmalloc_fault /proc/kallsyms ffffffff81046b40 t vmalloc_fault
[root@zoo ~]# grep -w _text /proc/kallsyms ffffffff81000000 T _text
[root@zoo ~]# python
Python 2.7.5 (default, Nov 3 2014, 14:26:24)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

0xffffffff81046b40 - 0xffffffff81000000

289600L

Quoted 1 line
Quoted 1 line
Quoted 1 line

[root@zoo ~]# readelf -s /lib/modules/3.19.0-rc6+/build/vmlinux | egrep -w _text\|vmalloc_fault

    3499: ffffffff81046b40   410 FUNC    LOCAL  DEFAULT    1 vmalloc_fault
   48873: ffffffff81000000     0 NOTYPE  GLOBAL DEFAULT    1 _text
Quoted 7 lines

[root@zoo ~]#

---------------------------------------------------------------------------

So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?

Ok, actually vmalloc_fault is marked as a nokprobe symbol.

$ grep vmalloc_fault -r arch/x86/
arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address)
arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault);
arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address)
arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault);
arch/x86/mm/fault.c:                    if (vmalloc_fault(address) >= 0)

All the symbols which marked by NOKPROBE_SYMBOL macro can not be kprobed. And we now have <debugfs>/kprobes/blacklist special file to get all nokprobe symbols.
I'll make another perf-probe patch which looks the blacklist before trying define new events.

Thank you,

Quoted 10 lines

Looking for one, will post here after objdump -dS finishes...

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#6 lore raw

Re: Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance

Em Sun, Feb 08, 2015 at 05:37:08PM +0900, Masami Hiramatsu escreveu:

Quoted 14 lines
Quoted 3 lines · Arnaldo Carvalho de Melo

So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized no-inline functions which have only function definition but no actual instance at that point"?

Ok, actually vmalloc_fault is marked as a nokprobe symbol.

$ grep vmalloc_fault -r arch/x86/
arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address)
arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault);
arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address)
arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault);
arch/x86/mm/fault.c:                    if (vmalloc_fault(address) >= 0)
Quoted 2 lines

All the symbols which marked by NOKPROBE_SYMBOL macro can not be kprobed. And we now have <debugfs>/kprobes/blacklist special file

Ok!

Quoted 4 lines

to get all nokprobe symbols.

I'll make another perf-probe patch which looks the blacklist before trying define new events.

Ok, that will help, improving the error messages is a big priority.

- Arnaldo

#7 lore raw

PATCH 1/1 perf-probe: Check kprobes blacklist when adding new events

Recent linux kernel provides a blacklist of the functions which can not be probed. perf probe can now check this blacklist before setting new events and indicate better error message for users.

Without this patch,
----

  # perf probe --add vmalloc_fault
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.
  ----
With this patch
  ----
  # perf probe --add vmalloc_fault
  Added new event:
  Warning: Skipped probing on blacklisted function: vmalloc_fault
  ----
---
 tools/perf/util/probe-event.c |  109 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 108 insertions(+), 1 deletion(-)
tools/perf/util/probe-event.c +108 -1
index 9dfbed9..662d454 100644
@@ -1903,6 +1903,95 @@ static struct strlist *get_probe_trace_command_rawlist(int fd)
 	return sl;
 }
 
+struct kprobe_blacklist_node {
+	struct list_head list;
+	unsigned long start;
+	unsigned long end;
+	char *symbol;
+};
+
+static void kprobe_blacklist__delete(struct list_head *blacklist)
+{
+	struct kprobe_blacklist_node *node;
+
+	while (!list_empty(blacklist)) {
+		node = list_first_entry(blacklist,
+					struct kprobe_blacklist_node, list);
+		list_del(&node->list);
+		free(node->symbol);
+		free(node);
+	}
+}
+
+static int kprobe_blacklist__load(struct list_head *blacklist)
+{
+	struct kprobe_blacklist_node *node;
+	const char *__debugfs = debugfs_find_mountpoint();
+	char buf[PATH_MAX], *p;
+	FILE *fp;
+	int ret;
+
+	if (__debugfs == NULL)
+		return -ENOTSUP;
+
+	ret = e_snprintf(buf, PATH_MAX, "%s/kprobes/blacklist", __debugfs);
+	if (ret < 0)
+		return ret;
+
+	fp = fopen(buf, "r");
+	if (!fp)
+		return -errno;
+
+	ret = 0;
+	while (fgets(buf, PATH_MAX, fp)) {
+		node = zalloc(sizeof(*node));
+		if (!node) {
+			ret = -ENOMEM;
+			break;
+		}
+		INIT_LIST_HEAD(&node->list);
+		list_add_tail(&node->list, blacklist);
+		if (sscanf(buf, "0x%lx-0x%lx", &node->start, &node->end) != 2) {
+			ret = -EINVAL;
+			break;
+		}
+		p = strchr(buf, '\t');
+		if (p) {
+			p++;
+			if (p[strlen(p) - 1] == '\n')
+				p[strlen(p) - 1] = '\0';
+		} else
+			p = (char *)"unknown";
+		node->symbol = strdup(p);
+		if (!node->symbol) {
+			ret = -ENOMEM;
+			break;
+		}
+		pr_debug2("Blacklist: 0x%lx-0x%lx, %s\n",
+			  node->start, node->end, node->symbol);
+		ret++;
+	}
+	if (ret < 0)
+		kprobe_blacklist__delete(blacklist);
+	fclose(fp);
+
+	return ret;
+}
+
+static struct kprobe_blacklist_node *
+kprobe_blacklist__find_by_address(struct list_head *blacklist,
+				  unsigned long address)
+{
+	struct kprobe_blacklist_node *node;
+
+	list_for_each_entry(node, blacklist, list) {
+		if (node->start <= address && address <= node->end)
+			return node;
+	}
+
+	return NULL;
+}
+
 /* Show an event */
 static int show_perf_probe_event(struct perf_probe_event *pev,
 				 const char *module)
@@ -2117,6 +2206,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 	char buf[64];
 	const char *event, *group;
 	struct strlist *namelist;
+	LIST_HEAD(blacklist);
+	struct kprobe_blacklist_node *node;
 
 	if (pev->uprobes)
 		fd = open_uprobe_events(true);
@@ -2134,11 +2225,25 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 		pr_debug("Failed to get current event list.\n");
 		return -EIO;
 	}
+	/* Get kprobe blacklist if exists */
+	if (!pev->uprobes) {
+		ret = kprobe_blacklist__load(&blacklist);
+		if (ret < 0)
+			pr_debug("No kprobe blacklist support, ignored\n");
+	}
 
 	ret = 0;
 	pr_info("Added new event%s\n", (ntevs > 1) ? "s:" : ":");
 	for (i = 0; i < ntevs; i++) {
 		tev = &tevs[i];
+		/* Ensure that the address is NOT blacklisted */
+		node = kprobe_blacklist__find_by_address(&blacklist,
+							 tev->point.address);
+		if (node) {
+			pr_warning("Warning: Skipped probing on blacklisted function: %s\n", node->symbol);
+			continue;
+		}
+
 		if (pev->event)
 			event = pev->event;
 		else
@@ -2189,13 +2294,15 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 		allow_suffix = true;
 	}
 
-	if (ret >= 0) {
+	/* Note that it is possible to skip all events because of blacklist */
+	if (ret >= 0 && tev->event) {
 		/* Show how to use the event. */
 		pr_info("\nYou can now use it in all perf tools, such as:\n\n");
 		pr_info("\tperf record -e %s:%s -aR sleep 1\n\n", tev->group,
 			 tev->event);
 	}
 
+	kprobe_blacklist__delete(&blacklist);
 	strlist__delete(namelist);
 	close(fd);
 	return ret;
#8 lore raw

Re: [PATCH perf/core ] perf-probe: Check kprobes blacklist when adding new events

Em Thu, Feb 19, 2015 at 11:31:13PM +0900, Masami Hiramatsu escreveu:

Quoted 18 lines

Recent linux kernel provides a blacklist of the functions which can not be probed. perf probe can now check this blacklist before setting new events and indicate better error message for users.

Without this patch,
----

  # perf probe --add vmalloc_fault
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.
  ----
With this patch
  ----
  # perf probe --add vmalloc_fault
  Added new event:
  Warning: Skipped probing on blacklisted function: vmalloc_fault
  ----
Quoted 2 lines

Ok, I really need to use a newer kernel on this RHEL7 box, a 4.0-rc one, as when trying this I saw no difference, but probably that is because it didn't find the kprobes blacklist, tried inserting the probe and then failed with the exact previous message...

A follow up patch could try to improve the situation by saying something like:

"Failed to insert the probe, maybe this is a blacklisted function and you're running an old kernel?

Please check the function definition, it should not be marked as, for instance:

NOKPROBE_SYMBOL(vmalloc_fault);"

Anyway, applying your patch, clearly an improvement when the blacklist is available.

Thanks,

- Arnaldo

#9 lore raw

[tip:perf/core] perf probe: Check kprobes blacklist when adding new events

In reply to Masami Hiramatsu
perf probe: Check kprobes blacklist when adding new events
Commit-ID:  9aaf5a5f479bd68699f2e6f6e5e5f1253377b6da
Gitweb:     http://git.kernel.org/tip/9aaf5a5f479bd68699f2e6f6e5e5f1253377b6da
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Thu, 19 Feb 2015 23:31:13 +0900
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Thu, 26 Feb 2015 11:59:05 -0300

perf probe: Check kprobes blacklist when adding new events

Recent linux kernel provides a blacklist of the functions which can not be probed. perf probe can now check this blacklist before setting new events and indicate better error message for users.

Without this patch,
----

  # perf probe --add vmalloc_fault
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.
  ----
With this patch
  ----
  # perf probe --add vmalloc_fault
  Added new event:
  Warning: Skipped probing on blacklisted function: vmalloc_fault
  ----
---
 tools/perf/util/probe-event.c | 109 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 108 insertions(+), 1 deletion(-)
tools/perf/util/probe-event.c +108 -1
index 9dfbed9..662d454 100644
@@ -1903,6 +1903,95 @@ static struct strlist *get_probe_trace_command_rawlist(int fd)
 	return sl;
 }
 
+struct kprobe_blacklist_node {
+	struct list_head list;
+	unsigned long start;
+	unsigned long end;
+	char *symbol;
+};
+
+static void kprobe_blacklist__delete(struct list_head *blacklist)
+{
+	struct kprobe_blacklist_node *node;
+
+	while (!list_empty(blacklist)) {
+		node = list_first_entry(blacklist,
+					struct kprobe_blacklist_node, list);
+		list_del(&node->list);
+		free(node->symbol);
+		free(node);
+	}
+}
+
+static int kprobe_blacklist__load(struct list_head *blacklist)
+{
+	struct kprobe_blacklist_node *node;
+	const char *__debugfs = debugfs_find_mountpoint();
+	char buf[PATH_MAX], *p;
+	FILE *fp;
+	int ret;
+
+	if (__debugfs == NULL)
+		return -ENOTSUP;
+
+	ret = e_snprintf(buf, PATH_MAX, "%s/kprobes/blacklist", __debugfs);
+	if (ret < 0)
+		return ret;
+
+	fp = fopen(buf, "r");
+	if (!fp)
+		return -errno;
+
+	ret = 0;
+	while (fgets(buf, PATH_MAX, fp)) {
+		node = zalloc(sizeof(*node));
+		if (!node) {
+			ret = -ENOMEM;
+			break;
+		}
+		INIT_LIST_HEAD(&node->list);
+		list_add_tail(&node->list, blacklist);
+		if (sscanf(buf, "0x%lx-0x%lx", &node->start, &node->end) != 2) {
+			ret = -EINVAL;
+			break;
+		}
+		p = strchr(buf, '\t');
+		if (p) {
+			p++;
+			if (p[strlen(p) - 1] == '\n')
+				p[strlen(p) - 1] = '\0';
+		} else
+			p = (char *)"unknown";
+		node->symbol = strdup(p);
+		if (!node->symbol) {
+			ret = -ENOMEM;
+			break;
+		}
+		pr_debug2("Blacklist: 0x%lx-0x%lx, %s\n",
+			  node->start, node->end, node->symbol);
+		ret++;
+	}
+	if (ret < 0)
+		kprobe_blacklist__delete(blacklist);
+	fclose(fp);
+
+	return ret;
+}
+
+static struct kprobe_blacklist_node *
+kprobe_blacklist__find_by_address(struct list_head *blacklist,
+				  unsigned long address)
+{
+	struct kprobe_blacklist_node *node;
+
+	list_for_each_entry(node, blacklist, list) {
+		if (node->start <= address && address <= node->end)
+			return node;
+	}
+
+	return NULL;
+}
+
 /* Show an event */
 static int show_perf_probe_event(struct perf_probe_event *pev,
 				 const char *module)
@@ -2117,6 +2206,8 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 	char buf[64];
 	const char *event, *group;
 	struct strlist *namelist;
+	LIST_HEAD(blacklist);
+	struct kprobe_blacklist_node *node;
 
 	if (pev->uprobes)
 		fd = open_uprobe_events(true);
@@ -2134,11 +2225,25 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 		pr_debug("Failed to get current event list.\n");
 		return -EIO;
 	}
+	/* Get kprobe blacklist if exists */
+	if (!pev->uprobes) {
+		ret = kprobe_blacklist__load(&blacklist);
+		if (ret < 0)
+			pr_debug("No kprobe blacklist support, ignored\n");
+	}
 
 	ret = 0;
 	pr_info("Added new event%s\n", (ntevs > 1) ? "s:" : ":");
 	for (i = 0; i < ntevs; i++) {
 		tev = &tevs[i];
+		/* Ensure that the address is NOT blacklisted */
+		node = kprobe_blacklist__find_by_address(&blacklist,
+							 tev->point.address);
+		if (node) {
+			pr_warning("Warning: Skipped probing on blacklisted function: %s\n", node->symbol);
+			continue;
+		}
+
 		if (pev->event)
 			event = pev->event;
 		else
@@ -2189,13 +2294,15 @@ static int __add_probe_trace_events(struct perf_probe_event *pev,
 		allow_suffix = true;
 	}
 
-	if (ret >= 0) {
+	/* Note that it is possible to skip all events because of blacklist */
+	if (ret >= 0 && tev->event) {
 		/* Show how to use the event. */
 		pr_info("\nYou can now use it in all perf tools, such as:\n\n");
 		pr_info("\tperf record -e %s:%s -aR sleep 1\n\n", tev->group,
 			 tev->event);
 	}
 
+	kprobe_blacklist__delete(&blacklist);
 	strlist__delete(namelist);
 	close(fd);
 	return ret;
#10 lore raw

[tip:perf/core] perf probe: Fix to handle optimized not-inlined functions

In reply to Masami Hiramatsu
perf probe: Fix to handle optimized not-inlined functions
Commit-ID:  e1ecbbc3fa834cc6b4b344edb1968e734d57189b
Gitweb:     http://git.kernel.org/tip/e1ecbbc3fa834cc6b4b344edb1968e734d57189b
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Fri, 30 Jan 2015 18:37:44 +0900
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Fri, 6 Feb 2015 11:46:36 +0100

perf probe: Fix to handle optimized not-inlined functions

Fix to handle optimized no-inline functions which have only function definition but no actual instance at that point.

To fix this problem, we need to find actual instance of the function.

Without this patch:
----

  # perf probe -a __up
  Failed to get entry address of __up.
    Error: Failed to add events.
  # perf probe -L __up
  Specified source line is not found.
    Error: Failed to show lines.
  ----

With this patch:
----

  # perf probe -a __up
  Added new event:
    probe:__up           (on __up)

You can now use it in all perf tools, such as:

perf record -e probe:__up -aR sleep 1

  # perf probe -L __up
  <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
        0  static noinline void __sched __up(struct semaphore *sem)
           {
                  struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
                                                          struct semaphore_waite
        4         list_del(&waiter->list);
        5         waiter->up = true;
        6         wake_up_process(waiter->task);
        7  }
  ----
---
 tools/perf/util/dwarf-aux.c    | 15 +++++++++++++++
 tools/perf/util/dwarf-aux.h    |  3 +++
 tools/perf/util/probe-finder.c | 12 ++++--------
 3 files changed, 22 insertions(+), 8 deletions(-)
tools/perf/util/dwarf-aux.c +15 -0
index cc66c40..780b2bc 100644
@@ -278,6 +278,21 @@ bool die_is_func_def(Dwarf_Die *dw_die)
 }
 
 /**
+ * die_is_func_instance - Ensure that this DIE is an instance of a subprogram
+ * @dw_die: a DIE
+ *
+ * Ensure that this DIE is an instance (which has an entry address).
+ * This returns true if @dw_die is a function instance. If not, you need to
+ * call die_walk_instances() to find actual instances.
+ **/
+bool die_is_func_instance(Dwarf_Die *dw_die)
+{
+	Dwarf_Addr tmp;
+
+	/* Actually gcc optimizes non-inline as like as inlined */
+	return !dwarf_func_inline(dw_die) && dwarf_entrypc(dw_die, &tmp) == 0;
+}
+/**
  * die_get_data_member_location - Get the data-member offset
  * @mb_die: a DIE of a member of a data structure
  * @offs: The offset of the member in the data structure
tools/perf/util/dwarf-aux.h +3 -0
index b4fe90c..af7dbcd 100644
@@ -41,6 +41,9 @@ extern int cu_walk_functions_at(Dwarf_Die *cu_die, Dwarf_Addr addr,
 /* Ensure that this DIE is a subprogram and definition (not declaration) */
 extern bool die_is_func_def(Dwarf_Die *dw_die);
 
+/* Ensure that this DIE is an instance of a subprogram */
+extern bool die_is_func_instance(Dwarf_Die *dw_die);
+
 /* Compare diename and tname */
 extern bool die_compare_name(Dwarf_Die *dw_die, const char *tname);
 
tools/perf/util/probe-finder.c +4 -8
index b5247d7..d141935 100644
@@ -915,17 +915,13 @@ static int probe_point_search_cb(Dwarf_Die *sp_die, void *data)
 		dwarf_decl_line(sp_die, &pf->lno);
 		pf->lno += pp->line;
 		param->retval = find_probe_point_by_line(pf);
-	} else if (!dwarf_func_inline(sp_die)) {
+	} else if (die_is_func_instance(sp_die)) {
+		/* Instances always have the entry address */
+		dwarf_entrypc(sp_die, &pf->addr);
 		/* Real function */
 		if (pp->lazy_line)
 			param->retval = find_probe_point_lazy(sp_die, pf);
 		else {
-			if (dwarf_entrypc(sp_die, &pf->addr) != 0) {
-				pr_warning("Failed to get entry address of "
-					   "%s.\n", dwarf_diename(sp_die));
-				param->retval = -ENOENT;
-				return DWARF_CB_ABORT;
-			}
 			pf->addr += pp->offset;
 			/* TODO: Check the address in this function */
 			param->retval = call_probe_finder(sp_die, pf);
@@ -1536,7 +1532,7 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)
 		pr_debug("New line range: %d to %d\n", lf->lno_s, lf->lno_e);
 		lr->start = lf->lno_s;
 		lr->end = lf->lno_e;
-		if (dwarf_func_inline(sp_die))
+		if (!die_is_func_instance(sp_die))
 			param->retval = die_walk_instances(sp_die,
 						line_range_inline_cb, lf);
 		else
#11 lore raw

PATCH 2/2 perf-probe: Update perf-probe document

In reply to Masami Hiramatsu

Update Documentation/perf-probe.txt to add descriptions of some newer options.

---
 tools/perf/Documentation/perf-probe.txt |   16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-probe.txt +14 -2
index aaa869b..239609c 100644
@@ -47,6 +47,12 @@ OPTIONS
 -v::
 --verbose::
         Be more verbose (show parsed arguments, etc).
+	Can not use with -q.
+
+-q::
+--quiet::
+	Be quiet (do not show any messages including errors).
+	Can not use with -v.
 
 -a::
 --add=::
@@ -96,7 +102,7 @@ OPTIONS
 	Dry run. With this option, --add and --del doesn't execute actual
 	adding and removal operations.
 
---max-probes::
+--max-probes=NUM::
 	Set the maximum number of probe points for an event. Default is 128.
 
 -x::
@@ -104,8 +110,13 @@ OPTIONS
 	Specify path to the executable or shared library file for user
 	space tracing. Can also be used with --funcs option.
 
+--demangle::
+	Demangle application symbols. --no-demangle is also available
+	for disabling demangling.
+
 --demangle-kernel::
-	Demangle kernel symbols.
+	Demangle kernel symbols. --no-demangle-kernel is also available
+	for disabling kernel demangling.
 
 In absence of -m/-x options, perf probe checks if the first argument after
 the options is an absolute path name. If its an absolute path, perf probe
@@ -137,6 +148,7 @@ Each probe argument follows below syntax.
  [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE]
 
 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.)
+'$vars' special argument is also available for NAME, it is expanded to the local variables which can access at given probe point.
 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo. You can specify 'string' type only for the local variable or structure member which is an array of or a pointer to 'char' or 'unsigned char' type.
 
 On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid.
#12 lore raw

[tip:perf/core] perf probe: Update man page

perf probe: Update man page
Commit-ID:  8b72805fd1dbfd697c5d4492d0cf1ebbd994950d
Gitweb:     http://git.kernel.org/tip/8b72805fd1dbfd697c5d4492d0cf1ebbd994950d
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Fri, 30 Jan 2015 18:37:46 +0900
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Fri, 6 Feb 2015 11:46:36 +0100

perf probe: Update man page

Update Documentation/perf-probe.txt to add descriptions of some newer options.

---
 tools/perf/Documentation/perf-probe.txt | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-probe.txt +14 -2
index aaa869b..239609c 100644
@@ -47,6 +47,12 @@ OPTIONS
 -v::
 --verbose::
         Be more verbose (show parsed arguments, etc).
+	Can not use with -q.
+
+-q::
+--quiet::
+	Be quiet (do not show any messages including errors).
+	Can not use with -v.
 
 -a::
 --add=::
@@ -96,7 +102,7 @@ OPTIONS
 	Dry run. With this option, --add and --del doesn't execute actual
 	adding and removal operations.
 
---max-probes::
+--max-probes=NUM::
 	Set the maximum number of probe points for an event. Default is 128.
 
 -x::
@@ -104,8 +110,13 @@ OPTIONS
 	Specify path to the executable or shared library file for user
 	space tracing. Can also be used with --funcs option.
 
+--demangle::
+	Demangle application symbols. --no-demangle is also available
+	for disabling demangling.
+
 --demangle-kernel::
-	Demangle kernel symbols.
+	Demangle kernel symbols. --no-demangle-kernel is also available
+	for disabling kernel demangling.
 
 In absence of -m/-x options, perf probe checks if the first argument after
 the options is an absolute path name. If its an absolute path, perf probe
@@ -137,6 +148,7 @@ Each probe argument follows below syntax.
  [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE]
 
 'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), local array with fixed index (e.g. array[1], var->array[0], var->pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.)
+'$vars' special argument is also available for NAME, it is expanded to the local variables which can access at given probe point.
 'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo. You can specify 'string' type only for the local variable or structure member which is an array of or a pointer to 'char' or 'unsigned char' type.
 
 On x86 systems %REG is always the short form of the register: for example %AX. %RAX or %EAX is not valid.