Skip to contentSkip to conversation

perf probe can't insert return kprobe

BUG Jiri Olsa 5 participants 10 messages Jan 9, 2015 – Jan 12, 2015

lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

thanks,
jirka

#1 lore raw

Em Fri, Jan 09, 2015 at 03:55:39PM +0100, Jiri Olsa escreveu:

Quoted 7 lines

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

Wonder if we have any way of forcing 'perf probe' to show what it will feed the kernel, perhaps -v?

Quoted 16 lines

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

thanks,
jirka

#2 lore raw
Quoted 11 lines · Arnaldo Carvalho de Melo

Em Fri, Jan 09, 2015 at 03:55:39PM +0100, Jiri Olsa escreveu:

Quoted 7 lines

hi,
I couldn't use following perf command to insert return probe:

   # perf probe -a fork_exit=do_fork%return
   Added new event:
   Failed to write event: Invalid argument
     Error: Failed to add events.

Wonder if we have any way of forcing 'perf probe' to show what it will feed the kernel, perhaps -v?

# strace -e write perf probe -a fork_exit=do_fork%return
write(1, "Added new event:\n", 17Added new event:
)      = 17
write(3, "r:probe/fork_exit do_fork+0", 27) = 27

Would be nice to add the output to perf -v though.

David

#3 lore raw
In reply to Jiri Olsa
Quoted 12 lines · Jiri Olsa

hi,
I couldn't use following perf command to insert return probe:

   # perf probe -a fork_exit=do_fork%return
   Added new event:
   Failed to write event: Invalid argument
     Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

Quoted 9 lines
   # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

   # uname -a
   Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
   # ./perf version
   perf version 3.18.g6a7d78

If you want to bisect it the perf-probe command works for me on FC20, 3.17.7-200.fc20.x86_64 with perf from 3.12 (my local modified version is based on 3.12).

David

#4 lore raw
In reply to Jiri Olsa
Quoted 12 lines · Jiri Olsa

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

Quoted 9 lines
  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

Is it just return probe? Did it work for normal kprobes? Maybe it's related to the below:

https://lkml.org/lkml/2014/12/31/15

Have you check the acme/perf/urgent too?

Thanks,
Namhyung

#5 lore raw
Quoted 6 lines · Namhyung Kim

Is it just return probe? Did it work for normal kprobes? Maybe it's related to the below:

https://lkml.org/lkml/2014/12/31/15

Have you check the acme/perf/urgent too?

I believe that commit impacts uprobes only.

David

author
#6 lore raw
In reply to Namhyung Kim
Quoted 17 lines · Namhyung Kim
Quoted 16 lines · Jiri Olsa

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

Quoted 7 lines
Quoted 5 lines
  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

Is it just return probe? Did it work for normal kprobes?

yep, works for normal probes

Quoted 5 lines

Maybe it's related to the below:

https://lkml.org/lkml/2014/12/31/15

Have you check the acme/perf/urgent too?

hum.. can't access lkml, I'll check, also with perf/urgent

thanks,
jirka

author
#7 lore raw
Quoted 28 lines · Jiri Olsa
Quoted 17 lines · Namhyung Kim
Quoted 16 lines · Jiri Olsa

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

Quoted 7 lines
Quoted 5 lines
  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

Is it just return probe? Did it work for normal kprobes?

yep, works for normal probes

Quoted 7 lines
Quoted 5 lines

Maybe it's related to the below:

https://lkml.org/lkml/2014/12/31/15

Have you check the acme/perf/urgent too?

hum.. can't access lkml, I'll check, also with perf/urgent

neither helped..

jirka

#8 lore raw

Hi Jiri,

Quoted 31 lines · Jiri Olsa
Quoted 30 lines · Jiri Olsa
Quoted 17 lines · Namhyung Kim
Quoted 16 lines · Jiri Olsa

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

Quoted 7 lines
Quoted 5 lines
  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

Is it just return probe? Did it work for normal kprobes?

yep, works for normal probes

Quoted 2 lines

Maybe it's related to the below:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

https://lkml.org/lkml/2014/12/31/15

Have you check the acme/perf/urgent too?

hum.. can't access lkml, I'll check, also with perf/urgent

neither helped..

I think I've found the reason.

The commit dfef99cd0b2c ("perf probe: Use ref_reloc_sym based address instead of the symbol name") converts kprobes to use ref_reloc_sym (i.e. _stext) and offset instead of using symbol's name directly. So on my system, adding do_fork ends up with like below:

  $ sudo perf probe -v --add do_fork%return
  probe-definition(0): do_fork%return
  symbol:do_fork file:(null) line:0 offset:0 return:1 lazy:(null)
  0 arguments
  Looking at the vmlinux_path (7 entries long)
  Using /lib/modules/3.17.6-1-ARCH/build/vmlinux for symbols
  Could not open debuginfo. Try to use symbols.
  Opening /sys/kernel/debug/tracing/kprobe_events write=1
  Added new event:
  Writing event: r:probe/do_fork _stext+456136
  Failed to write event: Invalid argument
    Error: Failed to add events. Reason: Operation not permitted (Code: -1)

As you can see, the do_fork was translated to _stext+456136. This was because to support (local) symbols that have same name. But the problem is that kretprobe requires to be inserted at function start point so it simply checks whether it's called with offset 0. And if not, it'll return with -EINVAL. You can see it with dmesg.

  $ dmesg | tail -1
  [125621.764103] Return probe must be used without offset.

So we need to use the symbol name instead of ref_reloc_sym in case of return probes. During the tracking down, I found a couple of problems in the code. I'll send fixes soon.

Thanks,
Namhyung

#9 lore raw
Quoted 69 lines · Namhyung Kim

Hi Jiri,

Quoted 31 lines · Jiri Olsa
Quoted 30 lines · Jiri Olsa
Quoted 17 lines · Namhyung Kim
Quoted 16 lines · Jiri Olsa

hi,
I couldn't use following perf command to insert return probe:

  # perf probe -a fork_exit=do_fork%return
  Added new event:
  Failed to write event: Invalid argument
    Error: Failed to add events.

I'm pretty sure I used this command before, so seems like it's broken. I can still use debugfs tracing interface to do that:

  # echo 'r:do_fork_entry do_fork' > kprobe_events

I used Arnaldo's latest perf/core and FC20 kernel:

Quoted 7 lines
Quoted 5 lines
  # uname -a
  Linux krava 3.17.7-200.fc20.x86_64 #1 SMP Wed Dec 17 03:35:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  # ./perf version
  perf version 3.18.g6a7d78

Is it just return probe? Did it work for normal kprobes?

yep, works for normal probes

Quoted 2 lines

Maybe it's related to the below:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

https://lkml.org/lkml/2014/12/31/15

Have you check the acme/perf/urgent too?

hum.. can't access lkml, I'll check, also with perf/urgent

neither helped..

I think I've found the reason.

The commit dfef99cd0b2c ("perf probe: Use ref_reloc_sym based address instead of the symbol name") converts kprobes to use ref_reloc_sym (i.e. _stext) and offset instead of using symbol's name directly. So on my system, adding do_fork ends up with like below:

  $ sudo perf probe -v --add do_fork%return
  probe-definition(0): do_fork%return
  symbol:do_fork file:(null) line:0 offset:0 return:1 lazy:(null)
  0 arguments
  Looking at the vmlinux_path (7 entries long)
  Using /lib/modules/3.17.6-1-ARCH/build/vmlinux for symbols
  Could not open debuginfo. Try to use symbols.
  Opening /sys/kernel/debug/tracing/kprobe_events write=1
  Added new event:
  Writing event: r:probe/do_fork _stext+456136
  Failed to write event: Invalid argument
    Error: Failed to add events. Reason: Operation not permitted (Code: -1)

As you can see, the do_fork was translated to _stext+456136. This was because to support (local) symbols that have same name. But the problem is that kretprobe requires to be inserted at function start point so it simply checks whether it's called with offset 0. And if not, it'll return with -EINVAL. You can see it with dmesg.

Quoted 6 lines
  $ dmesg | tail -1
  [125621.764103] Return probe must be used without offset.

So we need to use the symbol name instead of ref_reloc_sym in case of return probes. During the tracking down, I found a couple of problems in the code. I'll send fixes soon.

Oops, thank you for analyzing the problem:) I've completely forgot about return probes with KASLR. This also should be fixed in kprobe-tracer, which should convert given _stext+offset to actual symbol in kernel.

Thanks,

Quoted 4 lines

Thanks,
Namhyung

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]