uprobes support

Linus,

Please consider pulling the new 'user-space probes (uprobes)' instrumentation feature from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf-uprobes-for-linus

HEAD: 9cba26e66d09bf394ae5a739627a1dc8b7cae6f4 Merge branch 'perf/uprobes' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/uprobes

Questions were raised by you and Andrew two days ago, in this discussion:

[tip:perf/uprobes] uprobes, mm, x86: Add the ability to install and remove uprobes breakpoints

Srikar and Oleg addressed them - but if there's more to do please let us know.

The uprobes code originates from SystemTap and has been used for years in Fedora and RHEL kernels. This version is much rewritten, reviews from PeterZ, Oleg and myself shaped the end result.

This tree includes uprobes support in 'perf probe' - but SystemTap (and other tools) can take advantage of user probe points as well.

Sample usage of uprobes via perf, for example to profile malloc() calls without modifying user-space binaries.

First boot a new kernel with CONFIG_UPROBE_EVENT=y enabled.

If you don't know which function you want to probe you can pick one from 'perf top' or can get a list all functions that can be probed within libc (binaries can be specified as well):

  $ perf probe -F -x /lib/libc.so.6

To probe libc's malloc():

  $ perf probe -x /lib64/libc.so.6 malloc
  Added new event:
  probe_libc:malloc    (on 0x7eac0)

You can now use it in all perf tools, such as:

perf record -e probe_libc:malloc -aR sleep 1

Make use of it to create a call graph (as the flat profile is going to look very boring):

  $ perf record -e probe_libc:malloc -gR make
  [ perf record: Woken up 173 times to write data ]
  [ perf record: Captured and wrote 44.190 MB perf.data (~1930712 

  $ perf report | less

    32.03%            git  libc-2.15.so   [.] malloc
                      |
                      --- malloc

    29.49%            cc1  libc-2.15.so   [.] malloc
                      |
                      --- malloc
                         |          
                         |--0.95%-- 0x208eb1000000000
                         |          
                         |--0.63%-- htab_traverse_noresize


    11.04%             as  libc-2.15.so   [.] malloc
                       |
                       --- malloc
                          |          

     7.15%             ld  libc-2.15.so   [.] malloc
                       |
                       --- malloc
                          |          

     5.07%             sh  libc-2.15.so   [.] malloc
                       |
                       --- malloc
                          |          
     4.99%  python-config  libc-2.15.so   [.] malloc
            |
            --- malloc
               |          
     4.54%           make  libc-2.15.so   [.] malloc
                     |
                     --- malloc
                        |          
                        |--7.34%-- glob
                        |          |          
                        |          |--93.18%-- 0x41588f
                        |          |          
                        |           --6.82%-- glob
                        |                     0x41588f

     ...

Or:

  $ perf report -g flat | less

# Overhead Command Shared Object Symbol

# ........  .............  .............  ..........
#
    32.03%            git  libc-2.15.so   [.] malloc
            27.19%
                malloc

    29.49%            cc1  libc-2.15.so   [.] malloc
            24.77%
                malloc

    11.04%             as  libc-2.15.so   [.] malloc
            11.02%
                malloc

     7.15%             ld  libc-2.15.so   [.] malloc
             6.57%
                malloc

...

The core uprobes design is fairly straightforward: uprobes probe points register themselves at (inode:offset) addresses of libraries/binaries, after which all existing (or new) vmas that map that address will have a software breakpoint injected at that address. vmas are COW-ed to preserve original content. The probe points are kept in an rbtree.

If user-space executes the probed inode:offset instruction address then an event is generated which can be recovered from the regular perf event channels and mmap-ed ring-buffer.

Multiple probes at the same address are supported, they create a dynamic callback list of event consumers.

The basic model is further complicated by the XOL speedup: the original instruction that is probed is copied (in an architecture specific fashion) and executed out of line when the probe triggers. The XOL area is a single vma per process, with a fixed number of entries (which limits probe execution parallelism).

The API: uprobes are installed/removed via /sys/kernel/debug/tracing/uprobe_events, the API is integrated to align with the kprobes interface as much as possible, but is separate to it.

Injecting a probe point is privileged operation, which can be relaxed by setting perf_paranoid to -1.

You can use multiple probes as well and mix them with kprobes and regular PMU events or tracepoints, when instrumenting a task.

Thanks,

Ingo

------------------>
Ingo Molnar (3):
uprobes/core: Clean up, refactor and improve the code
uprobes: Move to kernel/events/
uprobes: Update copyright notices

Srikar Dronamraju (18):
uprobes, mm, x86: Add the ability to install and remove uprobes breakpoints uprobes/core: Make instruction tables volatile
uprobes/core: Remove uprobe_opcode_sz
uprobes/core: Move insn to arch specific structure
uprobes/core: Make macro names consistent
uprobes/core: Make order of function parameters consistent across functions uprobes/core: Rename bkpt to swbp
uprobes/core: Handle breakpoint and singlestep exceptions uprobes/core: Allocate XOL slots for uprobes use
uprobes/core: Optimize probe hits with the help of a counter uprobes/core: Make background page replacement logic account for rss_stat counters uprobes/core: Decrement uprobe count before the pages are unmapped
tracing: Modify is_delete, is_return from int to bool
tracing: Extract out common code for kprobes/uprobes trace events
tracing: Provide trace events interface for uprobes
tracing: Fix kconfig warning due to a typo
perf probe: Provide perf interface for uprobes
perf probe: Detect probe target when m/x options are absent

 Documentation/trace/uprobetracer.txt    |  113 +++
 arch/Kconfig                            |   17 +
 arch/x86/Kconfig                        |    5 +-
 arch/x86/include/asm/thread_info.h      |    2 +
 arch/x86/include/asm/uprobes.h          |   57 ++
 arch/x86/kernel/Makefile                |    1 +
 arch/x86/kernel/signal.c                |    6 +
 arch/x86/kernel/uprobes.c               |  674 +++++++++++++
 include/linux/mm_types.h                |    2 +
 include/linux/sched.h                   |    4 +
 include/linux/uprobes.h                 |  165 +++
 kernel/events/Makefile                  |    3 +
 kernel/events/uprobes.c                 | 1667 +++++++++++++++++++++++++++++++
 kernel/fork.c                           |    9 +
 kernel/signal.c                         |    4 +
 kernel/trace/Kconfig                    |   20 +
 kernel/trace/Makefile                   |    2 +
 kernel/trace/trace.h                    |    5 +
 kernel/trace/trace_kprobe.c             |  899 +----------------
 kernel/trace/trace_probe.c              |  839 ++++++++++++++++
 kernel/trace/trace_probe.h              |  161 +++
 kernel/trace/trace_uprobe.c             |  788 +++++++++++++++
 mm/memory.c                             |    3 +
 mm/mmap.c                               |   33 +-
 tools/perf/Documentation/perf-probe.txt |   19 +-
 tools/perf/builtin-probe.c              |   86 +-
 tools/perf/util/probe-event.c           |  422 ++++++--
 tools/perf/util/probe-event.h           |   12 +-
 tools/perf/util/symbol.c                |    8 +
 tools/perf/util/symbol.h                |    1 +
 30 files changed, 5048 insertions(+), 979 deletions(-)
 create mode 100644 Documentation/trace/uprobetracer.txt
 create mode 100644 arch/x86/include/asm/uprobes.h
 create mode 100644 arch/x86/kernel/uprobes.c
 create mode 100644 include/linux/uprobes.h
 create mode 100644 kernel/events/uprobes.c
 create mode 100644 kernel/trace/trace_probe.c
 create mode 100644 kernel/trace/trace_probe.h
 create mode 100644 kernel/trace/trace_uprobe.c

new file mode 100644
index 0000000..24ce682
@@ -0,0 +1,113 @@
+		Uprobe-tracer: Uprobe-based Event Tracing
+		=========================================
+                 Documentation written by Srikar Dronamraju
+
+Overview
+--------
+Uprobe based trace events are similar to kprobe based trace events.
+To enable this feature, build your kernel with CONFIG_UPROBE_EVENT=y.
+
+Similar to the kprobe-event tracer, this doesn't need to be activated via
+current_tracer. Instead of that, add probe points via
+/sys/kernel/debug/tracing/uprobe_events, and enable it via
+/sys/kernel/debug/tracing/events/uprobes/<EVENT>/enabled.
+
+However unlike kprobe-event tracer, the uprobe event interface expects the
+user to calculate the offset of the probepoint in the object
+
+Synopsis of uprobe_tracer
+-------------------------
+  p[:[GRP/]EVENT] PATH:SYMBOL[+offs] [FETCHARGS]	: Set a probe
+
+ GRP		: Group name. If omitted, use "uprobes" for it.
+ EVENT		: Event name. If omitted, the event name is generated
+		  based on SYMBOL+offs.
+ PATH		: path to an executable or a library.
+ SYMBOL[+offs]	: Symbol+offset where the probe is inserted.
+
+ FETCHARGS	: Arguments. Each probe can have up to 128 args.
+  %REG		: Fetch register REG
+
+Event Profiling
+---------------
+ You can check the total number of probe hits and probe miss-hits via
+/sys/kernel/debug/tracing/uprobe_profile.
+ The first column is event name, the second is the number of probe hits,
+the third is the number of probe miss-hits.
+
+Usage examples
+--------------
+To add a probe as a new event, write a new definition to uprobe_events
+as below.
+
+  echo 'p: /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events
+
+ This sets a uprobe at an offset of 0x4245c0 in the executable /bin/bash
+
+  echo > /sys/kernel/debug/tracing/uprobe_events
+
+ This clears all probe points.
+
+The following example shows how to dump the instruction pointer and %ax
+a register at the probed text address.  Here we are trying to probe
+function zfree in /bin/zsh
+
+    # cd /sys/kernel/debug/tracing/
+    # cat /proc/`pgrep  zsh`/maps | grep /bin/zsh | grep r-xp
+    00400000-0048a000 r-xp 00000000 08:03 130904 /bin/zsh
+    # objdump -T /bin/zsh | grep -w zfree
+    0000000000446420 g    DF .text  0000000000000012  Base        zfree
+
+0x46420 is the offset of zfree in object /bin/zsh that is loaded at
+0x00400000. Hence the command to probe would be :
+
+    # echo 'p /bin/zsh:0x46420 %ip %ax' > uprobe_events
+
+Please note: User has to explicitly calculate the offset of the probepoint
+in the object. We can see the events that are registered by looking at the
+uprobe_events file.
+
+    # cat uprobe_events
+    p:uprobes/p_zsh_0x46420 /bin/zsh:0x00046420 arg1=%ip arg2=%ax
+
+The format of events can be seen by viewing the file events/uprobes/p_zsh_0x46420/format
+
+    # cat events/uprobes/p_zsh_0x46420/format
+    name: p_zsh_0x46420
+    ID: 922
+    format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+	field:int common_padding;	offset:8;	size:4;	signed:1;
+
+	field:unsigned long __probe_ip;	offset:12;	size:4;	signed:0;
+	field:u32 arg1;	offset:16;	size:4;	signed:0;
+	field:u32 arg2;	offset:20;	size:4;	signed:0;
+
+    print fmt: "(%lx) arg1=%lx arg2=%lx", REC->__probe_ip, REC->arg1, REC->arg2
+
+Right after definition, each event is disabled by default. For tracing these
+events, you need to enable it by:
+
+    # echo 1 > events/uprobes/enable
+
+Lets disable the event after sleeping for some time.
+    # sleep 20
+    # echo 0 > events/uprobes/enable
+
+And you can see the traced information via /sys/kernel/debug/tracing/trace.
+
+    # cat trace
+    # tracer: nop
+    #
+    #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
+    #              | |       |          |         |
+                 zsh-24842 [006] 258544.995456: p_zsh_0x46420: (0x446420) arg1=446421 arg2=79
+                 zsh-24842 [007] 258545.000270: p_zsh_0x46420: (0x446420) arg1=446421 arg2=79
+                 zsh-24842 [002] 258545.043929: p_zsh_0x46420: (0x446420) arg1=446421 arg2=79
+                 zsh-24842 [004] 258547.046129: p_zsh_0x46420: (0x446420) arg1=446421 arg2=79
+
+Each line shows us probes were triggered for a pid 24842 with ip being
+0x446421 and contents of ax register being 79.

index 684eb5a..2880abf 100644
@@ -76,6 +76,23 @@ config OPTPROBES
 	depends on KPROBES && HAVE_OPTPROBES
 	depends on !PREEMPT
 
+config UPROBES
+	bool "Transparent user-space probes (EXPERIMENTAL)"
+	depends on UPROBE_EVENT && PERF_EVENTS
+	default n
+	help
+	  Uprobes is the user-space counterpart to kprobes: they
+	  enable instrumentation applications (such as 'perf probe')
+	  to establish unintrusive probes in user-space binaries and
+	  libraries, by executing handler functions when the probes
+	  are hit by user-space applications.
+
+	  ( These probes come in the form of single-byte breakpoints,
+	    managed by the kernel and kept transparent to the probed
+	    application. )
+
+	  If in doubt, say "N".
+
 config HAVE_EFFICIENT_UNALIGNED_ACCESS
 	bool
 	help

index 1324139..8443c50 100644
@@ -83,7 +83,7 @@ config X86
 	select DCACHE_WORD_ACCESS if !DEBUG_PAGEALLOC
 
 config INSTRUCTION_DECODER
-	def_bool (KPROBES || PERF_EVENTS)
+	def_bool (KPROBES || PERF_EVENTS || UPROBES)
 
 config OUTPUT_FORMAT
 	string

@@ -242,6 +242,9 @@ config ARCH_CPU_PROBE_RELEASE
 	def_bool y
 	depends on HOTPLUG_CPU
 
+config ARCH_SUPPORTS_UPROBES
+	def_bool y
+
 source "init/Kconfig"
 source "kernel/Kconfig.freezer"
 

index ad6df8c..0710c11 100644
@@ -85,6 +85,7 @@ struct thread_info {
 #define TIF_SECCOMP		8	/* secure computing */
 #define TIF_MCE_NOTIFY		10	/* notify userspace of an MCE */
 #define TIF_USER_RETURN_NOTIFY	11	/* notify kernel of userspace return */
+#define TIF_UPROBE		12	/* breakpointed or singlestepping */
 #define TIF_NOTSC		16	/* TSC is not accessible in userland */
 #define TIF_IA32		17	/* IA32 compatibility process */
 #define TIF_FORK		18	/* ret_from_fork */

@@ -109,6 +110,7 @@ struct thread_info {
 #define _TIF_SECCOMP		(1 << TIF_SECCOMP)
 #define _TIF_MCE_NOTIFY		(1 << TIF_MCE_NOTIFY)
 #define _TIF_USER_RETURN_NOTIFY	(1 << TIF_USER_RETURN_NOTIFY)
+#define _TIF_UPROBE		(1 << TIF_UPROBE)
 #define _TIF_NOTSC		(1 << TIF_NOTSC)
 #define _TIF_IA32		(1 << TIF_IA32)
 #define _TIF_FORK		(1 << TIF_FORK)

index 532d2e0..d23d835 100644
@@ -101,6 +101,7 @@ obj-$(CONFIG_X86_CHECK_BIOS_CORRUPTION) += check.o
 
 obj-$(CONFIG_SWIOTLB)			+= pci-swiotlb.o
 obj-$(CONFIG_OF)			+= devicetree.o
+obj-$(CONFIG_UPROBES)			+= uprobes.o
 
 ###
 # 64 bit specific files

index 115eac4..041af2f 100644
@@ -18,6 +18,7 @@
 #include <linux/personality.h>
 #include <linux/uaccess.h>
 #include <linux/user-return-notifier.h>
+#include <linux/uprobes.h>
 
 #include <asm/processor.h>
 #include <asm/ucontext.h>

@@ -824,6 +825,11 @@ do_notify_resume(struct pt_regs *regs, void *unused, __u32 thread_info_flags)
 		mce_notify_process();
 #endif /* CONFIG_X86_64 && CONFIG_X86_MCE */
 
+	if (thread_info_flags & _TIF_UPROBE) {
+		clear_thread_flag(TIF_UPROBE);
+		uprobe_notify_resume(regs);
+	}
+
 	/* deal with pending signal delivery */
 	if (thread_info_flags & _TIF_SIGPENDING)
 		do_signal(regs);

index 3cc3062..26574c7 100644
@@ -12,6 +12,7 @@
 #include <linux/completion.h>
 #include <linux/cpumask.h>
 #include <linux/page-debug-flags.h>
+#include <linux/uprobes.h>
 #include <asm/page.h>
 #include <asm/mmu.h>
 

@@ -388,6 +389,7 @@ struct mm_struct {
 #ifdef CONFIG_CPUMASK_OFFSTACK
 	struct cpumask cpumask_allocation;
 #endif
+	struct uprobes_state uprobes_state;
 };
 
 static inline void mm_init_cpumask(struct mm_struct *mm)

index 81a173c..cff94cd 100644
@@ -1617,6 +1617,10 @@ struct task_struct {
 #ifdef CONFIG_HAVE_HW_BREAKPOINT
 	atomic_t ptrace_bp_refcnt;
 #endif
+#ifdef CONFIG_UPROBES
+	struct uprobe_task *utask;
+	int uprobe_srcu_id;
+#endif
 };
 
 /* Future-safe accessor for struct task_struct's cpus_allowed. */

index 22d901f..103f5d1 100644
@@ -3,4 +3,7 @@ CFLAGS_REMOVE_core.o = -pg
 endif
 
 obj-y := core.o ring_buffer.o callchain.o
+
 obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
+obj-$(CONFIG_UPROBES) += uprobes.o
+

index b9372a0..ca9a384 100644
@@ -67,6 +67,7 @@
 #include <linux/oom.h>
 #include <linux/khugepaged.h>
 #include <linux/signalfd.h>
+#include <linux/uprobes.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>

@@ -421,6 +422,9 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 
 		if (retval)
 			goto out;
+
+		if (file && uprobe_mmap(tmp))
+			goto out;
 	}
 	/* a new mm has just been created */
 	arch_dup_mmap(oldmm, mm);

@@ -569,6 +573,7 @@ void mmput(struct mm_struct *mm)
 	might_sleep();
 
 	if (atomic_dec_and_test(&mm->mm_users)) {
+		uprobe_clear_state(mm);
 		exit_aio(mm);
 		ksm_exit(mm);
 		khugepaged_exit(mm); /* must run before exit_mmap */

@@ -747,6 +752,8 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm)
 		exit_pi_state_list(tsk);
 #endif
 
+	uprobe_free_utask(tsk);
+
 	/* Get rid of any cached register state */
 	deactivate_mm(tsk, mm);
 

@@ -801,6 +808,7 @@ struct mm_struct *dup_mm(struct task_struct *tsk)
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	mm->pmd_huge_pte = NULL;
 #endif
+	uprobe_reset_state(mm);
 
 	if (!mm_init(mm, tsk))
 		goto fail_nomem;

@@ -1342,6 +1350,7 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 	INIT_LIST_HEAD(&p->pi_state_list);
 	p->pi_state_cache = NULL;
 #endif
+	uprobe_copy_process(p);
 	/*
 	 * sigaltstack should be cleared when sharing the same VM
 	 */

index 17afcaf..60d80ab 100644
@@ -29,6 +29,7 @@
 #include <linux/pid_namespace.h>
 #include <linux/nsproxy.h>
 #include <linux/user_namespace.h>
+#include <linux/uprobes.h>
 #define CREATE_TRACE_POINTS
 #include <trace/events/signal.h>
 

@@ -2202,6 +2203,9 @@ int get_signal_to_deliver(siginfo_t *info, struct k_sigaction *return_ka,
 	struct signal_struct *signal = current->signal;
 	int signr;
 
+	if (unlikely(uprobe_deny_signal()))
+		return 0;
+
 relock:
 	/*
 	 * We'll jump back here after any time we were stopped in TASK_STOPPED.

index a1d2849..ea4bff6 100644
@@ -373,6 +373,7 @@ config KPROBE_EVENT
 	depends on HAVE_REGS_AND_STACK_ACCESS_API
 	bool "Enable kprobes-based dynamic events"
 	select TRACING
+	select PROBE_EVENTS
 	default y
 	help
 	  This allows the user to add tracing events (similar to tracepoints)

@@ -385,6 +386,25 @@ config KPROBE_EVENT
 	  This option is also required by perf-probe subcommand of perf tools.
 	  If you want to use perf tools, this option is strongly recommended.
 
+config UPROBE_EVENT
+	bool "Enable uprobes-based dynamic events"
+	depends on ARCH_SUPPORTS_UPROBES
+	depends on MMU
+	select UPROBES
+	select PROBE_EVENTS
+	select TRACING
+	default n
+	help
+	  This allows the user to add tracing events on top of userspace
+	  dynamic events (similar to tracepoints) on the fly via the trace
+	  events interface. Those events can be inserted wherever uprobes
+	  can probe, and record various registers.
+	  This option is required if you plan to use perf-probe subcommand
+	  of perf tools on user space applications.
+
+config PROBE_EVENTS
+	def_bool n
+
 config DYNAMIC_FTRACE
 	bool "enable/disable ftrace tracepoints dynamically"
 	depends on FUNCTION_TRACER

index 5f39a07..1734c03 100644
@@ -61,5 +61,7 @@ endif
 ifeq ($(CONFIG_TRACING),y)
 obj-$(CONFIG_KGDB_KDB) += trace_kdb.o
 endif
+obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o
+obj-$(CONFIG_UPROBE_EVENT) += trace_uprobe.o
 
 libftrace-y := ftrace.o

index 1c8b7c6..a7d28e0 100644
@@ -103,6 +103,11 @@ struct kretprobe_trace_entry_head {
 	unsigned long		ret_ip;
 };
 
+struct uprobe_trace_entry_head {
+	struct trace_entry	ent;
+	unsigned long		ip;
+};
+
 /*
  * trace_flag_type is an enumeration that holds different
  * states when a trace occurs. These are:

index 6105f47..bf8b403 100644

index 848ef52..b8c4072 100644

index 2780d9c..b715cb7 100644

index a7dee83..f9f3de8 100644

index c0a028c..caaf75a 100644

index 1f00388..5649d63 100644

Part of the diff is not shown here. View the raw message.