Skip to contentSkip to conversation

Documentation: tracing: Add documentation about eprobes

PATCH In review Steven Rostedt 3 participants 6 messages Jul 28, 2025 – Jul 29, 2025

Subsystems (2)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

PATCH 1/1 Documentation: tracing: Add documentation about eprobes

From: Steven Rostedt <[email protected]>

Eprobes was added back in 5.15, but was never documented. It became a "secret" interface even though it has been a topic of several presentations. For some reason, when eprobes was added, documenting it never became a priority, until now.

---
 Documentation/trace/eprobes.rst | 268 ++++++++++++++++++++++++++++++++
 Documentation/trace/index.rst   |   1 +
 2 files changed, 269 insertions(+)
 create mode 100644 Documentation/trace/eprobes.rst
Documentation/trace/eprobes.rst new file +268 -0
new file mode 100644
index 000000000000..c7aa7c867e9e
@@ -0,0 +1,268 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================
+Eprobe - Event probes
+=====================
+
+:Author: Steven Rostedt <[email protected]>
+
+- Written for v6.17
+
+Overview
+========
+
+Eprobes are dynamic events that are placed on existing events to eiter
+dereference a field that is a pointer, or simply to limit what fields get
+recorded in the trace event.
+
+Eprobes depend on kprobe events so to enable this feature, build your kernel
+with CONFIG_KPROBE_EVENTS=y.
+
+Eprobes are created via the /sys/kernel/tracing/dynamic_events file.
+
+Synopsis of eprobe_events
+-------------------------
+::
+
+  e[:[EGRP/][EEVENT]] GRP.EVENT [FETCHARGS]	: Set a probe
+  -:[EGRP/][EEVENT]				: Clear a probe
+
+ EGRP		: Group name of the new event. If omitted, use "eprobes" for it.
+ EEVENT		: Event name. If omitted, the event name is generated and will
+		  be the same event name as the event it attached to.
+ GRP		: Group name of the event to attach to.
+ EVENT		: Event name of the event to attach to.
+
+ FETCHARGS	: Arguments. Each probe can have up to 128 args.
+  $FIELD	: Fetch the value of the event field called FIELD.
+  @ADDR		: Fetch memory at ADDR (ADDR should be in kernel)
+  @SYM[+|-offs]	: Fetch memory at SYM +|- offs (SYM should be a data symbol)
+  $comm		: Fetch current task comm.
+  +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
+  \IMM		: Store an immediate value to the argument.
+  NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
+  FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
+		  (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
+		  (x8/x16/x32/x64), VFS layer common type(%pd/%pD), "char",
+                  "string", "ustring", "symbol", "symstr" and bitfield are
+                  supported.
+
+Types
+-----
+The FETCHARGS above is very similar to the kprobe events as described in 
+Documentation/trace/kprobetrace.rst.
+
+The difference between eprobes and kprobes FETCHARGS is that eprobes has a
+$FIELD command that returns the content of the event field of the event
+that is attached. Eprobes do not have access to registers, stacks and function
+arguments that kprobes has.
+
+If a field argument is a pointer, it may be dereferenced just like a memory
+address using the FETCHARGS syntax.
+
+
+Attaching to dynamic events
+---------------------------
+
+Note that eprobes may attach to dynamic events as well as to normal events. It
+may attach to a kprobe event, a synthetic event or a fprobe event. This is
+useful if the type of a field needs to be changed. See Example 2 below.
+
+Usage examples
+==============
+
+Example 1
+---------
+
+The basic usage of eprobes is to limit the data that is being recorded into
+the tracing buffer. For example, a common event to trace is the sched_switch
+trace event. That has a format of::
+
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:char prev_comm[16];	offset:8;	size:16;	signed:0;
+	field:pid_t prev_pid;	offset:24;	size:4;	signed:1;
+	field:int prev_prio;	offset:28;	size:4;	signed:1;
+	field:long prev_state;	offset:32;	size:8;	signed:1;
+	field:char next_comm[16];	offset:40;	size:16;	signed:0;
+	field:pid_t next_pid;	offset:56;	size:4;	signed:1;
+	field:int next_prio;	offset:60;	size:4;	signed:1;
+
+The first four fields are common to all events and can not be limited. But the
+rest of the event has 60 bytes of information. It records the names of the
+previous and next tasks being scheduled out and in, as well as their pids and
+priorities. It also records the state of the previous task. If only the pids
+of the tasks are of interest, why waste the ring buffer with all the other
+fields?
+
+An eprobe can limit what gets recorded. Note, it does not help in performance,
+as all the fields are recorded in a temporary buffer to process the eprobe.
+::
+
+ # echo 'e:sched/switch sched.sched_switch prev=$prev_pid:u32 next=$next_pid:u32' >> /sys/kernel/tracing/dynamic_events
+ # echo 1 > /sys/kernel/tracing/events/sched/switch/enable
+ # cat /sys/kernel/tracing/trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 2721/2721   #P:8
+ #
+ #                                _-----=> irqs-off/BH-disabled
+ #                               / _----=> need-resched
+ #                              | / _---=> hardirq/softirq
+ #                              || / _--=> preempt-depth
+ #                              ||| / _-=> migrate-disable
+ #                              |||| /     delay
+ #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
+ #              | |         |   |||||     |         |
+     sshd-session-1082    [004] d..4.  5041.239906: switch: (sched.sched_switch) prev=1082 next=0
+             bash-1085    [001] d..4.  5041.240198: switch: (sched.sched_switch) prev=1085 next=141
+    kworker/u34:5-141     [001] d..4.  5041.240259: switch: (sched.sched_switch) prev=141 next=1085
+           <idle>-0       [004] d..4.  5041.240354: switch: (sched.sched_switch) prev=0 next=1082
+             bash-1085    [001] d..4.  5041.240385: switch: (sched.sched_switch) prev=1085 next=141
+    kworker/u34:5-141     [001] d..4.  5041.240410: switch: (sched.sched_switch) prev=141 next=1085
+             bash-1085    [001] d..4.  5041.240478: switch: (sched.sched_switch) prev=1085 next=0
+     sshd-session-1082    [004] d..4.  5041.240526: switch: (sched.sched_switch) prev=1082 next=0
+           <idle>-0       [001] d..4.  5041.247524: switch: (sched.sched_switch) prev=0 next=90
+           <idle>-0       [002] d..4.  5041.247545: switch: (sched.sched_switch) prev=0 next=16
+      kworker/1:1-90      [001] d..4.  5041.247580: switch: (sched.sched_switch) prev=90 next=0
+        rcu_sched-16      [002] d..4.  5041.247591: switch: (sched.sched_switch) prev=16 next=0
+           <idle>-0       [002] d..4.  5041.257536: switch: (sched.sched_switch) prev=0 next=16
+        rcu_sched-16      [002] d..4.  5041.257573: switch: (sched.sched_switch) prev=16 next=0
+ 
+Note, without adding the "u32" after the prev_pid and next_pid, the values
+would default showing in hexadecimal.
+
+Example 2
+---------
+
+If syscall events are not enabled but the raw syscall are (systemcall
+events are not normal events, but are created from the raw_syscall events
+within the kernel). In order to trace the openat system call, one can create
+an event probe on top of the raw_syscall event:
+::
+
+ # cd /sys/kernel/tracing
+ # cat events/raw_syscalls/sys_enter/format
+ name: sys_enter
+ ID: 395
+ format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:long id;	offset:8;	size:8;	signed:1;
+	field:unsigned long args[6];	offset:16;	size:48;	signed:0;
+
+ print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]
+
+From the source code, the sys_openat() has:
+::
+
+ int sys_openat(int dirfd, const char *path, int flags, mode_t mode)
+ {
+	return my_syscall4(__NR_openat, dirfd, path, flags, mode);
+ }
+
+The path is the second parameter, and that is what is wanted.
+::
+
+ # echo 'e:openat raw_syscalls.sys_enter nr=$id filename=+8($args):ustring' >> dynamic_events
+
+This is being run on x86_64 where the word size is 8 bytes and the openat
+systemcall __NR_openat is set at 257.
+::
+
+ # echo 'nr == 257' > events/eprobes/openat/filter
+
+Now enable the event and look at the trace.
+::
+
+ # echo 1 > events/eprobes/openat/enable
+ # cat trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 4/4   #P:8
+ #
+ #                                _-----=> irqs-off/BH-disabled
+ #                               / _----=> need-resched
+ #                              | / _---=> hardirq/softirq
+ #                              || / _--=> preempt-depth
+ #                              ||| / _-=> migrate-disable
+ #                              |||| /     delay
+ #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
+ #              | |         |   |||||     |         |
+              cat-1298    [003] ...2.  2060.875970: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+              cat-1298    [003] ...2.  2060.876197: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+              cat-1298    [003] ...2.  2060.879126: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+              cat-1298    [003] ...2.  2060.879639: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+
+The filename shows "(fault)". This is likely because the filename has not been
+pulled into memory yet and currently trace events cannot fault in memory that
+is not present. When a eprobe tries to read memory that has not been faulted
+in yet, it will show the "(fault)" text.
+
+To get around this, as the kernel will likely pull in this filename and make
+it present, attaching it to a synthetic event that can pass the address of the
+filename from the entry of the event to the end of the event, this can be used
+to show the filename when the system call returns.
+
+Remove the old eprobe::
+
+ # echo 1 > events/eprobes/openat/enable
+ # echo '-:openat' >> dynamic_events
+
+This time make an eprobe where the address of the filename is saved::
+
+ # echo 'e:openat_start raw_syscalls.sys_enter nr=$id filename=+8($args):x64' >> dynamic_events
+
+Create a synthetic event that passes the address of the filename to the
+end of the event::
+
+ # echo 's:filename u64 file' >> dynamic_events
+ # echo 'hist:keys=common_pid:f=filename if nr == 257' > events/eprobes/openat_start/trigger
+ # echo 'hist:keys=common_pid:file=$f:onmatch(eprobes.openat_start).trace(filename,$file) if id == 257' > events/raw_syscalls/sys_exit/trigger
+
+Now that the address of the filename has been passed to the end of the
+systemcall, create another eprobe to attach to the exit event to show the
+string::
+
+ # echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
+ # echo 1 > events/eprobes/openat/enable
+ # cat trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 4/4   #P:8
+ #
+ #                                _-----=> irqs-off/BH-disabled
+ #                               / _----=> need-resched
+ #                              | / _---=> hardirq/softirq
+ #                              || / _--=> preempt-depth
+ #                              ||| / _-=> migrate-disable
+ #                              |||| /     delay
+ #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
+ #              | |         |   |||||     |         |
+              cat-1331    [001] ...5.  2944.787977: openat: (synthetic.filename) filename="/etc/ld.so.cache"
+              cat-1331    [001] ...5.  2944.788480: openat: (synthetic.filename) filename="/lib/x86_64-linux-gnu/libc.so.6"
+              cat-1331    [001] ...5.  2944.793426: openat: (synthetic.filename) filename="/usr/lib/locale/locale-archive"
+              cat-1331    [001] ...5.  2944.831362: openat: (synthetic.filename) filename="trace"
+
+Example 3
+---------
+
+If syscall trace events are available, the above would not need the first
+eprobe, but it would still need the last one::
+
+ # echo 's:filename u64 file' >> dynamic_events
+ # echo 'hist:keys=common_pid:f=filename' > events/syscalls/sys_enter_openat/trigger
+ # echo 'hist:keys=common_pid:file=$f:onmatch(syscalls.sys_enter_openat).trace(filename,$file)' > events/syscalls/sys_exit_openat/trigger
+ # echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
+ # echo 1 > events/eprobes/openat/enable
+
+And this would produce the same result as Example 2.
Documentation/trace/index.rst +1 -0
index cc1dc5a087e8..812d9a46dd94 100644
@@ -37,6 +37,7 @@ the Linux kernel.
    kprobetrace
    fprobetrace
    fprobe
+   eprobes
    ring-buffer-design
 
 Event Tracing and Analysis
Signature · 1 line
2.47.2
reviewer
#1 lore raw
Quoted 24 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

Eprobes was added back in 5.15, but was never documented. It became a "secret" interface even though it has been a topic of several presentations. For some reason, when eprobes was added, documenting it never became a priority, until now.

---
 Documentation/trace/eprobes.rst | 268 ++++++++++++++++++++++++++++++++
 Documentation/trace/index.rst   |   1 +
 2 files changed, 269 insertions(+)
 create mode 100644 Documentation/trace/eprobes.rst
Documentation/trace/eprobes.rst new file +4 -0
new file mode 100644
index 000000000000..c7aa7c867e9e
@@ -0,0 +1,268 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================
+Eprobe - Event probes

Documentation/trace/eprobes.rst line 14

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+=====================
+
+:Author: Steven Rostedt <[email protected]>
+
+- Written for v6.17
+
+Overview
+========
+
+Eprobes are dynamic events that are placed on existing events to eiter

either

Documentation/trace/eprobes.rst line 15

Quoted 1 line
Documentation/trace/eprobes.rst new file +1 -0
+dereference a field that is a pointer, or simply to limit what fields get

(preference:) are

Documentation/trace/eprobes.rst line 18

Quoted 3 lines
Documentation/trace/eprobes.rst new file +3 -0
+recorded in the trace event.
+
+Eprobes depend on kprobe events so to enable this feature, build your kernel
                                                         s/,/;/
Quoted 19 lines
Documentation/trace/eprobes.rst new file +19 -0
+with CONFIG_KPROBE_EVENTS=y.
+
+Eprobes are created via the /sys/kernel/tracing/dynamic_events file.
+
+Synopsis of eprobe_events
+-------------------------
+::
+
+  e[:[EGRP/][EEVENT]] GRP.EVENT [FETCHARGS]	: Set a probe
+  -:[EGRP/][EEVENT]				: Clear a probe
+
+ EGRP		: Group name of the new event. If omitted, use "eprobes" for it.
+ EEVENT		: Event name. If omitted, the event name is generated and will
+		  be the same event name as the event it attached to.
+ GRP		: Group name of the event to attach to.
+ EVENT		: Event name of the event to attach to.
+
+ FETCHARGS	: Arguments. Each probe can have up to 128 args.
+  $FIELD	: Fetch the value of the event field called FIELD.

Documentation/trace/eprobes.rst line 47

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+  @ADDR		: Fetch memory at ADDR (ADDR should be in kernel)
+  @SYM[+|-offs]	: Fetch memory at SYM +|- offs (SYM should be a data symbol)
+  $comm		: Fetch current task comm.
+  +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
+  \IMM		: Store an immediate value to the argument.
+  NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
+  FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
+		  (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
+		  (x8/x16/x32/x64), VFS layer common type(%pd/%pD), "char",
+                  "string", "ustring", "symbol", "symstr" and bitfield are

Should bitfield be quoted?

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+                  supported.
+
+Types
+-----
+The FETCHARGS above is very similar to the kprobe events as described in 
+Documentation/trace/kprobetrace.rst.
+
+The difference between eprobes and kprobes FETCHARGS is that eprobes has a
+$FIELD command that returns the content of the event field of the event
+that is attached. Eprobes do not have access to registers, stacks and function

Documentation/trace/eprobes.rst line 67

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+arguments that kprobes has.
+
+If a field argument is a pointer, it may be dereferenced just like a memory
+address using the FETCHARGS syntax.
+
+
+Attaching to dynamic events
+---------------------------
+
+Note that eprobes may attach to dynamic events as well as to normal events. It

Don't need "Note that".

Documentation/trace/eprobes.rst line 68

Quoted 1 line
Documentation/trace/eprobes.rst new file +1 -0
+may attach to a kprobe event, a synthetic event or a fprobe event. This is

I would say: an fprobe event.

Quoted 64 lines
Documentation/trace/eprobes.rst new file +64 -0
+useful if the type of a field needs to be changed. See Example 2 below.
+
+Usage examples
+==============
+
+Example 1
+---------
+
+The basic usage of eprobes is to limit the data that is being recorded into
+the tracing buffer. For example, a common event to trace is the sched_switch
+trace event. That has a format of::
+
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:char prev_comm[16];	offset:8;	size:16;	signed:0;
+	field:pid_t prev_pid;	offset:24;	size:4;	signed:1;
+	field:int prev_prio;	offset:28;	size:4;	signed:1;
+	field:long prev_state;	offset:32;	size:8;	signed:1;
+	field:char next_comm[16];	offset:40;	size:16;	signed:0;
+	field:pid_t next_pid;	offset:56;	size:4;	signed:1;
+	field:int next_prio;	offset:60;	size:4;	signed:1;
+
+The first four fields are common to all events and can not be limited. But the
+rest of the event has 60 bytes of information. It records the names of the
+previous and next tasks being scheduled out and in, as well as their pids and
+priorities. It also records the state of the previous task. If only the pids
+of the tasks are of interest, why waste the ring buffer with all the other
+fields?
+
+An eprobe can limit what gets recorded. Note, it does not help in performance,
+as all the fields are recorded in a temporary buffer to process the eprobe.
+::
+
+ # echo 'e:sched/switch sched.sched_switch prev=$prev_pid:u32 next=$next_pid:u32' >> /sys/kernel/tracing/dynamic_events
+ # echo 1 > /sys/kernel/tracing/events/sched/switch/enable
+ # cat /sys/kernel/tracing/trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 2721/2721   #P:8
+ #
+ #                                _-----=> irqs-off/BH-disabled
+ #                               / _----=> need-resched
+ #                              | / _---=> hardirq/softirq
+ #                              || / _--=> preempt-depth
+ #                              ||| / _-=> migrate-disable
+ #                              |||| /     delay
+ #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
+ #              | |         |   |||||     |         |
+     sshd-session-1082    [004] d..4.  5041.239906: switch: (sched.sched_switch) prev=1082 next=0
+             bash-1085    [001] d..4.  5041.240198: switch: (sched.sched_switch) prev=1085 next=141
+    kworker/u34:5-141     [001] d..4.  5041.240259: switch: (sched.sched_switch) prev=141 next=1085
+           <idle>-0       [004] d..4.  5041.240354: switch: (sched.sched_switch) prev=0 next=1082
+             bash-1085    [001] d..4.  5041.240385: switch: (sched.sched_switch) prev=1085 next=141
+    kworker/u34:5-141     [001] d..4.  5041.240410: switch: (sched.sched_switch) prev=141 next=1085
+             bash-1085    [001] d..4.  5041.240478: switch: (sched.sched_switch) prev=1085 next=0
+     sshd-session-1082    [004] d..4.  5041.240526: switch: (sched.sched_switch) prev=1082 next=0
+           <idle>-0       [001] d..4.  5041.247524: switch: (sched.sched_switch) prev=0 next=90
+           <idle>-0       [002] d..4.  5041.247545: switch: (sched.sched_switch) prev=0 next=16
+      kworker/1:1-90      [001] d..4.  5041.247580: switch: (sched.sched_switch) prev=90 next=0
+        rcu_sched-16      [002] d..4.  5041.247591: switch: (sched.sched_switch) prev=16 next=0

Documentation/trace/eprobes.rst line 142

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+           <idle>-0       [002] d..4.  5041.257536: switch: (sched.sched_switch) prev=0 next=16
+        rcu_sched-16      [002] d..4.  5041.257573: switch: (sched.sched_switch) prev=16 next=0
+ 
+Note, without adding the "u32" after the prev_pid and next_pid, the values
+would default showing in hexadecimal.
+
+Example 2
+---------
+
+If syscall events are not enabled but the raw syscall are (systemcall

syscalls are (system call

Documentation/trace/eprobes.rst line 144

Quoted 2 lines
Documentation/trace/eprobes.rst new file +2 -0
+events are not normal events, but are created from the raw_syscall events
+within the kernel). In order to trace the openat system call, one can create

^^ not a complete sentence.

Quoted 17 lines
Documentation/trace/eprobes.rst new file +17 -0
+an event probe on top of the raw_syscall event:
+::
+
+ # cd /sys/kernel/tracing
+ # cat events/raw_syscalls/sys_enter/format
+ name: sys_enter
+ ID: 395
+ format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:long id;	offset:8;	size:8;	signed:1;
+	field:unsigned long args[6];	offset:16;	size:48;	signed:0;
+
+ print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]

Documentation/trace/eprobes.rst line 171

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+
+From the source code, the sys_openat() has:
+::
+
+ int sys_openat(int dirfd, const char *path, int flags, mode_t mode)
+ {
+	return my_syscall4(__NR_openat, dirfd, path, flags, mode);
+ }
+
+The path is the second parameter, and that is what is wanted.

s/wanted/want/

Documentation/trace/eprobes.rst line 177

Quoted 6 lines
Documentation/trace/eprobes.rst new file +6 -0
+::
+
+ # echo 'e:openat raw_syscalls.sys_enter nr=$id filename=+8($args):ustring' >> dynamic_events
+
+This is being run on x86_64 where the word size is 8 bytes and the openat
+systemcall __NR_openat is set at 257.

system call

Quoted 20 lines
Documentation/trace/eprobes.rst new file +20 -0
+::
+
+ # echo 'nr == 257' > events/eprobes/openat/filter
+
+Now enable the event and look at the trace.
+::
+
+ # echo 1 > events/eprobes/openat/enable
+ # cat trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 4/4   #P:8
+ #
+ #                                _-----=> irqs-off/BH-disabled
+ #                               / _----=> need-resched
+ #                              | / _---=> hardirq/softirq
+ #                              || / _--=> preempt-depth
+ #                              ||| / _-=> migrate-disable
+ #                              |||| /     delay

Documentation/trace/eprobes.rst line 207

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+ #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
+ #              | |         |   |||||     |         |
+              cat-1298    [003] ...2.  2060.875970: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+              cat-1298    [003] ...2.  2060.876197: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+              cat-1298    [003] ...2.  2060.879126: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+              cat-1298    [003] ...2.  2060.879639: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+
+The filename shows "(fault)". This is likely because the filename has not been
+pulled into memory yet and currently trace events cannot fault in memory that
+is not present. When a eprobe tries to read memory that has not been faulted

an eprobe

Quoted 15 lines
Documentation/trace/eprobes.rst new file +15 -0
+in yet, it will show the "(fault)" text.
+
+To get around this, as the kernel will likely pull in this filename and make
+it present, attaching it to a synthetic event that can pass the address of the
+filename from the entry of the event to the end of the event, this can be used
+to show the filename when the system call returns.
+
+Remove the old eprobe::
+
+ # echo 1 > events/eprobes/openat/enable
+ # echo '-:openat' >> dynamic_events
+
+This time make an eprobe where the address of the filename is saved::
+
+ # echo 'e:openat_start raw_syscalls.sys_enter nr=$id filename=+8($args):x64' >> dynamic_events

Documentation/trace/eprobes.rst line 232

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+
+Create a synthetic event that passes the address of the filename to the
+end of the event::
+
+ # echo 's:filename u64 file' >> dynamic_events
+ # echo 'hist:keys=common_pid:f=filename if nr == 257' > events/eprobes/openat_start/trigger
+ # echo 'hist:keys=common_pid:file=$f:onmatch(eprobes.openat_start).trace(filename,$file) if id == 257' > events/raw_syscalls/sys_exit/trigger
+
+Now that the address of the filename has been passed to the end of the
+systemcall, create another eprobe to attach to the exit event to show the

system call,

Quoted 26 lines
Documentation/trace/eprobes.rst new file +26 -0
+string::
+
+ # echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
+ # echo 1 > events/eprobes/openat/enable
+ # cat trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 4/4   #P:8
+ #
+ #                                _-----=> irqs-off/BH-disabled
+ #                               / _----=> need-resched
+ #                              | / _---=> hardirq/softirq
+ #                              || / _--=> preempt-depth
+ #                              ||| / _-=> migrate-disable
+ #                              |||| /     delay
+ #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
+ #              | |         |   |||||     |         |
+              cat-1331    [001] ...5.  2944.787977: openat: (synthetic.filename) filename="/etc/ld.so.cache"
+              cat-1331    [001] ...5.  2944.788480: openat: (synthetic.filename) filename="/lib/x86_64-linux-gnu/libc.so.6"
+              cat-1331    [001] ...5.  2944.793426: openat: (synthetic.filename) filename="/usr/lib/locale/locale-archive"
+              cat-1331    [001] ...5.  2944.831362: openat: (synthetic.filename) filename="trace"
+
+Example 3
+---------
+

Documentation/trace/eprobes.rst line 268

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+If syscall trace events are available, the above would not need the first
+eprobe, but it would still need the last one::
+
+ # echo 's:filename u64 file' >> dynamic_events
+ # echo 'hist:keys=common_pid:f=filename' > events/syscalls/sys_enter_openat/trigger
+ # echo 'hist:keys=common_pid:file=$f:onmatch(syscalls.sys_enter_openat).trace(filename,$file)' > events/syscalls/sys_exit_openat/trigger
+ # echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
+ # echo 1 > events/eprobes/openat/enable
+
+And this would produce the same result as Example 2.

Thanks for the new documentation.

Signature · 1 line
~Randy
author maintainer
#2 lore raw
Quoted 9 lines · On Mon, 28 Jul 2025 18:02:37 -0700 Randy Dunlap <[email protected]> wrote:

Documentation/trace/eprobes.rst line 14

Quoted 4 lines
Documentation/trace/eprobes.rst new file +4 -0
+Overview
+========
+
+Eprobes are dynamic events that are placed on existing events to eiter  

either

Quoted 1 line

+dereference a field that is a pointer, or simply to limit what fields get

Quoted 7 lines

(preference:) are

Documentation/trace/eprobes.rst line 18

Quoted 3 lines
Documentation/trace/eprobes.rst new file +3 -0
+recorded in the trace event.
+
+Eprobes depend on kprobe events so to enable this feature, build your kernel  
                                                         s/,/;/

OK.

Quoted 6 lines

Documentation/trace/eprobes.rst line 47

Quoted 4 lines
Documentation/trace/eprobes.rst new file +4 -0
+  FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
+		  (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
+		  (x8/x16/x32/x64), VFS layer common type(%pd/%pD), "char",
+                  "string", "ustring", "symbol", "symstr" and bitfield are  

Should bitfield be quoted?

Hmm, probably. And we should fix kprobetrace.rst as that's where I copied it from.

Quoted 7 lines

Documentation/trace/eprobes.rst line 67

Quoted 5 lines
Documentation/trace/eprobes.rst new file +5 -0
+
+Attaching to dynamic events
+---------------------------
+
+Note that eprobes may attach to dynamic events as well as to normal events. It  

Don't need "Note that".

OK.

Quoted 5 lines
Quoted 2 lines

+may attach to a kprobe event, a synthetic event or a fprobe event. This is

I would say: an fprobe event.

OK.

Quoted 8 lines

Documentation/trace/eprobes.rst line 142

Quoted 5 lines
Documentation/trace/eprobes.rst new file +5 -0
+
+Example 2
+---------
+
+If syscall events are not enabled but the raw syscall are (systemcall  

syscalls are (system call

Quoted 4 lines

Documentation/trace/eprobes.rst line 144

Quoted 2 lines
Documentation/trace/eprobes.rst new file +2 -0
+events are not normal events, but are created from the raw_syscall events
+within the kernel). In order to trace the openat system call, one can create  

^^ not a complete sentence.

Ah, I'm thinking that "This example is for the case that syscall events are not enabled..."

But it came out as the above. Will fix.

Quoted 23 lines
Quoted 23 lines
Documentation/trace/eprobes.rst new file +21 -0
+an event probe on top of the raw_syscall event:
+::
+
+ # cd /sys/kernel/tracing
+ # cat events/raw_syscalls/sys_enter/format
+ name: sys_enter
+ ID: 395
+ format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:long id;	offset:8;	size:8;	signed:1;
+	field:unsigned long args[6];	offset:16;	size:48;	signed:0;
+
+ print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]
+
+From the source code, the sys_openat() has:
+::
+
Quoted 8 lines

Documentation/trace/eprobes.rst line 171

Quoted 6 lines

+ int sys_openat(int dirfd, const char *path, int flags, mode_t mode)
+ {

Documentation/trace/eprobes.rst new file +4 -0
+	return my_syscall4(__NR_openat, dirfd, path, flags, mode);
+ }
+
+The path is the second parameter, and that is what is wanted.  

s/wanted/want/

Really? That sounds funny to me:

The path is the second parameter and that is what is want.

??

Quoted 9 lines

Documentation/trace/eprobes.rst line 177

Quoted 7 lines

+::

Documentation/trace/eprobes.rst new file +5 -0
+
+ # echo 'e:openat raw_syscalls.sys_enter nr=$id filename=+8($args):ustring' >> dynamic_events
+
+This is being run on x86_64 where the word size is 8 bytes and the openat
+systemcall __NR_openat is set at 257.  

system call

OK.

Quoted 5 lines

Documentation/trace/eprobes.rst line 207

Quoted 3 lines
Documentation/trace/eprobes.rst new file +3 -0
+The filename shows "(fault)". This is likely because the filename has not been
+pulled into memory yet and currently trace events cannot fault in memory that
+is not present. When a eprobe tries to read memory that has not been faulted  

an eprobe

OK.

Quoted 4 lines

Documentation/trace/eprobes.rst line 232

Quoted 2 lines
Documentation/trace/eprobes.rst new file +2 -0
+Now that the address of the filename has been passed to the end of the
+systemcall, create another eprobe to attach to the exit event to show the  

system call,

OK.

Quoted 3 lines

Thanks for the new documentation.

It was a long time coming :-p

-- Steve

reviewer
#3 lore raw

Hi,

Quoted 12 lines
Quoted 11 lines

Documentation/trace/eprobes.rst line 171

Quoted 9 lines
Documentation/trace/eprobes.rst new file +9 -0
+From the source code, the sys_openat() has:
+::
+
+ int sys_openat(int dirfd, const char *path, int flags, mode_t mode)
+ {
+	return my_syscall4(__NR_openat, dirfd, path, flags, mode);
+ }
+
+The path is the second parameter, and that is what is wanted.  

s/wanted/want/

Quoted 5 lines

Really? That sounds funny to me:

The path is the second parameter and that is what is want.

??

Yeah, I don't know how I did that. Sorry.

Signature · 1 line
~Randy
#4 lore raw
In reply to Steven Rostedt
Quoted 7 lines · On Mon, 28 Jul 2025 17:15:22 -0400 Steven Rostedt <[email protected]> wrote:

From: Steven Rostedt <[email protected]>

Eprobes was added back in 5.15, but was never documented. It became a "secret" interface even though it has been a topic of several presentations. For some reason, when eprobes was added, documenting it never became a priority, until now.

Thanks for the document!

Documentation/trace/eprobes.rst

Quoted 12 lines
---
 Documentation/trace/eprobes.rst | 268 ++++++++++++++++++++++++++++++++
 Documentation/trace/index.rst   |   1 +
 2 files changed, 269 insertions(+)
 create mode 100644 Documentation/trace/eprobes.rst
Documentation/trace/eprobes.rst new file +0 -0
new file mode 100644
index 000000000000..c7aa7c867e9e

BTW, can't you rename it as 'eprobetrace.rst' as same as others? I usually name the doc of "a probe feature which provides only in-kernel APIs" as '*probe.rst' and the doc of "a probe *event* feature which can controlled via tracefs interface" as '*probetrace.rst'.

Documentation/trace/eprobes.rst line 4

Quoted 5 lines
Documentation/trace/eprobes.rst new file +4 -0
@@ -0,0 +1,268 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================
+Eprobe - Event probes

What about below title?

Eprobe - Event-based Probe Tracing

Quoted 5 lines
Documentation/trace/eprobes.rst new file +5 -0
+=====================
+
+:Author: Steven Rostedt <[email protected]>
+
+- Written for v6.17

Documentation/trace/eprobes.rst line 19

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+
+Overview
+========
+
+Eprobes are dynamic events that are placed on existing events to eiter
+dereference a field that is a pointer, or simply to limit what fields get
+recorded in the trace event.
+
+Eprobes depend on kprobe events so to enable this feature, build your kernel
+with CONFIG_KPROBE_EVENTS=y.

Is this correct? It seems that eprobe event only depends on event trigger (by implementation. Actually we should fix the kernel/trace/Kconfig.)

https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git/tree/kernel/trace/trace_eprobe.c?h=trace-v6.16-rc5#n576

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
author maintainer
#5 lore raw
Quoted 10 lines · On Tue, 29 Jul 2025 10:26:36 +0900 Masami Hiramatsu (Google)…

Documentation/trace/eprobes.rst

Quoted 5 lines
Documentation/trace/eprobes.rst new file +0 -0
new file mode 100644
index 000000000000..c7aa7c867e9e

BTW, can't you rename it as 'eprobetrace.rst' as same as others? I usually name the doc of "a probe feature which provides only in-kernel APIs" as '*probe.rst' and the doc of "a probe *event* feature which can controlled via tracefs interface" as '*probetrace.rst'.

Sure, no problem. Will update.

Quoted 10 lines

Documentation/trace/eprobes.rst line 4

Quoted 6 lines
Documentation/trace/eprobes.rst new file +4 -0
@@ -0,0 +1,268 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================
+Eprobe - Event probes  

What about below title?

Eprobe - Event-based Probe Tracing

OK.

Quoted 20 lines
Quoted 6 lines
Documentation/trace/eprobes.rst new file +5 -0
+=====================
+
+:Author: Steven Rostedt <[email protected]>
+
+- Written for v6.17

Documentation/trace/eprobes.rst line 19

Quoted 10 lines
Documentation/trace/eprobes.rst new file +10 -0
+
+Overview
+========
+
+Eprobes are dynamic events that are placed on existing events to eiter
+dereference a field that is a pointer, or simply to limit what fields get
+recorded in the trace event.
+
+Eprobes depend on kprobe events so to enable this feature, build your kernel
+with CONFIG_KPROBE_EVENTS=y.  

Is this correct? It seems that eprobe event only depends on event trigger (by implementation. Actually we should fix the kernel/trace/Kconfig.)

Quoted 1 line

https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git/tree/kernel/trace/trace_eprobe.c?h=trace-v6.16-rc5#n576

I noticed there was no config that selected it, and it only gets selected by CONFIG_PROBE_EVENTS which requires something else to select it. I guess adding an EPROBE config would be useful too. Let me do that too.

Thanks for the review.

-- Steve