Skip to contentSkip to conversation

tracing/boot: Support eprobe, fprobe, and tprobe events and add test framework

PATCH Superseded Masami Hiramatsu (Google) 1 participant 3 messages Aug 6

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Masami Hiramatsu (Google)

author maintainer
#0 lore raw

v2 cover tracing/boot: Support eprobe, fprobe, and tprobe events and add test framework

Hi,

This is 2nd version of series to extend kernel boot-time tracing to support newly added dynamic event types (eprobes, fprobes, and tracepoint probes) and introduces an automated QEMU-based test suite under tools/tracing/boottime/.
Previous version is here:

https://lore.kernel.org/all/178597841107.123558.13065821394773752510.stgit@devnote2/

This version fixes some issues reported by Sashiko [1], simplifying code, and improve test program.

[1/2]
- Fix raw command detection logic for eprobes, fprobes, and tprobes by requiring ':' or isspace() after type prefix.
- Consolidate duplicate loop logic into trace_boot_add_probe_event() helper function.
[2/2]
- Copy shared library dependencies using ldd if busybox is dynamically linked.
- Wrap QEMU execution with timeout command and add -T/--timeout CLI option.
- Add pre-flight checks for busybox architecture compatibility and cpio tool in run_boottime_test.sh.
- Fix POSIX syntax compatibility and allow bypassing HOST_ARCH skip when custom kernel/QEMU parameters are passed.
- Add boottime/run_boottime_test.sh to TEST_PROGS_EXTENDED in selftests/ftrace/Makefile.

[1] https://sashiko.dev/#/patchset/178597841107.123558.13065821394773752510.stgit%40devnote2

Thanks,

---
base-commit: ea694cdd92b6bfa123b3f5c563e47b4cf968046a

Masami Hiramatsu (Google) (2):
tracing/boot: Add support for eprobe, fprobe, and tprobe events selftests/ftrace: Add boot-time tracing test framework

 Documentation/trace/boottime-trace.rst             |   18 +
 kernel/trace/trace_boot.c                          |   86 +++++++
 kernel/trace/trace_dynevent.c                      |    5 
 tools/testing/selftests/ftrace/Makefile            |    4 
 tools/testing/selftests/ftrace/boottime/Makefile   |   10 +
 tools/testing/selftests/ftrace/boottime/README     |   58 ++++
 .../ftrace/boottime/bootconfigs/01-kprobe.bconf    |    5 
 .../ftrace/boottime/bootconfigs/02-synth.bconf     |    4 
 .../ftrace/boottime/bootconfigs/03-eprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/04-fprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/05-tprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/06-instance.bconf  |    5 
 .../selftests/ftrace/boottime/run_boottime_test.sh |  264 ++++++++++++++++++++
 .../selftests/ftrace/boottime/tests/01-kprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/02-synth.sh    |   18 +
 .../selftests/ftrace/boottime/tests/03-eprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/04-fprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/05-tprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/06-instance.sh |   23 ++
 tools/testing/selftests/ftrace/config              |    4 
 20 files changed, 583 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/boottime/Makefile
 create mode 100644 tools/testing/selftests/ftrace/boottime/README
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf
 create mode 100755 tools/testing/selftests/ftrace/boottime/run_boottime_test.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/02-synth.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/06-instance.sh

--
Masami Hiramatsu (Google) <[email protected]>

author maintainer
#1 lore raw

PATCH v2 1/2 tracing/boot: Add support for eprobe, fprobe, and tprobe events

From: Masami Hiramatsu (Google) <[email protected]>

Boot-time tracing currently supports kprobe-events and synthetic-events under per-event configuration options.

Extend boot-time tracing to support newly added dynamic probe types:
- event probes (eprobe) under the "eprobes" event group
- function probes (fprobe) under the "fprobes" event group
- tracepoint probes (tprobe) under the "tracepoints" or "tprobes" event group

To support this cleanly, update dyn_event_create() in trace_dynevent.c so that passing NULL as the type parameter delegates to create_dyn_event(), allowing generic creation of any registered dynamic event type from a raw command string.

Update Documentation/trace/boottime-trace.rst accordingly to describe the new per-event bootconfig options.

Assisted-by Antigravity:gemini-3.6-flash

---
Changes in v2:
- Fix raw command detection logic for eprobes, fprobes, and tprobes by requiring ':' or isspace() after type prefix.
- Consolidate duplicate loop logic into trace_boot_add_probe_event() helper function.

---
 Documentation/trace/boottime-trace.rst |   18 ++++++-
 kernel/trace/trace_boot.c              |   86 ++++++++++++++++++++++++++++++++
 kernel/trace/trace_dynevent.c          |    5 ++
 3 files changed, 106 insertions(+), 3 deletions(-)
Documentation/trace/boottime-trace.rst +15 -3
index 651f3a2c01de..2c3c1fedb92c 100644
@@ -121,9 +121,21 @@ ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]]
    Defines new synthetic event with FIELDs. Each field should be
    "type varname".
 
-Note that kprobe and synthetic event definitions can be written under
-instance node, but those are also visible from other instances. So please
-take care for event name conflict.
+ftrace.[instance.INSTANCE.]event.eprobes.EVENT.probes = PROBE[, PROBE2[...]]
+   Defines new event probe based on PROBEs. This option is available only
+   for the event which group name is "eprobes".
+
+ftrace.[instance.INSTANCE.]event.fprobes.EVENT.probes = PROBE[, PROBE2[...]]
+   Defines new fprobe event based on PROBEs. This option is available only
+   for the event which group name is "fprobes".
+
+ftrace.[instance.INSTANCE.]event.tracepoints.EVENT.probes = PROBE[, PROBE2[...]]
+   Defines new tracepoint probe based on PROBEs. This option is available only
+   for the event which group name is "tracepoints" or "tprobes".
+
+Note that dynamic event definitions can be written under instance node, but
+those are also visible from other instances. So please take care for event
+name conflict.
 
 Ftrace Histogram Options
 ------------------------
kernel/trace/trace_boot.c +86 -0

104 lines not on this page · raw message

kernel/trace/trace_dynevent.c +5 -0
index c4dfbc293bae..6f2e39b797e5 100644
@@ -116,6 +116,8 @@ int dyn_event_release(const char *raw_command, struct dyn_event_operations *type
 	return ret;
 }
 
+static int create_dyn_event(const char *raw_command);
+
 /*
  * Locked version of event creation. The event creation must be protected by
  * dyn_event_ops_mutex because of protecting trace_probe_log.
@@ -124,6 +126,9 @@ int dyn_event_create(const char *raw_command, struct dyn_event_operations *type)
 {
 	int ret;
 
+	if (!type)
+		return create_dyn_event(raw_command);
+
 	mutex_lock(&dyn_event_ops_mutex);
 	ret = type->create(raw_command);
 	mutex_unlock(&dyn_event_ops_mutex);
author maintainer
#2 lore raw

PATCH v2 2/2 selftests/ftrace: Add boot-time tracing test framework

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Add an automated test framework under tools/testing/selftests/ftrace/boottime/ to verify boot-time tracing configuration parsing.

The test runner (run_boottime_test.sh) sources ktap_helpers.sh to generate output in TAP version 13 format. It builds a minimal initramfs with busybox, attaches bootconfig examples using tools/bootconfig/bootconfig, boots QEMU, and runs corresponding tracefs checker scripts.

Assisted-by Antigravity:gemini-3.6-flash

---
Changes in v2:
- Copy shared library dependencies using ldd if busybox is dynamically linked.
- Wrap QEMU execution with timeout command and add -T/--timeout CLI option.
- Add pre-flight checks for busybox architecture compatibility and cpio tool in run_boottime_test.sh.
- Fix POSIX syntax compatibility and allow bypassing HOST_ARCH skip when custom kernel/QEMU parameters are passed.
- Add boottime/run_boottime_test.sh to TEST_PROGS_EXTENDED in selftests/ftrace/Makefile.

---
 tools/testing/selftests/ftrace/Makefile            |    4 
 tools/testing/selftests/ftrace/boottime/Makefile   |   10 +
 tools/testing/selftests/ftrace/boottime/README     |   58 ++++
 .../ftrace/boottime/bootconfigs/01-kprobe.bconf    |    5 
 .../ftrace/boottime/bootconfigs/02-synth.bconf     |    4 
 .../ftrace/boottime/bootconfigs/03-eprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/04-fprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/05-tprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/06-instance.bconf  |    5 
 .../selftests/ftrace/boottime/run_boottime_test.sh |  264 ++++++++++++++++++++
 .../selftests/ftrace/boottime/tests/01-kprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/02-synth.sh    |   18 +
 .../selftests/ftrace/boottime/tests/03-eprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/04-fprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/05-tprobe.sh   |   18 +
 .../selftests/ftrace/boottime/tests/06-instance.sh |   23 ++
 tools/testing/selftests/ftrace/config              |    4 
 17 files changed, 477 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/boottime/Makefile
 create mode 100644 tools/testing/selftests/ftrace/boottime/README
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf
 create mode 100755 tools/testing/selftests/ftrace/boottime/run_boottime_test.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/02-synth.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh
 create mode 100755 tools/testing/selftests/ftrace/boottime/tests/06-instance.sh
tools/testing/selftests/ftrace/Makefile +2 -2
index 7c12263f8260..f804fdc595db 100644
@@ -1,9 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
 all:
 
-TEST_PROGS_EXTENDED := ftracetest
+TEST_PROGS_EXTENDED := ftracetest boottime/run_boottime_test.sh
 TEST_PROGS := ftracetest-ktap
-TEST_FILES := test.d settings
+TEST_FILES := test.d settings boottime
 EXTRA_CLEAN := $(OUTPUT)/logs/*
 
 TEST_GEN_FILES := poll
tools/testing/selftests/ftrace/boottime/Makefile new file +10 -0
new file mode 100644
index 000000000000..fd0975350f10
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+
+all:
+
+run_tests:
+	@./run_boottime_test.sh
+
+clean:
+
+.PHONY: all run_tests clean
tools/testing/selftests/ftrace/boottime/README new file +58 -0

58 lines not on this page · raw message

tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf new file +5 -0
new file mode 100644
index 000000000000..f607840e3858
@@ -0,0 +1,5 @@
+ftrace.event.kprobes.vfs_read {
+	probes = "vfs_read $arg1 $arg2"
+	filter = "common_pid < 200"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf new file +4 -0
new file mode 100644
index 000000000000..b1f37713f33c
@@ -0,0 +1,4 @@
+ftrace.event.synthetic.boot_lat {
+	fields = "unsigned long id", "u64 delta"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..f954aa7fb8bd
@@ -0,0 +1,4 @@
+ftrace.event.eprobes.ep_read {
+	probes = "syscalls/sys_enter_openat"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..82ae2ac0629f
@@ -0,0 +1,4 @@
+ftrace.event.fprobes.fp_read {
+	probes = "vfs_read"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..02a2803c770a
@@ -0,0 +1,4 @@
+ftrace.event.tracepoints.tp_sched {
+	probes = "sched_switch"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf new file +5 -0
new file mode 100644
index 000000000000..31f81da570ea
@@ -0,0 +1,5 @@
+ftrace.instance.foo {
+	event.sched.sched_switch.enable
+	buffer_size = 1MB
+	tracing_on = 1
+}
tools/testing/selftests/ftrace/boottime/run_boottime_test.sh new file +264 -0

264 lines not on this page · raw message

tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh new file +18 -0
new file mode 100755
index 000000000000..a20e7e82d158
@@ -0,0 +1,18 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Check kprobe bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/events/kprobes/vfs_read" ]; then
+	echo "FAIL: kprobe event kprobes/vfs_read does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/kprobes/vfs_read/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: kprobe event kprobes/vfs_read is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 01-kprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/02-synth.sh new file +18 -0
new file mode 100755
index 000000000000..46d3b152cbea
@@ -0,0 +1,18 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Check synthetic event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/events/synthetic/boot_lat" ]; then
+	echo "FAIL: synthetic event synthetic/boot_lat does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/synthetic/boot_lat/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: synthetic event synthetic/boot_lat is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 02-synth"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh new file +18 -0
new file mode 100755
index 000000000000..108f9ba31c06
@@ -0,0 +1,18 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Check eprobe event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/events/eprobes/ep_read" ]; then
+	echo "FAIL: eprobe event eprobes/ep_read does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/eprobes/ep_read/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: eprobe event eprobes/ep_read is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 03-eprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh new file +18 -0
new file mode 100755
index 000000000000..c0a316bf7dde
@@ -0,0 +1,18 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Check fprobe event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/events/fprobes/fp_read" ]; then
+	echo "FAIL: fprobe event fprobes/fp_read does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/fprobes/fp_read/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: fprobe event fprobes/fp_read is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 04-fprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh new file +18 -0
new file mode 100755
index 000000000000..b29119b81e07
@@ -0,0 +1,18 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Check tracepoint probe event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/events/tracepoints/tp_sched" ]; then
+	echo "FAIL: tracepoint probe event tracepoints/tp_sched does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/tracepoints/tp_sched/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: tracepoint probe event tracepoints/tp_sched is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 05-tprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/06-instance.sh new file +23 -0
new file mode 100755
index 000000000000..a95fdddaa6a6
@@ -0,0 +1,23 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Check instance bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/instances/foo" ]; then
+	echo "FAIL: trace instance foo does not exist"
+	exit 1
+fi
+
+if [ ! -d "$TRACEDIR/instances/foo/events/sched/sched_switch" ]; then
+	echo "FAIL: event sched_switch does not exist in instance foo"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/instances/foo/events/sched/sched_switch/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: event sched_switch is not enabled in instance foo ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 06-instance"
+exit 0
tools/testing/selftests/ftrace/config +4 -0
index 544de0db5f58..fec41c6b290e 100644
@@ -1,3 +1,5 @@
+CONFIG_BOOT_CONFIG=y
+CONFIG_BOOTTIME_TRACING=y
 CONFIG_BPF_SYSCALL=y
 CONFIG_DEBUG_INFO_BTF=y
 CONFIG_DEBUG_INFO_DWARF4=y
@@ -8,6 +10,7 @@ CONFIG_FTRACE=y
 CONFIG_FTRACE_SYSCALLS=y
 CONFIG_FUNCTION_GRAPH_RETVAL=y
 CONFIG_FUNCTION_PROFILER=y
+CONFIG_FUNCTION_TRACER=y
 CONFIG_HIST_TRIGGERS=y
 CONFIG_IRQSOFF_TRACER=y
 CONFIG_KALLSYMS_ALL=y
@@ -24,6 +27,7 @@ CONFIG_SAMPLE_TRACE_EVENTS=m
 CONFIG_SAMPLE_TRACE_PRINTK=m
 CONFIG_SCHED_TRACER=y
 CONFIG_STACK_TRACER=y
+CONFIG_SYNTH_EVENTS=y
 CONFIG_TRACER_SNAPSHOT=y
 CONFIG_UPROBES=y
 CONFIG_UPROBE_EVENTS=y