Skip to contentSkip to conversation

selftests/ftrace: Add a boottime tracing test framework

PATCH In review Masami Hiramatsu (Google) 3 participants 10 messages Aug 12 – Aug 14

Subsystems (2)
lore
Share & tools
lore

Masami Hiramatsu (Google)

author maintainer
#0 lore raw

v3 cover selftests/ftrace: Add a boottime tracing test framework

Hi,

This is the v3 series of the boottime tracing test framework to selftests/ftrace. This test builds initramfs from busybox binary and test scripts, and run it on QEMU with modified cmdline and bootconfig.

The previous version is here:

https://lore.kernel.org/all/178646245730.428104.4400499381224001727.stgit@devnote2/

In this version, I fixed some issues pointed by Sashiko[1] and add "# APPLETS:" line support for test script. See below changelog for details.

[1] https://sashiko.dev/#/patchset/178646245730.428104.4400499381224001727.stgit%40devnote2

Changes in v3:
[1/4]
- Require file command and check kernel binary architecture via file -bL instead of matching kernel path string in run_boottime_test.sh.
- Use 'cp -L' when copying BUSYBOX into initramfs to dereference symbolic links.
- Use 'ln -sf' when creating BusyBox applet symlinks in initramfs.
- Add support for '# APPLETS: <applets>' header in test scripts to dynamically create BusyBox applet symlinks in initramfs.
- Update boottime test README to document file command dependency and '# APPLETS:' header syntax.
[3/4]
- Add '# APPLETS: awk' header to cmdline-03-trace-buf-size.sh to request awk applet in initramfs.

Changes in v2:
[1/4]
- Add top-level boottime-ktap wrapper script and include it in TEST_PROGS in Makefile.
- Define BUSYBOX_APPLETS as a global array variable including sleep and sed.
- Require timeout tool in run_boottime_test.sh and README, and always invoke QEMU under timeout command to prevent infinite execution loops.
- Add fallback sleep loop safeguard (sleep 100 || break) in guest init script.
- Support multi-line .qemuopts files using read -r -d '' -a qemu_args.
[3/4]
- Fix buffer_size_kb parsing in cmdline-03-trace-buf-size.sh to handle unexpanded buffer string output '1 (expanded: 2048)'.
- Use grep -qw in cmdline-04-trace-options.sh to match word options and avoid matching disabled 'no<option>' entries.
[4/4]
- Increase reserve_mem alignment to 32M (reserve_mem=12M:32M:trace) to ensure stable physical memory placement across KASLR reboots.

The new run_boottime_test.sh can run tests not only for boottime trace but also other kernel cmdline tracing options, including persistent ring buffer and its backup.

Overview
--------
The test runner (run_boottime_test.sh) constructs a lightweight initramfs with busybox and a tracefs checker script from tests/, attaches optional bootconfigs from bootconfigs/ using the bootconfig tool, applies optional command-line parameters from cmdlines/, boots the target kernel image under QEMU, and reports results according to TAP version 13 format.

Here is an example usage of the test.
-----

$ ./tools/testing/selftests/ftrace/boottime/run_boottime_test.sh -k ${BUILD_PATH}/arch/x86/boot/bzImage
TAP version 13
1..14
ok 1 01-kprobe
ok 2 02-synth
ok 3 03-eprobe
ok 4 04-fprobe
ok 5 05-tprobe
ok 6 06-instance
ok 7 cmdline-01-ftrace
ok 8 cmdline-02-trace-event
ok 9 cmdline-03-trace-buf-size
ok 10 cmdline-04-trace-options
ok 11 cmdline-05-trace-clock
ok 12 cmdline-06-trace-instance
ok 13 persistent-01-reserve-mem
ok 14 persistent-02-backup-instance
# Totals: pass:14 fail:0 xfail:0 xpass:0 skip:0 error:0
-----

Note that this requires QEMU, busybox and some other tools to run, see README for more information. If those tools are not installed, this test will be skipped.

Other discussion
----------------
BTW, I found other similar shell scripts under kselftest for other subsystems. I think we can make an integrated (standard) QEMU test framework in the kselftest common directory and reuse it.
Also, maybe we can reconsider using ktest.pl to run these tests.

Thanks,

---
base-commit: 29a86c5e6361caffa7e75e891169e813f82ff688

Masami Hiramatsu (Google) (4):
selftests/ftrace: Add generic boot tracing test framework selftests/ftrace: Add boot-time tracing testcases selftests/ftrace: Add kernel cmdline tracing testcases selftests/ftrace: Add persistent ring buffer testcases

 tools/testing/selftests/ftrace/Makefile            |    4 
 tools/testing/selftests/ftrace/boottime-ktap       |    6 
 tools/testing/selftests/ftrace/boottime/Makefile   |   10 
 tools/testing/selftests/ftrace/boottime/README     |   74 ++++
 .../ftrace/boottime/bootconfigs/01-kprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/02-synth.bconf     |    4 
 .../ftrace/boottime/bootconfigs/03-eprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/04-fprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/05-tprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/06-instance.bconf  |    5 
 .../boottime/cmdlines/cmdline-01-ftrace.cmdline    |    1 
 .../cmdlines/cmdline-02-trace-event.cmdline        |    1 
 .../cmdlines/cmdline-03-trace-buf-size.cmdline     |    1 
 .../cmdlines/cmdline-04-trace-options.cmdline      |    1 
 .../cmdlines/cmdline-05-trace-clock.cmdline        |    1 
 .../cmdlines/cmdline-06-trace-instance.cmdline     |    1 
 .../cmdlines/persistent-01-reserve-mem.cmdline     |    1 
 .../cmdlines/persistent-02-backup-instance.cmdline |    1 
 .../selftests/ftrace/boottime/run_boottime_test.sh |  403 ++++++++++++++++++++
 .../selftests/ftrace/boottime/tests/01-kprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/02-synth.sh    |   25 +
 .../selftests/ftrace/boottime/tests/03-eprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/04-fprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/05-tprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/06-instance.sh |   30 +
 .../ftrace/boottime/tests/cmdline-01-ftrace.sh     |   19 +
 .../boottime/tests/cmdline-02-trace-event.sh       |   26 +
 .../boottime/tests/cmdline-03-trace-buf-size.sh    |   29 +
 .../boottime/tests/cmdline-04-trace-options.sh     |   23 +
 .../boottime/tests/cmdline-05-trace-clock.sh       |   19 +
 .../boottime/tests/cmdline-06-trace-instance.sh    |   24 +
 .../boottime/tests/persistent-01-reserve-mem.sh    |   28 +
 .../tests/persistent-02-backup-instance.sh         |   40 ++
 tools/testing/selftests/ftrace/config              |    6 
 34 files changed, 897 insertions(+), 2 deletions(-)
 create mode 100755 tools/testing/selftests/ftrace/boottime-ktap
 create mode 100644 tools/testing/selftests/ftrace/boottime/Makefile
 create mode 100644 tools/testing/selftests/ftrace/boottime/README
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-01-ftrace.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-02-trace-event.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-03-trace-buf-size.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-04-trace-options.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-05-trace-clock.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-06-trace-instance.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/persistent-01-reserve-mem.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/persistent-02-backup-instance.cmdline
 create mode 100755 tools/testing/selftests/ftrace/boottime/run_boottime_test.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/02-synth.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/06-instance.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-01-ftrace.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-02-trace-event.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-03-trace-buf-size.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-04-trace-options.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-05-trace-clock.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-06-trace-instance.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh

--
Masami Hiramatsu (Google) <[email protected]>

author maintainer
#1 lore raw

PATCH v3 1/4 selftests/ftrace: Add generic boot tracing test framework

From: Masami Hiramatsu (Google) <[email protected]>

Add a generic test framework under tools/testing/selftests/ftrace/boottime/ to verify kernel boot tracing configurations (bootconfig, kernel command-line tracing options, and persistent ring buffers) during early boot.

The test runner (run_boottime_test.sh) sources ktap_helpers.sh to generate output in TAP version 13 format. It builds a minimal initramfs with busybox, dynamically resolves test specifications (.bconf for bootconfig, .cmdline for kernel boot parameters, .qemuopts for QEMU options, and # REBOOT: 1 for multi-boot crash/reboot tests with doubled timeout), boots QEMU, and runs corresponding tracefs checker scripts.

Assisted-by Antigravity:gemini-3.6-flash

---
Changes in v3:
- Require file command and check kernel binary architecture via file -bL instead of matching kernel path string in run_boottime_test.sh.
- Use 'cp -L' when copying BUSYBOX into initramfs to dereference symbolic links.
- Use 'ln -sf' when creating BusyBox applet symlinks in initramfs.
- Add support for '# APPLETS: <applets>' header in test scripts to dynamically create BusyBox applet symlinks in initramfs.
- Update boottime test README to document file command dependency and '# APPLETS:' header syntax.
Changes in v2:
- Add top-level boottime-ktap wrapper script and include it in TEST_PROGS in Makefile.
- Define BUSYBOX_APPLETS as a global array variable including sleep and sed.
- Require timeout tool in run_boottime_test.sh and README, and always invoke QEMU under timeout command to prevent infinite execution loops.
- Add fallback sleep loop safeguard (sleep 100 || break) in guest init script.
- Support multi-line .qemuopts files using read -r -d '' -a qemu_args.

---
 tools/testing/selftests/ftrace/Makefile            |    4 
 tools/testing/selftests/ftrace/boottime-ktap       |    6 
 tools/testing/selftests/ftrace/boottime/Makefile   |   10 
 tools/testing/selftests/ftrace/boottime/README     |   74 ++++
 .../selftests/ftrace/boottime/run_boottime_test.sh |  403 ++++++++++++++++++++
 tools/testing/selftests/ftrace/config              |    6 
 6 files changed, 501 insertions(+), 2 deletions(-)
 create mode 100755 tools/testing/selftests/ftrace/boottime-ktap
 create mode 100644 tools/testing/selftests/ftrace/boottime/Makefile
 create mode 100644 tools/testing/selftests/ftrace/boottime/README
 create mode 100755 tools/testing/selftests/ftrace/boottime/run_boottime_test.sh
tools/testing/selftests/ftrace/Makefile +2 -2
index 7c12263f8260..3d41f9545683 100644
@@ -2,8 +2,8 @@
 all:
 
 TEST_PROGS_EXTENDED := ftracetest
-TEST_PROGS := ftracetest-ktap
-TEST_FILES := test.d settings
+TEST_PROGS := ftracetest-ktap boottime-ktap
+TEST_FILES := test.d settings boottime
 EXTRA_CLEAN := $(OUTPUT)/logs/*
 
 TEST_GEN_FILES := poll
tools/testing/selftests/ftrace/boottime-ktap new file +6 -0
new file mode 100755
index 000000000000..1fed68c8a944
@@ -0,0 +1,6 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# boottime-ktap: Wrapper to integrate boottime tracing test framework with kselftest runner
+
+exec ./boottime/run_boottime_test.sh "$@"
tools/testing/selftests/ftrace/boottime/Makefile new file +10 -0
new file mode 100644
index 000000000000..fd0975350f10
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+
+all:
+
+run_tests:
+	@./run_boottime_test.sh
+
+clean:
+
+.PHONY: all run_tests clean
tools/testing/selftests/ftrace/boottime/README new file +74 -0

74 lines not on this page · raw message

tools/testing/selftests/ftrace/boottime/run_boottime_test.sh new file +403 -0

403 lines not on this page · raw message

tools/testing/selftests/ftrace/config +6 -0
index 544de0db5f58..29f05f7c6e2c 100644
@@ -1,3 +1,5 @@
+CONFIG_BOOT_CONFIG=y
+CONFIG_BOOTTIME_TRACING=y
 CONFIG_BPF_SYSCALL=y
 CONFIG_DEBUG_INFO_BTF=y
 CONFIG_DEBUG_INFO_DWARF4=y
@@ -8,22 +10,26 @@ CONFIG_FTRACE=y
 CONFIG_FTRACE_SYSCALLS=y
 CONFIG_FUNCTION_GRAPH_RETVAL=y
 CONFIG_FUNCTION_PROFILER=y
+CONFIG_FUNCTION_TRACER=y
 CONFIG_HIST_TRIGGERS=y
 CONFIG_IRQSOFF_TRACER=y
 CONFIG_KALLSYMS_ALL=y
 CONFIG_KPROBES=y
 CONFIG_KPROBE_EVENTS=y
+CONFIG_MAGIC_SYSRQ=y
 CONFIG_MODULES=y
 CONFIG_MODULE_UNLOAD=y
 CONFIG_PREEMPTIRQ_DELAY_TEST=m
 CONFIG_PREEMPT_TRACER=y
 CONFIG_PROBE_EVENTS_BTF_ARGS=y
+CONFIG_RESERVE_MEM=y
 CONFIG_SAMPLES=y
 CONFIG_SAMPLE_FTRACE_DIRECT=m
 CONFIG_SAMPLE_TRACE_EVENTS=m
 CONFIG_SAMPLE_TRACE_PRINTK=m
 CONFIG_SCHED_TRACER=y
 CONFIG_STACK_TRACER=y
+CONFIG_SYNTH_EVENTS=y
 CONFIG_TRACER_SNAPSHOT=y
 CONFIG_UPROBES=y
 CONFIG_UPROBE_EVENTS=y
author maintainer
#2 lore raw

PATCH v3 2/4 selftests/ftrace: Add boot-time tracing testcases

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Add test cases for boot-time tracing configuration via bootconfig (see Documentation/trace/boottime-trace.rst).

These test cases verify:
- 01-kprobe: Dynamic kprobe events via bootconfig
- 02-synth: Synthetic events via bootconfig
- 03-eprobe: Event probes (eprobes) via bootconfig
- 04-fprobe: Function probes (fprobes) via bootconfig
- 05-tprobe: Tracepoint probes via bootconfig
- 06-instance: Custom trace instances via bootconfig

Assisted-by Antigravity:gemini-3.6-flash
---
 .../ftrace/boottime/bootconfigs/01-kprobe.bconf    |    4 +++
 .../ftrace/boottime/bootconfigs/02-synth.bconf     |    4 +++
 .../ftrace/boottime/bootconfigs/03-eprobe.bconf    |    4 +++
 .../ftrace/boottime/bootconfigs/04-fprobe.bconf    |    4 +++
 .../ftrace/boottime/bootconfigs/05-tprobe.bconf    |    4 +++
 .../ftrace/boottime/bootconfigs/06-instance.bconf  |    5 +++
 .../selftests/ftrace/boottime/tests/01-kprobe.sh   |   25 +++++++++++++++++
 .../selftests/ftrace/boottime/tests/02-synth.sh    |   25 +++++++++++++++++
 .../selftests/ftrace/boottime/tests/03-eprobe.sh   |   25 +++++++++++++++++
 .../selftests/ftrace/boottime/tests/04-fprobe.sh   |   25 +++++++++++++++++
 .../selftests/ftrace/boottime/tests/05-tprobe.sh   |   25 +++++++++++++++++
 .../selftests/ftrace/boottime/tests/06-instance.sh |   30 ++++++++++++++++++++
 12 files changed, 180 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/02-synth.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/06-instance.sh
tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..13d30d1e51a6
@@ -0,0 +1,4 @@
+ftrace.event.kprobes.vfs_read {
+	probes = "vfs_read $arg1"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf new file +4 -0
new file mode 100644
index 000000000000..b1f37713f33c
@@ -0,0 +1,4 @@
+ftrace.event.synthetic.boot_lat {
+	fields = "unsigned long id", "u64 delta"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..c1a8138e605d
@@ -0,0 +1,4 @@
+ftrace.event.eprobes.ep_read {
+	probes = "sched/sched_switch"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..82ae2ac0629f
@@ -0,0 +1,4 @@
+ftrace.event.fprobes.fp_read {
+	probes = "vfs_read"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf new file +4 -0
new file mode 100644
index 000000000000..02a2803c770a
@@ -0,0 +1,4 @@
+ftrace.event.tracepoints.tp_sched {
+	probes = "sched_switch"
+	enable
+}
tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf new file +5 -0
new file mode 100644
index 000000000000..3fdb6717afe2
@@ -0,0 +1,5 @@
+ftrace.instance.foo {
+	event.sched.sched_switch {
+		enable
+	}
+}
tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh new file +25 -0
new file mode 100644
index 000000000000..6ed7caca4c5f
@@ -0,0 +1,25 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check kprobe bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ -f /proc/bootconfig ] && grep -q "dump_bconf" /proc/cmdline 2>/dev/null; then
+	echo "=== /proc/bootconfig ==="
+	cat /proc/bootconfig
+	echo "========================"
+fi
+
+if [ ! -d "$TRACEDIR/events/kprobes/vfs_read" ]; then
+	echo "FAIL: kprobe event kprobes/vfs_read does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/kprobes/vfs_read/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: kprobe event kprobes/vfs_read is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 01-kprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/02-synth.sh new file +25 -0
new file mode 100644
index 000000000000..745c328dde15
@@ -0,0 +1,25 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check synthetic event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ -f /proc/bootconfig ] && grep -q "dump_bconf" /proc/cmdline 2>/dev/null; then
+	echo "=== /proc/bootconfig ==="
+	cat /proc/bootconfig
+	echo "========================"
+fi
+
+if [ ! -d "$TRACEDIR/events/synthetic/boot_lat" ]; then
+	echo "FAIL: synthetic event synthetic/boot_lat does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/synthetic/boot_lat/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: synthetic event synthetic/boot_lat is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 02-synth"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh new file +25 -0
new file mode 100644
index 000000000000..9b3b7ab23c69
@@ -0,0 +1,25 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check eprobe event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ -f /proc/bootconfig ] && grep -q "dump_bconf" /proc/cmdline 2>/dev/null; then
+	echo "=== /proc/bootconfig ==="
+	cat /proc/bootconfig
+	echo "========================"
+fi
+
+if [ ! -d "$TRACEDIR/events/eprobes/ep_read" ]; then
+	echo "FAIL: eprobe event eprobes/ep_read does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/eprobes/ep_read/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: eprobe event eprobes/ep_read is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 03-eprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh new file +25 -0
new file mode 100644
index 000000000000..27e090c28a4b
@@ -0,0 +1,25 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check fprobe event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ -f /proc/bootconfig ] && grep -q "dump_bconf" /proc/cmdline 2>/dev/null; then
+	echo "=== /proc/bootconfig ==="
+	cat /proc/bootconfig
+	echo "========================"
+fi
+
+if [ ! -d "$TRACEDIR/events/fprobes/fp_read" ]; then
+	echo "FAIL: fprobe event fprobes/fp_read does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/fprobes/fp_read/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: fprobe event fprobes/fp_read is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 04-fprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh new file +25 -0
new file mode 100644
index 000000000000..810d34055f94
@@ -0,0 +1,25 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check tracepoint probe event bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ -f /proc/bootconfig ] && grep -q "dump_bconf" /proc/cmdline 2>/dev/null; then
+	echo "=== /proc/bootconfig ==="
+	cat /proc/bootconfig
+	echo "========================"
+fi
+
+if [ ! -d "$TRACEDIR/events/tracepoints/tp_sched" ]; then
+	echo "FAIL: tracepoint probe event tracepoints/tp_sched does not exist"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/events/tracepoints/tp_sched/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: tracepoint probe event tracepoints/tp_sched is not enabled ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 05-tprobe"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/06-instance.sh new file +30 -0
new file mode 100644
index 000000000000..f6e8519912e4
@@ -0,0 +1,30 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check instance bootconfig settings on tracefs
+TRACEDIR="/sys/kernel/tracing"
+
+if [ -f /proc/bootconfig ] && grep -q "dump_bconf" /proc/cmdline 2>/dev/null; then
+	echo "=== /proc/bootconfig ==="
+	cat /proc/bootconfig
+	echo "========================"
+fi
+
+if [ ! -d "$TRACEDIR/instances/foo" ]; then
+	echo "FAIL: trace instance foo does not exist"
+	exit 1
+fi
+
+if [ ! -d "$TRACEDIR/instances/foo/events/sched/sched_switch" ]; then
+	echo "FAIL: event sched_switch does not exist in instance foo"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/instances/foo/events/sched/sched_switch/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: event sched_switch is not enabled in instance foo ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: 06-instance"
+exit 0
author maintainer
#3 lore raw

PATCH v3 3/4 selftests/ftrace: Add kernel cmdline tracing testcases

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Add test cases for kernel command-line tracing options. It verifies various parameters like trace_buf_size and trace_options by reading the output of tracing files in sysfs.

Assisted-by Antigravity:gemini-3.6-flash

---
Changes in v3:
- Add '# APPLETS: awk' header to cmdline-03-trace-buf-size.sh to request awk applet in initramfs.
Changes in v2:
- Fix buffer_size_kb parsing in cmdline-03-trace-buf-size.sh to handle unexpanded buffer string output '1 (expanded: 2048)'.
- Use grep -qw in cmdline-04-trace-options.sh to match word options and avoid matching disabled 'no<option>' entries.

---
 .../boottime/cmdlines/cmdline-01-ftrace.cmdline    |    1 +
 .../cmdlines/cmdline-02-trace-event.cmdline        |    1 +
 .../cmdlines/cmdline-03-trace-buf-size.cmdline     |    1 +
 .../cmdlines/cmdline-04-trace-options.cmdline      |    1 +
 .../cmdlines/cmdline-05-trace-clock.cmdline        |    1 +
 .../cmdlines/cmdline-06-trace-instance.cmdline     |    1 +
 .../ftrace/boottime/tests/cmdline-01-ftrace.sh     |   19 +++++++++++++
 .../boottime/tests/cmdline-02-trace-event.sh       |   26 ++++++++++++++++++
 .../boottime/tests/cmdline-03-trace-buf-size.sh    |   29 ++++++++++++++++++++
 .../boottime/tests/cmdline-04-trace-options.sh     |   23 ++++++++++++++++
 .../boottime/tests/cmdline-05-trace-clock.sh       |   19 +++++++++++++
 .../boottime/tests/cmdline-06-trace-instance.sh    |   24 +++++++++++++++++
 12 files changed, 146 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-01-ftrace.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-02-trace-event.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-03-trace-buf-size.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-04-trace-options.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-05-trace-clock.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-06-trace-instance.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-01-ftrace.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-02-trace-event.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-03-trace-buf-size.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-04-trace-options.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-05-trace-clock.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-06-trace-instance.sh
tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-01-ftrace.cmdline new file +1 -0
new file mode 100644
index 000000000000..4f6fc54240c8
@@ -0,0 +1 @@
+ftrace=function
tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-02-trace-event.cmdline new file +1 -0
new file mode 100644
index 000000000000..66bcd29b14cb
@@ -0,0 +1 @@
+trace_event=sched:sched_switch,kmem:kmalloc
tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-03-trace-buf-size.cmdline new file +1 -0
new file mode 100644
index 000000000000..e5d1c4a8ca89
@@ -0,0 +1 @@
+trace_buf_size=2048K
tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-04-trace-options.cmdline new file +1 -0
new file mode 100644
index 000000000000..51d82f8e948d
@@ -0,0 +1 @@
+trace_options=sym-addr,verbose
tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-05-trace-clock.cmdline new file +1 -0
new file mode 100644
index 000000000000..f72a4ac808a3
@@ -0,0 +1 @@
+trace_clock=global
tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-06-trace-instance.cmdline new file +1 -0
new file mode 100644
index 000000000000..9618e3e2b88b
@@ -0,0 +1 @@
+trace_instance=bar,sched:sched_switch
tools/testing/selftests/ftrace/boottime/tests/cmdline-01-ftrace.sh new file +19 -0
new file mode 100644
index 000000000000..23a0b15f2dd3
@@ -0,0 +1,19 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check ftrace= kernel command-line tracer setting
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -f "$TRACEDIR/current_tracer" ]; then
+	echo "FAIL: current_tracer does not exist"
+	exit 1
+fi
+
+read -r TRACER _ < "$TRACEDIR/current_tracer"
+if [ "$TRACER" != "function" ]; then
+	echo "FAIL: current_tracer is '$TRACER', expected 'function'"
+	exit 1
+fi
+
+echo "PASS: cmdline-01-ftrace"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/cmdline-02-trace-event.sh new file +26 -0
new file mode 100644
index 000000000000..b793907c48cd
@@ -0,0 +1,26 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check trace_event= kernel command-line setting
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/events/sched/sched_switch" ]; then
+	echo "FAIL: event sched:sched_switch does not exist"
+	exit 1
+fi
+
+if [ ! -d "$TRACEDIR/events/kmem/kmalloc" ]; then
+	echo "FAIL: event kmem:kmalloc does not exist"
+	exit 1
+fi
+
+ENABLE1=$(cat "$TRACEDIR/events/sched/sched_switch/enable")
+ENABLE2=$(cat "$TRACEDIR/events/kmem/kmalloc/enable")
+
+if [ "$ENABLE1" != "1" ] || [ "$ENABLE2" != "1" ]; then
+	echo "FAIL: events not enabled (sched_switch=$ENABLE1, kmalloc=$ENABLE2)"
+	exit 1
+fi
+
+echo "PASS: cmdline-02-trace-event"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/cmdline-03-trace-buf-size.sh new file +29 -0
new file mode 100644
index 000000000000..4b936f84d4d7
@@ -0,0 +1,29 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# APPLETS: awk
+# Check trace_buf_size= kernel command-line setting
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -f "$TRACEDIR/buffer_size_kb" ]; then
+	echo "FAIL: buffer_size_kb does not exist"
+	exit 1
+fi
+
+BUF_RAW=$(cat "$TRACEDIR/buffer_size_kb")
+case "$BUF_RAW" in
+	*"expanded:"*)
+		BUFSIZE=$(echo "$BUF_RAW" | sed -n 's/.*expanded: *\([0-9]*\).*/\1/p')
+		;;
+	*)
+		BUFSIZE=$(echo "$BUF_RAW" | awk '{print $1}')
+		;;
+esac
+
+if [ -z "$BUFSIZE" ] || [ "$BUFSIZE" -lt 2048 ]; then
+	echo "FAIL: buffer_size_kb is '$BUF_RAW', expected >= 2048"
+	exit 1
+fi
+
+echo "PASS: cmdline-03-trace-buf-size"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/cmdline-04-trace-options.sh new file +23 -0
new file mode 100644
index 000000000000..53a65b55f7a7
@@ -0,0 +1,23 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check trace_options= kernel command-line setting
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -f "$TRACEDIR/trace_options" ]; then
+	echo "FAIL: trace_options file does not exist"
+	exit 1
+fi
+
+if ! grep -qw "sym-addr" "$TRACEDIR/trace_options"; then
+	echo "FAIL: sym-addr option is not set in trace_options"
+	exit 1
+fi
+
+if ! grep -qw "verbose" "$TRACEDIR/trace_options"; then
+	echo "FAIL: verbose option is not set in trace_options"
+	exit 1
+fi
+
+echo "PASS: cmdline-04-trace-options"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/cmdline-05-trace-clock.sh new file +19 -0
new file mode 100644
index 000000000000..d6de9f6e2090
@@ -0,0 +1,19 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check trace_clock= kernel command-line setting
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -f "$TRACEDIR/trace_clock" ]; then
+	echo "FAIL: trace_clock file does not exist"
+	exit 1
+fi
+
+if ! grep -q '\[global\]' "$TRACEDIR/trace_clock"; then
+	CLOCK=$(cat "$TRACEDIR/trace_clock")
+	echo "FAIL: trace_clock is not set to global ($CLOCK)"
+	exit 1
+fi
+
+echo "PASS: cmdline-05-trace-clock"
+exit 0
tools/testing/selftests/ftrace/boottime/tests/cmdline-06-trace-instance.sh new file +24 -0
new file mode 100644
index 000000000000..aae7f0a86d81
@@ -0,0 +1,24 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check trace_instance= kernel command-line setting
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -d "$TRACEDIR/instances/bar" ]; then
+	echo "FAIL: trace instance bar does not exist"
+	exit 1
+fi
+
+if [ ! -d "$TRACEDIR/instances/bar/events/sched/sched_switch" ]; then
+	echo "FAIL: event sched_switch does not exist in instance bar"
+	exit 1
+fi
+
+ENABLE=$(cat "$TRACEDIR/instances/bar/events/sched/sched_switch/enable")
+if [ "$ENABLE" != "1" ]; then
+	echo "FAIL: event sched_switch is not enabled in instance bar ($ENABLE)"
+	exit 1
+fi
+
+echo "PASS: cmdline-06-trace-instance"
+exit 0
author maintainer
#4 lore raw

PATCH v3 4/4 selftests/ftrace: Add persistent ring buffer testcases

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Add test cases for persistent ring buffer (reserve_mem= combined with trace_instance=) and backup instance (trace_instance=backup=boot_map).

These test cases verify:
- persistent-01-reserve-mem: Memory reservation via reserve_mem= and trace_instance=boot_map@trace. Verifies trace data retention in the boot_map instance across guest crash/reboot via sysrq-trigger.
- persistent-02-backup-instance: Backup instance creation via trace_instance=backup=boot_map. Verifies that previous boot trace log is preserved in the backup instance on the subsequent boot.

These test cases specify '# REBOOT: 1' to enable multi-boot guest crash/reboot testing with doubled QEMU timeout.

Assisted-by Antigravity:gemini-3.6-flash

---
Changes in v2:
- Increase reserve_mem alignment to 32M (reserve_mem=12M:32M:trace) to ensure stable physical memory placement across KASLR reboots.

---
 .../cmdlines/persistent-01-reserve-mem.cmdline     |    1 +
 .../cmdlines/persistent-02-backup-instance.cmdline |    1 +
 .../boottime/tests/persistent-01-reserve-mem.sh    |   28 ++++++++++++++
 .../tests/persistent-02-backup-instance.sh         |   40 ++++++++++++++++++++
 4 files changed, 70 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/persistent-01-reserve-mem.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/persistent-02-backup-instance.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh
tools/testing/selftests/ftrace/boottime/cmdlines/persistent-01-reserve-mem.cmdline new file +1 -0
new file mode 100644
index 000000000000..c69eeb27deb6
@@ -0,0 +1 @@
+reserve_mem=12M:32M:trace trace_instance=boot_map@trace
tools/testing/selftests/ftrace/boottime/cmdlines/persistent-02-backup-instance.cmdline new file +1 -0
new file mode 100644
index 000000000000..43217c986ca1
@@ -0,0 +1 @@
+reserve_mem=12M:32M:trace trace_instance=boot_map@trace trace_instance=backup=boot_map
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +28 -0
new file mode 100644
index 000000000000..49abaf518481
@@ -0,0 +1,28 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check persistent ring buffer across guest crash/reboot
+# REBOOT: 1
+TRACEDIR="/sys/kernel/tracing"
+
+if [ ! -f "$TRACEDIR/instances/boot_map/trace" ]; then
+	echo "FAIL: persistent trace instance boot_map/trace file missing"
+	exit 1
+fi
+
+# Check if Boot 1 marker was already written
+if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/boot_map/trace" 2>/dev/null; then
+	# Second boot: verify persistent ring buffer content from first boot
+	echo "PASS: persistent-01-reserve-mem"
+	exit 0
+fi
+
+# First boot: write marker to persistent buffer and trigger kernel crash/reboot
+echo "BOOT1_MARKER" > "$TRACEDIR/instances/boot_map/trace_marker"
+sync
+
+# Trigger reboot to restart into second boot
+echo b > /proc/sysrq-trigger 2>/dev/null || echo c > /proc/sysrq-trigger 2>/dev/null || true
+sleep 5
+echo "FAIL: reboot trigger failed on first boot"
+exit 1
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +40 -0
new file mode 100644
index 000000000000..0b2a2f03ce4f
@@ -0,0 +1,40 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2026, Google LLC.
+# Check persistent backup instance across guest crash/reboot
+# REBOOT: 1
+TRACEDIR="/sys/kernel/tracing"
+
+# Ensure both boot_map/trace and backup/trace files exist
+if [ ! -f "$TRACEDIR/instances/boot_map/trace" ]; then
+	echo "FAIL: boot_map/trace file missing"
+	exit 1
+fi
+
+if [ ! -f "$TRACEDIR/instances/backup/trace" ]; then
+	echo "FAIL: backup/trace file missing"
+	exit 1
+fi
+
+# Check if BOOT1_MARKER is in boot_map/trace (indicates second boot)
+if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/boot_map/trace" 2>/dev/null; then
+	# Second boot: verify BOOT1_MARKER was copied into backup/trace from Boot 1
+	if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/backup/trace" 2>/dev/null; then
+		echo "PASS: persistent-02-backup-instance"
+		exit 0
+	else
+		echo "FAIL: BOOT1_MARKER found in boot_map/trace" \
+			"but missing from backup/trace on second boot"
+		exit 1
+	fi
+fi
+
+# First boot: write BOOT1_MARKER to boot_map and reboot via sysrq-trigger
+echo "BOOT1_MARKER" > "$TRACEDIR/instances/boot_map/trace_marker"
+sync
+
+# Trigger reboot to restart into second boot
+echo b > /proc/sysrq-trigger 2>/dev/null || echo c > /proc/sysrq-trigger 2>/dev/null || true
+sleep 5
+echo "FAIL: reboot trigger failed on first boot"
+exit 1
#5 lore raw

Re: [PATCH v3 4/4] selftests/ftrace: Add persistent ring buffer testcases

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Test enters an infinite reboot loop if the persistent trace feature fails. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Test enters an infinite reboot loop if the persistent trace feature fails. --

--- Patch [4]: [PATCH v3 4/4] selftests/ftrace: Add persistent ring buffer testcases --- commit d365a5483f67b1295e536e6d26fd4ee7166e27b3
Author: Masami Hiramatsu (Google) <[email protected]>

selftests/ftrace: Add persistent ring buffer testcases

Add test cases for persistent ring buffer and backup instance across guest crash/reboot. These verify trace data retention in the boot_map instance and preservation in the backup instance on subsequent boot.

Quoted 6 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +0 -0
new file mode 100644
index 0000000000000..49abaf518481f
@@ -0,0 +1,28 @@

[ ... ]

Quoted 4 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +4 -0
+
+# Check if Boot 1 marker was already written
+if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/boot_map/trace" 2>/dev/null; then
+	# Second boot: verify persistent ring buffer content from first boot

tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh line 25

Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +10 -0
+	echo "PASS: persistent-01-reserve-mem"
+	exit 0
+fi
+
+# First boot: write marker to persistent buffer and trigger kernel crash/reboot
+echo "BOOT1_MARKER" > "$TRACEDIR/instances/boot_map/trace_marker"
+sync
+
+# Trigger reboot to restart into second boot
+echo b > /proc/sysrq-trigger 2>/dev/null || echo c > /proc/sysrq-trigger 2>/dev/null || true

[Severity: Medium]
If the persistent trace feature fails to retain the BOOT1_MARKER across reboot, could this script enter an infinite reboot loop?

Since the test falls through to the first-boot logic when the marker is missing, it appears it will write the marker and reboot again indefinitely. This might cause the test to run until it is killed by a global timeout rather than failing cleanly.

Quoted 9 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +3 -0
+sleep 5
+echo "FAIL: reboot trigger failed on first boot"
+exit 1
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +0 -0
new file mode 100644
index 0000000000000..0b2a2f03ce4f2
@@ -0,0 +1,40 @@

[ ... ]

Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +10 -0
+
+# Check if BOOT1_MARKER is in boot_map/trace (indicates second boot)
+if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/boot_map/trace" 2>/dev/null; then
+	# Second boot: verify BOOT1_MARKER was copied into backup/trace from Boot 1
+	if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/backup/trace" 2>/dev/null; then
+		echo "PASS: persistent-02-backup-instance"
+		exit 0
+	else
+		echo "FAIL: BOOT1_MARKER found in boot_map/trace" \
+			"but missing from backup/trace on second boot"

tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh line 37

Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +10 -0
+		exit 1
+	fi
+fi
+
+# First boot: write BOOT1_MARKER to boot_map and reboot via sysrq-trigger
+echo "BOOT1_MARKER" > "$TRACEDIR/instances/boot_map/trace_marker"
+sync
+
+# Trigger reboot to restart into second boot
+echo b > /proc/sysrq-trigger 2>/dev/null || echo c > /proc/sysrq-trigger 2>/dev/null || true

[Severity: Medium]
Similarly, does this script also risk an infinite reboot loop if the marker is missing from boot_map/trace on the second boot?

It looks like it would also fall back to triggering another reboot instead of reporting a failure.

Quoted 3 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +3 -0
+sleep 5
+echo "FAIL: reboot trigger failed on first boot"
+exit 1
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/178649540853.438282.4271870161169100680.stgit@devnote2?part=4
author maintainer
#6 lore raw

Re: [PATCH v3 4/4] selftests/ftrace: Add persistent ring buffer testcases

Quoted 42 lines · On Wed, 12 Aug 2026 00:57:31 +0000 [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Test enters an infinite reboot loop if the persistent trace feature fails. --

--- Patch [4]: [PATCH v3 4/4] selftests/ftrace: Add persistent ring buffer testcases --- commit d365a5483f67b1295e536e6d26fd4ee7166e27b3
Author: Masami Hiramatsu (Google) <[email protected]>

selftests/ftrace: Add persistent ring buffer testcases

Add test cases for persistent ring buffer and backup instance across guest crash/reboot. These verify trace data retention in the boot_map instance and preservation in the backup instance on subsequent boot.

Quoted 6 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +0 -0
new file mode 100644
index 0000000000000..49abaf518481f
@@ -0,0 +1,28 @@

[ ... ]

Quoted 4 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +4 -0
+
+# Check if Boot 1 marker was already written
+if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/boot_map/trace" 2>/dev/null; then
+	# Second boot: verify persistent ring buffer content from first boot

tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh line 25

Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +10 -0
+	echo "PASS: persistent-01-reserve-mem"
+	exit 0
+fi
+
+# First boot: write marker to persistent buffer and trigger kernel crash/reboot
+echo "BOOT1_MARKER" > "$TRACEDIR/instances/boot_map/trace_marker"
+sync
+
+# Trigger reboot to restart into second boot
+echo b > /proc/sysrq-trigger 2>/dev/null || echo c > /proc/sysrq-trigger 2>/dev/null || true

[Severity: Medium]
If the persistent trace feature fails to retain the BOOT1_MARKER across reboot, could this script enter an infinite reboot loop?

Quoted 4 lines

Since the test falls through to the first-boot logic when the marker is missing, it appears it will write the marker and reboot again indefinitely. This might cause the test to run until it is killed by a global timeout rather than failing cleanly.

Yes, that is why I made the timeout command required. Since the test uses only initramfs, we can not save the state on it. Of course we can use virtio-fs or something like it, but it is too much. If we can handle such failure via timeout, it is good enough.

Quoted 38 lines
Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh new file +3 -0
+sleep 5
+echo "FAIL: reboot trigger failed on first boot"
+exit 1
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +0 -0
new file mode 100644
index 0000000000000..0b2a2f03ce4f2
@@ -0,0 +1,40 @@

[ ... ]

Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +10 -0
+
+# Check if BOOT1_MARKER is in boot_map/trace (indicates second boot)
+if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/boot_map/trace" 2>/dev/null; then
+	# Second boot: verify BOOT1_MARKER was copied into backup/trace from Boot 1
+	if grep -q "BOOT1_MARKER" "$TRACEDIR/instances/backup/trace" 2>/dev/null; then
+		echo "PASS: persistent-02-backup-instance"
+		exit 0
+	else
+		echo "FAIL: BOOT1_MARKER found in boot_map/trace" \
+			"but missing from backup/trace on second boot"

tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh line 37

Quoted 10 lines
tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh new file +10 -0
+		exit 1
+	fi
+fi
+
+# First boot: write BOOT1_MARKER to boot_map and reboot via sysrq-trigger
+echo "BOOT1_MARKER" > "$TRACEDIR/instances/boot_map/trace_marker"
+sync
+
+# Trigger reboot to restart into second boot
+echo b > /proc/sysrq-trigger 2>/dev/null || echo c > /proc/sysrq-trigger 2>/dev/null || true

[Severity: Medium]
Similarly, does this script also risk an infinite reboot loop if the marker is missing from boot_map/trace on the second boot?

Quoted 2 lines

It looks like it would also fall back to triggering another reboot instead of reporting a failure.

Ditto. It is eventually handled by timeout failure.

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
author maintainer
#7 lore raw
In reply to Masami Hiramatsu (Google)

Hi Steve,

Would you have any comment on this? I would like to pick this to probes/for-next for ensuring boottime tracing works correctly.

Thank you,

Quoted 178 lines · On Wed, 12 Aug 2026 09:43:28 +0900 "Masami Hiramatsu (Google)"…

Hi,

This is the v3 series of the boottime tracing test framework to selftests/ftrace. This test builds initramfs from busybox binary and test scripts, and run it on QEMU with modified cmdline and bootconfig.

The previous version is here:

https://lore.kernel.org/all/178646245730.428104.4400499381224001727.stgit@devnote2/

In this version, I fixed some issues pointed by Sashiko[1] and add "# APPLETS:" line support for test script. See below changelog for details.

[1] https://sashiko.dev/#/patchset/178646245730.428104.4400499381224001727.stgit%40devnote2

Changes in v3:
[1/4]
- Require file command and check kernel binary architecture via file -bL instead of matching kernel path string in run_boottime_test.sh.
- Use 'cp -L' when copying BUSYBOX into initramfs to dereference symbolic links.
- Use 'ln -sf' when creating BusyBox applet symlinks in initramfs.
- Add support for '# APPLETS: <applets>' header in test scripts to dynamically create BusyBox applet symlinks in initramfs.
- Update boottime test README to document file command dependency and '# APPLETS:' header syntax.
[3/4]
- Add '# APPLETS: awk' header to cmdline-03-trace-buf-size.sh to request awk applet in initramfs.

Changes in v2:
[1/4]
- Add top-level boottime-ktap wrapper script and include it in TEST_PROGS in Makefile.
- Define BUSYBOX_APPLETS as a global array variable including sleep and sed.
- Require timeout tool in run_boottime_test.sh and README, and always invoke QEMU under timeout command to prevent infinite execution loops.
- Add fallback sleep loop safeguard (sleep 100 || break) in guest init script.
- Support multi-line .qemuopts files using read -r -d '' -a qemu_args.
[3/4]
- Fix buffer_size_kb parsing in cmdline-03-trace-buf-size.sh to handle unexpanded buffer string output '1 (expanded: 2048)'.
- Use grep -qw in cmdline-04-trace-options.sh to match word options and avoid matching disabled 'no<option>' entries.
[4/4]
- Increase reserve_mem alignment to 32M (reserve_mem=12M:32M:trace) to ensure stable physical memory placement across KASLR reboots.

The new run_boottime_test.sh can run tests not only for boottime trace but also other kernel cmdline tracing options, including persistent ring buffer and its backup.

Overview
--------
The test runner (run_boottime_test.sh) constructs a lightweight initramfs with busybox and a tracefs checker script from tests/, attaches optional bootconfigs from bootconfigs/ using the bootconfig tool, applies optional command-line parameters from cmdlines/, boots the target kernel image under QEMU, and reports results according to TAP version 13 format.

Here is an example usage of the test.
-----

$ ./tools/testing/selftests/ftrace/boottime/run_boottime_test.sh -k ${BUILD_PATH}/arch/x86/boot/bzImage
TAP version 13
1..14
ok 1 01-kprobe
ok 2 02-synth
ok 3 03-eprobe
ok 4 04-fprobe
ok 5 05-tprobe
ok 6 06-instance
ok 7 cmdline-01-ftrace
ok 8 cmdline-02-trace-event
ok 9 cmdline-03-trace-buf-size
ok 10 cmdline-04-trace-options
ok 11 cmdline-05-trace-clock
ok 12 cmdline-06-trace-instance
ok 13 persistent-01-reserve-mem
ok 14 persistent-02-backup-instance
# Totals: pass:14 fail:0 xfail:0 xpass:0 skip:0 error:0
-----

Note that this requires QEMU, busybox and some other tools to run, see README for more information. If those tools are not installed, this test will be skipped.

Other discussion
----------------
BTW, I found other similar shell scripts under kselftest for other subsystems. I think we can make an integrated (standard) QEMU test framework in the kselftest common directory and reuse it.
Also, maybe we can reconsider using ktest.pl to run these tests.

Thanks,

---
base-commit: 29a86c5e6361caffa7e75e891169e813f82ff688

Masami Hiramatsu (Google) (4):
selftests/ftrace: Add generic boot tracing test framework selftests/ftrace: Add boot-time tracing testcases selftests/ftrace: Add kernel cmdline tracing testcases selftests/ftrace: Add persistent ring buffer testcases

 tools/testing/selftests/ftrace/Makefile            |    4 
 tools/testing/selftests/ftrace/boottime-ktap       |    6 
 tools/testing/selftests/ftrace/boottime/Makefile   |   10 
 tools/testing/selftests/ftrace/boottime/README     |   74 ++++
 .../ftrace/boottime/bootconfigs/01-kprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/02-synth.bconf     |    4 
 .../ftrace/boottime/bootconfigs/03-eprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/04-fprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/05-tprobe.bconf    |    4 
 .../ftrace/boottime/bootconfigs/06-instance.bconf  |    5 
 .../boottime/cmdlines/cmdline-01-ftrace.cmdline    |    1 
 .../cmdlines/cmdline-02-trace-event.cmdline        |    1 
 .../cmdlines/cmdline-03-trace-buf-size.cmdline     |    1 
 .../cmdlines/cmdline-04-trace-options.cmdline      |    1 
 .../cmdlines/cmdline-05-trace-clock.cmdline        |    1 
 .../cmdlines/cmdline-06-trace-instance.cmdline     |    1 
 .../cmdlines/persistent-01-reserve-mem.cmdline     |    1 
 .../cmdlines/persistent-02-backup-instance.cmdline |    1 
 .../selftests/ftrace/boottime/run_boottime_test.sh |  403 ++++++++++++++++++++
 .../selftests/ftrace/boottime/tests/01-kprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/02-synth.sh    |   25 +
 .../selftests/ftrace/boottime/tests/03-eprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/04-fprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/05-tprobe.sh   |   25 +
 .../selftests/ftrace/boottime/tests/06-instance.sh |   30 +
 .../ftrace/boottime/tests/cmdline-01-ftrace.sh     |   19 +
 .../boottime/tests/cmdline-02-trace-event.sh       |   26 +
 .../boottime/tests/cmdline-03-trace-buf-size.sh    |   29 +
 .../boottime/tests/cmdline-04-trace-options.sh     |   23 +
 .../boottime/tests/cmdline-05-trace-clock.sh       |   19 +
 .../boottime/tests/cmdline-06-trace-instance.sh    |   24 +
 .../boottime/tests/persistent-01-reserve-mem.sh    |   28 +
 .../tests/persistent-02-backup-instance.sh         |   40 ++
 tools/testing/selftests/ftrace/config              |    6 
 34 files changed, 897 insertions(+), 2 deletions(-)
 create mode 100755 tools/testing/selftests/ftrace/boottime-ktap
 create mode 100644 tools/testing/selftests/ftrace/boottime/Makefile
 create mode 100644 tools/testing/selftests/ftrace/boottime/README
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/01-kprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/02-synth.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/03-eprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/04-fprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/05-tprobe.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/bootconfigs/06-instance.bconf
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-01-ftrace.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-02-trace-event.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-03-trace-buf-size.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-04-trace-options.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-05-trace-clock.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/cmdline-06-trace-instance.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/persistent-01-reserve-mem.cmdline
 create mode 100644 tools/testing/selftests/ftrace/boottime/cmdlines/persistent-02-backup-instance.cmdline
 create mode 100755 tools/testing/selftests/ftrace/boottime/run_boottime_test.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/01-kprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/02-synth.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/03-eprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/04-fprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/05-tprobe.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/06-instance.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-01-ftrace.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-02-trace-event.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-03-trace-buf-size.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-04-trace-options.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-05-trace-clock.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/cmdline-06-trace-instance.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/persistent-01-reserve-mem.sh
 create mode 100644 tools/testing/selftests/ftrace/boottime/tests/persistent-02-backup-instance.sh

--
Masami Hiramatsu (Google) <[email protected]>

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
maintainer
#8 lore raw
Quoted 4 lines · On Sat, 15 Aug 2026 01:05:24 +0900 Masami Hiramatsu (Google)…

Hi Steve,

Would you have any comment on this? I would like to pick this to probes/for-next for ensuring boottime tracing works correctly.

Hmm, it's a bit too close to the merge window for a good review of it. I'll be traveling next week and I'm trying to get all my pull requests ready before then. You sent v1 on Monday and v3 on Tuesday. Sounds to me a bit rushed when you have 3 versions during the -rc7 release week.

This isn't a trivial feature. I think we should let it settle during the next merge cycle to do it properly.

-- Steve

author maintainer
#9 lore raw
Quoted 8 lines · On Fri, 14 Aug 2026 13:36:10 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 4 lines · On Sat, 15 Aug 2026 01:05:24 +0900 Masami Hiramatsu (Google)…

Hi Steve,

Would you have any comment on this? I would like to pick this to probes/for-next for ensuring boottime tracing works correctly.

Quoted 7 lines

Hmm, it's a bit too close to the merge window for a good review of it. I'll be traveling next week and I'm trying to get all my pull requests ready before then. You sent v1 on Monday and v3 on Tuesday. Sounds to me a bit rushed when you have 3 versions during the -rc7 release week.

This isn't a trivial feature. I think we should let it settle during the next merge cycle to do it properly.

OK, then I'll drop this from for-next.

Thanks!

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>