Skip to contentSkip to conversation

tracing: Fix a kernel crash related to :mod: command

PATCH New Masami Hiramatsu (Google) 1 participant 3 messages Jul 17

Subsystems (2)
lore
Share & tools
lore

Masami Hiramatsu (Google)

author maintainer
#0 lore raw

cover tracing: Fix a kernel crash related to :mod: command

Hi,

I found a kernel crash bug in tracing with :mod: command and dynamic events. If we leaves a dynamic event and run :mod: command to set_event, the kernel gets an oops.

Actually, this bug found by testing wprobe trigger test case, because ftracetest has another bug which does not clean up the top-level event filters/triggers before starting instance tests. Thus if the last test case in the top-level instance fails and if it leaves a dynamic event, that will be kept and not cleaned up. Then, test.d/event/event-mod.tc hits this bug.

Here is the reproduce script.
-----
#!/bin/sh

TRACE_DIR="/sys/kernel/tracing"
if [ ! -d "$TRACE_DIR" ]; then
TRACE_DIR="/sys/kernel/debug/tracing"
fi

if [ ! -d "$TRACE_DIR" ]; then
echo "Error: Tracefs not found. Please mount tracefs." exit 1
fi

cd "$TRACE_DIR"

echo "=== 1. Clear existing triggers and dynamic events ===" echo > events/sched/sched_process_fork/trigger || true echo > dynamic_events || true

echo "=== 2. Create a kprobe event ==="
# Adding a kprobe on vfs_write
echo 'p:kprobes/my_kprobe vfs_write' >> dynamic_events cat dynamic_events

echo "=== 3. Attach enable_event trigger targeting my_kprobe to sched_process_fork ===" if [ -f "events/sched/sched_process_fork/trigger" ]; then
echo 'enable_event:kprobes:my_kprobe' > events/sched/sched_process_fork/trigger cat events/sched/sched_process_fork/trigger
else
echo "Error: sched_process_fork event trigger is not supported."
exit 1
fi

echo "=== 4. Try to delete the kprobe event while the trigger is active ==="
echo "Attempting: echo -:kprobes/my_kprobe >> dynamic_events"
if ! echo '-:kprobes/my_kprobe' >> dynamic_events 2>/dev/null; then
echo "Result: Successfully reproduced kprobe event deletion failure (Device or resource busy)!" echo "The 'my_kprobe' event remains in dynamic_events:"
cat dynamic_events
else
echo "Warning: Deletion succeeded? Check if the trigger was active."
fi

echo ""
echo "=== WARNING: The next step will crash the kernel if the union collision bug is not fixed! ===" echo "Press Ctrl+C to stop now, or press Enter to trigger the kernel panic (via set_event ':mod:...' write)." read tmp

echo "Triggering kernel panic..."
echo ':mod:trace-events-sample' > set_event
-----

Thanks,

---

Masami Hiramatsu (Google) (2):
tracing: Fix union collision of module and refcnt for dynamic events selftests/ftrace: Reset triggers at top level before instance loop

 kernel/trace/trace_events.c               |    4 +++-
 tools/testing/selftests/ftrace/ftracetest |    1 +
 2 files changed, 4 insertions(+), 1 deletion(-)

--
Masami Hiramatsu (Google) <[email protected]>

author maintainer
#1 lore raw

PATCH 1/2 tracing: Fix union collision of module and refcnt for dynamic events

From: Masami Hiramatsu (Google) <[email protected]>

In 'struct trace_event_call', the 'module' pointer and the 'refcnt' atomic variable share the same memory space in a union. For dynamic events, the union member is 'refcnt', which acts as an active reference counter.

When a dynamic event (such as kprobe, uprobe, fprobe, eprobe, or wprobe) has a non-zero reference count (e.g. due to active event triggers or perf attachments), its 'call->module' evaluates to a small non-zero integer instead of NULL.

When filtering or setting events for a specific module (e.g., writing ':mod:<module>' to 'set_event'), the code in '__ftrace_set_clr_event_nolock()' and 'update_event_fields()' reads 'call->module' directly without checking whether the event is dynamic. This causes the kernel to treat the small integer (refcnt) as a 'struct module' pointer, leading to a NULL/invalid pointer dereference (Oops) when dereferencing the module name.

Fix this by ensuring that the 'TRACE_EVENT_FL_DYNAMIC' flag is checked before treating 'call->module' as a valid pointer in these code paths.

Fixes 4c86bc531e60 ("tracing: Add :mod: command to enabled module events")
Assisted-by Antigravity:gemini-3.5-flash
---
 kernel/trace/trace_events.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/trace/trace_events.c +3 -1
index c46e623e7e0d..956692856fa8 100644
@@ -1350,7 +1350,9 @@ __ftrace_set_clr_event_nolock(struct trace_array *tr, const char *match,
 		call = file->event_call;
 
 		/* If a module is specified, skip events that are not that module */
-		if (module && (!call->module || strcmp(module_name(call->module), module)))
+		if (module &&
+		    ((call->flags & TRACE_EVENT_FL_DYNAMIC) ||
+		     !call->module || strcmp(module_name(call->module), module)))
 			continue;
 
 		name = trace_event_name(call);
author maintainer
#2 lore raw

PATCH 2/2 selftests/ftrace: Reset triggers at top level before instance loop

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

When running instance tests, 'ftracetest' creates a new ftrace instance and runs the tests inside it. Before starting each test, it executes 'initialize_system()' to reset the ftrace state to initial-state.

However, since 'initialize_system()' is executed in the context of the instance directory, it only cleans up triggers and filters of that instance.
Any triggers or dynamic events left behind in the top-level instance by previous failed top-level tests, are left completely untouched. These top-level leftovers can cause subsequent instance-based tests to fail or even crash the kernel.

Fix this by executing 'initialize_system()' in the top-level tracing directory once before entering the instance loop.

Fixes b5b77be812de ("selftests: ftrace: Allow some tests to be run in a tracing instance")
Assisted-by Antigravity:gemini-3.5-flash
---
 tools/testing/selftests/ftrace/ftracetest |    1 +
 1 file changed, 1 insertion(+)
tools/testing/selftests/ftrace/ftracetest +1 -0
index 0a56bf209f6c..8ad2c385407e 100755
@@ -503,6 +503,7 @@ for t in $TEST_CASES; do
 done
 
 # Test on instance loop
+(cd $TRACING_DIR; initialize_system)
 INSTANCE=" (instance) "
 for t in $TEST_CASES; do
   test_on_instance $t || continue