tracing: Fix a kernel crash related to :mod: command
Masami Hiramatsu (Google)
maintainercover tracing: Fix a kernel crash related to :mod: command
Hi,
I found a kernel crash bug in tracing with :mod: command and dynamic events. If we leaves a dynamic event and run :mod: command to set_event, the kernel gets an oops.
Actually, this bug found by testing wprobe trigger test case, because ftracetest has another bug which does not clean up the top-level event filters/triggers before starting instance tests. Thus if the last test case in the top-level instance fails and if it leaves a dynamic event, that will be kept and not cleaned up. Then, test.d/event/event-mod.tc hits this bug.
Here is the reproduce script.
-----
#!/bin/sh
TRACE_DIR="/sys/kernel/tracing"
if [ ! -d "$TRACE_DIR" ]; then
TRACE_DIR="/sys/kernel/debug/tracing"
fi
if [ ! -d "$TRACE_DIR" ]; then
echo "Error: Tracefs not found. Please mount tracefs." exit 1
fi
cd "$TRACE_DIR"
echo "=== 1. Clear existing triggers and dynamic events ===" echo > events/sched/sched_process_fork/trigger || true echo > dynamic_events || true
echo "=== 2. Create a kprobe event ==="
# Adding a kprobe on vfs_write
echo 'p:kprobes/my_kprobe vfs_write' >> dynamic_events cat dynamic_events
echo "=== 3. Attach enable_event trigger targeting my_kprobe to sched_process_fork ===" if [ -f "events/sched/sched_process_fork/trigger" ]; then
echo 'enable_event:kprobes:my_kprobe' > events/sched/sched_process_fork/trigger cat events/sched/sched_process_fork/trigger
else
echo "Error: sched_process_fork event trigger is not supported."
exit 1
fi
echo "=== 4. Try to delete the kprobe event while the trigger is active ==="
echo "Attempting: echo -:kprobes/my_kprobe >> dynamic_events"
if ! echo '-:kprobes/my_kprobe' >> dynamic_events 2>/dev/null; then
echo "Result: Successfully reproduced kprobe event deletion failure (Device or resource busy)!" echo "The 'my_kprobe' event remains in dynamic_events:"
cat dynamic_events
else
echo "Warning: Deletion succeeded? Check if the trigger was active."
fi
echo ""
echo "=== WARNING: The next step will crash the kernel if the union collision bug is not fixed! ===" echo "Press Ctrl+C to stop now, or press Enter to trigger the kernel panic (via set_event ':mod:...' write)." read tmp
echo "Triggering kernel panic..."
echo ':mod:trace-events-sample' > set_event
-----
Thanks,
---
Masami Hiramatsu (Google) (2):
tracing: Fix union collision of module and refcnt for dynamic events selftests/ftrace: Reset triggers at top level before instance loop
kernel/trace/trace_events.c | 4 +++- tools/testing/selftests/ftrace/ftracetest | 1 + 2 files changed, 4 insertions(+), 1 deletion(-)
--
Masami Hiramatsu (Google) <[email protected]>
Masami Hiramatsu (Google)
maintainerPATCH 1/2 tracing: Fix union collision of module and refcnt for dynamic events
From: Masami Hiramatsu (Google) <[email protected]>
In 'struct trace_event_call', the 'module' pointer and the 'refcnt' atomic variable share the same memory space in a union. For dynamic events, the union member is 'refcnt', which acts as an active reference counter.
When a dynamic event (such as kprobe, uprobe, fprobe, eprobe, or wprobe) has a non-zero reference count (e.g. due to active event triggers or perf attachments), its 'call->module' evaluates to a small non-zero integer instead of NULL.
When filtering or setting events for a specific module (e.g., writing ':mod:<module>' to 'set_event'), the code in '__ftrace_set_clr_event_nolock()' and 'update_event_fields()' reads 'call->module' directly without checking whether the event is dynamic. This causes the kernel to treat the small integer (refcnt) as a 'struct module' pointer, leading to a NULL/invalid pointer dereference (Oops) when dereferencing the module name.
Fix this by ensuring that the 'TRACE_EVENT_FL_DYNAMIC' flag is checked before treating 'call->module' as a valid pointer in these code paths.
--- kernel/trace/trace_events.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/trace/trace_events.c +3 -1
@@ -1350,7 +1350,9 @@ __ftrace_set_clr_event_nolock(struct trace_array *tr, const char *match, call = file->event_call;/* If a module is specified, skip events that are not that module */ -if (module && (!call->module ||strcmp (module_name (call->module), module))) +if (module && + ((call->flags &TRACE_EVENT_FL_DYNAMIC ) || + !call->module ||strcmp (module_name (call->module), module)))continue ; name =trace_event_name (call);
Masami Hiramatsu (Google)
maintainerPATCH 2/2 selftests/ftrace: Reset triggers at top level before instance loop
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
When running instance tests, 'ftracetest' creates a new ftrace instance and runs the tests inside it. Before starting each test, it executes 'initialize_system()' to reset the ftrace state to initial-state.
However, since 'initialize_system()' is executed in the context of the instance directory, it only cleans up triggers and filters of that instance.
Any triggers or dynamic events left behind in the top-level instance by previous failed top-level tests, are left completely untouched. These top-level leftovers can cause subsequent instance-based tests to fail or even crash the kernel.
Fix this by executing 'initialize_system()' in the top-level tracing directory once before entering the instance loop.
--- tools/testing/selftests/ftrace/ftracetest | 1 + 1 file changed, 1 insertion(+)
tools/testing/selftests/ftrace/ftracetest +1 -0
@@ -503,6 +503,7 @@ for t in $TEST_CASES; do done # Test on instance loop +(cd $TRACING_DIR; initialize_system) INSTANCE=" (instance) " for t in $TEST_CASES; do test_on_instance $t || continue