Skip to contentSkip to conversation

tracing: Track event ref in tracefs enable/disable

PATCH In review Beau Belgrave 2 participants 5 messages Oct 12, 2022 – Oct 13, 2022 Tracing

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tracing: Track event ref in tracefs enable/disable

When events are enabled via the "enable" file within tracefs there is no get or put ref. Add these to ensure modules and dynamic events do not unload while the event is enabled via tracefs.

---
 kernel/trace/trace_events.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
kernel/trace/trace_events.c +13 -3
index 0356cae0cf74..f0b17adba1bc 100644
@@ -634,6 +634,8 @@ static int __ftrace_event_enable_disable(struct trace_event_file *file,
 			}
 
 			call->class->reg(call, TRACE_REG_UNREGISTER, file);
+
+			trace_event_put_ref(call);
 		}
 		/* If in SOFT_MODE, just set the SOFT_DISABLE_BIT, else clear it */
 		if (file->flags & EVENT_FILE_FL_SOFT_MODE)
@@ -659,7 +661,7 @@ static int __ftrace_event_enable_disable(struct trace_event_file *file,
 		}
 
 		if (!(file->flags & EVENT_FILE_FL_ENABLED)) {
-			bool cmd = false, tgid = false;
+			bool cmd = false, tgid = false, put = false;
 
 			/* Keep the event disabled, when going to SOFT_MODE. */
 			if (soft_disable)
@@ -677,14 +679,22 @@ static int __ftrace_event_enable_disable(struct trace_event_file *file,
 				set_bit(EVENT_FILE_FL_RECORDED_TGID_BIT, &file->flags);
 			}
 
+			if (!trace_event_try_get_ref(call))
+				goto error_enabling;
+
+			put = true;
+
 			ret = call->class->reg(call, TRACE_REG_REGISTER, file);
 			if (ret) {
+error_enabling:
+				pr_info("event trace: Could not enable event "
+					"%s\n", trace_event_name(call));
 				if (cmd)
 					tracing_stop_cmdline_record();
 				if (tgid)
 					tracing_stop_tgid_record();
-				pr_info("event trace: Could not enable event "
-					"%s\n", trace_event_name(call));
+				if (put)
+					trace_event_put_ref(call);
 				break;
 			}
 			set_bit(EVENT_FILE_FL_ENABLED_BIT, &file->flags);

base-commit: 6c0f39e87b6ab1a3009e3a49d3e6f6db8dc756a8

Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 3 lines · On Wed, 12 Oct 2022 14:57:17 -0700 Beau Belgrave <[email protected]>…

When events are enabled via the "enable" file within tracefs there is no get or put ref. Add these to ensure modules and dynamic events do not unload while the event is enabled via tracefs.

Why is this an issue?

The events are only called from the module code, and when the module is unloaded, they are no longer called. Why keep the module from unloading when enabled?

-- Steve

#2 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 3 lines · On Wed, 12 Oct 2022 14:57:17 -0700 Beau Belgrave <[email protected]>…

When events are enabled via the "enable" file within tracefs there is no get or put ref. Add these to ensure modules and dynamic events do not unload while the event is enabled via tracefs.

Why is this an issue?

It depends on the scenario, it may or may not cause a issue.

Quoted 4 lines

The events are only called from the module code, and when the module is unloaded, they are no longer called. Why keep the module from unloading when enabled?

Won't the modules remove the event calls? At the very least the event call structure in memory goes away during module unload. If it gets reused odd things will happen, right? IE: trace_module_remove_events().

Maybe I have a bad assumption:
I thought the point of trace_event_try_get_ref()/put_ref() was to tell the system the call cannot go away. However, if ftrace enable doesn't use these the lifetime is ambigious in this case. If this was intentional, how are event call lifetimes described if not within the ref?

In my namespace patches I hit this case when user_events try to go away during namespace teardown. Since there is no reference to the event being used I removed the call. However, it was clearly being used within tracefs at that point. When I cat "enable" in this case instead of "0" or "1" I get "?". I suppose worse things could happen when the memory of the call gets reused?

Quoted 2 lines

-- Steve

Thanks,
-Beau

maintainer
#3 lore raw
Quoted 8 lines · On Wed, 12 Oct 2022 17:19:38 -0700 Beau Belgrave <[email protected]>…
Quoted 4 lines

The events are only called from the module code, and when the module is unloaded, they are no longer called. Why keep the module from unloading when enabled?

Won't the modules remove the event calls? At the very least the event call structure in memory goes away during module unload. If it gets reused odd things will happen, right? IE: trace_module_remove_events().

Yes it gets removed along with the module. But everything about the static event is part of the module.

Quoted 8 lines

Maybe I have a bad assumption:
I thought the point of trace_event_try_get_ref()/put_ref() was to tell the system the call cannot go away. However, if ftrace enable doesn't use these the lifetime is ambigious in this case. If this was intentional, how are event call lifetimes described if not within the ref?

The purpose of trace_event_try_get_ref and friends is for the case of dynamic events (eprobes and synthetic events) that can attach to any event (including module events), and the module removal does not remove the dynamic portion that was attached to them. And in this case, the removal would have dire results.

Quoted 7 lines

In my namespace patches I hit this case when user_events try to go away during namespace teardown. Since there is no reference to the event being used I removed the call. However, it was clearly being used within tracefs at that point. When I cat "enable" in this case instead of "0" or "1" I get "?". I suppose worse things could happen when the memory of the call gets reused?

Could you have a callback telling tracefs that it is going away (like the module notifier does)?

-- Steve

#4 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 5 lines · On Wed, 12 Oct 2022 17:19:38 -0700 Beau Belgrave <[email protected]>…
Quoted 4 lines

The events are only called from the module code, and when the module is unloaded, they are no longer called. Why keep the module from unloading when enabled?

Quoted 7 lines
Quoted 3 lines

Won't the modules remove the event calls? At the very least the event call structure in memory goes away during module unload. If it gets reused odd things will happen, right? IE: trace_module_remove_events().

Yes it gets removed along with the module. But everything about the static event is part of the module.

I see, trace_module_remove_events() does remove the trace call for module cases only.

Quoted 9 lines
Quoted 8 lines

Maybe I have a bad assumption:
I thought the point of trace_event_try_get_ref()/put_ref() was to tell the system the call cannot go away. However, if ftrace enable doesn't use these the lifetime is ambigious in this case. If this was intentional, how are event call lifetimes described if not within the ref?

Quoted 6 lines

The purpose of trace_event_try_get_ref and friends is for the case of dynamic events (eprobes and synthetic events) that can attach to any event (including module events), and the module removal does not remove the dynamic portion that was attached to them. And in this case, the removal would have dire results.

Got it, uprobes and user_events are also dynamic events. Here are things I found and am thinking about and questioning:

Uprobe specifically calls trace_event_dyn_busy() during unregister_trace_uprobe(). This will return false in this case, since there is no reference. Fortunately trace_uprobe_is_busy() used by dyn_events checks the event flags to see if something really hooked there.

User_events also tracks usage via the event.class.reg() REG/UNREG callback. It's dyn_events is_busy() op also checks internal flags.

So normally these are not issues for either above system.

However, once we have a tracing_namespace with put/get ref semantics for lifetime, those corner case checks won't keep us from removing the calls.

It appears if perf attaches to any event it will take a reference. This would imply the module won't be allowed to go away while perf is running. I'm not sure why we wouldn't want consistency, perhaps it might break some scenarios I'm unaware of around module unload.

Quoted 11 lines
Quoted 7 lines

In my namespace patches I hit this case when user_events try to go away during namespace teardown. Since there is no reference to the event being used I removed the call. However, it was clearly being used within tracefs at that point. When I cat "enable" in this case instead of "0" or "1" I get "?". I suppose worse things could happen when the memory of the call gets reused?

Could you have a callback telling tracefs that it is going away (like the module notifier does)?

I could, would it be limited to just tracefs? I don't believe perf or eBPF currently hooks into those notifiers?

If we had a consistent get/put ref model I don't think we would need the notifiers (unless we cannot trust all legacy code doing a get/put ref). It appears in my testing that perf is always doing get/put refs in these cases to ensure the call cannot go away underneath it.

Thanks,
-Beau