tracing: Export ftrace API for kernel modules
PATCH 2/2 tracing: Export ftrace API for kernel modules
Export register_ and unresgister_ftrace_function_probe to modules. This can be used by SystemTap.
--- kernel/trace/ftrace.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/trace/ftrace.c +2 -0
@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,return count; } +EXPORT_SYMBOL_GPL (register_ftrace_function_probe);enum {PROBE_TEST_FUNC =1 ,
@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,__unregister_ftrace_function_probe (glob, ops, data,PROBE_TEST_FUNC |PROBE_TEST_DATA ); } +EXPORT_SYMBOL_GPL (unregister_ftrace_function_probe);void unregister_ftrace_function_probe_func (char *glob,struct ftrace_probe_ops *ops)
Signature · 1 line
1.5.5.1
Quoted 2 lines · Atsushi Tsuji
Export
register_andunresgister_ftrace_function_probeto modules. This can be used by SystemTap.
Its unusual to export bits without an in-kernel user.
Quoted 25 lines
Signed-off-by Atsushi Tsuji <[email protected]>--- kernel/trace/ftrace.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)kernel/trace/ftrace.c +2
-0@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,return count; } +EXPORT_SYMBOL_GPL (register_ftrace_function_probe);enum {PROBE_TEST_FUNC =1 ,@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,__unregister_ftrace_function_probe (glob, ops, data,PROBE_TEST_FUNC |PROBE_TEST_DATA ); } +EXPORT_SYMBOL_GPL (unregister_ftrace_function_probe);void unregister_ftrace_function_probe_func (char *glob,struct ftrace_probe_ops *ops)
Steven Rostedt
maintainerQuoted 21 lines · Atsushi Tsuji
Export
register_andunresgister_ftrace_function_probeto modules. This can be used by SystemTap.Signed-off-by Atsushi Tsuji <[email protected]>--- kernel/trace/ftrace.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)kernel/trace/ftrace.c +1
-0@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,return count; } +EXPORT_SYMBOL_GPL (register_ftrace_function_probe);enum {PROBE_TEST_FUNC =1 ,
kernel/trace/ftrace.c unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops, · line 2112
Quoted 4 lines
kernel/trace/ftrace.c +1
-0@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,__unregister_ftrace_function_probe (glob, ops, data,PROBE_TEST_FUNC |PROBE_TEST_DATA ); } +EXPORT_SYMBOL_GPL (unregister_ftrace_function_probe);
I have to NAK this as is. There's a reason I never exported these to modules, and that is because they are not module safe. There's nothing to stop a module from registering a probe and then being removed. Then the function probe will still be called causing a kernel panic.
Now I know of two ways to fix this.
1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.
2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do). And the current function tracer will optimize that if only one function is registered to mcount, then the mcount caller will call that function directly.
It will still need to up the mod ref count when a probe is added, but it can also remove it.
The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.
1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!
-- Steve
Quoted 39 lines · Steven Rostedt
Quoted 21 lines · Atsushi Tsuji
Export
register_andunresgister_ftrace_function_probeto modules. This can be used by SystemTap.Signed-off-by Atsushi Tsuji <[email protected]>--- kernel/trace/ftrace.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)kernel/trace/ftrace.c +1
-0@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,return count; } +EXPORT_SYMBOL_GPL (register_ftrace_function_probe);enum {PROBE_TEST_FUNC =1 ,kernel/trace/ftrace.c unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops, · line 2112
Quoted 4 lines
kernel/trace/ftrace.c +1
-0@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,__unregister_ftrace_function_probe (glob, ops, data,PROBE_TEST_FUNC |PROBE_TEST_DATA ); } +EXPORT_SYMBOL_GPL (unregister_ftrace_function_probe);I have to NAK this as is. There's a reason I never exported these to modules, and that is because they are not module safe. There's nothing to stop a module from registering a probe and then being removed. Then the function probe will still be called causing a kernel panic.
Now I know of two ways to fix this.
1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.
Quoted 6 lines
2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do).
I think this is better solution.
Out of curiously, is disabling preemption so harmful?
Quoted 3 lines
And the current function
tracer will optimize that if only one function is registered to mcount, then the mcount caller will call that function directly.
Would you mean that current mcount hook supports only one handler?
Quoted 8 lines
It will still need to up the mod ref count when a probe is added, but it can also remove it.
The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.
Quoted 7 lines
1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!
Agreed, if mcount doesn't disable preemption, this will happen.
Thank you,
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
Steven Rostedt
maintainerQuoted 15 lines · Masami Hiramatsu
Quoted 6 lines · Steven Rostedt
Now I know of two ways to fix this.
1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.
Quoted 6 lines
2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do).
Quoted 2 lines
I think this is better solution.
Out of curiously, is disabling preemption so harmful?
Yes ;-)
I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.
Quoted 6 lines
Quoted 4 lines
And the current function
tracer will optimize that if only one function is registered to mcount, then the mcount caller will call that function directly.Would you mean that current mcount hook supports only one handler?
Yes, currently a branch can only got to a single function ;-)
What ftrace does, is if you register a single function, the mcount location will call that function directly. But if you register an second function, it changes the mcount call site to call a wrapper that loops to through all the functions that are registered.
Thus a module version would need to call that wrapper every time. Even if only one function is called. And this wrapper would have to disable preemption for the entire loop.
Quoted 17 lines
Quoted 9 lines
It will still need to up the mod ref count when a probe is added, but it can also remove it.
The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.
Quoted 7 lines
1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!
Quoted 1 line
Agreed, if mcount doesn't disable preemption, this will happen.
And it does not.
-- Steve
Quoted 2 lines · Atsushi Tsuji
Export
register_andunresgister_ftrace_function_probeto modules. This can be used by SystemTap.
And can be added to the kernel treee once we merge a reasonably rewritten version of it, if at all possible.
Remember that we do not add random crap for out of tree modules.
Quoted 18 lines · Steven Rostedt
Quoted 8 lines · Masami Hiramatsu
Quoted 6 lines · Steven Rostedt
Now I know of two ways to fix this.
1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.
Quoted 8 lines
Quoted 6 lines
2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do).
I think this is better solution.
Out of curiously, is disabling preemption so harmful?
Quoted 8 lines
Yes ;-)
I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.
Thank you for detailed explanation.
I may be wrong, but I think function_trace_probe_call using register_ftrace_function_probe is almost enough for modules, since it disables preemption while a probe is calling and it called every time even if only one probe function is registered. So is it enough to make a new registering function using it and upping module ref count for module safe?
Or should I make another handler for modules not using function_trace_probe_call?
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
It will still need to up the mod ref count when a probe is added, but it can also remove it.
The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.
1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!
Quoted 3 lines
Quoted 1 line
Agreed, if mcount doesn't disable preemption, this will happen.
And it does not.
I think the preemption is disabled in not register_ftrace_function but register_ftrace_function_probe, is that wrong?
Thanks,
Atsushi
Steven Rostedt
maintainerQuoted 11 lines · Atsushi Tsuji
Quoted 6 lines
I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.
Thank you for detailed explanation.
I may be wrong, but I think
function_trace_probe_callusingregister_ftrace_function_probeis almost enough for modules,
Heh, I forgot about function_probe. Yeah, looking at that, it does seem that it would be safe for modules.
Quoted 4 lines
since it disables preemption while a probe is calling and it called every time even if only one probe function is registered. So is it enough to make a new registering function using it and upping module ref count for module safe?
Yes, upping the module ref count for every function probe would be required. Unfortunately, this would require adding another variable to struct ftrace_func_probe, which I hate to do.
Quoted 3 lines
Or should I make another handler for modules not using
function_trace_probe_call?
Maybe another handler might be better. But it may be similar to function_probe.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
It will still need to up the mod ref count when a probe is added, but it can also remove it.
The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.
1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Agreed, if mcount doesn't disable preemption, this will happen.
And it does not.
I think the preemption is disabled in not
register_ftrace_functionbutregister_ftrace_function_probe, is that wrong?
No that's correct. It's been a while since I worked on the probe code, so I forgot about it :-)
-- Steve
Quoted 26 lines · Steven Rostedt
Quoted 10 lines · Atsushi Tsuji
Quoted 6 lines
I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.
Thank you for detailed explanation.
I may be wrong, but I think
function_trace_probe_callusingregister_ftrace_function_probeis almost enough for modules,Heh, I forgot about
function_probe. Yeah, looking at that, it does seem that it would be safe for modules.Quoted 4 lines
since it disables preemption while a probe is calling and it called every time even if only one probe function is registered. So is it enough to make a new registering function using it and upping module ref count for module safe?
Yes, upping the module ref count for every function probe would be required. Unfortunately, this would require adding another variable to struct
ftrace_func_probe, which I hate to do.
Quoted 5 lines
Quoted 2 lines
Or should I make another handler for modules not using
function_trace_probe_call?Maybe another handler might be better. But it may be similar to
function_probe.
I see. I'll try to make a new handler for modules.
Thanks,
Atsushi
Quoted 28 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
It will still need to up the mod ref count when a probe is added, but it can also remove it.
The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.
1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!Quoted 4 lines
Quoted 2 lines
Quoted 1 line
Agreed, if mcount doesn't disable preemption, this will happen.
And it does not.
I think the preemption is disabled in not
register_ftrace_functionbutregister_ftrace_function_probe, is that wrong?No that's correct. It's been a while since I worked on the probe code, so I forgot about it :-)
-- Steve