Skip to contentSkip to conversation

tracing: Export ftrace API for kernel modules

PATCH In review Atsushi Tsuji 5 participants 9 messages Sep 15, 2009 – Sep 17, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 2/2 tracing: Export ftrace API for kernel modules

Export register_ and unresgister_ftrace_function_probe to modules. This can be used by SystemTap.

---
 kernel/trace/ftrace.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/trace/ftrace.c +2 -0
index 5ef8f59..9c32291 100644
@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 
 	return count;
 }
+EXPORT_SYMBOL_GPL(register_ftrace_function_probe);
 
 enum {
 	PROBE_TEST_FUNC		= 1,
@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 	__unregister_ftrace_function_probe(glob, ops, data,
 					  PROBE_TEST_FUNC | PROBE_TEST_DATA);
 }
+EXPORT_SYMBOL_GPL(unregister_ftrace_function_probe);
 
 void
 unregister_ftrace_function_probe_func(char *glob, struct ftrace_probe_ops *ops)
Signature · 1 line
1.5.5.1
#1 lore raw
Quoted 2 lines · Atsushi Tsuji

Export register_ and unresgister_ftrace_function_probe to modules. This can be used by SystemTap.

Its unusual to export bits without an in-kernel user.

Quoted 25 lines
---
 kernel/trace/ftrace.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/trace/ftrace.c +2 -0
index 5ef8f59..9c32291 100644
@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 
 	return count;
 }
+EXPORT_SYMBOL_GPL(register_ftrace_function_probe);
 
 enum {
 	PROBE_TEST_FUNC		= 1,
@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 	__unregister_ftrace_function_probe(glob, ops, data,
 					  PROBE_TEST_FUNC | PROBE_TEST_DATA);
 }
+EXPORT_SYMBOL_GPL(unregister_ftrace_function_probe);
 
 void
 unregister_ftrace_function_probe_func(char *glob, struct ftrace_probe_ops *ops)
maintainer
#2 lore raw
In reply to Atsushi Tsuji
Quoted 21 lines · Atsushi Tsuji

Export register_ and unresgister_ftrace_function_probe to modules. This can be used by SystemTap.

---
 kernel/trace/ftrace.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/trace/ftrace.c +1 -0
index 5ef8f59..9c32291 100644
@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 
 	return count;
 }
+EXPORT_SYMBOL_GPL(register_ftrace_function_probe);
 
 enum {
 	PROBE_TEST_FUNC		= 1,

kernel/trace/ftrace.c unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops, · line 2112

Quoted 4 lines
kernel/trace/ftrace.c +1 -0
@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 	__unregister_ftrace_function_probe(glob, ops, data,
 					  PROBE_TEST_FUNC | PROBE_TEST_DATA);
 }
+EXPORT_SYMBOL_GPL(unregister_ftrace_function_probe);

I have to NAK this as is. There's a reason I never exported these to modules, and that is because they are not module safe. There's nothing to stop a module from registering a probe and then being removed. Then the function probe will still be called causing a kernel panic.

Now I know of two ways to fix this.

1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.

2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do). And the current function tracer will optimize that if only one function is registered to mcount, then the mcount caller will call that function directly.

It will still need to up the mod ref count when a probe is added, but it can also remove it.

The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.

1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!

-- Steve

#3 lore raw
Quoted 39 lines · Steven Rostedt
Quoted 21 lines · Atsushi Tsuji

Export register_ and unresgister_ftrace_function_probe to modules. This can be used by SystemTap.

---
 kernel/trace/ftrace.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/trace/ftrace.c +1 -0
index 5ef8f59..9c32291 100644
@@ -2042,6 +2042,7 @@ register_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 
 	return count;
 }
+EXPORT_SYMBOL_GPL(register_ftrace_function_probe);
 
 enum {
 	PROBE_TEST_FUNC		= 1,

kernel/trace/ftrace.c unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops, · line 2112

Quoted 4 lines
kernel/trace/ftrace.c +1 -0
@@ -2108,6 +2109,7 @@ unregister_ftrace_function_probe(char *glob, struct ftrace_probe_ops *ops,
 	__unregister_ftrace_function_probe(glob, ops, data,
 					  PROBE_TEST_FUNC | PROBE_TEST_DATA);
 }
+EXPORT_SYMBOL_GPL(unregister_ftrace_function_probe);

I have to NAK this as is. There's a reason I never exported these to modules, and that is because they are not module safe. There's nothing to stop a module from registering a probe and then being removed. Then the function probe will still be called causing a kernel panic.

Now I know of two ways to fix this.

1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.

Quoted 6 lines

2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do).

I think this is better solution.
Out of curiously, is disabling preemption so harmful?

Quoted 3 lines

And the current function
tracer will optimize that if only one function is registered to mcount, then the mcount caller will call that function directly.

Would you mean that current mcount hook supports only one handler?

Quoted 8 lines

It will still need to up the mod ref count when a probe is added, but it can also remove it.

The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.

Quoted 7 lines

1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!

Agreed, if mcount doesn't disable preemption, this will happen.

Thank you,

Signature · 7 lines
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: [email protected]
maintainer
#4 lore raw
Quoted 15 lines · Masami Hiramatsu
Quoted 6 lines · Steven Rostedt

Now I know of two ways to fix this.

1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.

Quoted 6 lines

2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do).

Quoted 2 lines

I think this is better solution.
Out of curiously, is disabling preemption so harmful?

Yes ;-)

I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.

Quoted 6 lines
Quoted 4 lines

And the current function
tracer will optimize that if only one function is registered to mcount, then the mcount caller will call that function directly.

Would you mean that current mcount hook supports only one handler?

Yes, currently a branch can only got to a single function ;-)

What ftrace does, is if you register a single function, the mcount location will call that function directly. But if you register an second function, it changes the mcount call site to call a wrapper that loops to through all the functions that are registered.

Thus a module version would need to call that wrapper every time. Even if only one function is called. And this wrapper would have to disable preemption for the entire loop.

Quoted 17 lines
Quoted 9 lines

It will still need to up the mod ref count when a probe is added, but it can also remove it.

The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.

Quoted 7 lines

1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!

Quoted 1 line

Agreed, if mcount doesn't disable preemption, this will happen.

And it does not.

-- Steve

#5 lore raw
Quoted 18 lines · Steven Rostedt
Quoted 8 lines · Masami Hiramatsu
Quoted 6 lines · Steven Rostedt

Now I know of two ways to fix this.

1) The simple way. Up the module ref count so once it registers a function it can never be disabled. Of course there's the "force module unload" but people should not do that anyway.

Quoted 8 lines
Quoted 6 lines

2) Create a second hook handler for modules. That is the function caller for modules will go to a wrapper first. This wrapper could disable interrupts or grab a lock or something that would prevent a module from being unloaded as the hooks are being called. Perhaps even disabling preemption while calling the hooks will be enough (this is not something I want the normal function caller to do).

I think this is better solution.
Out of curiously, is disabling preemption so harmful?

Quoted 8 lines

Yes ;-)

I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.

Thank you for detailed explanation.

I may be wrong, but I think function_trace_probe_call using register_ftrace_function_probe is almost enough for modules, since it disables preemption while a probe is calling and it called every time even if only one probe function is registered. So is it enough to make a new registering function using it and upping module ref count for module safe?

Or should I make another handler for modules not using function_trace_probe_call?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

It will still need to up the mod ref count when a probe is added, but it can also remove it.

The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.

1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!

Quoted 3 lines
Quoted 1 line

Agreed, if mcount doesn't disable preemption, this will happen.

And it does not.

I think the preemption is disabled in not register_ftrace_function but register_ftrace_function_probe, is that wrong?

Thanks,
Atsushi

maintainer
#6 lore raw
Quoted 11 lines · Atsushi Tsuji
Quoted 6 lines

I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.

Thank you for detailed explanation.

I may be wrong, but I think function_trace_probe_call using register_ftrace_function_probe is almost enough for modules,

Heh, I forgot about function_probe. Yeah, looking at that, it does seem that it would be safe for modules.

Quoted 4 lines

since it disables preemption while a probe is calling and it called every time even if only one probe function is registered. So is it enough to make a new registering function using it and upping module ref count for module safe?

Yes, upping the module ref count for every function probe would be required. Unfortunately, this would require adding another variable to struct ftrace_func_probe, which I hate to do.

Quoted 3 lines

Or should I make another handler for modules not using function_trace_probe_call?

Maybe another handler might be better. But it may be similar to function_probe.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

It will still need to up the mod ref count when a probe is added, but it can also remove it.

The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.

1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Agreed, if mcount doesn't disable preemption, this will happen.

And it does not.

I think the preemption is disabled in not register_ftrace_function but register_ftrace_function_probe, is that wrong?

No that's correct. It's been a while since I worked on the probe code, so I forgot about it :-)

-- Steve

#7 lore raw
Quoted 26 lines · Steven Rostedt
Quoted 10 lines · Atsushi Tsuji
Quoted 6 lines

I don't want to disable preemption when I don't have to. The function tracer that is called can. But actually, it's ever more that that. If you only register a single function, it will call that function directly. Then there will always be a race window between when the function gets called and disabling preemption, even if the called function disables preemption as the first thing it does.

Thank you for detailed explanation.

I may be wrong, but I think function_trace_probe_call using register_ftrace_function_probe is almost enough for modules,

Heh, I forgot about function_probe. Yeah, looking at that, it does seem that it would be safe for modules.

Quoted 4 lines

since it disables preemption while a probe is calling and it called every time even if only one probe function is registered. So is it enough to make a new registering function using it and upping module ref count for module safe?

Yes, upping the module ref count for every function probe would be required. Unfortunately, this would require adding another variable to struct ftrace_func_probe, which I hate to do.

Quoted 5 lines
Quoted 2 lines

Or should I make another handler for modules not using function_trace_probe_call?

Maybe another handler might be better. But it may be similar to function_probe.

I see. I'll try to make a new handler for modules.

Thanks,
Atsushi

Quoted 28 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

It will still need to up the mod ref count when a probe is added, but it can also remove it.

The problem with the current method, is that a probe can be executing at anytime. Here's an example if we did it your way.

1. module installed
2. module adds probe
3. function X in kernel calls probe but gets preempted.
4. module removes probe
5. module unistalled
6. function X in kernel continues to run probe but probe no longer exists --- Oops!

Quoted 4 lines
Quoted 2 lines
Quoted 1 line

Agreed, if mcount doesn't disable preemption, this will happen.

And it does not.

I think the preemption is disabled in not register_ftrace_function but register_ftrace_function_probe, is that wrong?

No that's correct. It's been a while since I worked on the probe code, so I forgot about it :-)

-- Steve

#8 lore raw
In reply to Atsushi Tsuji
Quoted 2 lines · Atsushi Tsuji

Export register_ and unresgister_ftrace_function_probe to modules. This can be used by SystemTap.

And can be added to the kernel treee once we merge a reasonably rewritten version of it, if at all possible.

Remember that we do not add random crap for out of tree modules.