ftrace: Use -mfentry when supported (this is for x86_64 right now)

Makefile +3 -0

+endif
+KBUILD_CFLAGS	+= -pg $(CC_HAS_FENTRY)
+KBUILD_AFLAGS	+= $(CC_HAS_FENTRY)
 ifdef CONFIG_DYNAMIC_FTRACE
 	ifdef CONFIG_HAVE_C_RECORDMCOUNT
 		BUILD_C_RECORDMCOUNT := y

Makefile KBUILD_CFLAGS += $(call cc-option, -femit-struct-debug-baseonly) · line 582

Quoted 4 lines · Steven Rostedt

ifdef CONFIG_FUNCTION_TRACER

Makefile +2 -1

-KBUILD_CFLAGS	+= -pg
+ifdef CONFIG_HAVE_FENTRY
+CC_HAS_FENTRY	:= $(call cc-option, -mfentry -DCC_HAS_FENTRY)

I should include a thanks to H. Peter Anvin in the change log for telling me about the cc-option trick.

Thanks to Andi Kleen gcc 4.6.0 now supports -mfentry with the -pg option to place a call to __fentry__ at the very beginning of the function instead of after the fact.

The old way:

00000000000000c4 <atomic_long_add>:

      c4:       55                      push   %rbp
      c5:       48 89 e5                mov    %rsp,%rbp
      c8:       e8 00 00 00 00          callq  cd <atomic_long_add+0x9>

c9: R_X86_64_PC32 mcount-0x4
cd: f0 48 01 3e lock add %rdi,(%rsi)
d1: c9 leaveq
d2: c3 retq

The new way:

000000000000009e <atomic_long_add>:
9e: e8 00 00 00 00 callq a3 <atomic_long_add+0x5>
9f: R_X86_64_PC32 __fentry__-0x4
a3: 55 push %rbp
a4: 48 89 e5 mov %rsp,%rbp
a7: f0 48 01 3e lock add %rdi,(%rsi)
ab: 5d pop %rbp
ac: c3 retq

Note, with -mfentry, frame pointers is no longer required by the function tracer. But this patch series still requires FRAME_POINTER to be set, since I need to figure out a good way to enable FRAME_POINTER only if gcc doesn't support this. But that can come later.

With the new __fentry__, we could possible record the parameters of a function call. This may take some work, and perhaps be a little like kprobes. But it is doable.

This is still just RFC. I only wrote the code to support x86_64 even though gcc 4.6.0 also supports i386. I figured I would post this first to get peoples reactions before converting i386 too. Other archs can soon follow.

-- Steve

The following patches are in:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git

branch: rfc/tracing/fentry

Steven Rostedt (4):
ftrace: Make recordmcount.c handle __fentry__
ftrace: Add -mfentry to Makefile on function tracer
ftrace: Do not test frame pointers if -mfentry is used ftrace/x86: Add support for -mfentry to x86_64

----

 Makefile                             |    6 +++++-
 arch/x86/Kconfig                     |    1 +
 arch/x86/include/asm/ftrace.h        |    7 ++++++-
 arch/x86/kernel/entry_64.S           |   17 ++++++++++++++++-
 arch/x86/kernel/x8664_ksyms_64.c     |    6 +++++-
 kernel/trace/Kconfig                 |    5 +++++
 kernel/trace/trace_functions_graph.c |    5 ++++-
 scripts/recordmcount.h               |    4 +++-
 8 files changed, 45 insertions(+), 6 deletions(-)

Thanks to Andi Kleen gcc 4.6.0 now supports -mfentry with the -pg option to place a call to __fentry__ at the very beginning of the function instead of after the fact.

The old way:

00000000000000c4 <atomic_long_add>:

      c4:       55                      push   %rbp
      c5:       48 89 e5                mov    %rsp,%rbp
      c8:       e8 00 00 00 00          callq  cd <atomic_long_add+0x9>

c9: R_X86_64_PC32 mcount-0x4
cd: f0 48 01 3e lock add %rdi,(%rsi)
d1: c9 leaveq
d2: c3 retq

The new way:

000000000000009e <atomic_long_add>:
9e: e8 00 00 00 00 callq a3 <atomic_long_add+0x5>
9f: R_X86_64_PC32 __fentry__-0x4
a3: 55 push %rbp
a4: 48 89 e5 mov %rsp,%rbp
a7: f0 48 01 3e lock add %rdi,(%rsi)
ab: 5d pop %rbp
ac: c3 retq

Note, with -mfentry, frame pointers is no longer required by the function tracer. But this patch series still requires FRAME_POINTER to be set, since I need to figure out a good way to enable FRAME_POINTER only if gcc doesn't support this. But that can come later.

With the new __fentry__, we could possible record the parameters of a function call. This may take some work, and perhaps be a little like kprobes. But it is doable.

This is still just RFC. I only wrote the code to support x86_64 even though gcc 4.6.0 also supports i386. I figured I would post this first to get peoples reactions before converting i386 too. Other archs can soon follow.

Oops! with this change, current kprobes might not be able to probe the entry of functions, because that is always reserved by ftrace! I think we need to have some new interface for replacing each other safely...

Quoted 4 lines · Masami Hiramatsu

Oops! with this change, current kprobes might not be able to probe the entry of functions, because that is always reserved by ftrace! I think we need to have some new interface for replacing each other safely...

Good point. I suspect that this wont be ready till .40 anyway. When I get a chance to work more an this, I'll also include patches where if -mfentry is activated kprobes will just hook to the mcount caller instead. Or ftrace itself :)

I'm also working on making the function tracer a bit more flexible. That is, to let multiple clients control the dynamic trace instead of just one big global one.

Quoted 7 lines · Steven Rostedt

Quoted 4 lines · Masami Hiramatsu

Oops! with this change, current kprobes might not be able to probe the entry of functions, because that is always reserved by ftrace! I think we need to have some new interface for replacing each other safely...

Quoted 4 lines

Good point. I suspect that this wont be ready till .40 anyway. When I get a chance to work more an this, I'll also include patches where if -mfentry is activated kprobes will just hook to the mcount caller instead. Or ftrace itself :)

Ah, that's a good idea! :) it could be done without -mfentry too. But is that possible to modify just one mcount entry? I also worry about the latency of enabling/disabling one entry.

BTW, without dynamic ftrace (no code modifying), I think we don't need to reserve mcount code, because no one modifies it.

Quoted 13 lines · Masami Hiramatsu

Quoted 11 lines · Steven Rostedt

Quoted 4 lines · Masami Hiramatsu

Oops! with this change, current kprobes might not be able to probe the entry of functions, because that is always reserved by ftrace! I think we need to have some new interface for replacing each other safely...

Good point. I suspect that this wont be ready till .40 anyway. When I get a chance to work more an this, I'll also include patches where if -mfentry is activated kprobes will just hook to the mcount caller instead. Or ftrace itself :)

Quoted 3 lines

Ah, that's a good idea! :) it could be done without -mfentry too. But is that possible to modify just one mcount entry? I also worry about the latency of enabling/disabling one entry.

I would have it go through the ftrace function tracing facility, which would handle which entry to enable/disable. It still does stopmachine. Is that an issue to enable/disable kprobes? The "fast" enable/disable could be done by the called function to just ignore the call.

Also note, if there's other callbacks that are attached to the function being traced, no stop machine is enabled. The callbacks are just a list and as long as a function has an associated callback, no code modification needs to be done to add or remove other callbacks.

Quoted 2 lines

BTW, without dynamic ftrace (no code modifying), I think we don't need to reserve mcount code, because no one modifies it.

Correct. And even today, you can remove any kprobe code that checks for mcount without dynamic ftrace enabled. But I'm not sure if anyone enables the function tracer without dynamic ftrace, except for debugging in archs that do not support dynamic ftrace. As the overhead of this is quite high even when function tracer is disabled.

I just thought that frequent stop-machine is not so good from the user's POV. I agree that disabled probe ignoring the call is enough. Maybe, it could be done with the similar mechanism of jump optimization.

Quoted 3 lines · Masami Hiramatsu

I just thought that frequent stop-machine is not so good from the user's POV. I agree that disabled probe ignoring the call is enough. Maybe, it could be done with the similar mechanism of jump optimization.

I thought jump optimization still calls stop_machine too?

Quoted 6 lines · Steven Rostedt

Quoted 3 lines · Masami Hiramatsu

I just thought that frequent stop-machine is not so good from the user's POV. I agree that disabled probe ignoring the call is enough. Maybe, it could be done with the similar mechanism of jump optimization.

Quoted 1 line

I thought jump optimization still calls stop_machine too?

Yes, but now it does batch optimization.
Even if hundreds kprobes are registered separately, jump optimization has been done in background with a stop_machine per every 256 probes. (Until optimizing, kprobes can use breakpoints instead)

Quoted 7 lines · Masami Hiramatsu

Quoted 6 lines · Steven Rostedt

Quoted 3 lines · Masami Hiramatsu

I just thought that frequent stop-machine is not so good from the user's POV. I agree that disabled probe ignoring the call is enough. Maybe, it could be done with the similar mechanism of jump optimization.

Quoted 6 lines

Quoted 1 line

I thought jump optimization still calls stop_machine too?

Yes, but now it does batch optimization.
Even if hundreds kprobes are registered separately, jump optimization has been done in background with a stop_machine per every 256 probes. (Until optimizing, kprobes can use breakpoints instead)

But a single optimized kprobe still must use stopmachine.

But it is true that the function tracer does it as one big shot. That is, it will do all functions in a single stop machine that needs to be changed. It too is batched, but there is not a limit to that batch.

I would be interested in hearing from users and real use cases that someone would like to trace functions but stopmachine is too big of a hammer.

-----Original Message-----
From: Mathieu Desnoyers [mailto:[email protected]]
Sent: 18-Feb-11 10:20
To: Steven Rostedt; Dominique Toupin
Cc: Masami Hiramatsu; [email protected]; Ingo Molnar; Andrew Morton; Thomas Gleixner; Frederic Weisbecker; H. Peter Anvin; Andi Kleen; [email protected]
Subject: Re: [RFC][PATCH 0/4] ftrace: Use -mfentry when supported (this is for x86_64 right now)

[Adding Dominique Toupin, from Ericsson, to CC list]

Quoted 5 lines · * Steven Rostedt ([email protected]) wrote:

Quoted 4 lines · Masami Hiramatsu

Quoted 3 lines · Steven Rostedt

Quoted 1 line · Masami Hiramatsu

I just thought that frequent stop-machine is not so

good from the

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

user's POV. I agree that disabled probe ignoring the

call is enough.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Maybe, it could be done with the similar mechanism of

jump optimization.

Quoted 6 lines

Quoted 6 lines

Quoted 2 lines

I thought jump optimization still calls stop_machine too?

Yes, but now it does batch optimization.
Even if hundreds kprobes are registered separately, jump optimization has been done in background with a

stop_machine per every 256 probes.

Quoted 5 lines

Quoted 1 line

(Until optimizing, kprobes can use breakpoints instead)

But a single optimized kprobe still must use stopmachine.

But it is true that the function tracer does it as one big

shot. That

Quoted 1 line

is, it will do all functions in a single stop machine that

needs to be

Quoted 4 lines

changed. It too is batched, but there is not a limit to that batch.

I would be interested in hearing from users and real use cases that someone would like to trace functions but stopmachine is

too big of a

Quoted 1 line

hammer.

Hi Steven,

Telecom end users are one of such cases where the latency induced by stop machine while the system is running is a problem. Dominique Toupin could certainly tell us more about Ericsson's use-cases.

Thanks,

Mathieu

--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant EfficiOS Inc.
http://www.efficios.com

My understanding is stop_machine will stop all processors for many ms.

Even if most of our systems are not hard real-time they are soft real-time and stopping all cores for a few ms is not allowed. We can stop a few threads while we are jump patching but all processors is too much for us.

-----Original Message-----
From: Steven Rostedt [mailto:[email protected]]
Sent: 18-Feb-11 15:37
To: Dominique Toupin
Cc: Mathieu Desnoyers; Masami Hiramatsu; [email protected]; Ingo Molnar; Andrew Morton; Thomas Gleixner; Frederic Weisbecker; H. Peter Anvin; Andi Kleen; [email protected]
Subject: RE: [RFC][PATCH 0/4] ftrace: Use -mfentry when supported (this is for x86_64 right now)

Quoted 1 line · Dominique Toupin

My understanding is stop_machine will stop all processors

for many ms.

s/ms/us/

Quoted 1 line

Even if most of our systems are not hard real-time they are

soft real-time and stopping all cores for a few ms is not allowed.

Quoted 1 line

We can stop a few threads while we are jump patching but

all processors is too much for us.

I think I could hit a single ms if we enable full function tracing which disables ~22,000 functions in one shot. But if you enable full function tracing, the kernel can slow down quite drastically, and that would even be more problematic than a single ms hic-up. As hackbench showed a %150 slowdown when function tracer was running.

Now the last measurements I took was a few years ago and it was on a 4 CPU box. Perhaps stop_machine() may be a bit more expensive on a 1024 CPU box.

-- Steve

My understanding is stop_machine will stop all processors for many ms.

Quoted 2 lines · Dominique Toupin

My understanding is stop_machine will stop all processors for many ms.

I haven't measured it recently, but as long as the callback inside stop machine is short it definitely shouldn't be "many ms". The latency is bound by how long each CPU needs to answer to an interrupt, so if you have some code that disables interrupts for a long time it will take long -- but then your realtime response will be already bad.

The interrupts are also done in parallel, so the interrupt latencies don't add up.

If all the CPUs answer in a reasonable time it's still not a cheap operation, but nothing that takes "many ms". Most likely it's fine for most soft real time purposes.

We should also be able to use the breakpoint hack to avoid holding all the CPUs. They still need to be interrupted, but that skips the rendezvous operation.

Quoted 3 lines · H. Peter Anvin

We should also be able to use the breakpoint hack to avoid holding all the CPUs. They still need to be interrupted, but that skips the rendezvous operation.

As this is about the ftrace code, I'm in the process of analyzing and updating how the function tracer works. I can look to see if I can design it so we don't have to always use stop_machine() if a breakpoint method is in place.

Basically what is needed is to convert a "nop" into a "call" or maybe the other way around, safely.

Now is it safe to insert a breakpoint (usually a byte I believe), modify the rest of the instruction and then replace the breakpoint to the new code? Since the instruction that is being replaced or the instruction being added is always a nop, the breakpoint handler needs to do nothing but return to the location after the nop/call.

Is there any synchronization that needs to be done when doing this? Or can it just be:

	insert_breakpoint();
	update_instruction();
	remove_breakpoint();

Because we need to do this for 22,000 calls in a row.

Is there any synchronization that needs to be done when doing this? Or can it just be:

	insert_breakpoint();
	update_instruction();
	remove_breakpoint();

Because we need to do this for 22,000 calls in a row.