ftrace: Add use of -mfentry for x86_64

From: Steven Rostedt <[email protected]>

With gcc 4.6.0 the -mfentry feature places the function profiling call at the start of the function. When this is used, the call is to __fentry__ and not mcount.

Change recordmcount.c to record both callers to __fentry__ and mcount.

If the user changes compilers (or changes CFLAGS by insert/remove "-mfentry") without doing a "make clean", then recordmcount will omit some calls.

Those restrictions are easy to guess, and recovery is easy. Therefore, Ack'ed by: John Reiser <[email protected]>

From: Steven Rostedt <[email protected]>

The function graph has a test to check if the frame pointer is corrupted, which can happen with various options of gcc with mcount. But this is not an issue with -mfentry as -mfentry does not need nor use frame pointers for function graph tracing.

Cc Andi Kleen <[email protected]>

Signed-off-by Steven Rostedt <[email protected]>

---
 kernel/trace/trace_functions_graph.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

kernel/trace/trace_functions_graph.c +0 -0

index ce27c8b..99b4378 100644

kernel/trace/trace_functions_graph.c ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, · line 146

Quoted 3 lines

kernel/trace/trace_functions_graph.c +1 -1

-#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST
+#if defined(CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST) && !defined(CC_USING_FENTRY)

I think CONFIG_HAVE_FENTRY would better unselect CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST in arch/x86/Kconfig explicitly.

Quoted 6 lines · Masami Hiramatsu

kernel/trace/trace_functions_graph.c ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, · line 146

Quoted 3 lines

kernel/trace/trace_functions_graph.c +1 -1

-#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST
+#if defined(CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST) && !defined(CC_USING_FENTRY)

I think CONFIG_HAVE_FENTRY would better unselect CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST in arch/x86/Kconfig explicitly.

No, CONFIG_HAVE_FENTRY just means fentry is supported, it does not mean that it is being used. It only gets used if CC_USING_FENTRY is set, which is set by the Makefile at time of compile.

If CONFIG_HAVE_FENTRY is defined, a test is done to see if the gcc compiling the kernel supports -mfentry. If it does, then it defines the CC_USING_FENTRY macro, if not, the macro is not defined and the old way is performed.

If the old way is performed, even if CONFIG_HAVE_FENTRY is defined, then we still need the above test. We can not have CONFIG_HAVE_FENTRY unselect CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST.

No, CONFIG_HAVE_FENTRY just means fentry is supported, it does not mean that it is being used. It only gets used if CC_USING_FENTRY is set, which is set by the Makefile at time of compile.

Quoted 4 lines · Steven Rostedt

No, CONFIG_HAVE_FENTRY just means fentry is supported, it does not mean that it is being used. It only gets used if CC_USING_FENTRY is set, which is set by the Makefile at time of compile.

Btw, it might be lovely to consider the concept of "Kconfig variables set by shell-scripts".

We currently have a metric sh*t-ton of Makefile magic for testing various things like this, and integrating it into Kconfig might be a good idea. That way you would be able to use the Kconfig logic on these things.

Kconfig already has the "option env=XYZ" syntax for importing values from the environment variables. Extending it to some kind of "option shell=xyz" would allow for more complex interactions like this (imagine testing compiler options and version dependencies in the Kconfig files instead of the Makefiles)?

Btw, it might be lovely to consider the concept of "Kconfig variables set by shell-scripts".

We currently have a metric sh*t-ton of Makefile magic for testing various things like this, and integrating it into Kconfig might be a good idea. That way you would be able to use the Kconfig logic on these things.

Kconfig already has the "option env=XYZ" syntax for importing values from the environment variables. Extending it to some kind of "option shell=xyz" would allow for more complex interactions like this (imagine testing compiler options and version dependencies in the Kconfig files instead of the Makefiles)?

Quoted 4 lines · Steven Rostedt

No, CONFIG_HAVE_FENTRY just means fentry is supported, it does not mean that it is being used. It only gets used if CC_USING_FENTRY is set, which is set by the Makefile at time of compile.

Btw, it might be lovely to consider the concept of "Kconfig variables set by shell-scripts".

We currently have a metric sh*t-ton of Makefile magic for testing various things like this, and integrating it into Kconfig might be a good idea. That way you would be able to use the Kconfig logic on these things.

From: Steven Rostedt <[email protected]>

If the kernel is compiled with gcc 4.6.0 which supports -mfentry, then use that instead of mcount.

With mcount, frame pointers are forced with the -pg option and we get something like:

<can_vma_merge_before>:

       55                      push   %rbp
       48 89 e5                mov    %rsp,%rbp
       53                      push   %rbx
       41 51                   push   %r9
       e8 fe 6a 39 00          callq  ffffffff81483d00 <mcount>
       31 c0                   xor    %eax,%eax
       48 89 fb                mov    %rdi,%rbx
       48 89 d7                mov    %rdx,%rdi
       48 33 73 30             xor    0x30(%rbx),%rsi
       48 f7 c6 ff ff ff f7    test   $0xfffffffff7ffffff,%rsi

With -mfentry, frame pointers are no longer forced and the call looks like this:

<can_vma_merge_before>:

       e8 33 af 37 00          callq  ffffffff81461b40 <__fentry__>
       53                      push   %rbx
       48 89 fb                mov    %rdi,%rbx
       31 c0                   xor    %eax,%eax
       48 89 d7                mov    %rdx,%rdi
       41 51                   push   %r9
       48 33 73 30             xor    0x30(%rbx),%rsi
       48 f7 c6 ff ff ff f7    test   $0xfffffffff7ffffff,%rsi

This adds the ftrace hook at the beginning of the function before a frame is set up, and allows the function callbacks to be able to access parameters. As kprobes now can use function tracing (at least on x86) this speeds up the kprobe hooks that are at the beginning of the function.

Cc Masami Hiramatsu <[email protected]>

Cc Andi Kleen <[email protected]>

Signed-off-by Steven Rostedt <[email protected]>

---
 arch/x86/Kconfig                 |    1 +
 arch/x86/include/asm/ftrace.h    |    7 ++++++-
 arch/x86/kernel/entry_64.S       |   18 +++++++++++++++++-
 arch/x86/kernel/x8664_ksyms_64.c |    6 +++++-
 4 files changed, 29 insertions(+), 3 deletions(-)

arch/x86/Kconfig +1 -0

index c70684f..bbbf5d8 100644
@@ -36,6 +36,7 @@ config X86
 	select HAVE_KRETPROBES
 	select HAVE_OPTPROBES
 	select HAVE_FTRACE_MCOUNT_RECORD
+	select HAVE_FENTRY if X86_64
 	select HAVE_C_RECORDMCOUNT
 	select HAVE_DYNAMIC_FTRACE
 	select HAVE_FUNCTION_TRACER

arch/x86/include/asm/ftrace.h +6 -1

index a6cae0c..9a25b52 100644
@@ -35,7 +35,11 @@
 #endif
 
 #ifdef CONFIG_FUNCTION_TRACER
-#define MCOUNT_ADDR		((long)(mcount))
+#ifdef CC_USING_FENTRY
+# define MCOUNT_ADDR		((long)(__fentry__))
+#else
+# define MCOUNT_ADDR		((long)(mcount))
+#endif
 #define MCOUNT_INSN_SIZE	5 /* sizeof mcount call */
 
 #ifdef CONFIG_DYNAMIC_FTRACE

@@ -46,6 +50,7 @@
 #ifndef __ASSEMBLY__
 extern void mcount(void);
 extern atomic_t modifying_ftrace_code;
+extern void __fentry__(void);
 
 static inline unsigned long ftrace_call_adjust(unsigned long addr)
 {

arch/x86/kernel/entry_64.S +17 -1

index 38308fa..2add3bb 100644
@@ -69,9 +69,16 @@
 
 #ifdef CONFIG_FUNCTION_TRACER
 #ifdef CONFIG_DYNAMIC_FTRACE
+
+#ifdef CC_USING_FENTRY
+ENTRY(__fentry__)
+	retq
+END(__fentry__)
+#else
 ENTRY(mcount)
 	retq
 END(mcount)
+#endif
 
 /* skip is set if stack has been adjusted */
 .macro ftrace_caller_setup skip=0

@@ -84,7 +91,11 @@ END(mcount)
 	movq RIP(%rsp), %rdi
 	subq $MCOUNT_INSN_SIZE, %rdi
 	/* Load the parent_ip into the second parameter */
+#ifdef CC_USING_FENTRY
+	movq SS+16(%rsp), %rsi
+#else
 	movq 8(%rbp), %rsi
+#endif
 .endm
 
 ENTRY(ftrace_caller)

@@ -215,9 +226,14 @@ END(mcount)
 ENTRY(ftrace_graph_caller)
 	MCOUNT_SAVE_FRAME
 
+#ifdef CC_USING_FENTRY
+	leaq SS+16(%rsp), %rdi
+	movq $0, %rdx	/* No framepointers needed */
+#else
 	leaq 8(%rbp), %rdi
-	movq RIP(%rsp), %rsi
 	movq (%rbp), %rdx
+#endif
+	movq RIP(%rsp), %rsi
 	subq $MCOUNT_INSN_SIZE, %rsi
 
 	call	prepare_ftrace_return

arch/x86/kernel/x8664_ksyms_64.c +5 -1

index 9796c2f..643b236 100644
@@ -13,9 +13,13 @@
 #include <asm/ftrace.h>
 
 #ifdef CONFIG_FUNCTION_TRACER
-/* mcount is defined in assembly */
+/* mcount and __fentry__ are defined in assembly */
+#ifdef CC_USING_FENTRY
+EXPORT_SYMBOL(__fentry__);
+#else
 EXPORT_SYMBOL(mcount);
 #endif
+#endif
 
 EXPORT_SYMBOL(__get_user_1);
 EXPORT_SYMBOL(__get_user_2);

From: Steven Rostedt <[email protected]>

If the kernel is compiled with gcc 4.6.0 which supports -mfentry, then use that instead of mcount.

With mcount, frame pointers are forced with the -pg option and we get something like:

<can_vma_merge_before>:

       55                      push   %rbp
       48 89 e5                mov    %rsp,%rbp
       53                      push   %rbx
       41 51                   push   %r9
       e8 fe 6a 39 00          callq  ffffffff81483d00 <mcount>
       31 c0                   xor    %eax,%eax
       48 89 fb                mov    %rdi,%rbx
       48 89 d7                mov    %rdx,%rdi
       48 33 73 30             xor    0x30(%rbx),%rsi
       48 f7 c6 ff ff ff f7    test   $0xfffffffff7ffffff,%rsi

With -mfentry, frame pointers are no longer forced and the call looks like this:

<can_vma_merge_before>:

       e8 33 af 37 00          callq  ffffffff81461b40 <__fentry__>
       53                      push   %rbx
       48 89 fb                mov    %rdi,%rbx
       31 c0                   xor    %eax,%eax
       48 89 d7                mov    %rdx,%rdi
       41 51                   push   %r9
       48 33 73 30             xor    0x30(%rbx),%rsi
       48 f7 c6 ff ff ff f7    test   $0xfffffffff7ffffff,%rsi

This adds the ftrace hook at the beginning of the function before a frame is set up, and allows the function callbacks to be able to access parameters. As kprobes now can use function tracing (at least on x86) this speeds up the kprobe hooks that are at the beginning of the function.

Cc Masami Hiramatsu <[email protected]>

Cc Andi Kleen <[email protected]>

Signed-off-by Steven Rostedt <[email protected]>

Peter and Masami

During my final tests, I found that this change breaks the !DYNAMIC_FTRACE config. That is, when we don't do the run-time updates of mcount calls to nops, the compiler will use fentry but the code still uses mcount.

I fixed this in the patch below. But as you two have acked and reviewed it, I can't add your tags if I have changed the code. Can you ack/review it again.

Thanks!

Peter and Masami

During my final tests, I found that this change breaks the !DYNAMIC_FTRACE config. That is, when we don't do the run-time updates of mcount calls to nops, the compiler will use fentry but the code still uses mcount.

I fixed this in the patch below. But as you two have acked and reviewed it, I can't add your tags if I have changed the code. Can you ack/review it again.

Thanks!

-- Steve

Quoted 39 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

If the kernel is compiled with gcc 4.6.0 which supports -mfentry, then use that instead of mcount.

With mcount, frame pointers are forced with the -pg option and we get something like:

<can_vma_merge_before>:

       55                      push   %rbp
       48 89 e5                mov    %rsp,%rbp
       53                      push   %rbx
       41 51                   push   %r9
       e8 fe 6a 39 00          callq  ffffffff81483d00 <mcount>
       31 c0                   xor    %eax,%eax
       48 89 fb                mov    %rdi,%rbx
       48 89 d7                mov    %rdx,%rdi
       48 33 73 30             xor    0x30(%rbx),%rsi
       48 f7 c6 ff ff ff f7    test   $0xfffffffff7ffffff,%rsi

With -mfentry, frame pointers are no longer forced and the call looks like this:

<can_vma_merge_before>:

       e8 33 af 37 00          callq  ffffffff81461b40 <__fentry__>
       53                      push   %rbx
       48 89 fb                mov    %rdi,%rbx
       31 c0                   xor    %eax,%eax
       48 89 d7                mov    %rdx,%rdi
       41 51                   push   %r9
       48 33 73 30             xor    0x30(%rbx),%rsi
       48 f7 c6 ff ff ff f7    test   $0xfffffffff7ffffff,%rsi

This adds the ftrace hook at the beginning of the function before a frame is set up, and allows the function callbacks to be able to access parameters. As kprobes now can use function tracing (at least on x86) this speeds up the kprobe hooks that are at the beginning of the function.

Quoted 3 lines

Cc Masami Hiramatsu <[email protected]>

Cc Andi Kleen <[email protected]>

Signed-off-by Steven Rostedt <[email protected]>

(change log kept the same)

arch/x86/Kconfig +1 -0

index c70684f..bbbf5d8 100644
@@ -36,6 +36,7 @@ config X86
 	select HAVE_KRETPROBES
 	select HAVE_OPTPROBES
 	select HAVE_FTRACE_MCOUNT_RECORD
+	select HAVE_FENTRY if X86_64
 	select HAVE_C_RECORDMCOUNT
 	select HAVE_DYNAMIC_FTRACE
 	select HAVE_FUNCTION_TRACER

arch/x86/include/asm/ftrace.h +6 -1

index a6cae0c..9a25b52 100644
@@ -35,7 +35,11 @@
 #endif
 
 #ifdef CONFIG_FUNCTION_TRACER
-#define MCOUNT_ADDR		((long)(mcount))
+#ifdef CC_USING_FENTRY
+# define MCOUNT_ADDR		((long)(__fentry__))
+#else
+# define MCOUNT_ADDR		((long)(mcount))
+#endif
 #define MCOUNT_INSN_SIZE	5 /* sizeof mcount call */
 
 #ifdef CONFIG_DYNAMIC_FTRACE

@@ -46,6 +50,7 @@
 #ifndef __ASSEMBLY__
 extern void mcount(void);
 extern atomic_t modifying_ftrace_code;
+extern void __fentry__(void);
 
 static inline unsigned long ftrace_call_adjust(unsigned long addr)
 {

arch/x86/kernel/entry_64.S +27 -5

index 38308fa..a698521 100644
@@ -68,10 +68,18 @@
 	.section .entry.text, "ax"
 
 #ifdef CONFIG_FUNCTION_TRACER
+
+#ifdef CC_USING_FENTRY
+# define function_hook	__fentry__
+#else
+# define function_hook	mcount
+#endif
+
 #ifdef CONFIG_DYNAMIC_FTRACE
-ENTRY(mcount)
+
+ENTRY(function_hook)
 	retq
-END(mcount)
+END(function_hook)
 
 /* skip is set if stack has been adjusted */
 .macro ftrace_caller_setup skip=0

@@ -84,7 +92,11 @@ END(mcount)
 	movq RIP(%rsp), %rdi
 	subq $MCOUNT_INSN_SIZE, %rdi
 	/* Load the parent_ip into the second parameter */
+#ifdef CC_USING_FENTRY
+	movq SS+16(%rsp), %rsi
+#else
 	movq 8(%rbp), %rsi
+#endif
 .endm
 
 ENTRY(ftrace_caller)

@@ -177,7 +189,8 @@ END(ftrace_regs_caller)
 
 
 #else /* ! CONFIG_DYNAMIC_FTRACE */
-ENTRY(mcount)
+
+ENTRY(function_hook)
 	cmpl $0, function_trace_stop
 	jne  ftrace_stub
 

@@ -199,7 +212,11 @@ trace:
 	MCOUNT_SAVE_FRAME
 
 	movq RIP(%rsp), %rdi
+#ifdef CC_USING_FENTRY
+	movq SS+16(%rsp), %rsi
+#else
 	movq 8(%rbp), %rsi
+#endif
 	subq $MCOUNT_INSN_SIZE, %rdi
 
 	call   *ftrace_trace_function

@@ -207,7 +224,7 @@ trace:
 	MCOUNT_RESTORE_FRAME
 
 	jmp ftrace_stub
-END(mcount)
+END(function_hook)
 #endif /* CONFIG_DYNAMIC_FTRACE */
 #endif /* CONFIG_FUNCTION_TRACER */
 

@@ -215,9 +232,14 @@ END(mcount)
 ENTRY(ftrace_graph_caller)
 	MCOUNT_SAVE_FRAME
 
+#ifdef CC_USING_FENTRY
+	leaq SS+16(%rsp), %rdi
+	movq $0, %rdx	/* No framepointers needed */
+#else
 	leaq 8(%rbp), %rdi
-	movq RIP(%rsp), %rsi
 	movq (%rbp), %rdx
+#endif
+	movq RIP(%rsp), %rsi
 	subq $MCOUNT_INSN_SIZE, %rsi
 
 	call	prepare_ftrace_return

arch/x86/kernel/x8664_ksyms_64.c +5 -2

index 9796c2f..643b236 100644
@@ -13,9 +13,13 @@
 #include <asm/ftrace.h>
 
 #ifdef CONFIG_FUNCTION_TRACER
-/* mcount is defined in assembly */
+/* mcount and __fentry__ are defined in assembly */
+#ifdef CC_USING_FENTRY
+EXPORT_SYMBOL(__fentry__);
+#else
 EXPORT_SYMBOL(mcount);
 #endif
+#endif
 
 EXPORT_SYMBOL(__get_user_1);
 EXPORT_SYMBOL(__get_user_2);
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

This is an RFC patch set that makes gcc use the -mfentry option with -pg. This will set the ftrace 'hook' to the beginning of the function and also remove the requirement that -pg enables frame pointers.

This has a couple of benefits (and probably more).

1) removal of the frame pointer requirement makes for smaller and faster code.

2) Having the function trace hook at the beginning of the function instead of after the frame is set up, gives the function tracing callbacks access to the parameters. This means that kprobes can take advantage of this. When a kprobe is set on top of a ftrace hook (nop), it will automatically use the function tracing callback. This makes it into an 'optimized' probe as there's no need to hit a breakpoint and trigger the probe that way. The function tracing code can do the work for it. Note, optimized probes are only allowed with !PREEMPT, but a ftrace optimize probe is allowed in any context (another benefit).

This only implements fentry for x86_64.

Quoted 19 lines · Steven Rostedt

This is an RFC patch set that makes gcc use the -mfentry option with -pg. This will set the ftrace 'hook' to the beginning of the function and also remove the requirement that -pg enables frame pointers.

This has a couple of benefits (and probably more).

1) removal of the frame pointer requirement makes for smaller and faster code.

2) Having the function trace hook at the beginning of the function instead of after the frame is set up, gives the function tracing callbacks access to the parameters. This means that kprobes can take advantage of this. When a kprobe is set on top of a ftrace hook (nop), it will automatically use the function tracing callback. This makes it into an 'optimized' probe as there's no need to hit a breakpoint and trigger the probe that way. The function tracing code can do the work for it. Note, optimized probes are only allowed with !PREEMPT, but a ftrace optimize probe is allowed in any context (another benefit).

Quoted 2 lines

This only implements fentry for x86_64.

Acked-by H. Peter Anvin <[email protected]>