tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
Subsystems (16)
Masami Hiramatsu (Google)
maintainerv10 cover tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
Hi,
Here is the 10th version of the series to re-implement the fprobe on function-graph tracer. The previous version is;
https://lore.kernel.org/all/171318533841.254850.15841395205784342850.stgit@devnote2/
This version is ported on the latest kernel (v6.9-rc6 + probes/for-next) and fixed some bugs + performance optimizations.
- [7/36] Fix terminology in comments and code. Use "offset" instead of "index" for shadow stack. This also update macros.
- [18/36] Fix supported data size bug.
- [29/36] Define bpf_kprobe_multi_pt_regs only if it is used.
- [36/36] Add likely() to skip timestamp.
Overview
--------
This series does major 2 changes, enable multiple function-graphs on the ftrace (e.g. allow function-graph on sub instances) and rewrite the fprobe on this function-graph.
The former changes had been sent from Steven Rostedt 4 years ago (*), which allows users to set different setting function-graph tracer (and other tracers based on function-graph) in each trace-instances at the same time.
(*) https://lore.kernel.org/all/[email protected]/
The purpose of latter change are;
1) Remove dependency of the rethook from fprobe so that we can reduce the return hook code and shadow stack.
2) Make 'ftrace_regs' the common trace interface for the function boundary.
1) Currently we have 2(or 3) different function return hook codes, the function-graph tracer and rethook (and legacy kretprobe). But since this is redundant and needs double maintenance cost, I would like to unify those. From the user's viewpoint, function- graph tracer is very useful to grasp the execution path. For this purpose, it is hard to use the rethook in the function-graph tracer, but the opposite is possible. (Strictly speaking, kretprobe can not use it because it requires 'pt_regs' for historical reasons.)
2) Now the fprobe provides the 'pt_regs' for its handler, but that is wrong for the function entry and exit. Moreover, depending on the architecture, there is no way to accurately reproduce 'pt_regs' outside of interrupt or exception handlers. This means fprobe should not use 'pt_regs' because it does not use such exceptions. (Conversely, kprobe should use 'pt_regs' because it is an abstract interface of the software breakpoint exception.)
This series changes fprobe to use function-graph tracer for tracing function entry and exit, instead of mixture of ftrace and rethook. Unlike the rethook which is a per-task list of system-wide allocated nodes, the function graph's ret_stack is a per-task shadow stack. Thus it does not need to set 'nr_maxactive' (which is the number of pre-allocated nodes).
Also the handlers will get the 'ftrace_regs' instead of 'pt_regs'. Since eBPF mulit_kprobe/multi_kretprobe events still use 'pt_regs' as their register interface, this changes it to convert 'ftrace_regs' to 'pt_regs'. Of course this conversion makes an incomplete 'pt_regs', so users must access only registers for function parameters or return value.
Design
------
Instead of using ftrace's function entry hook directly, the new fprobe is built on top of the function-graph's entry and return callbacks with 'ftrace_regs'.
Since the fprobe requires access to 'ftrace_regs', the architecture must support CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS and CONFIG_HAVE_FTRACE_GRAPH_FUNC, which enables to call function-graph entry callback with 'ftrace_regs', and also CONFIG_HAVE_FUNCTION_GRAPH_FREGS, which passes the ftrace_regs to return_to_handler.
All fprobes share a single function-graph ops (means shares a common ftrace filter) similar to the kprobe-on-ftrace. This needs another layer to find corresponding fprobe in the common function-graph callbacks, but has much better scalability, since the number of registered function-graph ops is limited.
In the entry callback, the fprobe runs its entry_handler and saves the address of 'fprobe' on the function-graph's shadow stack as data. The return callback decodes the data to get the 'fprobe' address, and runs the exit_handler.
The fprobe introduces two hash-tables, one is for entry callback which searches fprobes related to the given function address passed by entry callback. The other is for a return callback which checks if the given 'fprobe' data structure pointer is still valid. Note that it is possible to unregister fprobe before the return callback runs. Thus the address validation must be done before using it in the return callback.
This series can be applied against the probes/for-next branch, which is based on v6.9-rc6.
This series can also be found below branch.
https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/fprobe-on-fgraph
Thank you,
---
Masami Hiramatsu (Google) (21):
tracing: Add a comment about ftrace_regs definition
tracing: Rename ftrace_regs_return_value to ftrace_regs_get_return_value
x86: tracing: Add ftrace_regs definition in the header
function_graph: Use a simple LRU for fgraph_array index number
ftrace: Add multiple fgraph storage selftest
function_graph: Pass ftrace_regs to entryfunc
function_graph: Replace fgraph_ret_regs with ftrace_regs
function_graph: Pass ftrace_regs to retfunc
fprobe: Use ftrace_regs in fprobe entry handler
fprobe: Use ftrace_regs in fprobe exit handler
tracing: Add ftrace_partial_regs() for converting ftrace_regs to pt_regs
tracing: Add ftrace_fill_perf_regs() for perf event tracing/fprobe: Enable fprobe events with CONFIG_DYNAMIC_FTRACE_WITH_ARGS
bpf: Enable kprobe_multi feature if CONFIG_FPROBE is enabled
ftrace: Add CONFIG_HAVE_FTRACE_GRAPH_FUNC
fprobe: Rewrite fprobe on function-graph tracer tracing/fprobe: Remove nr_maxactive from fprobe
selftests: ftrace: Remove obsolate maxactive syntax check selftests/ftrace: Add a test case for repeating register/unregister fprobe
Documentation: probes: Update fprobe on function-graph tracer
fgraph: Skip recording calltime/rettime if it is not nneeded
Steven Rostedt (VMware) (15):
function_graph: Convert ret_stack to a series of longs
fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic around ftrace_graph_entry and return ftrace/function_graph: Pass fgraph_ops to function graph callbacks
ftrace: Allow function_graph tracer to be enabled in instances
ftrace: Allow ftrace startup flags exist without dynamic ftrace
function_graph: Have the instances use their own ftrace_ops for filtering
function_graph: Add "task variables" per task for fgraph_ops
function_graph: Move set_graph_function tests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data()
function_graph: Add selftest for passing local variables
Documentation/trace/fprobe.rst | 42 + arch/arm64/Kconfig | 3 arch/arm64/include/asm/ftrace.h | 47 + arch/arm64/kernel/asm-offsets.c | 12 arch/arm64/kernel/entry-ftrace.S | 32 - arch/arm64/kernel/ftrace.c | 21 arch/loongarch/Kconfig | 4 arch/loongarch/include/asm/ftrace.h | 32 - arch/loongarch/kernel/asm-offsets.c | 12 arch/loongarch/kernel/ftrace_dyn.c | 15 arch/loongarch/kernel/mcount.S | 17 arch/loongarch/kernel/mcount_dyn.S | 14 arch/powerpc/Kconfig | 1 arch/powerpc/include/asm/ftrace.h | 15 arch/powerpc/kernel/trace/ftrace.c | 3 arch/powerpc/kernel/trace/ftrace_64_pg.c | 10 arch/riscv/Kconfig | 3 arch/riscv/include/asm/ftrace.h | 21 arch/riscv/kernel/ftrace.c | 15 arch/riscv/kernel/mcount.S | 24 arch/s390/Kconfig | 3 arch/s390/include/asm/ftrace.h | 39 - arch/s390/kernel/asm-offsets.c | 6 arch/s390/kernel/mcount.S | 9 arch/x86/Kconfig | 4 arch/x86/include/asm/ftrace.h | 43 - arch/x86/kernel/ftrace.c | 51 + arch/x86/kernel/ftrace_32.S | 15 arch/x86/kernel/ftrace_64.S | 17 include/linux/fprobe.h | 57 + include/linux/ftrace.h | 172 +++ include/linux/sched.h | 2 include/linux/trace_recursion.h | 39 - kernel/trace/Kconfig | 23 kernel/trace/bpf_trace.c | 19 kernel/trace/fgraph.c | 1024 ++++++++++++++++---- kernel/trace/fprobe.c | 637 +++++++++--- kernel/trace/ftrace.c | 13 kernel/trace/ftrace_internal.h | 2 kernel/trace/trace.h | 96 ++ kernel/trace/trace_fprobe.c | 147 +-- kernel/trace/trace_functions.c | 8 kernel/trace/trace_functions_graph.c | 98 +- kernel/trace/trace_irqsoff.c | 12 kernel/trace/trace_probe_tmpl.h | 2 kernel/trace/trace_sched_wakeup.c | 12 kernel/trace/trace_selftest.c | 262 +++++ lib/test_fprobe.c | 51 - samples/fprobe/fprobe_example.c | 4 .../test.d/dynevent/add_remove_fprobe_repeat.tc | 19 .../ftrace/test.d/dynevent/fprobe_syntax_errors.tc | 4 51 files changed, 2347 insertions(+), 886 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc
--
Masami Hiramatsu (Google) <[email protected]>
Masami Hiramatsu (Google)
maintainerPATCH v10 1/36 tracing: Add a comment about ftrace_regs definition
From: Masami Hiramatsu (Google) <[email protected]>
To clarify what will be expected on ftrace_regs, add a comment to the architecture independent definition of the ftrace_regs.
---
Changes in v8:
- Update that the saved registers depends on the context.
Changes in v3:
- Add instruction pointer
Changes in v2:
- newly added.
--- include/linux/ftrace.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
include/linux/ftrace.h +26 -0
@@ -118,6 +118,32 @@ extern int ftrace_enabled;#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/** +* ftrace_regs - ftrace partial/optimal register set +* +* ftrace_regs represents a group of registers which is used at the +* function entry and exit. There are three types of registers. +* +* - Registers for passing the parameters to callee, including the stack +* pointer. (e.g. rcx, rdx, rdi, rsi, r8, r9 and rsp on x86_64) +* - Registers for passing the return values to caller. +* (e.g. rax and rdx on x86_64) +* - Registers for hooking the function call and return including the +* frame pointer (the frame pointer is architecture/config dependent) +* (e.g. rip, rbp and rsp for x86_64) +* +* Also, architecture dependent fields can be used for internal process. +* (e.g. orig_ax on x86_64) +* +* On the function entry, those registers will be restored except for +* the stack pointer, so that user can change the function parameters +* and instruction pointer (e.g. live patching.) +* On the function exit, only registers which is used for return values +* are restored. +* +* NOTE: user *must not* access regs directly, only do it via APIs, because +* the member can be changed according to the architecture. +*/ struct ftrace_regs {struct pt_regs regs; };
Steven Rostedt
maintainerRe: [PATCH v10 01/36] tracing: Add a comment about ftrace_regs definition
Quoted 38 lines · On Tue, 7 May 2024 23:08:12 +0900 "Masami Hiramatsu (Google)"…
From: Masami Hiramatsu (Google) <[email protected]>
To clarify what will be expected on
ftrace_regs, add a comment to the architecture independent definition of theftrace_regs.Signed-off-by Masami Hiramatsu (Google) <[email protected]>Acked-by Mark Rutland <[email protected]>---
Changes in v8:
- Update that the saved registers depends on the context.
Changes in v3:
- Add instruction pointer
Changes in v2:
- newly added.--- include/linux/ftrace.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)include/linux/ftrace.h +12
-0@@ -118,6 +118,32 @@ extern int ftrace_enabled;#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/** +* ftrace_regs - ftrace partial/optimal register set +* +* ftrace_regs represents a group of registers which is used at the +* function entry and exit. There are three types of registers. +* +* - Registers for passing the parameters to callee, including the stack +* pointer. (e.g. rcx, rdx, rdi, rsi, r8, r9 and rsp on x86_64) +* - Registers for passing the return values to caller. +* (e.g. rax and rdx on x86_64) +* - Registers for hooking the function call and return including the +* frame pointer (the frame pointer is architecture/config dependent)
include/linux/ftrace.h extern int ftrace_enabled; · line 142
Quoted 10 lines
include/linux/ftrace.h +10
-0+* (e.g. rip, rbp and rsp for x86_64) +* +* Also, architecture dependent fields can be used for internal process. +* (e.g. orig_ax on x86_64) +* +* On the function entry, those registers will be restored except for +* the stack pointer, so that user can change the function parameters +* and instruction pointer (e.g. live patching.) +* On the function exit, only registers which is used for return values +* are restored.
I wonder if we should also add a note about some architectures in some circumstances may store all pt_regs in ftrace_regs. For example, if an architecture supports FTRACE_WITH_REGS, it may pass the pt_regs within the ftrace_regs. If that is the case, then ftrace_get_regs() called on it will return a pointer to a valid pt_regs, or NULL if it is not supported or the ftrace_regs does not have a all the registers.
-- Steve
Quoted 7 lines
include/linux/ftrace.h +4
-0+* +* NOTE: user *must not* access regs directly, only do it via APIs, because +* the member can be changed according to the architecture. +*/ struct ftrace_regs {struct pt_regs regs; };
Masami Hiramatsu (Google)
maintainerRe: [PATCH v10 01/36] tracing: Add a comment about ftrace_regs definition
Quoted 50 lines · On Thu, 23 May 2024 19:10:31 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 47 lines · On Tue, 7 May 2024 23:08:12 +0900 "Masami Hiramatsu (Google)"…
From: Masami Hiramatsu (Google) <[email protected]>
To clarify what will be expected on
ftrace_regs, add a comment to the architecture independent definition of theftrace_regs.Signed-off-by Masami Hiramatsu (Google) <[email protected]>Acked-by Mark Rutland <[email protected]>---
Changes in v8:
- Update that the saved registers depends on the context.
Changes in v3:
- Add instruction pointer
Changes in v2:
- newly added.--- include/linux/ftrace.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)include/linux/ftrace.h +21
-0@@ -118,6 +118,32 @@ extern int ftrace_enabled;#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/** +* ftrace_regs - ftrace partial/optimal register set +* +* ftrace_regs represents a group of registers which is used at the +* function entry and exit. There are three types of registers. +* +* - Registers for passing the parameters to callee, including the stack +* pointer. (e.g. rcx, rdx, rdi, rsi, r8, r9 and rsp on x86_64) +* - Registers for passing the return values to caller. +* (e.g. rax and rdx on x86_64) +* - Registers for hooking the function call and return including the +* frame pointer (the frame pointer is architecture/config dependent) +* (e.g. rip, rbp and rsp for x86_64) +* +* Also, architecture dependent fields can be used for internal process. +* (e.g. orig_ax on x86_64) +* +* On the function entry, those registers will be restored except for +* the stack pointer, so that user can change the function parameters +* and instruction pointer (e.g. live patching.) +* On the function exit, only registers which is used for return values
Quoted 8 lines
include/linux/ftrace.h extern int ftrace_enabled; · line 142
Quoted 1 line
include/linux/ftrace.h +1
-0+* are restored. I wonder if we should also add a note about some architectures in some circumstances may store all
pt_regsinftrace_regs. For example, if an architecture supportsFTRACE_WITH_REGS, it may pass thept_regswithin theftrace_regs. If that is the case, thenftrace_get_regs()called on it will return a pointer to a validpt_regs, or NULL if it is not supported or theftrace_regsdoes not have a all the registers.
Agreed. That case also should be noted. Thanks for pointing!
Quoted 12 lines
-- Steve
Quoted 7 lines
include/linux/ftrace.h +4
-0+* +* NOTE: user *must not* access regs directly, only do it via APIs, because +* the member can be changed according to the architecture. +*/ struct ftrace_regs {struct pt_regs regs; };
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Masami Hiramatsu (Google)
maintainerPATCH v10 2/36 tracing: Rename ftrace_regs_return_value to ftrace_regs_get_return_value
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Rename ftrace_regs_return_value to ftrace_regs_get_return_value as same as other ftrace_regs_get/set_* APIs.
---
Changes in v6:
- Moved to top of the series.
Changes in v3:
- Newly added.
--- arch/loongarch/include/asm/ftrace.h | 2 +- arch/powerpc/include/asm/ftrace.h | 2 +- arch/s390/include/asm/ftrace.h | 2 +- arch/x86/include/asm/ftrace.h | 2 +- include/linux/ftrace.h | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
arch/loongarch/include/asm/ftrace.h +1 -1
@@ -70,7 +70,7 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) regs_get_kernel_argument(&(fregs)->regs, n)#define ftrace_regs_get_stack_pointer (fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value (fregs) \ +#define ftrace_regs_get_return_value (fregs) \ regs_return_value(&(fregs)->regs)#define ftrace_regs_set_return_value (fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret)
arch/powerpc/include/asm/ftrace.h +1 -1
@@ -61,7 +61,7 @@ ftrace_regs_get_instruction_pointer(struct ftrace_regs *fregs) regs_get_kernel_argument(&(fregs)->regs, n)#define ftrace_regs_get_stack_pointer (fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value (fregs) \ +#define ftrace_regs_get_return_value (fregs) \ regs_return_value(&(fregs)->regs)#define ftrace_regs_set_return_value (fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret)
arch/s390/include/asm/ftrace.h +1 -1
@@ -88,7 +88,7 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, regs_get_kernel_argument(&(fregs)->regs, n)#define ftrace_regs_get_stack_pointer (fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value (fregs) \ +#define ftrace_regs_get_return_value (fregs) \ regs_return_value(&(fregs)->regs)#define ftrace_regs_set_return_value (fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret)
arch/x86/include/asm/ftrace.h +1 -1
@@ -58,7 +58,7 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) regs_get_kernel_argument(&(fregs)->regs, n)#define ftrace_regs_get_stack_pointer (fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value (fregs) \ +#define ftrace_regs_get_return_value (fregs) \ regs_return_value(&(fregs)->regs)#define ftrace_regs_set_return_value (fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret)
include/linux/ftrace.h +1 -1
@@ -184,7 +184,7 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) regs_get_kernel_argument(ftrace_get_regs(fregs), n)#define ftrace_regs_get_stack_pointer (fregs) \ kernel_stack_pointer(ftrace_get_regs(fregs)) -#define ftrace_regs_return_value (fregs) \ +#define ftrace_regs_get_return_value (fregs) \ regs_return_value(ftrace_get_regs(fregs))#define ftrace_regs_set_return_value (fregs, ret) \ regs_set_return_value(ftrace_get_regs(fregs), ret)
Masami Hiramatsu (Google)
maintainerPATCH v10 3/36 x86: tracing: Add ftrace_regs definition in the header
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Add ftrace_regs definition for x86_64 in the ftrace header to clarify what register will be accessible from ftrace_regs.
---
Changes in v3:
- Add rip to be saved.
Changes in v2:
- Newly added.
--- arch/x86/include/asm/ftrace.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/include/asm/ftrace.h +6 -0
@@ -36,6 +36,12 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS struct ftrace_regs { +/* +* On the x86_64, the ftrace_regs saves; +* rax, rcx, rdx, rdi, rsi, r8, r9, rbp, rip and rsp. +* Also orig_ax is used for passing direct trampoline address. +* x86_32 doesn't support ftrace_regs. +*/ struct pt_regs regs; };
Steven Rostedt
maintainerRe: [PATCH v10 03/36] x86: tracing: Add ftrace_regs definition in the header
Quoted 20 lines · On Tue, 7 May 2024 23:08:35 +0900 "Masami Hiramatsu (Google)"…
From: Masami Hiramatsu (Google) <[email protected]>
Add
ftrace_regsdefinition forx86_64in the ftrace header to clarify what register will be accessible fromftrace_regs.Signed-off-by Masami Hiramatsu (Google) <[email protected]>---
Changes in v3:
- Add rip to be saved.
Changes in v2:
- Newly added.--- arch/x86/include/asm/ftrace.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/ftrace.h +0
-0
arch/x86/include/asm/ftrace.h static inline unsigned long ftrace_call_adjust(unsigned long addr) · line 43
Quoted 8 lines
arch/x86/include/asm/ftrace.h +5
-0@@ -36,6 +36,12 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS struct ftrace_regs { +/* +* On the x86_64, the ftrace_regs saves; +* rax, rcx, rdx, rdi, rsi, r8, r9, rbp, rip and rsp. +* Also orig_ax is used for passing direct trampoline address. +* x86_32 doesn't support ftrace_regs.
Should add a comment that if fregs->regs.cs is set, then all of the pt_regs is valid. And x86_32 does support ftrace_regs, it just doesn't support having a subset of it.
-- Steve
Quoted 4 lines
arch/x86/include/asm/ftrace.h +1
-0+*/ struct pt_regs regs; };
Masami Hiramatsu (Google)
maintainerRe: [PATCH v10 03/36] x86: tracing: Add ftrace_regs definition in the header
Quoted 24 lines · On Thu, 23 May 2024 19:14:59 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 21 lines · On Tue, 7 May 2024 23:08:35 +0900 "Masami Hiramatsu (Google)"…
From: Masami Hiramatsu (Google) <[email protected]>
Add
ftrace_regsdefinition forx86_64in the ftrace header to clarify what register will be accessible fromftrace_regs.Signed-off-by Masami Hiramatsu (Google) <[email protected]>---
Changes in v3:
- Add rip to be saved.
Changes in v2:
- Newly added.--- arch/x86/include/asm/ftrace.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/ftrace.h +0
-0@@ -36,6 +36,12 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)
Quoted 10 lines
arch/x86/include/asm/ftrace.h static inline unsigned long ftrace_call_adjust(unsigned long addr) · line 43
Quoted 7 lines
arch/x86/include/asm/ftrace.h +5
-0#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS struct ftrace_regs { +/* +* On the x86_64, the ftrace_regs saves; +* rax, rcx, rdx, rdi, rsi, r8, r9, rbp, rip and rsp. +* Also orig_ax is used for passing direct trampoline address. +* x86_32 doesn't support ftrace_regs. Should add a comment that if
fregs->regs.csis set, then all of thept_regsis valid.
But what about rbx and r1*? Only regs->cs should be care for pt_regs? Or, did you mean "the ftrace_regs is valid"?
Quoted 2 lines
And
x86_32does supportftrace_regs, it just doesn't support having a subset of it.
Oh, thanks. I'll update the comment about x86_32.
Thank you,
Quoted 10 lines
-- Steve
Quoted 4 lines
arch/x86/include/asm/ftrace.h +1
-0+*/ struct pt_regs regs; };
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Steven Rostedt
maintainerRe: [PATCH v10 03/36] x86: tracing: Add ftrace_regs definition in the header
Quoted 12 lines · On Fri, 24 May 2024 10:37:54 +0900 Masami Hiramatsu (Google)…
Quoted 11 lines
arch/x86/include/asm/ftrace.h static inline unsigned long ftrace_call_adjust(unsigned long addr) · line 43
Quoted 8 lines
#ifdef
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS
structftrace_regs{arch/x86/include/asm/ftrace.h +5
-0+/* +* On the x86_64, the ftrace_regs saves; +* rax, rcx, rdx, rdi, rsi, r8, r9, rbp, rip and rsp. +* Also orig_ax is used for passing direct trampoline address. +* x86_32 doesn't support ftrace_regs. Should add a comment that if
fregs->regs.csis set, then all of thept_regsis valid.
Quoted 2 lines
But what about rbx and r1*? Only
regs->csshould be care forpt_regs? Or, did you mean "theftrace_regsis valid"?
Yeah, on x86_64 ftrace_regs uses regs.cs to denote if it is valid or not:
static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *fregs)
{
/* Only when FL_SAVE_REGS is set, cs will be non zero */
if (!fregs->regs.cs)
return NULL;
return &fregs->regs;
}-- Steve
Masami Hiramatsu (Google)
maintainerPATCH v10 4/36 function_graph: Convert ret_stack to a series of longs
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
In order to make it possible to have multiple callbacks registered with the function_graph tracer, the retstack needs to be converted from an array of ftrace_ret_stack structures to an array of longs. This will allow to store the list of callbacks on the stack for the return side of the functions.
--- include/linux/sched.h | 2 - kernel/trace/fgraph.c | 124 ++++++++++++++++++++++++++++--------------------- 2 files changed, 71 insertions(+), 55 deletions(-)
include/linux/sched.h +1 -1
@@ -1396,7 +1396,7 @@ struct task_struct {int curr_ret_depth;/* Stack of return addresses for return function tracing: */ -struct ftrace_ret_stack *ret_stack; +unsigned long *ret_stack;/* Timestamp for last schedule: */ unsigned long long ftrace_timestamp;
kernel/trace/fgraph.c +70 -54
307 lines not on this page · raw message
Masami Hiramatsu (Google)
maintainerPATCH v10 5/36 fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Instead of using "ALIGN()", use BUILD_BUG_ON() as the structures should always be divisible by sizeof(long).
---
Changes in v7:
- Use DIV_ROUND_UP() to calculate FGRAPH_RET_INDEX
--- kernel/trace/fgraph.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
kernel/trace/fgraph.c +6 -3
@@ -26,10 +26,9 @@#endif #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) -#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define FGRAPH_RET_INDEX DIV_ROUND_UP(FGRAPH_RET_SIZE, sizeof(long))#define SHADOW_STACK_SIZE (PAGE_SIZE) -#define SHADOW_STACK_INDEX \ - (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long))/* Leave on a buffer at the end */ #define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX)
@@ -91,6 +90,8 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func,if (!current->ret_stack)return -EBUSY ; +BUILD_BUG_ON (SHADOW_STACK_SIZE %sizeof (long )); +/* * We must make sure the ret_stack is tested before we read * anything else.
@@ -325,6 +326,8 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) {int index = task->curr_ret_stack; +BUILD_BUG_ON (FGRAPH_RET_SIZE %sizeof (long )); + index -=FGRAPH_RET_INDEX * (idx +1 );if (index <0 )return NULL ;
Masami Hiramatsu (Google)
maintainerPATCH v10 6/36 function_graph: Add an array structure that will allow multiple callbacks
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Add an array structure that will eventually allow the function graph tracer to have up to 16 simultaneous callbacks attached. It's an array of 16 fgraph_ops pointers, that is assigned when one is registered. On entry of a function the entry of the first item in the array is called, and if it returns zero, then the callback returns non zero if it wants the return callback to be called on exit of the function.
The array will simplify the process of having more than one callback attached to the same function, as its index into the array can be stored on the shadow stack. We need to only save the index, because this will allow the fgraph_ops to be freed before the function returns (which may happen if the function call schedule for a long time).
---
Changes in v2:
- Remove unneeded brace.
--- kernel/trace/fgraph.c | 114 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 81 insertions(+), 33 deletions(-)
kernel/trace/fgraph.c +81 -33
@@ -39,6 +39,11 @@DEFINE_STATIC_KEY_FALSE (kill_ftrace_graph);int ftrace_graph_active; +static int fgraph_array_cnt; +#define FGRAPH_ARRAY_SIZE 16 + +static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE ]; +/* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true;
@@ -62,6 +67,20 @@ int __weak ftrace_disable_ftrace_graph_caller(void) }#endif +int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace) +{ +return 0 ; +} + +static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace) +{ +} + +static struct fgraph_ops fgraph_stub = { + .entryfunc = ftrace_graph_entry_stub, + .retfunc = ftrace_graph_ret_stub, +}; +/** * ftrace_graph_stop - set to permanently disable function graph tracing *
@@ -159,7 +178,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, goto out;/* Only trace if the calling function expects to */ -if (!ftrace_graph_entry (&trace)) +if (!fgraph_array[0 ]->entryfunc (&trace)) goto out_ret;return 0 ;
@@ -274,7 +293,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval =fgraph_ret_regs_return_value (ret_regs);#endif trace.rettime =trace_clock_local (); -ftrace_graph_return (&trace); + fgraph_array[0 ]->retfunc (&trace);/* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of
@@ -410,11 +429,6 @@ void ftrace_graph_sleep_time_control(bool enable) fgraph_sleep_time = enable; } -int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace) -{ -return 0 ; -} -/* * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h
@@ -652,37 +666,54 @@ static int start_graph_tracing(void)int register_ftrace_graph (struct fgraph_ops *gops) {int ret =0 ; +int i;mutex_lock (&ftrace_lock); -/* we currently allow only one tracer registered at a time */ -if (ftrace_graph_active) { +if (!fgraph_array[0 ]) { +/* The array must always have real data on it */ +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) + fgraph_array[i] = &fgraph_stub; + } + +/* Look for an available spot */ +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) { +if (fgraph_array[i] == &fgraph_stub) +break ; + } +if (i >=FGRAPH_ARRAY_SIZE ) { ret = -EBUSY ; goto out; } -register_pm_notifier (&ftrace_suspend_notifier); + fgraph_array[i] = gops; +if (i +1 > fgraph_array_cnt) + fgraph_array_cnt = i +1 ; ftrace_graph_active++; - ret =start_graph_tracing (); -if (ret) { - ftrace_graph_active--; - goto out; - } - ftrace_graph_return = gops->retfunc; +if (ftrace_graph_active ==1 ) { +register_pm_notifier (&ftrace_suspend_notifier); + ret =start_graph_tracing (); +if (ret) { + ftrace_graph_active--; + goto out; + } + + ftrace_graph_return = gops->retfunc; -/* -* Update the indirect function to the entryfunc, and the -* function that gets called to the entry_test first. Then -* call the update fgraph entry function to determine if -* the entryfunc should be called directly or not. -*/ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; -update_function_graph_func (); +/* +* Update the indirect function to the entryfunc, and the +* function that gets called to the entry_test first. Then +* call the update fgraph entry function to determine if +* the entryfunc should be called directly or not. +*/ + __ftrace_graph_entry = gops->entryfunc; + ftrace_graph_entry = ftrace_graph_entry_test; +update_function_graph_func (); - ret =ftrace_startup (&graph_ops,FTRACE_START_FUNC_RET ); + ret =ftrace_startup (&graph_ops,FTRACE_START_FUNC_RET ); + } out:mutex_unlock (&ftrace_lock);return ret;
@@ -690,19 +721,36 @@ int register_ftrace_graph(struct fgraph_ops *gops)void unregister_ftrace_graph (struct fgraph_ops *gops) { +int i; +mutex_lock (&ftrace_lock);if (unlikely (!ftrace_graph_active)) goto out; - ftrace_graph_active--; - ftrace_graph_return = ftrace_stub_graph; - ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; -ftrace_shutdown (&graph_ops,FTRACE_STOP_FUNC_RET ); -unregister_pm_notifier (&ftrace_suspend_notifier); -unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL ); +for (i =0 ; i < fgraph_array_cnt; i++) +if (gops == fgraph_array[i]) +break ; +if (i >= fgraph_array_cnt) + goto out; + fgraph_array[i] = &fgraph_stub; +if (i +1 == fgraph_array_cnt) { +for (; i >=0 ; i--) +if (fgraph_array[i] != &fgraph_stub) +break ; + fgraph_array_cnt = i +1 ; + } + + ftrace_graph_active--; +if (!ftrace_graph_active) { + ftrace_graph_return = ftrace_stub_graph; + ftrace_graph_entry = ftrace_graph_entry_stub; + __ftrace_graph_entry = ftrace_graph_entry_stub; +ftrace_shutdown (&graph_ops,FTRACE_STOP_FUNC_RET ); +unregister_pm_notifier (&ftrace_suspend_notifier); +unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL ); + } out:mutex_unlock (&ftrace_lock); }
Masami Hiramatsu (Google)
maintainerPATCH v10 7/36 function_graph: Allow multiple users to attach to function graph
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Allow for multiple users to attach to function graph tracer at the same time. Only 16 simultaneous users can attach to the tracer. This is because there's an array that stores the pointers to the attached fgraph_ops. When a function being traced is entered, each of the ftrace_ops entryfunc is called and if it returns non zero, its index into the array will be added to the shadow stack.
On exit of the function being traced, the shadow stack will contain the indexes of the ftrace_ops on the array that want their retfunc to be called.
Because a function may sleep for a long time (if a task sleeps itself), the return of the function may be literally days later. If the ftrace_ops is removed, its place on the array is replaced with a ftrace_ops that contains the stub functions and that will be called when the function finally returns.
If another ftrace_ops is added that happens to get the same index into the array, its return function may be called. But that's actually the way things current work with the old function graph tracer. If one tracer is removed and another is added, the new one will get the return calls of the function traced by the previous one, thus this is not a regression. This can be fixed by adding a counter to each time the array item is updated and save that on the shadow stack as well, such that it won't be called if the index saved does not match the index on the array.
Note, being able to filter functions when both are called is not completely handled yet, but that shouldn't be too hard to manage.
---
Changes in v10:
- Rephrase all "index" on shadow stack to "offset" and some "ret_stack" to "frame".
- Macros are renamed;
FGRAPH_RET_SIZE to FGRAPH_FRAME_SIZE
FGRAPH_RET_INDEX to FGRAPH_FRAME_OFFSET FGRAPH_RET_INDEX_SIZE to FGRAPH_FRAME_OFFSET_SIZE FGRAPH_RET_INDEX_MASK to FGRAPH_FRAME_OFFSET_MASK SHADOW_STACK_INDEX to SHADOW_STACK_IN_WORD SHADOW_STACK_MAX_INDEX to SHADOW_STACK_MAX_OFFSET
- Remove unused FGRAPH_MAX_INDEX macro.
- Fix wrong explanation of the shadow stack entry.
Changes in v7:
- Fix max limitation check in ftrace_graph_push_return().
- Rewrite the shadow stack implementation using bitmap entry. This allows us to detect recursive call/tail call easier. (this implementation is moved from later patch in the series.
Changes in v2:
- Check return value of the ftrace_pop_return_trace() instead of 'ret' since 'ret' is set to the address of panic().
- Fix typo and make lines shorter than 76 chars in description.
--- include/linux/ftrace.h | 3 kernel/trace/fgraph.c | 378 +++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 310 insertions(+), 71 deletions(-)
include/linux/ftrace.h +2 -1
@@ -1066,6 +1066,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace);struct fgraph_ops {trace_func_graph_ent_t entryfunc;trace_func_graph_ret_t retfunc; +int idx; };/*
@@ -1100,7 +1101,7 @@ function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp);struct ftrace_ret_stack * -ftrace_graph_get_ret_stack (struct task_struct *task,int idx); +ftrace_graph_get_ret_stack (struct task_struct *task,int skip);unsigned long ftrace_graph_ret_addr (struct task_struct *task,int *idx,unsigned long ret,unsigned long *retp);
kernel/trace/fgraph.c +308 -70
579 lines not on this page · raw message
Steven Rostedt
maintainerRe: [PATCH v10 07/36] function_graph: Allow multiple users to attach to function graph
Quoted 10 lines · On Tue, 7 May 2024 23:09:22 +0900 "Masami Hiramatsu (Google)"…
kernel/trace/fgraph.c +6
-0@@ -109,6 +244,21 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func,if (!current->ret_stack)return -EBUSY ; +/* +* At first, check whether the previous fgraph callback is pushed by +* the fgraph on the same function entry. +* But if @func is the self tail-call function, we also need to ensure +* the ret_stack is not for the previous call by checking whether the +* bit of @fgraph_idx is set or not.
kernel/trace/fgraph.c ftrace_push_return_trace(unsigned long ret, unsigned long func, · line 262
Quoted 10 lines
kernel/trace/fgraph.c +9
-0+*/ + ret_stack =get_ret_stack (current, current->curr_ret_stack, &offset); +if (ret_stack && ret_stack->func == func && +get_fgraph_type (current, offset +FGRAPH_FRAME_OFFSET ) ==FGRAPH_TYPE_BITMAP && + !is_fgraph_index_set (current, offset +FGRAPH_FRAME_OFFSET , fgraph_idx)) +return offset +FGRAPH_FRAME_OFFSET ; + + val = (FGRAPH_TYPE_RESERVED <<FGRAPH_TYPE_SHIFT ) |FGRAPH_FRAME_OFFSET ; +BUILD_BUG_ON (SHADOW_STACK_SIZE %sizeof (long ));
I'm trying to figure out what the above is trying to do. This gets called once in function_graph_enter() (or function_graph_enter_ops()). What exactly are you trying to catch here?
Is it from this email:
https://lore.kernel.org/all/[email protected]/
As that's the last version before you added the above code.
But you also noticed it may not be needed, but triggered a crash without it in v3:
https://lore.kernel.org/all/[email protected]/
I removed this code in my version and it runs just fine. Perhaps there was another bug that this was hiding that you fixed in later versions?
-- Steve
Masami Hiramatsu (Google)
maintainerRe: [PATCH v10 07/36] function_graph: Allow multiple users to attach to function graph
Quoted 19 lines · On Fri, 24 May 2024 21:32:08 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 16 lines · On Tue, 7 May 2024 23:09:22 +0900 "Masami Hiramatsu (Google)"…
kernel/trace/fgraph.c +12
-0@@ -109,6 +244,21 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func,if (!current->ret_stack)return -EBUSY ; +/* +* At first, check whether the previous fgraph callback is pushed by +* the fgraph on the same function entry. +* But if @func is the self tail-call function, we also need to ensure +* the ret_stack is not for the previous call by checking whether the +* bit of @fgraph_idx is set or not. +*/ + ret_stack =get_ret_stack (current, current->curr_ret_stack, &offset); +if (ret_stack && ret_stack->func == func && +get_fgraph_type (current, offset +FGRAPH_FRAME_OFFSET ) ==FGRAPH_TYPE_BITMAP && + !is_fgraph_index_set (current, offset +FGRAPH_FRAME_OFFSET , fgraph_idx)) +return offset +FGRAPH_FRAME_OFFSET ;
Quoted 8 lines
kernel/trace/fgraph.c ftrace_push_return_trace(unsigned long ret, unsigned long func, · line 262
Quoted 4 lines
kernel/trace/fgraph.c +3
-0+ + val = (FGRAPH_TYPE_RESERVED <<FGRAPH_TYPE_SHIFT ) |FGRAPH_FRAME_OFFSET ; +BUILD_BUG_ON (SHADOW_STACK_SIZE %sizeof (long ));I'm trying to figure out what the above is trying to do. This gets called once in
function_graph_enter()(orfunction_graph_enter_ops()). What exactly are you trying to catch here?
Aah, good catch! This was originally for catching the self tail-call case with multiple fgraph callback on the same function, but it was my misread. In later patch ([12/36]), we introduced function_graph_enter_ops() so that we can skip checking hash table and directly pass the fgraph_ops to user callback. I thought this function_graph_enter_ops() is used even if multiple fgraph is set on the same function. In this case, we always need to check the stack can be reused(pushed by other fgraph_ops on the same function) or not. But as we discussed, the function_graph_enter_ops() is used only when only one fgraph is set on the function (if there are multiple fgraphs are set on the same function, use function_graph_enter() ), we are sure that ftrace_push_return_trace() is called only once on hooking the function entry. Thus we don't need to reuse it.
Quoted 7 lines
Is it from this email:
https://lore.kernel.org/all/[email protected]/
As that's the last version before you added the above code.
Quoted 7 lines
But you also noticed it may not be needed, but triggered a crash without it in v3:
https://lore.kernel.org/all/[email protected]/
I removed this code in my version and it runs just fine. Perhaps there was another bug that this was hiding that you fixed in later versions?
No problem. I think we can remove this block safely.
Thank you,
Quoted 3 lines
-- Steve
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Masami Hiramatsu (Google)
maintainerPATCH v10 8/36 function_graph: Remove logic around ftrace_graph_entry and return
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
The function pointers ftrace_graph_entry and ftrace_graph_return are no longer called via the function_graph tracer. Instead, an array structure is now used that will allow for multiple users of the function_graph infrastructure. The variables are still used by the architecture code for non dynamic ftrace configs, where a test is made against them to see if they point to the default stub function or not. This is how the static function tracing knows to call into the function graph tracer infrastructure or not.
Two new stub functions are made. entry_run() and return_run(). The ftrace_graph_entry and ftrace_graph_return are set to them respectively when the function graph tracer is enabled, and this will trigger the architecture specific function graph code to be executed.
This also requires checking the global_ops hash for all calls into the function_graph tracer.
---
Changes in v2:
- Fix typo and make lines shorter than 76 chars in the description.
- Remove unneeded return from return_run() function.
--- kernel/trace/fgraph.c | 67 +++++++++------------------------------- kernel/trace/ftrace.c | 2 - kernel/trace/ftrace_internal.h | 2 - 3 files changed, 15 insertions(+), 56 deletions(-)
kernel/trace/fgraph.c +15 -52
@@ -143,6 +143,17 @@ add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) t->ret_stack[offset] |= (bitmap <<FGRAPH_INDEX_SHIFT ); } +/* ftrace_graph_entry set to this to tell some archs to run function graph */ +static int entry_run (struct ftrace_graph_ent *trace) +{ +return 0 ; +} + +/* ftrace_graph_return set to this to tell some archs to run function graph */ +static void return_run (struct ftrace_graph_ret *trace) +{ +} +/* * @offset: The offset into @t->ret_stack to find the ret_stack entry * @frame_offset: Where to place the offset into @t->ret_stack of that entry
@@ -673,7 +684,6 @@ extern void ftrace_stub_graph(struct ftrace_graph_ret *);/* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; -static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub;/* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ static int alloc_retstack_tasklist (unsigned long **ret_stack_list)
@@ -756,46 +766,6 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, } } -static int ftrace_graph_entry_test (struct ftrace_graph_ent *trace) -{ -if (!ftrace_ops_test (&global_ops, trace->func,NULL )) -return 0 ; -return __ftrace_graph_entry (trace); -} - -/* -* The function graph tracer should only trace the functions defined -* by set_ftrace_filter and set_ftrace_notrace. If another function -* tracer ops is registered, the graph tracer requires testing the -* function against the global ops, and not just trace any function -* that any ftrace_ops registered. -*/ -void update_function_graph_func (void ) -{ -struct ftrace_ops *op; -bool do_test = false; - -/* -* The graph and global ops share the same set of functions -* to test. If any other ops is on the list, then -* the graph tracing needs to test if its the function -* it should call. -*/ -do_for_each_ftrace_op (op, ftrace_ops_list) { -if (op != &global_ops && op != &graph_ops && - op != &ftrace_list_end) { - do_test = true; -/* in double loop, break out with goto */ - goto out; - } - }while_for_each_ftrace_op (op); - out: -if (do_test) - ftrace_graph_entry = ftrace_graph_entry_test; -else - ftrace_graph_entry = __ftrace_graph_entry; -} - static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void
@@ -937,18 +907,12 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_active--; goto out; } - - ftrace_graph_return = gops->retfunc; -/* -* Update the indirect function to the entryfunc, and the -* function that gets called to the entry_test first. Then -* call the update fgraph entry function to determine if -* the entryfunc should be called directly or not. +* Some archs just test to see if these are not +* the default function */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; -update_function_graph_func (); + ftrace_graph_return = return_run; + ftrace_graph_entry = entry_run; ret =ftrace_startup (&graph_ops,FTRACE_START_FUNC_RET ); }
@@ -984,7 +948,6 @@ void unregister_ftrace_graph(struct fgraph_ops *gops)if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub;ftrace_shutdown (&graph_ops,FTRACE_STOP_FUNC_RET );unregister_pm_notifier (&ftrace_suspend_notifier);unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL );
kernel/trace/ftrace.c +0 -2
@@ -235,8 +235,6 @@ static void update_ftrace_function(void) func = ftrace_ops_list_func; } -update_function_graph_func (); -/* If there's no change, then do nothing more here */ if (ftrace_trace_function == func)return ;
kernel/trace/ftrace_internal.h +0 -2
@@ -42,10 +42,8 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs)#ifdef CONFIG_FUNCTION_GRAPH_TRACER extern int ftrace_graph_active; -void update_function_graph_func (void );#else /* !CONFIG_FUNCTION_GRAPH_TRACER */ # define ftrace_graph_active 0 -static inline void update_function_graph_func (void ) { }#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ #else /* !CONFIG_FUNCTION_TRACER */
Masami Hiramatsu (Google)
maintainerPATCH v10 9/36 ftrace/function_graph: Pass fgraph_ops to function graph callbacks
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Pass the fgraph_ops structure to the function graph callbacks. This will allow callbacks to add a descriptor to a fgraph_ops private field that wil be added in the future and use it for the callbacks. This will be useful when more than one callback can be registered to the function graph tracer.
---
Changes in v2:
- cleanup to set argument name on function prototype.
--- include/linux/ftrace.h | 10 +++++++--- kernel/trace/fgraph.c | 16 +++++++++------- kernel/trace/ftrace.c | 6 ++++-- kernel/trace/trace.h | 4 ++-- kernel/trace/trace_functions_graph.c | 11 +++++++---- kernel/trace/trace_irqsoff.c | 6 ++++-- kernel/trace/trace_sched_wakeup.c | 6 ++++-- kernel/trace/trace_selftest.c | 5 +++-- 8 files changed, 40 insertions(+), 24 deletions(-)
include/linux/ftrace.h +7 -3
@@ -1055,11 +1055,15 @@ struct ftrace_graph_ret {unsigned long long rettime; } __packed; +struct fgraph_ops ; +/* Type of the callback handlers for tracing function graph*/ -typedef void (*trace_func_graph_ret_t )(struct ftrace_graph_ret *);/* return */ -typedef int (*trace_func_graph_ent_t )(struct ftrace_graph_ent *);/* entry */ +typedef void (*trace_func_graph_ret_t )(struct ftrace_graph_ret *, +struct fgraph_ops *);/* return */ +typedef int (*trace_func_graph_ent_t )(struct ftrace_graph_ent *, +struct fgraph_ops *);/* entry */ -extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace); +extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace,struct fgraph_ops *gops);#ifdef CONFIG_FUNCTION_GRAPH_TRACER
kernel/trace/fgraph.c +9 -7
@@ -144,13 +144,13 @@ add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) }/* ftrace_graph_entry set to this to tell some archs to run function graph */ -static int entry_run (struct ftrace_graph_ent *trace) +static int entry_run (struct ftrace_graph_ent *trace,struct fgraph_ops *ops) {return 0 ; }/* ftrace_graph_return set to this to tell some archs to run function graph */ -static void return_run (struct ftrace_graph_ret *trace) +static void return_run (struct ftrace_graph_ret *trace,struct fgraph_ops *ops) { }
@@ -211,12 +211,14 @@ int __weak ftrace_disable_ftrace_graph_caller(void) }#endif -int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace) +int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {return 0 ; } -static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace) +static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) { }
@@ -377,7 +379,7 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (gops == &fgraph_stub)continue ; -if (gops->entryfunc (&trace)) +if (gops->entryfunc (&trace, gops)) bitmap |=BIT (i); }
@@ -525,7 +527,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regsif (gops == &fgraph_stub)continue ; - gops->retfunc (&trace); + gops->retfunc (&trace, gops); }/*
@@ -679,7 +681,7 @@ void ftrace_graph_sleep_time_control(bool enable)* Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h */ -extern void ftrace_stub_graph (struct ftrace_graph_ret *); +void ftrace_stub_graph (struct ftrace_graph_ret *trace,struct fgraph_ops *gops);/* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;
kernel/trace/ftrace.c +4 -2
@@ -815,7 +815,8 @@ void ftrace_graph_graph_time_control(bool enable) fgraph_graph_time = enable; } -static int profile_graph_entry (struct ftrace_graph_ent *trace) +static int profile_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct ftrace_ret_stack *ret_stack;
@@ -832,7 +833,8 @@ static int profile_graph_entry(struct ftrace_graph_ent *trace)return 1 ; } -static void profile_graph_return (struct ftrace_graph_ret *trace) +static void profile_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct ftrace_ret_stack *ret_stack;struct ftrace_profile_stat *stat;
kernel/trace/trace.h +2 -2
@@ -678,8 +678,8 @@ void trace_latency_header(struct seq_file *m);void trace_default_header (struct seq_file *m);void print_trace_header (struct seq_file *m,struct trace_iterator *iter); -void trace_graph_return (struct ftrace_graph_ret *trace); -int trace_graph_entry (struct ftrace_graph_ent *trace); +void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops); +int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops);void set_graph_array (struct trace_array *tr);void tracing_start_cmdline_record (void );
kernel/trace/trace_functions_graph.c +7 -4
@@ -129,7 +129,8 @@ static inline int ftrace_graph_ignore_irqs(void)return in_hardirq (); } -int trace_graph_entry (struct ftrace_graph_ent *trace) +int trace_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct trace_array *tr = graph_array;struct trace_array_cpu *data;
@@ -238,7 +239,8 @@ void __trace_graph_return(struct trace_array *tr,trace_buffer_unlock_commit_nostack (buffer, event); } -void trace_graph_return (struct ftrace_graph_ret *trace) +void trace_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct trace_array *tr = graph_array;struct trace_array_cpu *data;
@@ -275,7 +277,8 @@ void set_graph_array(struct trace_array *tr)smp_mb (); } -static void trace_graph_thresh_return (struct ftrace_graph_ret *trace) +static void trace_graph_thresh_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {ftrace_graph_addr_finish (trace);
@@ -288,7 +291,7 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) (trace->rettime - trace->calltime < tracing_thresh)) return; else -trace_graph_return (trace ); +trace_graph_return (trace ,gops ); }static struct fgraph_ops funcgraph_thresh_ops = {
kernel/trace/trace_irqsoff.c +4 -2
@@ -175,7 +175,8 @@ static int irqsoff_display_graph(struct trace_array *tr, int set)return start_irqsoff_tracer (irqsoff_trace, set); } -static int irqsoff_graph_entry (struct ftrace_graph_ent *trace) +static int irqsoff_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct trace_array *tr = irqsoff_trace;struct trace_array_cpu *data;
@@ -205,7 +206,8 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace)return ret; } -static void irqsoff_graph_return (struct ftrace_graph_ret *trace) +static void irqsoff_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct trace_array *tr = irqsoff_trace;struct trace_array_cpu *data;
kernel/trace/trace_sched_wakeup.c +4 -2
@@ -112,7 +112,8 @@ static int wakeup_display_graph(struct trace_array *tr, int set)return start_func_tracer (tr, set); } -static int wakeup_graph_entry (struct ftrace_graph_ent *trace) +static int wakeup_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct trace_array *tr = wakeup_trace;struct trace_array_cpu *data;
@@ -141,7 +142,8 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace)return ret; } -static void wakeup_graph_return (struct ftrace_graph_ret *trace) +static void wakeup_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct trace_array *tr = wakeup_trace;struct trace_array_cpu *data;
kernel/trace/trace_selftest.c +3 -2
@@ -762,7 +762,8 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)static unsigned int graph_hang_thresh;/* Wrap the real function entry probe to avoid possible hanging */ -static int trace_graph_entry_watchdog (struct ftrace_graph_ent *trace) +static int trace_graph_entry_watchdog (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {/* This is harmlessly racy, we want to approximately detect a hang */ if (unlikely (++graph_hang_thresh >GRAPH_MAX_FUNC_TEST )) {
@@ -776,7 +777,7 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace)return 0 ; } -return trace_graph_entry (trace); +return trace_graph_entry (trace, gops); }static struct fgraph_ops fgraph_ops__initdata = {
Masami Hiramatsu (Google)
maintainerPATCH v10 10/36 ftrace: Allow function_graph tracer to be enabled in instances
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Now that function graph tracing can handle more than one user, allow it to be enabled in the ftrace instances. Note, the filtering of the functions is still joined by the top level set_ftrace_filter and friends, as well as the graph and nograph files.
---
Changes in v2:
- Fix to remove set_graph_array() completely.
--- include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 1 + kernel/trace/trace.h | 13 ++++++- kernel/trace/trace_functions.c | 8 ++++ kernel/trace/trace_functions_graph.c | 65 +++++++++++++++++++++------------- kernel/trace/trace_selftest.c | 4 +- 6 files changed, 64 insertions(+), 28 deletions(-)
include/linux/ftrace.h +1 -0
@@ -1070,6 +1070,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraphstruct fgraph_ops {trace_func_graph_ent_t entryfunc;trace_func_graph_ret_t retfunc; +void *private;int idx; };
kernel/trace/ftrace.c +1 -0
@@ -7326,6 +7326,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr;ftrace_init_trace_array (tr); +init_array_fgraph_ops (tr); }void ftrace_init_array_ops (struct trace_array *tr,ftrace_func_t func)
kernel/trace/trace.h +12 -1
@@ -396,6 +396,9 @@ struct trace_array {struct ftrace_ops *ops;struct trace_pid_list __rcu *function_pids;struct trace_pid_list __rcu *function_no_pids; +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +struct fgraph_ops *gops; +#endif #ifdef CONFIG_DYNAMIC_FTRACE /* All of these are protected by the ftrace_lock */ struct list_head func_probes;
@@ -680,7 +683,6 @@ void print_trace_header(struct seq_file *m, struct trace_iterator *iter);void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops);int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops); -void set_graph_array (struct trace_array *tr);void tracing_start_cmdline_record (void );void tracing_stop_cmdline_record (void );
@@ -891,6 +893,9 @@ extern int __trace_graph_entry(struct trace_array *tr,extern void __trace_graph_return (struct trace_array *tr,struct ftrace_graph_ret *trace,unsigned int trace_ctx); +extern void init_array_fgraph_ops (struct trace_array *tr); +extern int allocate_fgraph_ops (struct trace_array *tr); +extern void free_fgraph_ops (struct trace_array *tr);#ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash;
@@ -1003,6 +1008,12 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags) {return TRACE_TYPE_UNHANDLED ; } +static inline void init_array_fgraph_ops (struct trace_array *tr) { } +static inline int allocate_fgraph_ops (struct trace_array *tr) +{ +return 0 ; +} +static inline void free_fgraph_ops (struct trace_array *tr) { }#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids;
kernel/trace/trace_functions.c +8 -0
@@ -80,6 +80,7 @@ void ftrace_free_ftrace_ops(struct trace_array *tr)int ftrace_create_function_files (struct trace_array *tr,struct dentry *parent) { +int ret;/* * The top level array uses the "global_ops", and the files are * created on boot up.
@@ -90,6 +91,12 @@ int ftrace_create_function_files(struct trace_array *tr,if (!tr->ops)return -EINVAL ; + ret =allocate_fgraph_ops (tr); +if (ret) { +kfree (tr->ops); +return ret; + } +ftrace_create_filter_files (tr->ops, parent);return 0 ;
@@ -99,6 +106,7 @@ void ftrace_destroy_function_files(struct trace_array *tr) {ftrace_destroy_filter_files (tr->ops);ftrace_free_ftrace_ops (tr); +free_fgraph_ops (tr); }static ftrace_func_t select_trace_function (u32 flags_val)
kernel/trace/trace_functions_graph.c +40 -25
118 lines not on this page · raw message
kernel/trace/trace_selftest.c +2 -2
@@ -813,7 +813,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,* to detect and recover from possible hangs */ tracing_reset_online_cpus (&tr->array_buffer); -set_graph_array (tr); + fgraph_ops.private = tr; ret =register_ftrace_graph (&fgraph_ops);if (ret) {warn_failed_init_tracer (trace, ret);
@@ -856,7 +856,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,cond_resched ();tracing_reset_online_cpus (&tr->array_buffer); -set_graph_array (tr); + fgraph_ops.private = tr;/* * Some archs *cough*PowerPC*cough* add characters to the
Masami Hiramatsu (Google)
maintainerPATCH v10 11/36 ftrace: Allow ftrace startup flags exist without dynamic ftrace
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Some of the flags for ftrace_startup() may be exposed even when CONFIG_DYNAMIC_FTRACE is not configured in. This is fine as the difference between dynamic ftrace and static ftrace is done within the internals of ftrace itself. No need to have use cases fail to compile because dynamic ftrace is disabled.
This change is needed to move some of the logic of what is passed to ftrace_startup() out of the parameters of ftrace_startup().
--- include/linux/ftrace.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
include/linux/ftrace.h +9 -9
@@ -538,6 +538,15 @@ static inline void stack_tracer_disable(void) { }static inline void stack_tracer_enable (void ) { }#endif +enum { +FTRACE_UPDATE_CALLS = (1 <<0 ), +FTRACE_DISABLE_CALLS = (1 <<1 ), +FTRACE_UPDATE_TRACE_FUNC = (1 <<2 ), +FTRACE_START_FUNC_RET = (1 <<3 ), +FTRACE_STOP_FUNC_RET = (1 <<4 ), +FTRACE_MAY_SLEEP = (1 <<5 ), +}; +#ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare (void );
@@ -632,15 +641,6 @@ void ftrace_set_global_notrace(unsigned char *buf, int len, int reset);void ftrace_free_filter (struct ftrace_ops *ops);void ftrace_ops_set_global_filter (struct ftrace_ops *ops); -enum { -FTRACE_UPDATE_CALLS = (1 <<0 ), -FTRACE_DISABLE_CALLS = (1 <<1 ), -FTRACE_UPDATE_TRACE_FUNC = (1 <<2 ), -FTRACE_START_FUNC_RET = (1 <<3 ), -FTRACE_STOP_FUNC_RET = (1 <<4 ), -FTRACE_MAY_SLEEP = (1 <<5 ), -}; -/* * The FTRACE_UPDATE_* enum is used to pass information back * from the ftrace_update_record() and ftrace_test_record()
Masami Hiramatsu (Google)
maintainerPATCH v10 12/36 function_graph: Have the instances use their own ftrace_ops for filtering
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Allow for instances to have their own ftrace_ops part of the fgraph_ops that makes the funtion_graph tracer filter on the set_ftrace_filter file of the instance and not the top instance.
Note that this also requires to update ftrace_graph_func() to call new function_graph_enter_ops() instead of function_graph_enter() so that it avoid pushing on shadow stack multiple times on the same function.
---
Changes in v10:
- Use "offset" for shadow stack instead of "index".
Changes in v9:
- Fix to clear fgraph_array correctly when ftrace_startup() fails.
- Return -ENOSPC if fgraph_array is full.
Changes in v8:
- Fix a compilation error in loongarch implementation.
- Update riscv implementation of ftrace_graph_func().
Changes in v7:
- Move FGRAPH_TYPE_BITMAP type implementation to earlier patch ( which implements FGRAPH_TYPE_ARRAY) so that it does not need to replace the FGRAPH_TYPE_ARRAY type.
- Update loongarch and powerpc implementation of ftrace_graph_func().
- Update description.
Changes in v6:
- Fix to check whether the fgraph_ops is already unregistered in function_graph_enter_ops().
- Fix stack unwinder error on arm64 because of passing wrong value as retp. Thanks Mark!
Changes in v4:
- Simplify get_ret_stack() sanity check and use WARN_ON_ONCE() for obviously wrong value.
- Do not check ret == return_to_handler but always read the previous ret_stack in ftrace_push_return_trace() to check it is reusable.
- Set the bit 0 of the bitmap entry always in function_graph_enter() because it uses bit 0 to check re-usability.
- Fix to ensure the ret_stack entry is bitmap type when checking the bitmap.
Changes in v3:
- Pass current fgraph_ops to the new entry handler (function_graph_enter_ops) if fgraph use ftrace.
- Add fgraph_ops::idx in this patch.
- Replace the array type with the bitmap type so that it can record which fgraph is called.
- Fix some helper function to use passed task_struct instead of current.
- Reduce the ret-index size to 1024 words.
- Make the ret-index directly points the ret_stack.
- Fix ftrace_graph_ret_addr() to handle tail-call case correctly.
Changes in v2:
- Use ftrace_graph_func and FTRACE_OPS_GRAPH_STUB instead of ftrace_stub and FTRACE_OPS_FL_STUB for new ftrace based fgraph.
--- arch/arm64/kernel/ftrace.c | 21 +++++- arch/loongarch/kernel/ftrace_dyn.c | 15 ++++ arch/powerpc/kernel/trace/ftrace.c | 3 + arch/riscv/kernel/ftrace.c | 15 ++++ arch/x86/kernel/ftrace.c | 19 +++++ include/linux/ftrace.h | 6 ++ kernel/trace/fgraph.c | 125 +++++++++++++++++++++++++--------- kernel/trace/ftrace.c | 4 + kernel/trace/trace.h | 16 ++-- kernel/trace/trace_functions.c | 2 - kernel/trace/trace_functions_graph.c | 8 ++ 11 files changed, 183 insertions(+), 51 deletions(-)
arch/arm64/kernel/ftrace.c +20 -1
@@ -481,7 +481,26 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent,void ftrace_graph_func (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs) { -prepare_ftrace_return (ip, &fregs->lr, fregs->fp); + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); +unsigned long frame_pointer = fregs->fp; +unsigned long *parent = &fregs->lr; +int bit; + +if (unlikely (ftrace_graph_is_dead ())) +return ; + +if (unlikely (atomic_read (¤t->tracing_graph_pause))) +return ; + + bit =ftrace_test_recursion_trylock (ip, *parent); +if (bit <0 ) +return ; + +if (!function_graph_enter_ops (*parent, ip, frame_pointer, + (void *)frame_pointer, gops)) + *parent = (unsigned long )&return_to_handler; + +ftrace_test_recursion_unlock (bit); }#else /*
arch/loongarch/kernel/ftrace_dyn.c +13 -2
@@ -241,10 +241,21 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent)void ftrace_graph_func (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs) { + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); +unsigned long return_hooker = (unsigned long )&return_to_handler;struct pt_regs *regs = &fregs->regs; -unsigned long *parent = (unsigned long *)®s->regs[1 ]; +unsigned long *parent; +unsigned long old; + + parent = (unsigned long *)®s->regs[1 ]; -prepare_ftrace_return (ip, (unsigned long *)parent); +if (unlikely (atomic_read (¤t->tracing_graph_pause))) +return ; + + old = *parent; + +if (!function_graph_enter_ops (old, ip,0 , parent, gops)) + *parent = return_hooker; }#else static int ftrace_modify_graph_caller (bool enable)
arch/powerpc/kernel/trace/ftrace.c +2 -1
@@ -421,6 +421,7 @@ int __init ftrace_dyn_arch_init(void)void ftrace_graph_func (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs) { + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops);unsigned long sp = fregs->regs.gpr[1 ];int bit;
@@ -434,7 +435,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,if (bit <0 ) goto out; -if (!function_graph_enter (parent_ip, ip,0 , (unsigned long *)sp)) +if (!function_graph_enter_ops (parent_ip, ip,0 , (unsigned long *)sp, gops)) parent_ip =ppc_function_entry (return_to_handler);ftrace_test_recursion_unlock (bit);
arch/riscv/kernel/ftrace.c +14 -1
@@ -182,10 +182,23 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr,void ftrace_graph_func (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs) { + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); +unsigned long return_hooker = (unsigned long )&return_to_handler;struct pt_regs *regs =arch_ftrace_get_regs (fregs);unsigned long *parent = (unsigned long *)®s->ra; +unsigned long old; + +if (unlikely (atomic_read (¤t->tracing_graph_pause))) +return ; + +/* +* We don't suffer access faults, so no extra fault-recovery assembly +* is needed here. +*/ + old = *parent; -prepare_ftrace_return (parent, ip,frame_pointer (regs)); +if (!function_graph_enter_ops (old, ip,frame_pointer (regs), parent, gops)) + *parent = return_hooker; }#else /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */ extern void ftrace_graph_call (void );
arch/x86/kernel/ftrace.c +17 -2
@@ -658,9 +658,24 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { struct pt_regs *regs = &fregs->regs; -unsigned long *stack = (unsigned long *)kernel_stack_pointer (regs); +unsigned long *parent = (unsigned long *)kernel_stack_pointer (regs); + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); +int bit; + +if (unlikely (ftrace_graph_is_dead ())) +return ; + +if (unlikely (atomic_read (¤t->tracing_graph_pause))) +return ; -prepare_ftrace_return (ip, (unsigned long *)stack,0 ); + bit =ftrace_test_recursion_trylock (ip, *parent); +if (bit <0 ) +return ; + +if (!function_graph_enter_ops (*parent, ip,0 , parent, gops)) + *parent = (unsigned long )&return_to_handler; + +ftrace_test_recursion_unlock (bit); }#endif
include/linux/ftrace.h +6 -0
@@ -1070,6 +1070,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraphstruct fgraph_ops {trace_func_graph_ent_t entryfunc;trace_func_graph_ret_t retfunc; +struct ftrace_ops ops;/* for the hash lists */ void *private;int idx; };
@@ -1105,6 +1106,11 @@ extern int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp); +extern int +function_graph_enter_ops (unsigned long ret,unsigned long func, +unsigned long frame_pointer,unsigned long *retp, +struct fgraph_ops *gops); +struct ftrace_ret_stack *ftrace_graph_get_ret_stack (struct task_struct *task,int skip);
kernel/trace/fgraph.c +92 -33
203 lines not on this page · raw message
kernel/trace/ftrace.c +3 -1
@@ -3017,6 +3017,8 @@ int ftrace_startup(struct ftrace_ops *ops, int command)if (unlikely (ftrace_disabled))return -ENODEV ; +ftrace_ops_init (ops); + ret =__register_ftrace_function (ops);if (ret)return ret;
@@ -7326,7 +7328,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr;ftrace_init_trace_array (tr); -init_array_fgraph_ops (tr); +init_array_fgraph_ops (tr, tr->ops); }void ftrace_init_array_ops (struct trace_array *tr,ftrace_func_t func)
kernel/trace/trace.h +9 -7
@@ -893,8 +893,8 @@ extern int __trace_graph_entry(struct trace_array *tr,extern void __trace_graph_return (struct trace_array *tr,struct ftrace_graph_ret *trace,unsigned int trace_ctx); -extern void init_array_fgraph_ops (struct trace_array *tr); -extern int allocate_fgraph_ops (struct trace_array *tr); +extern void init_array_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops); +extern int allocate_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops);extern void free_fgraph_ops (struct trace_array *tr);#ifdef CONFIG_DYNAMIC_FTRACE
@@ -977,6 +977,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr)preempt_enable_notrace (); return ret; } +#else static inline int ftrace_graph_addr (struct ftrace_graph_ent *trace) {
@@ -1002,18 +1003,19 @@ static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) (fgraph_max_depth && trace->depth >= fgraph_max_depth); } +void fgraph_init_ops (struct ftrace_ops *dst_ops, +struct ftrace_ops *src_ops); +#else /* CONFIG_FUNCTION_GRAPH_TRACER */ static inline enum print_line_t print_graph_function_flags (struct trace_iterator *iter,u32 flags) {return TRACE_TYPE_UNHANDLED ; } -static inline void init_array_fgraph_ops (struct trace_array *tr) { } -static inline int allocate_fgraph_ops (struct trace_array *tr) -{ -return 0 ; -}static inline void free_fgraph_ops (struct trace_array *tr) { } +/* ftrace_ops may not be defined */ +#define init_array_fgraph_ops (tr, ops) do { } while (0) +#define allocate_fgraph_ops (tr, ops) ({ 0; })#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids;
kernel/trace/trace_functions.c +1 -1
@@ -91,7 +91,7 @@ int ftrace_create_function_files(struct trace_array *tr,if (!tr->ops)return -EINVAL ; - ret =allocate_fgraph_ops (tr); + ret =allocate_fgraph_ops (tr, tr->ops);if (ret) {kfree (tr->ops);return ret;
kernel/trace/trace_functions_graph.c +6 -2
@@ -288,7 +288,7 @@ static struct fgraph_ops funcgraph_ops = { .retfunc = &trace_graph_return, }; -int allocate_fgraph_ops (struct trace_array *tr) +int allocate_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops) {struct fgraph_ops *gops;
@@ -301,6 +301,9 @@ int allocate_fgraph_ops(struct trace_array *tr) tr->gops = gops; gops->private = tr; + +fgraph_init_ops (&gops->ops, ops); +return 0 ; }
@@ -309,10 +312,11 @@ void free_fgraph_ops(struct trace_array *tr)kfree (tr->gops); } -__init void init_array_fgraph_ops (struct trace_array *tr) +__init void init_array_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops) { tr->gops = &funcgraph_ops; funcgraph_ops.private = tr; +fgraph_init_ops (&tr->gops->ops, ops); }static int graph_trace_init (struct trace_array *tr)
Masami Hiramatsu (Google)
maintainerPATCH v10 13/36 function_graph: Use a simple LRU for fgraph_array index number
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Since the fgraph_array index is used for the bitmap on the shadow stack, it may leave some entries after a function_graph instance is removed. Thus if another instance reuses the fgraph_array index soon after releasing it, the fgraph may confuse to call the newer callback for the entries which are pushed by the older instance.
To avoid reusing the fgraph_array index soon after releasing, introduce a simple LRU table for managing the index number. This will reduce the possibility of this confusion.
---
Changes in v8:
- Add a WARN_ON_ONCE() if fgraph_lru_table[] is broken when releasing index, and remove WARN_ON_ONCE() from unregister_ftrace_graph()
- Fix to release allocated index if register_ftrace_graph() fails.
- Add comments and code cleanup.
Changes in v5:
- Fix the underflow bug in fgraph_lru_release_index() and return 0 if the release is succeded.
Changes in v4:
- Newly added.
--- kernel/trace/fgraph.c | 71 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 50 insertions(+), 21 deletions(-)
kernel/trace/fgraph.c +50 -21
@@ -97,10 +97,48 @@ enum {DEFINE_STATIC_KEY_FALSE (kill_ftrace_graph);int ftrace_graph_active; -static int fgraph_array_cnt; -static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE ]; +/* LRU index table for fgraph_array */ +static int fgraph_lru_table[FGRAPH_ARRAY_SIZE ]; +static int fgraph_lru_next; +static int fgraph_lru_last; + +/* Initialize fgraph_lru_table with unused index */ +static void fgraph_lru_init (void ) +{ +int i; + +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) + fgraph_lru_table[i] = i; +} + +/* Release the used index to the LRU table */ +static int fgraph_lru_release_index (int idx) +{ +if (idx <0 || idx >=FGRAPH_ARRAY_SIZE || +WARN_ON_ONCE (fgraph_lru_table[fgraph_lru_last] !=-1 )) +return -1 ; + + fgraph_lru_table[fgraph_lru_last] = idx; + fgraph_lru_last = (fgraph_lru_last +1 ) %FGRAPH_ARRAY_SIZE ; +return 0 ; +} + +/* Allocate a new index from LRU table */ +static int fgraph_lru_alloc_index (void ) +{ +int idx = fgraph_lru_table[fgraph_lru_next]; + +/* No id is available */ +if (idx ==-1 ) +return -1 ; + + fgraph_lru_table[fgraph_lru_next] =-1 ; + fgraph_lru_next = (fgraph_lru_next +1 ) %FGRAPH_ARRAY_SIZE ; +return idx; +} +static inline int get_frame_offset (struct task_struct *t,int offset) {return t->ret_stack[offset] &FGRAPH_FRAME_OFFSET_MASK ;
@@ -365,7 +403,7 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (offset <0 ) goto out; -for (i =0 ; i < fgraph_array_cnt; i++) { +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) {struct fgraph_ops *gops = fgraph_array[i];if (gops == &fgraph_stub)
@@ -917,7 +955,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) {int command =0 ;int ret =0 ; -int i; +int i =-1 ;mutex_lock (&ftrace_lock);
@@ -933,21 +971,16 @@ int register_ftrace_graph(struct fgraph_ops *gops)/* The array must always have real data on it */ for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) fgraph_array[i] = &fgraph_stub; +fgraph_lru_init (); } -/* Look for an available spot */ -for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) { -if (fgraph_array[i] == &fgraph_stub) -break ; - } -if (i >=FGRAPH_ARRAY_SIZE ) { + i =fgraph_lru_alloc_index (); +if (i <0 ||WARN_ON_ONCE (fgraph_array[i] != &fgraph_stub)) { ret = -ENOSPC ; goto out; } fgraph_array[i] = gops; -if (i +1 > fgraph_array_cnt) - fgraph_array_cnt = i +1 ; gops->idx = i; ftrace_graph_active++;
@@ -971,6 +1004,7 @@ int register_ftrace_graph(struct fgraph_ops *gops)if (ret) { fgraph_array[i] = &fgraph_stub; ftrace_graph_active--; +fgraph_lru_release_index (i); } out:mutex_unlock (&ftrace_lock);
@@ -980,25 +1014,20 @@ int register_ftrace_graph(struct fgraph_ops *gops)void unregister_ftrace_graph (struct fgraph_ops *gops) {int command =0 ; -int i;mutex_lock (&ftrace_lock);if (unlikely (!ftrace_graph_active)) goto out; -if (unlikely (gops->idx <0 || gops->idx >= fgraph_array_cnt)) +if (unlikely (gops->idx <0 || gops->idx >=FGRAPH_ARRAY_SIZE || + fgraph_array[gops->idx] != gops)) goto out; -WARN_ON_ONCE (fgraph_array[gops->idx] != gops); +if (fgraph_lru_release_index (gops->idx) <0 ) + goto out; fgraph_array[gops->idx] = &fgraph_stub; -if (gops->idx +1 == fgraph_array_cnt) { - i = gops->idx; -while (i >=0 && fgraph_array[i] == &fgraph_stub) - i--; - fgraph_array_cnt = i +1 ; - } ftrace_graph_active--;
Masami Hiramatsu (Google)
maintainerPATCH v10 14/36 function_graph: Add "task variables" per task for fgraph_ops
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Add a "task variables" array on the tasks shadow ret_stack that is the size of longs for each possible registered fgraph_ops. That's a total of 16, taking up 8 * 16 = 128 bytes (out of a page size 4k).
This will allow for fgraph_ops to do specific features on a per task basis having a way to maintain state for each task.
---
Changes in v10:
- Explain where the task vars is placed in shadow stack.
Changes in v3:
- Move fgraph_ops::idx to previous patch in the series.
Changes in v2:
- Make description lines shorter than 76 chars.
--- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 1 deletion(-)
include/linux/ftrace.h +1 -0
@@ -1116,6 +1116,7 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int skip);unsigned long ftrace_graph_ret_addr (struct task_struct *task,int *idx,unsigned long ret,unsigned long *retp); +unsigned long *fgraph_get_task_var (struct fgraph_ops *gops);/* * Sometimes we don't want to trace a function with the function
kernel/trace/fgraph.c +73 -1
119 lines not on this page · raw message
Masami Hiramatsu (Google)
maintainerPATCH v10 15/36 function_graph: Move set_graph_function tests to shadow stack global var
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
The use of the task->trace_recursion for the logic used for the set_graph_funnction was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead.
--- include/linux/trace_recursion.h | 5 +---- kernel/trace/trace.h | 32 +++++++++++++++++++++----------- kernel/trace/trace_functions_graph.c | 6 +++--- kernel/trace/trace_irqsoff.c | 4 ++-- kernel/trace/trace_sched_wakeup.c | 4 ++-- 5 files changed, 29 insertions(+), 22 deletions(-)
include/linux/trace_recursion.h +1 -4
@@ -44,9 +44,6 @@ enum {*/ TRACE_IRQ_BIT, -/* Set if the function is in the set_graph_function file */ - TRACE_GRAPH_BIT, -/* * In the very unlikely case that an interrupt came in * at a start of graph tracing, and we want to trace
@@ -60,7 +57,7 @@ enum {* that preempted a softirq start of a function that * preempted normal context!!!! Luckily, it can't be * greater than 3, so the next two bits are a mask -* of what the depth is when we set TRACE_GRAPH_BIT +* of what the depth is when we set TRACE_GRAPH_FL */ TRACE_GRAPH_DEPTH_START_BIT,
kernel/trace/trace.h +21 -11
@@ -897,11 +897,16 @@ extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *opsextern int allocate_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops);extern void free_fgraph_ops (struct trace_array *tr); +enum { +TRACE_GRAPH_FL =1 , +}; +#ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash;extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; -static inline int ftrace_graph_addr (struct ftrace_graph_ent *trace) +static inline int +ftrace_graph_addr (unsigned long *task_var,struct ftrace_graph_ent *trace) {unsigned long addr = trace->func;int ret =0 ;
@@ -923,12 +928,11 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) }if (ftrace_lookup_ip (hash, addr)) { -/* * This needs to be cleared on the return functions * when the depth is zero. */ -trace_recursion_set (TRACE_GRAPH_BIT ); + *task_var |=TRACE_GRAPH_FL ;trace_recursion_set_depth (trace->depth);/*
@@ -948,11 +952,14 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace)return ret; } -static inline void ftrace_graph_addr_finish (struct ftrace_graph_ret *trace) +static inline void +ftrace_graph_addr_finish (struct fgraph_ops *gops,struct ftrace_graph_ret *trace) { -if (trace_recursion_test (TRACE_GRAPH_BIT ) && +unsigned long *task_var =fgraph_get_task_var (gops); + +if ((*task_var &TRACE_GRAPH_FL ) && trace->depth ==trace_recursion_depth ()) -trace_recursion_clear (TRACE_GRAPH_BIT ); + *task_var &= ~TRACE_GRAPH_FL ; }static inline int ftrace_graph_notrace_addr (unsigned long addr)
@@ -979,7 +986,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) }#else -static inline int ftrace_graph_addr (struct ftrace_graph_ent *trace) +static inline int ftrace_graph_addr (unsigned long *task_var,struct ftrace_graph_ent *trace) {return 1 ; }
@@ -988,17 +995,20 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) {return 0 ; } -static inline void ftrace_graph_addr_finish (struct ftrace_graph_ret *trace) +static inline void ftrace_graph_addr_finish (struct fgraph_ops *gops,struct ftrace_graph_ret *trace) { }#endif /* CONFIG_DYNAMIC_FTRACE */ extern unsigned int fgraph_max_depth; -static inline bool ftrace_graph_ignore_func (struct ftrace_graph_ent *trace) +static inline bool +ftrace_graph_ignore_func (struct fgraph_ops *gops,struct ftrace_graph_ent *trace) { +unsigned long *task_var =fgraph_get_task_var (gops); +/* trace it when it is-nested-in or is a function enabled. */ -return !(trace_recursion_test (TRACE_GRAPH_BIT ) || -ftrace_graph_addr (trace)) || +return !((*task_var &TRACE_GRAPH_FL ) || +ftrace_graph_addr (task_var, trace)) || (trace->depth <0 ) || (fgraph_max_depth && trace->depth >= fgraph_max_depth); }
kernel/trace/trace_functions_graph.c +3 -3
@@ -160,7 +160,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,if (!ftrace_trace_task (tr))return 0 ; -if (ftrace_graph_ignore_func (trace)) +if (ftrace_graph_ignore_func (gops, trace))return 0 ;if (ftrace_graph_ignore_irqs ())
@@ -247,7 +247,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace,long disabled;int cpu; -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) {trace_recursion_clear (TRACE_GRAPH_NOTRACE_BIT );
@@ -269,7 +269,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace,static void trace_graph_thresh_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops) { -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) {trace_recursion_clear (TRACE_GRAPH_NOTRACE_BIT );
kernel/trace/trace_irqsoff.c +2 -2
@@ -184,7 +184,7 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace,unsigned int trace_ctx;int ret; -if (ftrace_graph_ignore_func (trace)) +if (ftrace_graph_ignore_func (gops, trace))return 0 ;/* * Do not trace a function if it's filtered by set_graph_notrace.
@@ -214,7 +214,7 @@ static void irqsoff_graph_return(struct ftrace_graph_ret *trace,unsigned long flags;unsigned int trace_ctx; -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (!func_prolog_dec (tr, &data, &flags))return ;
kernel/trace/trace_sched_wakeup.c +2 -2
@@ -120,7 +120,7 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace,unsigned int trace_ctx;int ret =0 ; -if (ftrace_graph_ignore_func (trace)) +if (ftrace_graph_ignore_func (gops, trace))return 0 ;/* * Do not trace a function if it's filtered by set_graph_notrace.
@@ -149,7 +149,7 @@ static void wakeup_graph_return(struct ftrace_graph_ret *trace,struct trace_array_cpu *data;unsigned int trace_ctx; -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (!func_prolog_preempt_disable (tr, &data, &trace_ctx))return ;
Masami Hiramatsu (Google)
maintainerPATCH v10 16/36 function_graph: Move graph depth stored data to shadow stack global var
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
The use of the task->trace_recursion for the logic used for the function graph depth was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead.
--- include/linux/trace_recursion.h | 29 ----------------------------- kernel/trace/trace.h | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 31 deletions(-)
include/linux/trace_recursion.h +0 -29
@@ -44,25 +44,6 @@ enum {*/ TRACE_IRQ_BIT, -/* -* In the very unlikely case that an interrupt came in -* at a start of graph tracing, and we want to trace -* the function in that interrupt, the depth can be greater -* than zero, because of the preempted start of a previous -* trace. In an even more unlikely case, depth could be 2 -* if a softirq interrupted the start of graph tracing, -* followed by an interrupt preempting a start of graph -* tracing in the softirq, and depth can even be 3 -* if an NMI came in at the start of an interrupt function -* that preempted a softirq start of a function that -* preempted normal context!!!! Luckily, it can't be -* greater than 3, so the next two bits are a mask -* of what the depth is when we set TRACE_GRAPH_FL -*/ - -TRACE_GRAPH_DEPTH_START_BIT , -TRACE_GRAPH_DEPTH_END_BIT , -/* * To implement set_graph_notrace, if this bit is set, we ignore * function graph tracing of called functions, until the return
@@ -78,16 +59,6 @@ enum {#define trace_recursion_clear (bit) do { (current)->trace_recursion &= ~(1<<(bit)); } while (0)#define trace_recursion_test (bit) ((current)->trace_recursion & (1<<(bit))) -#define trace_recursion_depth () \ - (((current)->trace_recursion >> TRACE_GRAPH_DEPTH_START_BIT) & 3) -#define trace_recursion_set_depth (depth) \ - do { \ - current->trace_recursion &= \ - ~(3 << TRACE_GRAPH_DEPTH_START_BIT); \ - current->trace_recursion |= \ - ((depth) & 3) << TRACE_GRAPH_DEPTH_START_BIT; \ - } while (0) -#define TRACE_CONTEXT_BITS 4#define TRACE_FTRACE_START TRACE_FTRACE_BIT
kernel/trace/trace.h +32 -2
@@ -899,8 +899,38 @@ extern void free_fgraph_ops(struct trace_array *tr);enum {TRACE_GRAPH_FL =1 , + +/* +* In the very unlikely case that an interrupt came in +* at a start of graph tracing, and we want to trace +* the function in that interrupt, the depth can be greater +* than zero, because of the preempted start of a previous +* trace. In an even more unlikely case, depth could be 2 +* if a softirq interrupted the start of graph tracing, +* followed by an interrupt preempting a start of graph +* tracing in the softirq, and depth can even be 3 +* if an NMI came in at the start of an interrupt function +* that preempted a softirq start of a function that +* preempted normal context!!!! Luckily, it can't be +* greater than 3, so the next two bits are a mask +* of what the depth is when we set TRACE_GRAPH_FL +*/ + +TRACE_GRAPH_DEPTH_START_BIT , +TRACE_GRAPH_DEPTH_END_BIT , }; +static inline unsigned long ftrace_graph_depth (unsigned long *task_var) +{ +return (*task_var >>TRACE_GRAPH_DEPTH_START_BIT ) &3 ; +} + +static inline void ftrace_graph_set_depth (unsigned long *task_var,int depth) +{ + *task_var &= ~(3 <<TRACE_GRAPH_DEPTH_START_BIT ); + *task_var |= (depth &3 ) <<TRACE_GRAPH_DEPTH_START_BIT ; +} +#ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash;extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash;
@@ -933,7 +963,7 @@ ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace)* when the depth is zero. */ *task_var |=TRACE_GRAPH_FL ; -trace_recursion_set_depth (trace->depth); +ftrace_graph_set_depth (task_var, trace->depth);/* * If no irqs are to be traced, but a set_graph_function
@@ -958,7 +988,7 @@ ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *traceunsigned long *task_var =fgraph_get_task_var (gops);if ((*task_var &TRACE_GRAPH_FL ) && - trace->depth ==trace_recursion_depth ()) + trace->depth ==ftrace_graph_depth (task_var)) *task_var &= ~TRACE_GRAPH_FL ; }
Masami Hiramatsu (Google)
maintainerPATCH v10 17/36 function_graph: Move graph notrace bit to shadow stack global var
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
The use of the task->trace_recursion for the logic used for the function graph no-trace was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead.
---
Changes in v2:
- Make description lines shorter than 76 chars.
--- include/linux/trace_recursion.h | 7 ------- kernel/trace/trace.h | 9 +++++++++ kernel/trace/trace_functions_graph.c | 10 ++++++---- 3 files changed, 15 insertions(+), 11 deletions(-)
include/linux/trace_recursion.h +0 -7
@@ -44,13 +44,6 @@ enum {*/ TRACE_IRQ_BIT , -/* -* To implement set_graph_notrace, if this bit is set, we ignore -* function graph tracing of called functions, until the return -* function is called to clear it. -*/ -TRACE_GRAPH_NOTRACE_BIT , -/* Used to prevent recursion recording from recursing. */ TRACE_RECORD_RECURSION_BIT, };
kernel/trace/trace.h +9 -0
@@ -918,8 +918,17 @@ enum {TRACE_GRAPH_DEPTH_START_BIT ,TRACE_GRAPH_DEPTH_END_BIT , + +/* +* To implement set_graph_notrace, if this bit is set, we ignore +* function graph tracing of called functions, until the return +* function is called to clear it. +*/ + TRACE_GRAPH_NOTRACE_BIT, }; +#define TRACE_GRAPH_NOTRACE (1 << TRACE_GRAPH_NOTRACE_BIT) +static inline unsigned long ftrace_graph_depth (unsigned long *task_var) {return (*task_var >>TRACE_GRAPH_DEPTH_START_BIT ) &3 ;
kernel/trace/trace_functions_graph.c +6 -4
@@ -130,6 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void)int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops) { +unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;struct trace_array_cpu *data;unsigned long flags;
@@ -138,7 +139,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,int ret;int cpu; -if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) +if (*task_var &TRACE_GRAPH_NOTRACE )return 0 ;/*
@@ -149,7 +150,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,* returning from the function. */ if (ftrace_graph_notrace_addr (trace->func)) { -trace_recursion_set (TRACE_GRAPH_NOTRACE_BIT ); + *task_var |=TRACE_GRAPH_NOTRACE_BIT ;/* * Need to return 1 to have the return called * that will clear the NOTRACE bit.
@@ -240,6 +241,7 @@ void __trace_graph_return(struct trace_array *tr,void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops) { +unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;struct trace_array_cpu *data;unsigned long flags;
@@ -249,8 +251,8 @@ void trace_graph_return(struct ftrace_graph_ret *trace,ftrace_graph_addr_finish (gops, trace); -if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) { -trace_recursion_clear (TRACE_GRAPH_NOTRACE_BIT ); +if (*task_var &TRACE_GRAPH_NOTRACE ) { + *task_var &= ~TRACE_GRAPH_NOTRACE ;return ; }
Masami Hiramatsu (Google)
maintainerPATCH v10 18/36 function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data()
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Added functions that can be called by a fgraph_ops entryfunc and retfunc to store state between the entry of the function being traced to the exit of the same function. The fgraph_ops entryfunc() may call fgraph_reserve_data() to store up to 32 words onto the task's shadow ret_stack and this then can be retrieved by fgraph_retrieve_data() called by the corresponding retfunc().
---
Changes in v10:
- Fix to support data size up to 32 words (previously it only support up to 31 words but the max size check missed to check it is 32 words)
- Use "offset" instead of "index".
Changes in v8:
- Avoid using DIV_ROUND_UP() in the hot path.
Changes in v3:
- Store fgraph_array index to the data entry.
- Both function requires fgraph_array index to store/retrieve data.
- Reserve correct size of the data.
- Return correct data area.
Changes in v2:
- Retrieve the reserved size by fgraph_retrieve_data().
- Expand the maximum data size to 32 words.
- Update stack index with __get_index(val) if FGRAPH_TYPE_ARRAY entry.
- fix typos and make description lines shorter than 76 chars.
--- include/linux/ftrace.h | 3 + kernel/trace/fgraph.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 175 insertions(+), 7 deletions(-)
include/linux/ftrace.h +3 -0
@@ -1075,6 +1075,9 @@ struct fgraph_ops {int idx; }; +void *fgraph_reserve_data (int idx,int size_bytes); +void *fgraph_retrieve_data (int idx,int *size_bytes); +/* * Stack of return addresses for functions * of a thread.
kernel/trace/fgraph.c +172 -7
272 lines not on this page · raw message
Masami Hiramatsu (Google)
maintainerPATCH v10 19/36 function_graph: Add selftest for passing local variables
In reply to Masami Hiramatsu (Google)From: Steven Rostedt (VMware) <[email protected]>
Add boot up selftest that passes variables from a function entry to a function exit, and make sure that they do get passed around.
---
Changes in v2:
- Add reserved size test.
- Use pr_*() instead of printk(KERN_*).
--- kernel/trace/trace_selftest.c | 169 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+)
kernel/trace/trace_selftest.c +169 -0
@@ -756,6 +756,173 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)#ifdef CONFIG_FUNCTION_GRAPH_TRACER +#ifdef CONFIG_DYNAMIC_FTRACE + +#define BYTE_NUMBER 123 +#define SHORT_NUMBER 12345 +#define WORD_NUMBER 1234567890 +#define LONG_NUMBER 1234567890123456789LL + +static int fgraph_store_size__initdata ; +static const char *fgraph_store_type_name__initdata ; +static char *fgraph_error_str__initdata ; +static char fgraph_error_str_buf[128 ]__initdata ; + +static __init int store_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) +{ +const char *type = fgraph_store_type_name; +int size = fgraph_store_size; +void *p; + + p =fgraph_reserve_data (gops->idx, size); +if (!p) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Failed to reserve %s\n" , type); + fgraph_error_str = fgraph_error_str_buf; +return 0 ; + } + +switch (fgraph_store_size) { +case 1 : + *(char *)p =BYTE_NUMBER ; +break ; +case 2 : + *(short *)p =SHORT_NUMBER ; +break ; +case 4 : + *(int *)p =WORD_NUMBER ; +break ; +case 8 : + *(long long *)p =LONG_NUMBER ; +break ; + } + +return 1 ; +} + +static __init void store_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) +{ +const char *type = fgraph_store_type_name; +long long expect =0 ; +long long found =-1 ; +int size; +char *p; + + p =fgraph_retrieve_data (gops->idx, &size); +if (!p) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Failed to retrieve %s\n" , type); + fgraph_error_str = fgraph_error_str_buf; +return ; + } +if (fgraph_store_size > size) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Retrieved size %d is smaller than expected %d\n" , + size, (int )fgraph_store_size); + fgraph_error_str = fgraph_error_str_buf; +return ; + } + +switch (fgraph_store_size) { +case 1 : + expect =BYTE_NUMBER ; + found = *(char *)p; +break ; +case 2 : + expect =SHORT_NUMBER ; + found = *(short *)p; +break ; +case 4 : + expect =WORD_NUMBER ; + found = *(int *)p; +break ; +case 8 : + expect =LONG_NUMBER ; + found = *(long long *)p; +break ; + } + +if (found != expect) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"%s returned not %lld but %lld\n" , type, expect, found); + fgraph_error_str = fgraph_error_str_buf; +return ; + } + fgraph_error_str =NULL ; +} + +static struct fgraph_ops store_bytes__initdata = { + .entryfunc = store_entry, + .retfunc = store_return, +}; + +static int __init test_graph_storage_type (const char *name,int size) +{ +char *func_name; +int len; +int ret; + + fgraph_store_type_name = name; + fgraph_store_size = size; + +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Failed to execute storage %s\n" , name); + fgraph_error_str = fgraph_error_str_buf; + +pr_cont ("PASSED\n" ); +pr_info ("Testing fgraph storage of %d byte%s: " , size, size >1 ?"s" :"" ); + + func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); + len =strlen (func_name); + + ret =ftrace_set_filter (&store_bytes.ops, func_name, len,1 ); +if (ret && ret != -ENODEV ) { +pr_cont ("*Could not set filter* " ); +return -1 ; + } + + ret =register_ftrace_graph (&store_bytes); +if (ret) { +pr_warn ("Failed to init store_bytes fgraph tracing\n" ); +return -1 ; + } + +DYN_FTRACE_TEST_NAME (); + +unregister_ftrace_graph (&store_bytes); + +if (fgraph_error_str) { +pr_cont ("*** %s ***" , fgraph_error_str); +return -1 ; + } + +return 0 ; +} +/* Test the storage passed across function_graph entry and return */ +static __init int test_graph_storage (void ) +{ +int ret; + + ret =test_graph_storage_type ("byte" ,1 ); +if (ret) +return ret; + ret =test_graph_storage_type ("short" ,2 ); +if (ret) +return ret; + ret =test_graph_storage_type ("word" ,4 ); +if (ret) +return ret; + ret =test_graph_storage_type ("long long" ,8 ); +if (ret) +return ret; +return 0 ; +} +#else +static inline int test_graph_storage (void ) {return 0 ; } +#endif /* CONFIG_DYNAMIC_FTRACE */ +/* Maximum number of functions to trace before diagnosing a hang */ #define GRAPH_MAX_FUNC_TEST 100000000
@@ -913,6 +1080,8 @@ trace_selftest_startup_function_graph(struct tracer *trace,ftrace_set_global_filter (NULL ,0 ,1 );#endif + ret =test_graph_storage (); +/* Don't test dynamic tracing, the function tracer already did */ out:/* Stop it if we failed */
Masami Hiramatsu (Google)
maintainerPATCH v10 20/36 ftrace: Add multiple fgraph storage selftest
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Add a selftest for multiple function graph tracer with storage on a same function. In this case, the shadow stack entry will be shared among those fgraph with different data storage. So this will ensure the fgraph will not mixed those storage data.
---
Changes in v8:
- Newly added.
--- kernel/trace/trace_selftest.c | 171 ++++++++++++++++++++++++++++++----------- 1 file changed, 126 insertions(+), 45 deletions(-)
kernel/trace/trace_selftest.c +126 -45
@@ -762,28 +762,32 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)#define SHORT_NUMBER 12345#define WORD_NUMBER 1234567890#define LONG_NUMBER 1234567890123456789LL - -static int fgraph_store_size__initdata ; -static const char *fgraph_store_type_name__initdata ; -static char *fgraph_error_str__initdata ; -static char fgraph_error_str_buf[128 ]__initdata ; +#define ERRSTR_BUFLEN 128 + +struct fgraph_fixture { +struct fgraph_ops gops; +int store_size; +const char *store_type_name; +char error_str_buf[ERRSTR_BUFLEN ]; +char *error_str; +};static __init int store_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops) { -const char *type = fgraph_store_type_name; -int size = fgraph_store_size; + struct fgraph_fixture *fixture = container_of(gops, struct fgraph_fixture, gops); +const char *type = fixture->store_type_name; +int size = fixture->store_size;void *p; p =fgraph_reserve_data (gops->idx, size);if (!p) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"Failed to reserve %s\n" , type); - fgraph_error_str = fgraph_error_str_buf;return 0 ; } -switch (fgraph_store_size) { +switch (size) {case 1 : *(char *)p =BYTE_NUMBER ;break ;
@@ -804,7 +808,8 @@ static __init int store_entry(struct ftrace_graph_ent *trace,static __init void store_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops) { -const char *type = fgraph_store_type_name; + struct fgraph_fixture *fixture = container_of(gops, struct fgraph_fixture, gops); +const char *type = fixture->store_type_name;long long expect =0 ;long long found =-1 ;int size;
@@ -812,20 +817,18 @@ static __init void store_return(struct ftrace_graph_ret *trace, p =fgraph_retrieve_data (gops->idx, &size);if (!p) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"Failed to retrieve %s\n" , type); - fgraph_error_str = fgraph_error_str_buf;return ; } -if (fgraph_store_size > size) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +if (fixture->store_size > size) { +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"Retrieved size %d is smaller than expected %d\n" , - size, (int )fgraph_store_size); - fgraph_error_str = fgraph_error_str_buf; + size, (int )fixture->store_size);return ; } -switch (fgraph_store_size) { +switch (fixture->store_size) {case 1 : expect =BYTE_NUMBER ; found = *(char *)p;
@@ -845,45 +848,44 @@ static __init void store_return(struct ftrace_graph_ret *trace, }if (found != expect) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"%s returned not %lld but %lld\n" , type, expect, found); - fgraph_error_str = fgraph_error_str_buf;return ; } - fgraph_error_str =NULL ; + fixture->error_str =NULL ; } -static struct fgraph_ops store_bytes__initdata = { - .entryfunc = store_entry, - .retfunc = store_return, -}; - -static int __init test_graph_storage_type (const char *name,int size) +static int __init init_fgraph_fixture (struct fgraph_fixture *fixture) {char *func_name;int len; -int ret; - fgraph_store_type_name = name; - fgraph_store_size = size; +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN , +"Failed to execute storage %s\n" , fixture->store_type_name); + fixture->error_str = fixture->error_str_buf; -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), -"Failed to execute storage %s\n" , name); - fgraph_error_str = fgraph_error_str_buf; + func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); + len =strlen (func_name); + +return ftrace_set_filter (&fixture->gops.ops, func_name, len,1 ); +} + +/* Test fgraph storage for each size */ +static int __init test_graph_storage_single (struct fgraph_fixture *fixture) +{ +int size = fixture->store_size; +int ret;pr_cont ("PASSED\n" );pr_info ("Testing fgraph storage of %d byte%s: " , size, size >1 ?"s" :"" ); - func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); - len =strlen (func_name); - - ret =ftrace_set_filter (&store_bytes.ops, func_name, len,1 ); + ret =init_fgraph_fixture (fixture);if (ret && ret != -ENODEV ) {pr_cont ("*Could not set filter* " );return -1 ; } - ret =register_ftrace_graph (&store_bytes); + ret =register_ftrace_graph (&fixture->gops);if (ret) {pr_warn ("Failed to init store_bytes fgraph tracing\n" );return -1 ;
@@ -891,30 +893,109 @@ static int __init test_graph_storage_type(const char *name, int size)DYN_FTRACE_TEST_NAME (); -unregister_ftrace_graph (&store_bytes); +unregister_ftrace_graph (&fixture->gops); -if (fgraph_error_str) { -pr_cont ("*** %s ***" , fgraph_error_str); +if (fixture->error_str) { +pr_cont ("*** %s ***" , fixture->error_str);return -1 ; }return 0 ; } + +static struct fgraph_fixture store_bytes[4 ]__initdata = { + [0 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =1 , + .store_type_name ="byte" , + }, + [1 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =2 , + .store_type_name ="short" , + }, + [2 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =4 , + .store_type_name ="word" , + }, + [3 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =8 , + .store_type_name ="long long" , + }, +}; + +static __init int test_graph_storage_multi (void ) +{ +struct fgraph_fixture *fixture; +bool printed = false; +int i, ret; + +pr_cont ("PASSED\n" ); +pr_info ("Testing multiple fgraph storage on a function: " ); + +for (i =0 ; i <ARRAY_SIZE (store_bytes); i++) { + fixture = &store_bytes[i]; + ret =init_fgraph_fixture (fixture); +if (ret && ret != -ENODEV ) { +pr_cont ("*Could not set filter* " ); + printed = true; + goto out; + } + + ret =register_ftrace_graph (&fixture->gops); +if (ret) { +pr_warn ("Failed to init store_bytes fgraph tracing\n" ); + printed = true; + goto out; + } + } + +DYN_FTRACE_TEST_NAME (); +out: +while (--i >=0 ) { + fixture = &store_bytes[i]; +unregister_ftrace_graph (&fixture->gops); + +if (fixture->error_str && !printed) { +pr_cont ("*** %s ***" , fixture->error_str); + printed = true; + } + } +return printed ?-1 :0 ; +} +/* Test the storage passed across function_graph entry and return */ static __init int test_graph_storage (void ) {int ret; - ret =test_graph_storage_type ("byte" ,1 ); + ret =test_graph_storage_single (&store_bytes[0 ]); +if (ret) +return ret; + ret =test_graph_storage_single (&store_bytes[1 ]);if (ret)return ret; - ret =test_graph_storage_type ("short" ,2 ); + ret =test_graph_storage_single (&store_bytes[2 ]);if (ret)return ret; - ret =test_graph_storage_type ("word" ,4 ); + ret =test_graph_storage_single (&store_bytes[3 ]);if (ret)return ret; - ret =test_graph_storage_type ("long long" ,8 ); + ret =test_graph_storage_multi ();if (ret)return ret;return 0 ;
Masami Hiramatsu (Google)
maintainerPATCH v10 21/36 function_graph: Pass ftrace_regs to entryfunc
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Pass ftrace_regs to the fgraph_ops::entryfunc(). If ftrace_regs is not available, it passes a NULL instead. User callback function can access some registers (including return address) via this ftrace_regs.
---
Changes in v8:
- Just pass ftrace_regs to the handler instead of adding a new entryregfunc.
- Update riscv ftrace_graph_func().
Changes in v3:
- Update for new multiple fgraph.
--- arch/arm64/kernel/ftrace.c | 2 + arch/loongarch/kernel/ftrace_dyn.c | 2 + arch/powerpc/kernel/trace/ftrace.c | 2 + arch/powerpc/kernel/trace/ftrace_64_pg.c | 10 ++++--- arch/riscv/kernel/ftrace.c | 2 + arch/x86/kernel/ftrace.c | 42 ++++++++++++++++-------------- include/linux/ftrace.h | 20 +++++++++++--- kernel/trace/fgraph.c | 21 +++++++++------ kernel/trace/ftrace.c | 3 +- kernel/trace/trace.h | 3 +- kernel/trace/trace_functions_graph.c | 3 +- kernel/trace/trace_irqsoff.c | 3 +- kernel/trace/trace_sched_wakeup.c | 3 +- kernel/trace/trace_selftest.c | 8 ++++-- 14 files changed, 76 insertions(+), 48 deletions(-)
arch/arm64/kernel/ftrace.c +1 -1
@@ -497,7 +497,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,return ;if (!function_graph_enter_ops (*parent, ip, frame_pointer, - (void *)frame_pointer, gops)) + (void *)frame_pointer, fregs, gops)) *parent = (unsigned long )&return_to_handler;ftrace_test_recursion_unlock (bit);
arch/loongarch/kernel/ftrace_dyn.c +1 -1
@@ -254,7 +254,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, old = *parent; -if (!function_graph_enter_ops (old, ip,0 , parent, gops)) +if (!function_graph_enter_ops (old, ip,0 , parent, fregs, gops)) *parent = return_hooker; }#else
arch/powerpc/kernel/trace/ftrace.c +1 -1
@@ -435,7 +435,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,if (bit <0 ) goto out; -if (!function_graph_enter_ops (parent_ip, ip,0 , (unsigned long *)sp, gops)) +if (!function_graph_enter_ops (parent_ip, ip,0 , (unsigned long *)sp, fregs, gops)) parent_ip =ppc_function_entry (return_to_handler);ftrace_test_recursion_unlock (bit);
arch/powerpc/kernel/trace/ftrace_64_pg.c +6 -4
@@ -800,7 +800,8 @@ int ftrace_disable_ftrace_graph_caller(void)* in current thread info. Return the address we want to divert to. */ static unsigned long -__prepare_ftrace_return (unsigned long parent,unsigned long ip,unsigned long sp) +__prepare_ftrace_return (unsigned long parent,unsigned long ip,unsigned long sp, +struct ftrace_regs *fregs) {unsigned long return_hooker;int bit;
@@ -817,7 +818,7 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp return_hooker =ppc_function_entry (return_to_handler); -if (!function_graph_enter (parent, ip,0 , (unsigned long *)sp)) +if (!function_graph_enter_regs (parent, ip,0 , (unsigned long *)sp, fregs)) parent = return_hooker;ftrace_test_recursion_unlock (bit);
@@ -829,13 +830,14 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long spvoid ftrace_graph_func (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs) { - fregs->regs.link =__prepare_ftrace_return (parent_ip, ip, fregs->regs.gpr[1 ]); + fregs->regs.link =__prepare_ftrace_return (parent_ip, ip, + fregs->regs.gpr[1 ], fregs); }#else unsigned long prepare_ftrace_return (unsigned long parent,unsigned long ip,unsigned long sp) { -return __prepare_ftrace_return (parent, ip, sp); +return __prepare_ftrace_return (parent, ip, sp,NULL ); }#endif #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
arch/riscv/kernel/ftrace.c +1 -1
@@ -197,7 +197,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,*/ old = *parent; -if (!function_graph_enter_ops (old, ip,frame_pointer (regs), parent, gops)) +if (!function_graph_enter_ops (old, ip,frame_pointer (regs), parent, fregs, gops)) *parent = return_hooker; }#else /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */
arch/x86/kernel/ftrace.c +23 -19
69 lines not on this page · raw message
include/linux/ftrace.h +15 -5
@@ -1061,9 +1061,12 @@ struct fgraph_ops;typedef void (*trace_func_graph_ret_t )(struct ftrace_graph_ret *,struct fgraph_ops *);/* return */ typedef int (*trace_func_graph_ent_t )(struct ftrace_graph_ent *, -struct fgraph_ops *);/* entry */ +struct fgraph_ops *, +struct ftrace_regs *);/* entry */ -extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace,struct fgraph_ops *gops); +extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops, +struct ftrace_regs *fregs);#ifdef CONFIG_FUNCTION_GRAPH_TRACER
@@ -1106,13 +1109,20 @@ struct ftrace_ret_stack {extern void return_to_handler (void );extern int -function_graph_enter (unsigned long ret,unsigned long func, -unsigned long frame_pointer,unsigned long *retp); +function_graph_enter_regs (unsigned long ret,unsigned long func, +unsigned long frame_pointer,unsigned long *retp, +struct ftrace_regs *fregs); + +static inline int function_graph_enter (unsigned long ret,unsigned long func, +unsigned long fp,unsigned long *retp) +{ +return function_graph_enter_regs (ret, func, fp, retp,NULL ); +}extern int function_graph_enter_ops (unsigned long ret,unsigned long func,unsigned long frame_pointer,unsigned long *retp, -struct fgraph_ops *gops); +struct ftrace_regs *fregs,struct fgraph_ops *gops);struct ftrace_ret_stack *ftrace_graph_get_ret_stack (struct task_struct *task,int skip);
kernel/trace/fgraph.c +13 -8
@@ -257,7 +257,8 @@ static inline unsigned long make_fgraph_data(int idx, int size, int offset) }/* ftrace_graph_entry set to this to tell some archs to run function graph */ -static int entry_run (struct ftrace_graph_ent *trace,struct fgraph_ops *ops) +static int entry_run (struct ftrace_graph_ent *trace,struct fgraph_ops *ops, +struct ftrace_regs *fregs) {return 0 ; }
@@ -448,7 +449,7 @@ int __weak ftrace_disable_ftrace_graph_caller(void)#endif int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops,struct ftrace_regs *fregs) {return 0 ; }
@@ -582,8 +583,9 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func,#endif /* If the caller does not use ftrace, call this function. */ -int function_graph_enter (unsigned long ret,unsigned long func, -unsigned long frame_pointer,unsigned long *retp) +int function_graph_enter_regs (unsigned long ret,unsigned long func, +unsigned long frame_pointer,unsigned long *retp, +struct ftrace_regs *fregs) {struct ftrace_graph_ent trace;unsigned long bitmap =0 ;
@@ -618,7 +620,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, save_curr_ret_stack = current->curr_ret_stack;if (ftrace_ops_test (&gops->ops, func,NULL ) && - gops->entryfunc (&trace, gops)) + gops->entryfunc (&trace, gops, fregs)) bitmap |=BIT (i);else /* Clear out any saved storage */
@@ -645,6 +647,7 @@ int function_graph_enter(unsigned long ret, unsigned long func,/* This is called from ftrace_graph_func() via ftrace */ int function_graph_enter_ops (unsigned long ret,unsigned long func,unsigned long frame_pointer,unsigned long *retp, +struct ftrace_regs *fregs,struct fgraph_ops *gops) {struct ftrace_graph_ent trace;
@@ -669,7 +672,7 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, trace.func = func; trace.depth = current->curr_ret_depth; save_curr_ret_stack = current->curr_ret_stack; -if (gops->entryfunc (&trace, gops)) { +if (gops->entryfunc (&trace, gops, fregs)) {if (type ==FGRAPH_TYPE_RESERVED )set_fgraph_index_bitmap (current, offset,BIT (gops->idx));else
@@ -950,7 +953,8 @@ void fgraph_init_ops(struct ftrace_ops *dst_ops, struct ftrace_ops *src_ops) { dst_ops->func = ftrace_graph_func; - dst_ops->flags =FTRACE_OPS_FL_PID |FTRACE_OPS_GRAPH_STUB ; + dst_ops->flags =FTRACE_OPS_FL_PID |FTRACE_OPS_GRAPH_STUB | +FTRACE_OPS_FL_SAVE_ARGS ;#ifdef FTRACE_GRAPH_TRAMP_ADDR dst_ops->trampoline =FTRACE_GRAPH_TRAMP_ADDR ;
@@ -1195,7 +1199,8 @@ int register_ftrace_graph(struct fgraph_ops *gops)mutex_lock (&ftrace_lock);if (!gops->ops.func) { - gops->ops.flags |=FTRACE_OPS_GRAPH_STUB ; + gops->ops.flags |=FTRACE_OPS_GRAPH_STUB | +FTRACE_OPS_FL_SAVE_ARGS ; gops->ops.func = ftrace_graph_func;#ifdef FTRACE_GRAPH_TRAMP_ADDR gops->ops.trampoline =FTRACE_GRAPH_TRAMP_ADDR ;
kernel/trace/ftrace.c +2 -1
@@ -816,7 +816,8 @@ void ftrace_graph_graph_time_control(bool enable) }static int profile_graph_entry (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {struct ftrace_ret_stack *ret_stack;
kernel/trace/trace.h +2 -1
@@ -682,7 +682,8 @@ void trace_default_header(struct seq_file *m);void print_trace_header (struct seq_file *m,struct trace_iterator *iter);void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops); -int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops); +int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops, +struct ftrace_regs *fregs);void tracing_start_cmdline_record (void );void tracing_stop_cmdline_record (void );
kernel/trace/trace_functions_graph.c +2 -1
@@ -128,7 +128,8 @@ static inline int ftrace_graph_ignore_irqs(void) }int trace_graph_entry (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;
kernel/trace/trace_irqsoff.c +2 -1
@@ -176,7 +176,8 @@ static int irqsoff_display_graph(struct trace_array *tr, int set) }static int irqsoff_graph_entry (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {struct trace_array *tr = irqsoff_trace;struct trace_array_cpu *data;
kernel/trace/trace_sched_wakeup.c +2 -1
@@ -113,7 +113,8 @@ static int wakeup_display_graph(struct trace_array *tr, int set) }static int wakeup_graph_entry (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {struct trace_array *tr = wakeup_trace;struct trace_array_cpu *data;
kernel/trace/trace_selftest.c +5 -3
@@ -773,7 +773,8 @@ struct fgraph_fixture { };static __init int store_entry (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) { struct fgraph_fixture *fixture = container_of(gops, struct fgraph_fixture, gops);const char *type = fixture->store_type_name;
@@ -1011,7 +1012,8 @@ static unsigned int graph_hang_thresh;/* Wrap the real function entry probe to avoid possible hanging */ static int trace_graph_entry_watchdog (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {/* This is harmlessly racy, we want to approximately detect a hang */ if (unlikely (++graph_hang_thresh >GRAPH_MAX_FUNC_TEST )) {
@@ -1025,7 +1027,7 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace,return 0 ; } -return trace_graph_entry (trace, gops); +return trace_graph_entry (trace, gops, fregs); }static struct fgraph_ops fgraph_ops__initdata = {
Masami Hiramatsu (Google)
maintainerPATCH v10 22/36 function_graph: Replace fgraph_ret_regs with ftrace_regs
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Use ftrace_regs instead of fgraph_ret_regs for tracing return value on function_graph tracer because of simplifying the callback interface.
The CONFIG_HAVE_FUNCTION_GRAPH_RETVAL is also replaced by CONFIG_HAVE_FUNCTION_GRAPH_FREGS.
---
Changes in v8:
- Newly added.
--- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/ftrace.h | 23 ++++++----------------- arch/arm64/kernel/asm-offsets.c | 12 ------------ arch/arm64/kernel/entry-ftrace.S | 32 ++++++++++++++++++-------------- arch/loongarch/Kconfig | 2 +- arch/loongarch/include/asm/ftrace.h | 24 ++---------------------- arch/loongarch/kernel/asm-offsets.c | 12 ------------ arch/loongarch/kernel/mcount.S | 17 ++++++++++------- arch/loongarch/kernel/mcount_dyn.S | 14 +++++++------- arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/ftrace.h | 21 --------------------- arch/riscv/kernel/mcount.S | 24 +++++++++++++----------- arch/s390/Kconfig | 2 +- arch/s390/include/asm/ftrace.h | 26 +++++++++----------------- arch/s390/kernel/asm-offsets.c | 6 ------ arch/s390/kernel/mcount.S | 9 +++++---- arch/x86/Kconfig | 2 +- arch/x86/include/asm/ftrace.h | 22 ++-------------------- arch/x86/kernel/ftrace_32.S | 15 +++++++++------ arch/x86/kernel/ftrace_64.S | 17 +++++++++-------- include/linux/ftrace.h | 14 +++++++++++--- kernel/trace/Kconfig | 4 ++-- kernel/trace/fgraph.c | 21 +++++++++------------ 23 files changed, 117 insertions(+), 206 deletions(-)
arch/arm64/Kconfig +1 -1
@@ -209,7 +209,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS select HAVE_FUNCTION_GRAPH_TRACER select HAVE_GCC_PLUGINS select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && \
arch/arm64/include/asm/ftrace.h +6 -17
@@ -137,6 +137,12 @@ ftrace_override_function_with_return(struct ftrace_regs *fregs) fregs->pc = fregs->lr; } +static __always_inline unsigned long +ftrace_regs_get_frame_pointer (const struct ftrace_regs *fregs) +{ +return fregs->fp; +} +int ftrace_regs_query_register_offset (const char *name);int ftrace_init_nop (struct module *mod,struct dyn_ftrace *rec);
@@ -194,23 +200,6 @@ static inline bool arch_syscall_match_sym_name(const char *sym,#ifndef __ASSEMBLY__#ifdef CONFIG_FUNCTION_GRAPH_TRACER -struct fgraph_ret_regs { -/* x0 - x7 */ -unsigned long regs[8 ]; - -unsigned long fp; -unsigned long __unused; -}; - -static inline unsigned long fgraph_ret_regs_return_value (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->regs[0 ]; -} - -static inline unsigned long fgraph_ret_regs_frame_pointer (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->fp; -}void prepare_ftrace_return (unsigned long self_addr,unsigned long *parent,unsigned long frame_pointer);
arch/arm64/kernel/asm-offsets.c +0 -12
@@ -200,18 +200,6 @@ int main(void)DEFINE (FTRACE_OPS_FUNC , offsetof(struct ftrace_ops , func));#endif BLANK (); -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - DEFINE(FGRET_REGS_X0, offsetof(struct fgraph_ret_regs, regs[0])); - DEFINE(FGRET_REGS_X1, offsetof(struct fgraph_ret_regs, regs[1])); - DEFINE(FGRET_REGS_X2, offsetof(struct fgraph_ret_regs, regs[2])); - DEFINE(FGRET_REGS_X3, offsetof(struct fgraph_ret_regs, regs[3])); - DEFINE(FGRET_REGS_X4, offsetof(struct fgraph_ret_regs, regs[4])); - DEFINE(FGRET_REGS_X5, offsetof(struct fgraph_ret_regs, regs[5])); - DEFINE(FGRET_REGS_X6, offsetof(struct fgraph_ret_regs, regs[6])); - DEFINE(FGRET_REGS_X7, offsetof(struct fgraph_ret_regs, regs[7])); -DEFINE (FGRET_REGS_FP , offsetof(struct fgraph_ret_regs , fp)); -DEFINE (FGRET_REGS_SIZE ,sizeof (struct fgraph_ret_regs )); -#endif #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS DEFINE (FTRACE_OPS_DIRECT_CALL , offsetof(struct ftrace_ops , direct_call));#endif
arch/arm64/kernel/entry-ftrace.S +18 -14
@@ -329,24 +329,28 @@ SYM_FUNC_END(ftrace_stub_graph) * @fp is checked against the value passed by ftrace_graph_caller(). */ SYM_CODE_START(return_to_handler) - /* save return value regs */ - sub sp, sp, #FGRET_REGS_SIZE - stp x0, x1, [sp, #FGRET_REGS_X0] - stp x2, x3, [sp, #FGRET_REGS_X2] - stp x4, x5, [sp, #FGRET_REGS_X4] - stp x6, x7, [sp, #FGRET_REGS_X6] - str x29, [sp, #FGRET_REGS_FP] // parent's fp + /* Make room for ftrace_regs */ + sub sp, sp, #FREGS_SIZE + + /* Save return value regs */ + stp x0, x1, [sp, #FREGS_X0] + stp x2, x3, [sp, #FREGS_X2] + stp x4, x5, [sp, #FREGS_X4] + stp x6, x7, [sp, #FREGS_X6] + + /* Save the callsite's FP */ + str x29, [sp, #FREGS_FP] mov x0, sp - bl ftrace_return_to_handler // addr = ftrace_return_to_hander(regs); + bl ftrace_return_to_handler // addr = ftrace_return_to_hander(fregs); mov x30, x0 // restore the original return address - /* restore return value regs */ - ldp x0, x1, [sp, #FGRET_REGS_X0] - ldp x2, x3, [sp, #FGRET_REGS_X2] - ldp x4, x5, [sp, #FGRET_REGS_X4] - ldp x6, x7, [sp, #FGRET_REGS_X6] - add sp, sp, #FGRET_REGS_SIZE + /* Restore return value regs */ + ldp x0, x1, [sp, #FREGS_X0] + ldp x2, x3, [sp, #FREGS_X2] + ldp x4, x5, [sp, #FREGS_X4] + ldp x6, x7, [sp, #FREGS_X6] + add sp, sp, #FREGS_SIZE ret SYM_CODE_END(return_to_handler)
arch/loongarch/Kconfig +1 -1
@@ -123,7 +123,7 @@ config LOONGARCH select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS
arch/loongarch/include/asm/ftrace.h +2 -22
@@ -78,6 +78,8 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) override_function_with_return(&(fregs)->regs)#define ftrace_regs_query_register_offset (name) \ regs_query_register_offset(name) +#define ftrace_regs_get_frame_pointer (fregs) \ + ((fregs)->regs.regs[22])#define ftrace_graph_func ftrace_graph_func void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
@@ -100,26 +102,4 @@ __arch_ftrace_set_direct_caller(struct pt_regs *regs, unsigned long addr)#endif /* CONFIG_FUNCTION_TRACER */ -#ifndef __ASSEMBLY__ -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -struct fgraph_ret_regs { -/* a0 - a1 */ -unsigned long regs[2 ]; - -unsigned long fp; -unsigned long __unused; -}; - -static inline unsigned long fgraph_ret_regs_return_value (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->regs[0 ]; -} - -static inline unsigned long fgraph_ret_regs_frame_pointer (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->fp; -} -#endif /* ifdef CONFIG_FUNCTION_GRAPH_TRACER */ -#endif -#endif /* _ASM_LOONGARCH_FTRACE_H */
arch/loongarch/kernel/asm-offsets.c +0 -12
@@ -279,18 +279,6 @@ static void __used output_pbe_defines(void) }#endif -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -static void __used output_fgraph_ret_regs_defines (void ) -{ -COMMENT ("LoongArch fgraph_ret_regs offsets." ); -OFFSET (FGRET_REGS_A0 , fgraph_ret_regs, regs[0 ]); -OFFSET (FGRET_REGS_A1 , fgraph_ret_regs, regs[1 ]); -OFFSET (FGRET_REGS_FP , fgraph_ret_regs, fp); -DEFINE (FGRET_REGS_SIZE ,sizeof (struct fgraph_ret_regs )); -BLANK (); -} -#endif -static void __used output_kvm_defines (void ) {COMMENT ("KVM/LoongArch Specific offsets." );
arch/loongarch/kernel/mcount.S +10 -7
@@ -79,10 +79,11 @@ SYM_FUNC_START(ftrace_graph_caller) SYM_FUNC_END(ftrace_graph_caller) SYM_FUNC_START(return_to_handler) - PTR_ADDI sp, sp, -FGRET_REGS_SIZE - PTR_S a0, sp, FGRET_REGS_A0 - PTR_S a1, sp, FGRET_REGS_A1 - PTR_S zero, sp, FGRET_REGS_FP + /* Save return value regs */ + PTR_ADDI sp, sp, -PT_SIZE + PTR_S a0, sp, PT_R4 + PTR_S a1, sp, PT_R5 + PTR_S zero, sp, PT_R22 move a0, sp bl ftrace_return_to_handler
@@ -90,9 +91,11 @@ SYM_FUNC_START(return_to_handler) /* Restore the real parent address: a0 -> ra */ move ra, a0 - PTR_L a0, sp, FGRET_REGS_A0 - PTR_L a1, sp, FGRET_REGS_A1 - PTR_ADDI sp, sp, FGRET_REGS_SIZE + /* Restore return value regs */ + PTR_L a0, sp, PT_R4 + PTR_L a1, sp, PT_R5 + PTR_ADDI sp, sp, PT_SIZE + jr ra SYM_FUNC_END(return_to_handler) #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
arch/loongarch/kernel/mcount_dyn.S +7 -7
@@ -140,19 +140,19 @@ SYM_CODE_END(ftrace_graph_caller) SYM_CODE_START(return_to_handler) UNWIND_HINT_UNDEFINED /* Save return value regs */ - PTR_ADDI sp, sp, -FGRET_REGS_SIZE - PTR_S a0, sp, FGRET_REGS_A0 - PTR_S a1, sp, FGRET_REGS_A1 - PTR_S zero, sp, FGRET_REGS_FP + PTR_ADDI sp, sp, -PT_SIZE + PTR_S a0, sp, PT_R4 + PTR_S a1, sp, PT_R5 + PTR_S zero, sp, PT_R22 move a0, sp bl ftrace_return_to_handler move ra, a0 /* Restore return value regs */ - PTR_L a0, sp, FGRET_REGS_A0 - PTR_L a1, sp, FGRET_REGS_A1 - PTR_ADDI sp, sp, FGRET_REGS_SIZE + PTR_L a0, sp, PT_R4 + PTR_L a1, sp, PT_R5 + PTR_ADDI sp, sp, PT_SIZE jr ra SYM_CODE_END(return_to_handler)
arch/riscv/Kconfig +1 -1
@@ -129,7 +129,7 @@ config RISCV select HAVE_DYNAMIC_FTRACE_WITH_REGS if HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL select HAVE_FUNCTION_GRAPH_TRACER - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION select HAVE_EBPF_JIT if MMU select HAVE_FAST_GUP if MMU
arch/riscv/include/asm/ftrace.h +0 -21
@@ -143,25 +143,4 @@ static inline void __arch_ftrace_set_direct_caller(struct pt_regs *regs, unsigne#endif /* CONFIG_DYNAMIC_FTRACE */ -#ifndef __ASSEMBLY__ -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -struct fgraph_ret_regs { -unsigned long a1; -unsigned long a0; -unsigned long s0; -unsigned long ra; -}; - -static inline unsigned long fgraph_ret_regs_return_value (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->a0; -} - -static inline unsigned long fgraph_ret_regs_frame_pointer (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->s0; -} -#endif /* ifdef CONFIG_FUNCTION_GRAPH_TRACER */ -#endif -#endif /* _ASM_RISCV_FTRACE_H */
arch/riscv/kernel/mcount.S +13 -11
@@ -12,6 +12,8 @@ #include <asm/asm-offsets.h> #include <asm/ftrace.h> +#define ABI_SIZE_ON_STACK 80 + .text .macro SAVE_ABI_STATE
@@ -26,12 +28,12 @@ * register if a0 was not saved. */ .macro SAVE_RET_ABI_STATE - addi sp, sp, -4*SZREG - REG_S s0, 2*SZREG(sp) - REG_S ra, 3*SZREG(sp) - REG_S a0, 1*SZREG(sp) - REG_S a1, 0*SZREG(sp) - addi s0, sp, 4*SZREG + addi sp, sp, -ABI_SIZE_ON_STACK + REG_S ra, 1*SZREG(sp) + REG_S s0, 8*SZREG(sp) + REG_S a0, 10*SZREG(sp) + REG_S a1, 11*SZREG(sp) + addi s0, sp, ABI_SIZE_ON_STACK .endm .macro RESTORE_ABI_STATE
@@ -41,11 +43,11 @@ .endm .macro RESTORE_RET_ABI_STATE - REG_L ra, 3*SZREG(sp) - REG_L s0, 2*SZREG(sp) - REG_L a0, 1*SZREG(sp) - REG_L a1, 0*SZREG(sp) - addi sp, sp, 4*SZREG + REG_L ra, 1*SZREG(sp) + REG_L s0, 8*SZREG(sp) + REG_L a0, 10*SZREG(sp) + REG_L a1, 11*SZREG(sp) + addi sp, sp, ABI_SIZE_ON_STACK .endm SYM_TYPED_FUNC_START(ftrace_stub)
arch/s390/Kconfig +1 -1
@@ -179,7 +179,7 @@ config S390 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_RETVAL + select HAVE_FUNCTION_GRAPH_FREGS select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS
arch/s390/include/asm/ftrace.h +9 -17
@@ -54,23 +54,6 @@ static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *return NULL ; } -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -struct fgraph_ret_regs { -unsigned long gpr2; -unsigned long fp; -}; - -static __always_inline unsigned long fgraph_ret_regs_return_value (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->gpr2; -} - -static __always_inline unsigned long fgraph_ret_regs_frame_pointer (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->fp; -} -#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ -static __always_inline unsigned long ftrace_regs_get_instruction_pointer (const struct ftrace_regs *fregs) {
@@ -97,6 +80,15 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs,#define ftrace_regs_query_register_offset (name) \ regs_query_register_offset(name) +static __always_inline unsigned long +ftrace_regs_get_frame_pointer (struct ftrace_regs *fregs) +{ +unsigned long *sp; + + sp = (void *)ftrace_regs_get_stack_pointer (fregs); +return sp[0 ];/* return backchain */ +} +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * When an ftrace registered caller is tracing a function that is
arch/s390/kernel/asm-offsets.c +0 -6
@@ -178,12 +178,6 @@ int main(void)DEFINE (OLDMEM_SIZE ,PARMAREA + offsetof(struct parmarea , oldmem_size));DEFINE (COMMAND_LINE ,PARMAREA + offsetof(struct parmarea , command_line));DEFINE (MAX_COMMAND_LINE_SIZE ,PARMAREA + offsetof(struct parmarea , max_command_line_size)); -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -/* function graph return value tracing */ -OFFSET (__FGRAPH_RET_GPR2, fgraph_ret_regs, gpr2); -OFFSET (__FGRAPH_RET_FP, fgraph_ret_regs, fp); -DEFINE (__FGRAPH_RET_SIZE,sizeof (struct fgraph_ret_regs )); -#endif OFFSET (__FTRACE_REGS_PT_REGS, ftrace_regs, regs);DEFINE (__FTRACE_REGS_SIZE,sizeof (struct ftrace_regs ));return 0 ;
arch/s390/kernel/mcount.S +5 -4
@@ -133,14 +133,15 @@ SYM_CODE_END(ftrace_common) SYM_FUNC_START(return_to_handler) stmg %r2,%r5,32(%r15) lgr %r1,%r15 - aghi %r15,-(STACK_FRAME_OVERHEAD+__FGRAPH_RET_SIZE) +# Allocate ftrace_regs + backchain on the stack + aghi %r15,-STACK_FRAME_SIZE_FREGS stg %r1,__SF_BACKCHAIN(%r15) la %r3,STACK_FRAME_OVERHEAD(%r15) - stg %r1,__FGRAPH_RET_FP(%r3) - stg %r2,__FGRAPH_RET_GPR2(%r3) + stg %r2,(__SF_GPRS+2*8)(%r15) + stg %r15,(__SF_GPRS+15*8)(%r15) lgr %r2,%r3 brasl %r14,ftrace_return_to_handler - aghi %r15,STACK_FRAME_OVERHEAD+__FGRAPH_RET_SIZE + aghi %r15,STACK_FRAME_SIZE_FREGS lgr %r14,%r2 lmg %r2,%r5,32(%r15) BR_EX %r14
arch/x86/Kconfig +1 -1
@@ -225,7 +225,7 @@ config X86 select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_GRAPH_TRACER if X86_32 || (X86_64 && DYNAMIC_FTRACE) select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS
arch/x86/include/asm/ftrace.h +2 -20
@@ -72,6 +72,8 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) override_function_with_return(&(fregs)->regs)#define ftrace_regs_query_register_offset (name) \ regs_query_register_offset(name) +#define ftrace_regs_get_frame_pointer (fregs) \ + frame_pointer(&(fregs)->regs)struct ftrace_ops ;#define ftrace_graph_func ftrace_graph_func
@@ -156,24 +158,4 @@ static inline bool arch_trace_is_compat_syscall(struct pt_regs *regs)#endif /* !COMPILE_OFFSETS */ #endif /* !__ASSEMBLY__ */ -#ifndef __ASSEMBLY__ -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -struct fgraph_ret_regs { -unsigned long ax; -unsigned long dx; -unsigned long bp; -}; - -static inline unsigned long fgraph_ret_regs_return_value (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->ax; -} - -static inline unsigned long fgraph_ret_regs_frame_pointer (struct fgraph_ret_regs *ret_regs) -{ -return ret_regs->bp; -} -#endif /* ifdef CONFIG_FUNCTION_GRAPH_TRACER */ -#endif -#endif /* _ASM_X86_FTRACE_H */
arch/x86/kernel/ftrace_32.S +9 -6
@@ -23,6 +23,8 @@ SYM_FUNC_START(__fentry__) SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) +#define FRAME_SIZE PT_OLDSS+4 + SYM_CODE_START(ftrace_caller) #ifdef CONFIG_FRAME_POINTER
@@ -187,14 +189,15 @@ SYM_CODE_END(ftrace_graph_caller) .globl return_to_handler return_to_handler: - pushl $0 - pushl %edx - pushl %eax + subl $(FRAME_SIZE), %esp + movl $0, PT_EBP(%esp) + movl %edx, PT_EDX(%esp) + movl %eax, PT_EAX(%esp) movl %esp, %eax call ftrace_return_to_handler movl %eax, %ecx - popl %eax - popl %edx - addl $4, %esp # skip ebp + movl %eax, PT_EAX(%esp) + movl %edx, PT_EDX(%esp) + addl $(FRAME_SIZE), %esp JMP_NOSPEC ecx #endif
arch/x86/kernel/ftrace_64.S +9 -8
@@ -348,21 +348,22 @@ STACK_FRAME_NON_STANDARD_FP(__fentry__) SYM_CODE_START(return_to_handler) UNWIND_HINT_UNDEFINED ANNOTATE_NOENDBR - subq $24, %rsp - /* Save the return values */ - movq %rax, (%rsp) - movq %rdx, 8(%rsp) - movq %rbp, 16(%rsp) + /* Save ftrace_regs for function exit context */ + subq $(FRAME_SIZE), %rsp + + movq %rax, RAX(%rsp) + movq %rdx, RDX(%rsp) + movq %rbp, RBP(%rsp) movq %rsp, %rdi call ftrace_return_to_handler movq %rax, %rdi - movq 8(%rsp), %rdx - movq (%rsp), %rax + movq RDX(%rsp), %rdx + movq RAX(%rsp), %rax - addq $24, %rsp + addq $(FRAME_SIZE), %rsp /* * Jump back to the old return address. This cannot be JMP_NOSPEC rdi * since IBT would demand that contain ENDBR, which simply isn't so for
include/linux/ftrace.h +11 -3
@@ -43,9 +43,8 @@ struct dyn_ftrace;char *arch_ftrace_match_adjust (char *str,const char *search); -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL -struct fgraph_ret_regs ; -unsigned long ftrace_return_to_handler (struct fgraph_ret_regs *ret_regs); +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler (struct ftrace_regs *fregs);#else unsigned long ftrace_return_to_handler (unsigned long frame_pointer);#endif
@@ -135,6 +134,13 @@ extern int ftrace_enabled;* Also, architecture dependent fields can be used for internal process. * (e.g. orig_ax on x86_64) * +* Basically, ftrace_regs stores the registers related to the context. +* On function entry, registers for function parameters and hooking the +* function call are stored, and on function exit, registers for function +* return value and frame pointers are stored. +* +* And also, it dpends on the context that which registers are restored +* from the ftrace_regs. * On the function entry, those registers will be restored except for * the stack pointer, so that user can change the function parameters * and instruction pointer (e.g. live patching.)
@@ -192,6 +198,8 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) override_function_with_return(ftrace_get_regs(fregs))#define ftrace_regs_query_register_offset (name) \ regs_query_register_offset(name) +#define ftrace_regs_get_frame_pointer (fregs) \ + frame_pointer(&(fregs)->regs)#endif typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip,
kernel/trace/Kconfig +2 -2
@@ -31,7 +31,7 @@ config HAVE_FUNCTION_GRAPH_TRACER help See Documentation/trace/ftrace-design.rst -config HAVE_FUNCTION_GRAPH_RETVAL +config HAVE_FUNCTION_GRAPH_FREGS bool config HAVE_DYNAMIC_FTRACE
@@ -232,7 +232,7 @@ config FUNCTION_GRAPH_TRACER config FUNCTION_GRAPH_RETVAL bool "Kernel Function Graph Return Value" - depends on HAVE_FUNCTION_GRAPH_RETVAL + depends on HAVE_FUNCTION_GRAPH_FREGS depends on FUNCTION_GRAPH_TRACER default n help
kernel/trace/fgraph.c +9 -12
@@ -775,15 +775,12 @@ static struct notifier_block ftrace_suspend_notifier = { .notifier_call = ftrace_suspend_notifier_call, }; -/* fgraph_ret_regs is not defined without CONFIG_FUNCTION_GRAPH_RETVAL */ -struct fgraph_ret_regs ; -/* * Send the trace to the ring-buffer. * @return the original return address. */ -static unsigned long __ftrace_return_to_handler (struct fgraph_ret_regs *ret_regs, -unsigned long frame_pointer) +static inline unsigned long +__ftrace_return_to_handler (struct ftrace_regs *fregs,unsigned long frame_pointer) {struct ftrace_ret_stack *ret_stack;struct ftrace_graph_ret trace;
@@ -803,7 +800,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.rettime =trace_clock_local ();#ifdef CONFIG_FUNCTION_GRAPH_RETVAL - trace.retval =fgraph_ret_regs_return_value (ret_regs); + trace.retval =ftrace_regs_get_return_value (fregs);#endif bitmap =get_fgraph_index_bitmap (current, offset);
@@ -831,14 +828,14 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs }/* -* After all architecures have selected HAVE_FUNCTION_GRAPH_RETVAL, we can -* leave only ftrace_return_to_handler(ret_regs). +* After all architecures have selected HAVE_FUNCTION_GRAPH_FREGS, we can +* leave only ftrace_return_to_handler(fregs). */ -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL -unsigned long ftrace_return_to_handler (struct fgraph_ret_regs *ret_regs) +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler (struct ftrace_regs *fregs) { -return __ftrace_return_to_handler (ret_regs, -fgraph_ret_regs_frame_pointer (ret_regs)); +return __ftrace_return_to_handler (fregs, +ftrace_regs_get_frame_pointer (fregs)); }#else unsigned long ftrace_return_to_handler (unsigned long frame_pointer)
Masami Hiramatsu (Google)
maintainerPATCH v10 23/36 function_graph: Pass ftrace_regs to retfunc
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Pass ftrace_regs to the fgraph_ops::retfunc(). If ftrace_regs is not available, it passes a NULL instead. User callback function can access some registers (including return address) via this ftrace_regs.
---
Changes in v8:
- Pass ftrace_regs to retfunc, instead of adding retregfunc.
Changes in v6:
- update to use ftrace_regs_get_return_value() because of reordering patches.
Changes in v3:
- Update for new multiple fgraph.
- Save the return address to instruction pointer in ftrace_regs.
--- include/linux/ftrace.h | 3 ++- kernel/trace/fgraph.c | 14 ++++++++++---- kernel/trace/ftrace.c | 3 ++- kernel/trace/trace.h | 3 ++- kernel/trace/trace_functions_graph.c | 7 ++++--- kernel/trace/trace_irqsoff.c | 3 ++- kernel/trace/trace_sched_wakeup.c | 3 ++- kernel/trace/trace_selftest.c | 3 ++- 8 files changed, 26 insertions(+), 13 deletions(-)
include/linux/ftrace.h +2 -1
@@ -1067,7 +1067,8 @@ struct fgraph_ops;/* Type of the callback handlers for tracing function graph*/ typedef void (*trace_func_graph_ret_t )(struct ftrace_graph_ret *, -struct fgraph_ops *);/* return */ +struct fgraph_ops *, +struct ftrace_regs *);/* return */ typedef int (*trace_func_graph_ent_t )(struct ftrace_graph_ent *,struct fgraph_ops *,struct ftrace_regs *);/* entry */
kernel/trace/fgraph.c +10 -4
@@ -264,7 +264,8 @@ static int entry_run(struct ftrace_graph_ent *trace, struct fgraph_ops *ops, }/* ftrace_graph_return set to this to tell some archs to run function graph */ -static void return_run (struct ftrace_graph_ret *trace,struct fgraph_ops *ops) +static void return_run (struct ftrace_graph_ret *trace,struct fgraph_ops *ops, +struct ftrace_regs *fregs) { }
@@ -455,7 +456,8 @@ int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, }static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) { }
@@ -799,6 +801,9 @@ __ftrace_return_to_handler(struct ftrace_regs *fregs, unsigned long frame_pointe } trace.rettime =trace_clock_local (); +if (fregs) +ftrace_regs_set_instruction_pointer (fregs, ret); +#ifdef CONFIG_FUNCTION_GRAPH_RETVAL trace.retval =ftrace_regs_get_return_value (fregs);#endif
@@ -812,7 +817,7 @@ __ftrace_return_to_handler(struct ftrace_regs *fregs, unsigned long frame_pointeif (gops == &fgraph_stub)continue ; - gops->retfunc (&trace, gops); + gops->retfunc (&trace, gops, fregs); }/*
@@ -976,7 +981,8 @@ void ftrace_graph_sleep_time_control(bool enable)* Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h */ -void ftrace_stub_graph (struct ftrace_graph_ret *trace,struct fgraph_ops *gops); +void ftrace_stub_graph (struct ftrace_graph_ret *trace,struct fgraph_ops *gops, +struct ftrace_regs *fregs);/* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;
kernel/trace/ftrace.c +2 -1
@@ -835,7 +835,8 @@ static int profile_graph_entry(struct ftrace_graph_ent *trace, }static void profile_graph_return (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {struct ftrace_ret_stack *ret_stack;struct ftrace_profile_stat *stat;
kernel/trace/trace.h +2 -1
@@ -681,7 +681,8 @@ void trace_latency_header(struct seq_file *m);void trace_default_header (struct seq_file *m);void print_trace_header (struct seq_file *m,struct trace_iterator *iter); -void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops); +void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops, +struct ftrace_regs *fregs);int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops,struct ftrace_regs *fregs);
kernel/trace/trace_functions_graph.c +4 -3
@@ -240,7 +240,7 @@ void __trace_graph_return(struct trace_array *tr, }void trace_graph_return (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops,struct ftrace_regs *fregs) {unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;
@@ -270,7 +270,8 @@ void trace_graph_return(struct ftrace_graph_ret *trace, }static void trace_graph_thresh_return (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {ftrace_graph_addr_finish (gops, trace);
@@ -283,7 +284,7 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, (trace->rettime - trace->calltime < tracing_thresh)) return; else -trace_graph_return (trace ,gops ); +trace_graph_return (trace ,gops ,fregs ); }static struct fgraph_ops funcgraph_ops = {
kernel/trace/trace_irqsoff.c +2 -1
@@ -208,7 +208,8 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, }static void irqsoff_graph_return (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {struct trace_array *tr = irqsoff_trace;struct trace_array_cpu *data;
kernel/trace/trace_sched_wakeup.c +2 -1
@@ -144,7 +144,8 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace, }static void wakeup_graph_return (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {struct trace_array *tr = wakeup_trace;struct trace_array_cpu *data;
kernel/trace/trace_selftest.c +2 -1
@@ -807,7 +807,8 @@ static __init int store_entry(struct ftrace_graph_ent *trace, }static __init void store_return (struct ftrace_graph_ret *trace, -struct fgraph_ops *gops) +struct fgraph_ops *gops, +struct ftrace_regs *fregs) { struct fgraph_fixture *fixture = container_of(gops, struct fgraph_fixture, gops);const char *type = fixture->store_type_name;
Masami Hiramatsu (Google)
maintainerPATCH v10 24/36 fprobe: Use ftrace_regs in fprobe entry handler
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
This allows fprobes to be available with CONFIG_DYNAMIC_FTRACE_WITH_ARGS instead of CONFIG_DYNAMIC_FTRACE_WITH_REGS, then we can enable fprobe on arm64.
---
Changes in v6:
- Keep using SAVE_REGS flag to avoid breaking bpf kprobe-multi test.
--- include/linux/fprobe.h | 2 +- kernel/trace/Kconfig | 3 ++- kernel/trace/bpf_trace.c | 10 +++++++--- kernel/trace/fprobe.c | 3 ++- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 4 ++-- samples/fprobe/fprobe_example.c | 2 +- 7 files changed, 20 insertions(+), 10 deletions(-)
include/linux/fprobe.h +1 -1
@@ -10,7 +10,7 @@struct fprobe ;typedef int (*fprobe_entry_cb )(struct fprobe *fp,unsigned long entry_ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *regs,void *entry_data); typedef void (*fprobe_exit_cb)(struct fprobe *fp, unsigned long entry_ip,
kernel/trace/Kconfig +2 -1
@@ -287,7 +287,7 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS + depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n
@@ -672,6 +672,7 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS + depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and
kernel/trace/bpf_trace.c +7 -3
@@ -2577,7 +2577,7 @@ static int __init bpf_event_init(void)fs_initcall (bpf_event_init);#endif /* CONFIG_MODULES */ -#ifdef CONFIG_FPROBE +#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp;
@@ -2822,10 +2822,14 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link,static int kprobe_multi_link_handler (struct fprobe *fp,unsigned long fentry_ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *fregs,void *data) {struct bpf_kprobe_multi_link *link; +struct pt_regs *regs =ftrace_get_regs (fregs); + +if (!regs) +return 0 ; link = container_of(fp, struct bpf_kprobe_multi_link, fp);kprobe_multi_link_prog_run (link,get_entry_ip (fentry_ip), regs);
@@ -3099,7 +3103,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prkvfree (cookies); return err; } -#else /* !CONFIG_FPROBE */ +#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ int bpf_kprobe_multi_link_attach (const union bpf_attr *attr,struct bpf_prog *prog) {return -EOPNOTSUPP ;
kernel/trace/fprobe.c +2 -1
@@ -46,7 +46,7 @@ static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, }if (fp->entry_handler) - ret = fp->entry_handler (fp, ip, parent_ip,ftrace_get_regs (fregs), entry_data); + ret = fp->entry_handler (fp, ip, parent_ip, fregs, entry_data);/* If entry_handler returns !0, nmissed is not counted. */ if (rh) {
@@ -182,6 +182,7 @@ static void fprobe_init(struct fprobe *fp) fp->ops.func = fprobe_kprobe_handler; else fp->ops.func = fprobe_handler; + fp->ops.flags |=FTRACE_OPS_FL_SAVE_REGS ; }
kernel/trace/trace_fprobe.c +5 -1
@@ -338,12 +338,16 @@ NOKPROBE_SYMBOL(fexit_perf_func);#endif /* CONFIG_PERF_EVENTS */ static int fentry_dispatcher (struct fprobe *fp,unsigned long entry_ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *fregs,void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); +struct pt_regs *regs =ftrace_get_regs (fregs);int ret =0 ; +if (!regs) +return 0 ; +if (trace_probe_test_flag (&tf->tp,TP_FLAG_TRACE ))fentry_trace_func (tf, entry_ip, regs);#ifdef CONFIG_PERF_EVENTS
lib/test_fprobe.c +2 -2
@@ -40,7 +40,7 @@ static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip,unsigned long ret_ip, -struct pt_regs *regs,void *data) +struct ftrace_regs *fregs,void *data) {KUNIT_EXPECT_FALSE (current_test,preemptible ());/* This can be called on the fprobe_selftest_target and the fprobe_selftest_target2 */
@@ -81,7 +81,7 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip,unsigned long ret_ip, -struct pt_regs *regs,void *data) +struct ftrace_regs *fregs,void *data) {KUNIT_EXPECT_FALSE (current_test,preemptible ());return 0 ;
samples/fprobe/fprobe_example.c +1 -1
@@ -50,7 +50,7 @@ static void show_backtrace(void)static int sample_entry_handler (struct fprobe *fp,unsigned long ip,unsigned long ret_ip, -struct pt_regs *regs,void *data) +struct ftrace_regs *fregs,void *data) {if (use_trace)/*
Masami Hiramatsu (Google)
maintainerPATCH v10 25/36 fprobe: Use ftrace_regs in fprobe exit handler
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Change the fprobe exit handler to use ftrace_regs structure instead of pt_regs. This also introduce HAVE_PT_REGS_TO_FTRACE_REGS_CAST which means the ftrace_regs's memory layout is equal to the pt_regs so that those are able to cast. Fprobe introduces a new dependency with that.
---
Changes in v3:
- Use ftrace_regs_get_return_value()
Changes from previous series: NOTHING, just forward ported.
--- arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/fprobe.h | 2 +- include/linux/ftrace.h | 6 ++++++ kernel/trace/Kconfig | 8 ++++++++ kernel/trace/bpf_trace.c | 6 +++++- kernel/trace/fprobe.c | 3 ++- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 6 +++--- samples/fprobe/fprobe_example.c | 2 +- 11 files changed, 34 insertions(+), 8 deletions(-)
arch/loongarch/Kconfig +1 -0
@@ -114,6 +114,7 @@ config LOONGARCH select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT
arch/s390/Kconfig +1 -0
@@ -170,6 +170,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT if HAVE_MARCH_Z196_FEATURES
arch/x86/Kconfig +1 -0
@@ -215,6 +215,7 @@ config X86 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64
include/linux/fprobe.h +1 -1
@@ -14,7 +14,7 @@ typedef int (*fprobe_entry_cb)(struct fprobe *fp, unsigned long entry_ip, void *entry_data);typedef void (*fprobe_exit_cb )(struct fprobe *fp,unsigned long entry_ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *regs,void *entry_data);/**
include/linux/ftrace.h +6 -0
@@ -163,6 +163,12 @@ struct ftrace_regs {#define ftrace_regs_set_instruction_pointer (fregs, ip) do { } while (0)#endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ +#ifdef CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST + +static_assert (sizeof (struct pt_regs ) ==sizeof (struct ftrace_regs )); + +#endif /* CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ + static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) {if (!fregs)
kernel/trace/Kconfig +8 -0
@@ -57,6 +57,13 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of ftrace_regs_get_argument() and ftrace_regs_get_stack_pointer(). +config HAVE_PT_REGS_TO_FTRACE_REGS_CAST + bool + help + If this is set, the memory layout of the ftrace_regs data structure + is the same as the pt_regs. So the pt_regs is possible to be casted + to ftrace_regs. + config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE bool help
@@ -288,6 +295,7 @@ config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS + depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n
kernel/trace/bpf_trace.c +5 -1
@@ -2838,10 +2838,14 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip,static void kprobe_multi_link_exit_handler (struct fprobe *fp,unsigned long fentry_ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *fregs,void *data) {struct bpf_kprobe_multi_link *link; +struct pt_regs *regs =ftrace_get_regs (fregs); + +if (!regs) +return ; link = container_of(fp, struct bpf_kprobe_multi_link, fp);kprobe_multi_link_prog_run (link,get_entry_ip (fentry_ip), regs);
kernel/trace/fprobe.c +2 -1
@@ -124,6 +124,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, {struct fprobe *fp = (struct fprobe *)data;struct fprobe_rethook_node *fpr; +struct ftrace_regs *fregs = (struct ftrace_regs *)regs;int bit;if (!fp ||fprobe_disabled (fp))
@@ -141,7 +142,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data,return ; } - fp->exit_handler (fp, fpr->entry_ip, ret_ip, regs, + fp->exit_handler (fp, fpr->entry_ip, ret_ip, fregs, fp->entry_data_size ? (void *)fpr->data :NULL );ftrace_test_recursion_unlock (bit); }
kernel/trace/trace_fprobe.c +5 -1
@@ -359,10 +359,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip,NOKPROBE_SYMBOL (fentry_dispatcher);static void fexit_dispatcher (struct fprobe *fp,unsigned long entry_ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *fregs,void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); +struct pt_regs *regs =ftrace_get_regs (fregs); + +if (!regs) +return ;if (trace_probe_test_flag (&tf->tp,TP_FLAG_TRACE ))fexit_trace_func (tf, entry_ip, ret_ip, regs, entry_data);
lib/test_fprobe.c +3 -3
@@ -59,9 +59,9 @@ static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip,unsigned long ret_ip, -struct pt_regs *regs,void *data) +struct ftrace_regs *fregs,void *data) { -unsigned long ret =regs_return_value (regs); +unsigned long ret =ftrace_regs_get_return_value (fregs);KUNIT_EXPECT_FALSE (current_test,preemptible ());if (ip != target_ip) {
@@ -89,7 +89,7 @@ static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip,unsigned long ret_ip, -struct pt_regs *regs,void *data) +struct ftrace_regs *fregs,void *data) {KUNIT_EXPECT_FALSE (current_test,preemptible ());KUNIT_EXPECT_EQ (current_test, ip, target_nest_ip);
samples/fprobe/fprobe_example.c +1 -1
@@ -67,7 +67,7 @@ static int sample_entry_handler(struct fprobe *fp, unsigned long ip, }static void sample_exit_handler (struct fprobe *fp,unsigned long ip, -unsigned long ret_ip,struct pt_regs *regs, +unsigned long ret_ip,struct ftrace_regs *regs,void *data) {unsigned long rip = ret_ip;
Masami Hiramatsu (Google)
maintainerPATCH v10 26/36 tracing: Add ftrace_partial_regs() for converting ftrace_regs to pt_regs
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Add ftrace_partial_regs() which converts the ftrace_regs to pt_regs. This is for the eBPF which needs this to keep the same pt_regs interface to access registers.
Thus when replacing the pt_regs with ftrace_regs in fprobes (which is used by kprobe_multi eBPF event), this will be used.
If the architecture defines its own ftrace_regs, this copies partial registers to pt_regs and returns it. If not, ftrace_regs is the same as pt_regs and ftrace_partial_regs() will return ftrace_regs::regs.
---
Changes in v8:
- Add the reason why this required in changelog.
Changes from previous series: NOTHING, just forward ported.
--- arch/arm64/include/asm/ftrace.h | 11 +++++++++++ include/linux/ftrace.h | 17 +++++++++++++++++ 2 files changed, 28 insertions(+)
arch/arm64/include/asm/ftrace.h +11 -0
@@ -143,6 +143,17 @@ ftrace_regs_get_frame_pointer(const struct ftrace_regs *fregs)return fregs->fp; } +static __always_inline struct pt_regs * +ftrace_partial_regs (const struct ftrace_regs *fregs,struct pt_regs *regs) +{ +memcpy (regs->regs, fregs->regs,sizeof (u64) *9 ); + regs->sp = fregs->sp; + regs->pc = fregs->pc; + regs->regs[29 ] = fregs->fp; + regs->regs[30 ] = fregs->lr; +return regs; +} +int ftrace_regs_query_register_offset (const char *name);int ftrace_init_nop (struct module *mod,struct dyn_ftrace *rec);
include/linux/ftrace.h +17 -0
@@ -177,6 +177,23 @@ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregsreturn arch_ftrace_get_regs (fregs); } +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS ) || \ + defined(CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST ) + +static __always_inline struct pt_regs * +ftrace_partial_regs (struct ftrace_regs *fregs,struct pt_regs *regs) +{ +/* +* If CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y, ftrace_regs memory +* layout is the same as pt_regs. So always returns that address. +* Since arch_ftrace_get_regs() will check some members and may return +* NULL, we can not use it. +*/ +return &fregs->regs; +} + +#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ +/* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs.
Masami Hiramatsu (Google)
maintainerPATCH v10 27/36 tracing: Add ftrace_fill_perf_regs() for perf event
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Add ftrace_fill_perf_regs() which should be compatible with the perf_fetch_caller_regs(). In other words, the pt_regs returned from the ftrace_fill_perf_regs() must satisfy 'user_mode(regs) == false' and can be used for stack tracing.
---
Changes from previous series: NOTHING, just forward ported.
--- arch/arm64/include/asm/ftrace.h | 7 +++++++ arch/powerpc/include/asm/ftrace.h | 7 +++++++ arch/s390/include/asm/ftrace.h | 5 +++++ arch/x86/include/asm/ftrace.h | 7 +++++++ include/linux/ftrace.h | 31 +++++++++++++++++++++++++++++++ 5 files changed, 57 insertions(+)
arch/arm64/include/asm/ftrace.h +7 -0
@@ -154,6 +154,13 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs)return regs; } +#define arch_ftrace_fill_perf_regs (fregs, _regs) do { \ + (_regs)->pc = (fregs)->pc; \ + (_regs)->regs[29] = (fregs)->fp; \ + (_regs)->sp = (fregs)->sp; \ + (_regs)->pstate = PSR_MODE_EL1h; \ + } while (0) +int ftrace_regs_query_register_offset (const char *name);int ftrace_init_nop (struct module *mod,struct dyn_ftrace *rec);
arch/powerpc/include/asm/ftrace.h +7 -0
@@ -44,6 +44,13 @@ static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *return fregs->regs.msr ? &fregs->regs :NULL ; } +#define arch_ftrace_fill_perf_regs (fregs, _regs) do { \ + (_regs)->result = 0; \ + (_regs)->nip = (fregs)->regs.nip; \ + (_regs)->gpr[1] = (fregs)->regs.gpr[1]; \ + asm volatile("mfmsr %0" : "=r" ((_regs)->msr)); \ + } while (0) + static __always_inline voidftrace_regs_set_instruction_pointer (struct ftrace_regs *fregs,unsigned long ip)
arch/s390/include/asm/ftrace.h +5 -0
@@ -89,6 +89,11 @@ ftrace_regs_get_frame_pointer(struct ftrace_regs *fregs)return sp[0 ];/* return backchain */ } +#define arch_ftrace_fill_perf_regs (fregs, _regs) do { \ + (_regs)->psw.addr = (fregs)->regs.psw.addr; \ + (_regs)->gprs[15] = (fregs)->regs.gprs[15]; \ + } while (0) +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * When an ftrace registered caller is tracing a function that is
arch/x86/include/asm/ftrace.h +7 -0
@@ -54,6 +54,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs)return &fregs->regs; } +#define arch_ftrace_fill_perf_regs (fregs, _regs) do { \ + (_regs)->ip = (fregs)->regs.ip; \ + (_regs)->sp = (fregs)->regs.sp; \ + (_regs)->cs = __KERNEL_CS; \ + (_regs)->flags = 0; \ + } while (0) +#define ftrace_regs_set_instruction_pointer (fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0)
include/linux/ftrace.h +31 -0
@@ -194,6 +194,37 @@ ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs)#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS + +/* +* Please define arch dependent pt_regs which compatible to the +* perf_arch_fetch_caller_regs() but based on ftrace_regs. +* This requires +* - user_mode(_regs) returns false (always kernel mode). +* - able to use the _regs for stack trace. +*/ +#ifndef arch_ftrace_fill_perf_regs +/* As same as perf_arch_fetch_caller_regs(), do nothing by default */ +#define arch_ftrace_fill_perf_regs (fregs, _regs) do {} while (0) +#endif + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs (struct ftrace_regs *fregs,struct pt_regs *regs) +{ +arch_ftrace_fill_perf_regs (fregs, regs); +return regs; +} + +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs (struct ftrace_regs *fregs,struct pt_regs *regs) +{ +return &fregs->regs; +} + +#endif +/* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs.
Masami Hiramatsu (Google)
maintainerPATCH v10 28/36 tracing/fprobe: Enable fprobe events with CONFIG_DYNAMIC_FTRACE_WITH_ARGS
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Allow fprobe events to be enabled with CONFIG_DYNAMIC_FTRACE_WITH_ARGS. With this change, fprobe events mostly use ftrace_regs instead of pt_regs. Note that if the arch doesn't enable HAVE_PT_REGS_COMPAT_FTRACE_REGS, fprobe events will not be able to be used from perf.
---
Changes in v9:
- Copy store_trace_entry_data() as store_fprobe_entry_data() for fprobe.
Chagnes in v3:
- Use ftrace_regs_get_return_value().
Changes in v2:
- Define ftrace_regs_get_kernel_stack_nth() for !CONFIG_HAVE_REGS_AND_STACK_ACCESS_API.
Changes from previous series: Update against the new series.
--- include/linux/ftrace.h | 17 ++++++ kernel/trace/Kconfig | 1 kernel/trace/trace_fprobe.c | 107 +++++++++++++++++++++++++-------------- kernel/trace/trace_probe_tmpl.h | 2 - 4 files changed, 86 insertions(+), 41 deletions(-)
include/linux/ftrace.h +17 -0
@@ -256,6 +256,23 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) frame_pointer(&(fregs)->regs)#endif +#ifdef CONFIG_HAVE_REGS_AND_STACK_ACCESS_API +static __always_inline unsigned long +ftrace_regs_get_kernel_stack_nth (struct ftrace_regs *fregs,unsigned int nth) +{ +unsigned long *stackp; + + stackp = (unsigned long *)ftrace_regs_get_stack_pointer (fregs); +if (((unsigned long )(stackp + nth) & ~(THREAD_SIZE -1 )) == + ((unsigned long )stackp & ~(THREAD_SIZE -1 ))) +return *(stackp + nth); + +return 0 ; +} +#else /* !CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ +#define ftrace_regs_get_kernel_stack_nth (fregs, nth) (0L) +#endif /* CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ +typedef void (*ftrace_func_t )(unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs);
kernel/trace/Kconfig +0 -1
@@ -680,7 +680,6 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS - depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and
kernel/trace/trace_fprobe.c +68 -39
271 lines not on this page · raw message
kernel/trace/trace_probe_tmpl.h +1 -1
@@ -232,7 +232,7 @@ process_fetch_insn_bottom(struct fetch_insn *code, unsigned long val,/* Sum up total data length for dynamic arrays (strings) */ static nokprobe_inline int -__get_data_size (struct trace_probe *tp,struct pt_regs *regs,void *edata) +__get_data_size (struct trace_probe *tp,void *regs,void *edata) {struct probe_arg *arg;int i, len, ret =0 ;
Masami Hiramatsu (Google)
maintainerPATCH v10 29/36 bpf: Enable kprobe_multi feature if CONFIG_FPROBE is enabled
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Enable kprobe_multi feature if CONFIG_FPROBE is enabled. The pt_regs is converted from ftrace_regs by ftrace_partial_regs(), thus some registers may always returns 0. But it should be enough for function entry (access arguments) and exit (access return value).
---
Changes in v9:
- Avoid wasting memory for bpf_kprobe_multi_pt_regs when CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y
--- kernel/trace/bpf_trace.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-)
kernel/trace/bpf_trace.c +14 -13
@@ -2577,7 +2577,7 @@ static int __init bpf_event_init(void)fs_initcall (bpf_event_init);#endif /* CONFIG_MODULES */ -#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) +#ifdef CONFIG_FPROBE struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp;
@@ -2600,6 +2600,13 @@ struct user_syms {char *buf; }; +#ifndef CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST +static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); +#define bpf_kprobe_multi_pt_regs_ptr () this_cpu_ptr(&bpf_kprobe_multi_pt_regs) +#else +#define bpf_kprobe_multi_pt_regs_ptr () (NULL) +#endif +static int copy_user_syms (struct user_syms *us,unsigned long __user *usyms,u32 cnt) {unsigned long __user usymbol;
@@ -2792,13 +2799,14 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)static int kprobe_multi_link_prog_run (struct bpf_kprobe_multi_link *link, -unsigned long entry_ip,struct pt_regs *regs) +unsigned long entry_ip,struct ftrace_regs *fregs) {struct bpf_kprobe_multi_run_ctx run_ctx = { .link = link, .entry_ip = entry_ip, };struct bpf_run_ctx *old_run_ctx; +struct pt_regs *regs;int err;if (unlikely (__this_cpu_inc_return (bpf_prog_active) !=1 )) {
@@ -2809,6 +2817,7 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link,migrate_disable ();rcu_read_lock (); + regs =ftrace_partial_regs (fregs,bpf_kprobe_multi_pt_regs_ptr ()); old_run_ctx =bpf_set_run_ctx (&run_ctx.run_ctx); err =bpf_prog_run (link->link.prog, regs);bpf_reset_run_ctx (old_run_ctx);
@@ -2826,13 +2835,9 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) {struct bpf_kprobe_multi_link *link; -struct pt_regs *regs =ftrace_get_regs (fregs); - -if (!regs) -return 0 ; link = container_of(fp, struct bpf_kprobe_multi_link, fp); -kprobe_multi_link_prog_run (link,get_entry_ip (fentry_ip), regs); +kprobe_multi_link_prog_run (link,get_entry_ip (fentry_ip), fregs);return 0 ; }
@@ -2842,13 +2847,9 @@ kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) {struct bpf_kprobe_multi_link *link; -struct pt_regs *regs =ftrace_get_regs (fregs); - -if (!regs) -return ; link = container_of(fp, struct bpf_kprobe_multi_link, fp); -kprobe_multi_link_prog_run (link,get_entry_ip (fentry_ip), regs); +kprobe_multi_link_prog_run (link,get_entry_ip (fentry_ip), fregs); }static int symbols_cmp_r (const void *a,const void *b,const void *priv)
@@ -3107,7 +3108,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prkvfree (cookies); return err; } -#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ +#else /* !CONFIG_FPROBE */ int bpf_kprobe_multi_link_attach (const union bpf_attr *attr,struct bpf_prog *prog) {return -EOPNOTSUPP ;
Masami Hiramatsu (Google)
maintainerPATCH v10 30/36 ftrace: Add CONFIG_HAVE_FTRACE_GRAPH_FUNC
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Add CONFIG_HAVE_FTRACE_GRAPH_FUNC kconfig in addition to ftrace_graph_func macro check. This is for the other feature (e.g. FPROBE) which requires to access ftrace_regs from fgraph_ops::entryfunc() can avoid compiling if the fgraph can not pass the valid ftrace_regs.
---
Changes in v8:
- Newly added.
--- arch/arm64/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/powerpc/Kconfig | 1 + arch/riscv/Kconfig | 1 + arch/x86/Kconfig | 1 + kernel/trace/Kconfig | 5 +++++ 6 files changed, 10 insertions(+)
arch/arm64/Kconfig +1 -0
@@ -206,6 +206,7 @@ config ARM64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FAST_GUP + select HAVE_FTRACE_GRAPH_FUNC select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION
arch/loongarch/Kconfig +1 -0
@@ -121,6 +121,7 @@ config LOONGARCH select HAVE_EFFICIENT_UNALIGNED_ACCESS if !ARCH_STRICT_ALIGN select HAVE_EXIT_THREAD select HAVE_FAST_GUP + select HAVE_FTRACE_GRAPH_FUNC select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION
arch/powerpc/Kconfig +1 -0
@@ -237,6 +237,7 @@ config PPC select HAVE_EBPF_JIT select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FAST_GUP + select HAVE_FTRACE_GRAPH_FUNC select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_DESCRIPTORS if PPC64_ELF_ABI_V1
arch/riscv/Kconfig +1 -0
@@ -127,6 +127,7 @@ config RISCV select HAVE_DYNAMIC_FTRACE if !XIP_KERNEL && MMU && (CLANG_SUPPORTS_DYNAMIC_FTRACE || GCC_SUPPORTS_DYNAMIC_FTRACE) select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS if HAVE_DYNAMIC_FTRACE + select HAVE_FTRACE_GRAPH_FUNC select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_GRAPH_FREGS
arch/x86/Kconfig +1 -0
@@ -225,6 +225,7 @@ config X86 select HAVE_EXIT_THREAD select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE + select HAVE_FTRACE_GRAPH_FUNC if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_FREGS if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_GRAPH_TRACER if X86_32 || (X86_64 && DYNAMIC_FTRACE)
kernel/trace/Kconfig +5 -0
@@ -34,6 +34,11 @@ config HAVE_FUNCTION_GRAPH_TRACER config HAVE_FUNCTION_GRAPH_FREGS bool +config HAVE_FTRACE_GRAPH_FUNC + bool + help + True if ftrace_graph_func() is defined. + config HAVE_DYNAMIC_FTRACE bool help
Masami Hiramatsu (Google)
maintainerPATCH v10 31/36 fprobe: Rewrite fprobe on function-graph tracer
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Rewrite fprobe implementation on function-graph tracer. Major API changes are:
- 'nr_maxactive' field is deprecated.
- This depends on CONFIG_DYNAMIC_FTRACE_WITH_ARGS or !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS, and CONFIG_HAVE_FUNCTION_GRAPH_FREGS. So currently works only on x86_64.
- Currently the entry size is limited in 15 * sizeof(long).
- If there is too many fprobe exit handler set on the same function, it will fail to probe.
---
Changes in v9:
- Remove unneeded prototype of ftrace_regs_get_return_address().
- Fix entry data address calculation.
- Remove DIV_ROUND_UP() from hotpath.
Changes in v8:
- Use trace_func_graph_ret/ent_t for fgraph_ops.
- Update CONFIG_FPROBE dependencies.
- Add ftrace_regs_get_return_address() for each arch.
Changes in v3:
- Update for new reserve_data/retrieve_data API.
- Fix internal push/pop on fgraph data logic so that it can correctly save/restore the returning fprobes.
Changes in v2:
- Add more lockdep_assert_held(fprobe_mutex)
- Use READ_ONCE() and WRITE_ONCE() for fprobe_hlist_node::fp.
- Add NOKPROBE_SYMBOL() for the functions which is called from entry/exit callback.
--- arch/arm64/include/asm/ftrace.h | 6 arch/loongarch/include/asm/ftrace.h | 6 arch/powerpc/include/asm/ftrace.h | 6 arch/s390/include/asm/ftrace.h | 6 arch/x86/include/asm/ftrace.h | 6 include/linux/fprobe.h | 53 ++- kernel/trace/Kconfig | 8 kernel/trace/fprobe.c | 638 +++++++++++++++++++++++++---------- lib/test_fprobe.c | 45 -- 9 files changed, 529 insertions(+), 245 deletions(-)
arch/arm64/include/asm/ftrace.h +6 -0
@@ -143,6 +143,12 @@ ftrace_regs_get_frame_pointer(const struct ftrace_regs *fregs)return fregs->fp; } +static __always_inline unsigned long +ftrace_regs_get_return_address (const struct ftrace_regs *fregs) +{ +return fregs->lr; +} + static __always_inline struct pt_regs *ftrace_partial_regs (const struct ftrace_regs *fregs,struct pt_regs *regs) {
arch/loongarch/include/asm/ftrace.h +6 -0
@@ -81,6 +81,12 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip)#define ftrace_regs_get_frame_pointer (fregs) \ ((fregs)->regs.regs[22]) +static __always_inline unsigned long +ftrace_regs_get_return_address (struct ftrace_regs *fregs) +{ +return *(unsigned long *)(fregs->regs.regs[1 ]); +} +#define ftrace_graph_func ftrace_graph_funcvoid ftrace_graph_func (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs);
arch/powerpc/include/asm/ftrace.h +6 -0
@@ -77,6 +77,12 @@ ftrace_regs_get_instruction_pointer(struct ftrace_regs *fregs)#define ftrace_regs_query_register_offset (name) \ regs_query_register_offset(name) +static __always_inline unsigned long +ftrace_regs_get_return_address (struct ftrace_regs *fregs) +{ +return fregs->regs.link; +} +struct ftrace_ops ;#define ftrace_graph_func ftrace_graph_func
arch/s390/include/asm/ftrace.h +6 -0
@@ -89,6 +89,12 @@ ftrace_regs_get_frame_pointer(struct ftrace_regs *fregs)return sp[0 ];/* return backchain */ } +static __always_inline unsigned long +ftrace_regs_get_return_address (const struct ftrace_regs *fregs) +{ +return fregs->regs.gprs[14 ]; +} +#define arch_ftrace_fill_perf_regs (fregs, _regs) do { \ (_regs)->psw.addr = (fregs)->regs.psw.addr; \ (_regs)->gprs[15] = (fregs)->regs.gprs[15]; \
arch/x86/include/asm/ftrace.h +6 -0
@@ -82,6 +82,12 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs)#define ftrace_regs_get_frame_pointer (fregs) \ frame_pointer(&(fregs)->regs) +static __always_inline unsigned long +ftrace_regs_get_return_address (struct ftrace_regs *fregs) +{ +return *(unsigned long *)ftrace_regs_get_stack_pointer (fregs); +} +struct ftrace_ops ;#define ftrace_graph_func ftrace_graph_func void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
include/linux/fprobe.h +38 -15
83 lines not on this page · raw message
kernel/trace/Kconfig +3 -5
@@ -298,11 +298,9 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" - depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_RETHOOK - select RETHOOK + depends on FUNCTION_GRAPH_TRACER + depends on HAVE_FUNCTION_GRAPH_FREGS && HAVE_FTRACE_GRAPH_FUNC + depends on DYNAMIC_FTRACE_WITH_ARGS default n help This option enables kernel function probe (fprobe) based on ftrace.
kernel/trace/fprobe.c +458 -180
768 lines not on this page · raw message
lib/test_fprobe.c +0 -45
85 lines not on this page · raw message
Masami Hiramatsu (Google)
maintainerPATCH v10 32/36 tracing/fprobe: Remove nr_maxactive from fprobe
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Remove depercated fprobe::nr_maxactive. This involves fprobe events to rejects the maxactive number.
---
Changes in v2:
- Newly added.
--- include/linux/fprobe.h | 2 -- kernel/trace/trace_fprobe.c | 44 ++++++------------------------------------- 2 files changed, 6 insertions(+), 40 deletions(-)
include/linux/fprobe.h +0 -2
@@ -54,7 +54,6 @@ struct fprobe_hlist {* @nmissed: The counter for missing events. * @flags: The status flag. * @entry_data_size: The private data storage size. -* @nr_maxactive: The max number of active functions. (*deprecated) * @entry_handler: The callback function for function entry. * @exit_handler: The callback function for function exit. * @hlist_array: The fprobe_hlist for fprobe search from IP hash table.
@@ -63,7 +62,6 @@ struct fprobe {unsigned long nmissed;unsigned int flags;size_t entry_data_size; -int nr_maxactive;fprobe_entry_cb entry_handler;fprobe_exit_cb exit_handler;
kernel/trace/trace_fprobe.c +6 -38
93 lines not on this page · raw message
Masami Hiramatsu (Google)
maintainerPATCH v10 33/36 selftests: ftrace: Remove obsolate maxactive syntax check
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Since the fprobe event does not support maxactive anymore, stop testing the maxactive syntax error checking.
--- .../ftrace/test.d/dynevent/fprobe_syntax_errors.tc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc +1 -3
@@ -16,9 +16,7 @@ aarch64) REG=%r0 ;; esac -check_error 'f^100 vfs_read' # MAXACT_NO_KPROBE -check_error 'f^1a111 vfs_read' # BAD_MAXACT -check_error 'f^100000 vfs_read' # MAXACT_TOO_BIG +check_error 'f^100 vfs_read' # BAD_MAXACT check_error 'f ^non_exist_func' # BAD_PROBE_ADDR (enoent) check_error 'f ^vfs_read+10' # BAD_PROBE_ADDR
Masami Hiramatsu (Google)
maintainerPATCH v10 34/36 selftests/ftrace: Add a test case for repeating register/unregister fprobe
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
This test case repeats define and undefine the fprobe dynamic event to ensure that the fprobe does not cause any issue with such operations.
--- .../test.d/dynevent/add_remove_fprobe_repeat.tc | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc
tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc new file +19 -0
@@ -0,0 +1,19 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: Generic dynamic event - Repeating add/remove fprobe events +# requires: dynamic_events "f[:[<group>/][<event>]] <func-name>[%return] [<args>]":README + +echo 0 > events/enable +echo > dynamic_events + +PLACE=$FUNCTION_FORK +REPEAT_TIMES=64 + +for i in `seq 1 $REPEAT_TIMES`; do + echo "f:myevent $PLACE" >> dynamic_events + grep -q myevent dynamic_events + test -d events/fprobes/myevent + echo > dynamic_events +done + +clear_trace
Masami Hiramatsu (Google)
maintainerPATCH v10 35/36 Documentation: probes: Update fprobe on function-graph tracer
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Update fprobe documentation for the new fprobe on function-graph tracer. This includes some bahvior changes and pt_regs to ftrace_regs interface change.
---
Changes in v2:
- Update @fregs parameter explanation.
--- Documentation/trace/fprobe.rst | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-)
Documentation/trace/fprobe.rst +27 -15
@@ -9,9 +9,10 @@ Fprobe - Function entry/exit probe Introduction ============ -Fprobe is a function entry/exit probe mechanism based on ftrace. -Instead of using ftrace full feature, if you only want to attach callbacks -on function entry and exit, similar to the kprobes and kretprobes, you can +Fprobe is a function entry/exit probe mechanism based on the function-graph +tracer. +Instead of tracing all functions, if you want to attach callbacks on specific +function entry and exit, similar to the kprobes and kretprobes, you can use fprobe. Compared with kprobes and kretprobes, fprobe gives faster instrumentation for multiple functions with single handler. This document describes how to use fprobe.
@@ -91,12 +92,14 @@ The prototype of the entry/exit callback function are as follows: .. code-block:: c - int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); - void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); -Note that the @entry_ip is saved at function entry and passed to exit handler. -If the entry callback function returns !0, the corresponding exit callback will be cancelled. +Note that the @entry_ip is saved at function entry and passed to exit +handler. +If the entry callback function returns !0, the corresponding exit callback +will be cancelled. @fp This is the address of `fprobe` data structure related to this handler.
@@ -112,12 +115,10 @@ If the entry callback function returns !0, the corresponding exit callback will This is the return address that the traced function will return to, somewhere in the caller. This can be used at both entry and exit. -@regs - This is the `pt_regs` data structure at the entry and exit. Note that - the instruction pointer of @regs may be different from the @entry_ip - in the entry_handler. If you need traced instruction pointer, you need - to use @entry_ip. On the other hand, in the exit_handler, the instruction - pointer of @regs is set to the current return address. +@fregs + This is the `ftrace_regs` data structure at the entry and exit. This + includes the function parameters, or the return values. So user can + access thos values via appropriate `ftrace_regs_*` APIs. @entry_data This is a local storage to share the data between entry and exit handlers.
@@ -125,6 +126,17 @@ If the entry callback function returns !0, the corresponding exit callback will and `entry_data_size` field when registering the fprobe, the storage is allocated and passed to both `entry_handler` and `exit_handler`. +Entry data size and exit handlers on the same function +====================================================== + +Since the entry data is passed via per-task stack and it is has limited size, +the entry data size per probe is limited to `15 * sizeof(long)`. You also need +to take care that the different fprobes are probing on the same function, this +limit becomes smaller. The entry data size is aligned to `sizeof(long)` and +each fprobe which has exit handler uses a `sizeof(long)` space on the stack, +you should keep the number of fprobes on the same function as small as +possible. + Share the callbacks with kprobes ================================
@@ -165,8 +177,8 @@ This counter counts up when; - fprobe fails to take ftrace_recursion lock. This usually means that a function which is traced by other ftrace users is called from the entry_handler. - - fprobe fails to setup the function exit because of the shortage of rethook - (the shadow stack for hooking the function return.) + - fprobe fails to setup the function exit because of failing to allocate the + data buffer from the per-task shadow stack. The `fprobe::nmissed` field counts up in both cases. Therefore, the former skips both of entry and exit callback and the latter skips the exit
Masami Hiramatsu (Google)
maintainerPATCH v10 36/36 fgraph: Skip recording calltime/rettime if it is not nneeded
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Skip recording calltime and rettime if the fgraph_ops does not need it. This is a kind of performance optimization for fprobe. Since the fprobe user does not use these entries, recording timestamp in fgraph is just a overhead (e.g. eBPF, ftrace). So introduce the skip_timestamp flag, and all fgraph_ops sets this flag, skip recording calltime and rettime.
---
Changes in v10:
- Add likely() to skipping timestamp.
Changes in v9:
- Newly added.
--- include/linux/ftrace.h | 2 ++ kernel/trace/fgraph.c | 51 +++++++++++++++++++++++++++++++++++++++++------- kernel/trace/fprobe.c | 1 + 3 files changed, 47 insertions(+), 7 deletions(-)
include/linux/ftrace.h +2 -0
@@ -1156,6 +1156,8 @@ struct fgraph_ops {struct ftrace_ops ops;/* for the hash lists */ void *private;int idx; +/* If skip_timestamp is true, this does not record timestamps. */ +bool skip_timestamp; };void *fgraph_reserve_data (int idx,int size_bytes);
kernel/trace/fgraph.c +44 -7
131 lines not on this page · raw message
kernel/trace/fprobe.c +1 -0
@@ -345,6 +345,7 @@ NOKPROBE_SYMBOL(fprobe_return);static struct fgraph_ops fprobe_graph_ops = { .entryfunc = fprobe_entry, .retfunc = fprobe_return, + .skip_timestamp = true, };static int fprobe_graph_active;
Steven Rostedt
maintainerQuoted 8 lines · On Tue, 7 May 2024 23:08:00 +0900 "Masami Hiramatsu (Google)"…
Steven Rostedt (VMware) (15):
function_graph: Convertret_stackto a series of longs
fgraph: UseBUILD_BUG_ON()to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic aroundftrace_graph_entryand return ftrace/function_graph: Passfgraph_opsto function graph callbacks
ftrace: Allowfunction_graphtracer to be enabled in instances
Quoted 8 lines
ftrace: Allow ftrace startup flags exist without dynamic ftrace
function_graph: Have the instances use their ownftrace_opsfor filtering
function_graph: Add "task variables" per task forfgraph_ops
function_graph: Moveset_graph_functiontests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implementfgraph_reserve_data()andfgraph_retrieve_data()
function_graph: Add selftest for passing local variables
Hi Masami,
While reviewing these patches, I realized there's several things I dislike about the patches I wrote. So I took these patches and started cleaning them up a little. Mostly renaming functions and adding comments.
As this is a major change to the function graph tracer, and I feel nervous about building something on top of this, how about I take over these patches and push them out for the next merge window. I'm hoping to get them into linux-next by v6.10-rc2 (I spent the day working on them, and it's mostly minor tweaks).
Then I can push it out to 6.11 and get some good testing against it. Then we can add your stuff on top and get that merged in 6.12.
If all goes well, I'm hoping to get a series on just these patches (and your selftest addition) by tonight.
Thoughts?
-- Steve
Masami Hiramatsu (Google)
maintainerQuoted 20 lines · On Fri, 24 May 2024 18:41:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 8 lines · On Tue, 7 May 2024 23:08:00 +0900 "Masami Hiramatsu (Google)"…
Steven Rostedt (VMware) (15):
function_graph: Convertret_stackto a series of longs
fgraph: UseBUILD_BUG_ON()to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic aroundftrace_graph_entryand return ftrace/function_graph: Passfgraph_opsto function graph callbacks
ftrace: Allowfunction_graphtracer to be enabled in instancesQuoted 8 lines
ftrace: Allow ftrace startup flags exist without dynamic ftrace
function_graph: Have the instances use their ownftrace_opsfor filtering
function_graph: Add "task variables" per task forfgraph_ops
function_graph: Moveset_graph_functiontests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implementfgraph_reserve_data()andfgraph_retrieve_data()
function_graph: Add selftest for passing local variables
Quoted 5 lines
Hi Masami,
While reviewing these patches, I realized there's several things I dislike about the patches I wrote. So I took these patches and started cleaning them up a little. Mostly renaming functions and adding comments.
Thanks for cleaning up the patches!!
Quoted 6 lines
As this is a major change to the function graph tracer, and I feel nervous about building something on top of this, how about I take over these patches and push them out for the next merge window. I'm hoping to get them into linux-next by v6.10-rc2 (I spent the day working on them, and it's mostly minor tweaks).
OK.
Quoted 2 lines
Then I can push it out to 6.11 and get some good testing against it. Then we can add your stuff on top and get that merged in 6.12.
Yeah, it is reasonable plan. I also concerns about the stability. Especially, this involves fprobe side changes too. If we introduce both at once, it may mess up many things.
Quoted 5 lines
If all goes well, I'm hoping to get a series on just these patches (and your selftest addition) by tonight.
Thoughts?
I agree with you.
Thank you,
Quoted 2 lines
-- Steve
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>