ftrace: Show all functions with addresses in available_filter_functions_addrs
Subsystems (4)
Jiri Olsa
reviewerPATCH v2 1/1 ftrace: Show all functions with addresses in available_filter_functions_addrs
Adding new available_filter_functions_addrs file that shows all available functions (same as available_filter_functions) together with addresses, like:
# cat available_filter_functions_addrs | head ffffffff81000770 __traceiter_initcall_level ffffffff810007c0 __traceiter_initcall_start ffffffff81000810 __traceiter_initcall_finish ffffffff81000860 trace_initcall_finish_cb ...
Note displayed address is the patch-site address and can differ from /proc/kallsyms address.
It's useful to have address avilable for traceable symbols, so we don't need to allways cross check kallsyms with available_filter_functions (or the other way around) and have all the data in single file.
For backwards compatibility reasons we can't change the existing available_filter_functions file output, but we need to add new file.
--- Documentation/trace/ftrace.rst | 6 ++++++ include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 37 ++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+)
v2 changes:
- simplified address print [Steven]
- added doc entry for the new file
Documentation/trace/ftrace.rst +6 -0
@@ -324,6 +324,12 @@ of ftrace. Here is a list of some of the key files: "set_graph_function", or "set_graph_notrace". (See the section "dynamic ftrace" below for more details.) + available_filter_functions_addrs: + + Similar to available_filter_functions, but with address displayed + for each function. The displayed address is the patch-site address + and can differ from /proc/kallsyms address. + dyn_ftrace_total_info: This file is for debugging purposes. The number of functions that
include/linux/ftrace.h +1 -0
@@ -633,6 +633,7 @@ enum {FTRACE_ITER_MOD = (1 <<5 ),FTRACE_ITER_ENABLED = (1 <<6 ),FTRACE_ITER_TOUCHED = (1 <<7 ), + FTRACE_ITER_ADDRS = (1 << 8), };void arch_ftrace_update_code (int command);
kernel/trace/ftrace.c +37 -0
@@ -3861,6 +3861,9 @@ static int t_show(struct seq_file *m, void *v)if (!rec)return 0 ; +if (iter->flags &FTRACE_ITER_ADDRS ) +seq_printf (m,"%lx " , rec->ip); +if (print_rec (m, rec->ip)) {/* This should only happen when a rec is disabled */ WARN_ON_ONCE (!(rec->flags &FTRACE_FL_DISABLED ));
@@ -3996,6 +3999,30 @@ ftrace_touched_open(struct inode *inode, struct file *file)return 0 ; } +static int +ftrace_avail_addrs_open (struct inode *inode,struct file *file) +{ +struct ftrace_iterator *iter; +int ret; + + ret =security_locked_down (LOCKDOWN_TRACEFS ); +if (ret) +return ret; + +if (unlikely (ftrace_disabled)) +return -ENODEV ; + + iter =__seq_open_private (file, &show_ftrace_seq_ops,sizeof (*iter)); +if (!iter) +return -ENOMEM ; + + iter->pg = ftrace_pages_start; + iter->flags =FTRACE_ITER_ADDRS ; + iter->ops = &global_ops; + +return 0 ; +} +/** * ftrace_regex_open - initialize function tracer filter files * @ops: The ftrace_ops that hold the hash filters
@@ -5916,6 +5943,13 @@ static const struct file_operations ftrace_touched_fops = { .release = seq_release_private, }; +static const struct file_operations ftrace_avail_addrs_fops = { + .open = ftrace_avail_addrs_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release_private, +}; +static const struct file_operations ftrace_filter_fops = { .open = ftrace_filter_open, .read = seq_read,
@@ -6377,6 +6411,9 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)trace_create_file ("available_filter_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_avail_fops); +trace_create_file ("available_filter_functions_addrs" ,TRACE_MODE_READ , + d_tracer,NULL , &ftrace_avail_addrs_fops); +trace_create_file ("enabled_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_enabled_fops);
Signature · 1 line
2.40.1
Quoted 11 lines · Jiri Olsa
Adding new
available_filter_functions_addrsfile that shows all available functions (same asavailable_filter_functions) together with addresses, like:# cat available_filter_functions_addrs | head ffffffff81000770 __traceiter_initcall_level ffffffff810007c0 __traceiter_initcall_start ffffffff81000810 __traceiter_initcall_finish ffffffff81000860 trace_initcall_finish_cb ...
Quoted 2 lines
Note displayed address is the patch-site address and can differ from
/proc/kallsymsaddress.
Could you explain how these addresses will be used in kernel, esp. since these addresses are different from kallsyms addresses?
Also, if there are multiple same static functions with different addresses, user space might need to check dwarf or proposed BTF_KIND_KFUNC (which encode kallsyms addresses) to find whether entry in available_filter_functions_addrs to be used. But addresses may not match. How this issue could be resolved?
Quoted 119 lines
It's useful to have address avilable for traceable symbols, so we don't need to allways cross check kallsyms with
available_filter_functions(or the other way around) and have all the data in single file.For backwards compatibility reasons we can't change the existing
available_filter_functionsfile output, but we need to add new file.Suggested-by Steven Rostedt (Google) <[email protected]>Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- Documentation/trace/ftrace.rst | 6 ++++++ include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 37 ++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+)v2 changes:
- simplified address print [Steven]
- added doc entry for the new fileDocumentation/trace/ftrace.rst +6
-0@@ -324,6 +324,12 @@ of ftrace. Here is a list of some of the key files: "set_graph_function", or "set_graph_notrace". (See the section "dynamic ftrace" below for more details.) + available_filter_functions_addrs: + + Similar to available_filter_functions, but with address displayed + for each function. The displayed address is the patch-site address + and can differ from /proc/kallsyms address. + dyn_ftrace_total_info: This file is for debugging purposes. The number of functions thatinclude/linux/ftrace.h +1
-0@@ -633,6 +633,7 @@ enum {FTRACE_ITER_MOD = (1 <<5 ),FTRACE_ITER_ENABLED = (1 <<6 ),FTRACE_ITER_TOUCHED = (1 <<7 ), + FTRACE_ITER_ADDRS = (1 << 8), };void arch_ftrace_update_code (int command);kernel/trace/ftrace.c +37
-0@@ -3861,6 +3861,9 @@ static int t_show(struct seq_file *m, void *v)if (!rec)return 0 ; +if (iter->flags &FTRACE_ITER_ADDRS ) +seq_printf (m,"%lx " , rec->ip); +if (print_rec (m, rec->ip)) {/* This should only happen when a rec is disabled */ WARN_ON_ONCE (!(rec->flags &FTRACE_FL_DISABLED ));@@ -3996,6 +3999,30 @@ ftrace_touched_open(struct inode *inode, struct file *file)return 0 ; } +static int +ftrace_avail_addrs_open (struct inode *inode,struct file *file) +{ +struct ftrace_iterator *iter; +int ret; + + ret =security_locked_down (LOCKDOWN_TRACEFS ); +if (ret) +return ret; + +if (unlikely (ftrace_disabled)) +return -ENODEV ; + + iter =__seq_open_private (file, &show_ftrace_seq_ops,sizeof (*iter)); +if (!iter) +return -ENOMEM ; + + iter->pg = ftrace_pages_start; + iter->flags =FTRACE_ITER_ADDRS ; + iter->ops = &global_ops; + +return 0 ; +} +/** * ftrace_regex_open - initialize function tracer filter files * @ops: The ftrace_ops that hold the hash filters @@ -5916,6 +5943,13 @@ static const struct file_operations ftrace_touched_fops = { .release = seq_release_private, }; +static const struct file_operations ftrace_avail_addrs_fops = { + .open = ftrace_avail_addrs_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release_private, +}; +static const struct file_operations ftrace_filter_fops = { .open = ftrace_filter_open, .read = seq_read,@@ -6377,6 +6411,9 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)trace_create_file ("available_filter_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_avail_fops); +trace_create_file ("available_filter_functions_addrs" ,TRACE_MODE_READ , + d_tracer,NULL , &ftrace_avail_addrs_fops); +trace_create_file ("enabled_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_enabled_fops);
Steven Rostedt
maintainerQuoted 5 lines · On Sun, 11 Jun 2023 17:22:29 -0700 Yonghong Song <[email protected]> wrote:
Quoted 2 lines
Note displayed address is the patch-site address and can differ from
/proc/kallsymsaddress.Could you explain how these addresses will be used in kernel, esp. since these addresses are different from kallsyms addresses?
These are the addresses that will be patched by the ftrace infrastructure. It's more efficient to use these addresses when setting a function filter with ftrace_set_filter_ip(), as it doesn't need to call into the kallsyms infrastructure to look to see if the passed in ip lands on a function. That is, if rec->ip matches the ip, it uses it. If you look at ftrace_location() (which is used to find the record to be patched), it first does a search of these addresses. If it doesn't find it, it then has to call into kallsyms to find the start and end of the function, and do another search into that range.
Quoted 7 lines
Also, if there are multiple same static functions with different addresses, user space might need to check dwarf or proposed
BTF_KIND_KFUNC(which encode kallsyms addresses) to find whether entry inavailable_filter_functions_addrsto be used. But addresses may not match. How this issue could be resolved?
Easy, you use the address between two other addresses in kallsyms. The address is still in the function. The addresses in kallsyms is the starting address, but there's cases that the patch location is not at the start.
-- Steve
Steven Rostedt
maintainerQuoted 10 lines · On Sun, 11 Jun 2023 22:54:07 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 6 lines
Also, if there are multiple same static functions with different addresses, user space might need to check dwarf or proposed
BTF_KIND_KFUNC(which encode kallsyms addresses) to find whether entry inavailable_filter_functions_addrsto be used. But addresses may not match. How this issue could be resolved?Easy, you use the address between two other addresses in kallsyms. The address is still in the function. The addresses in kallsyms is the starting address, but there's cases that the patch location is not at the start.
Not to mention, you can still use the kallsyms address. If you did the work to find it, then use it (it may not be as efficient as I mentioned before). That's basically what is done today (so I am told), and this patch was to create a file where you don't need to look up kallsyms when you know which function to use. The functions are sorted by address, so if you know of a unique function near the duplicate, you just find the duplicate that's near the unique function name.
-- Steve
Quoted 14 lines · Steven Rostedt
Quoted 10 lines · On Sun, 11 Jun 2023 22:54:07 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 6 lines
Also, if there are multiple same static functions with different addresses, user space might need to check dwarf or proposed
BTF_KIND_KFUNC(which encode kallsyms addresses) to find whether entry inavailable_filter_functions_addrsto be used. But addresses may not match. How this issue could be resolved?Easy, you use the address between two other addresses in kallsyms. The address is still in the function. The addresses in kallsyms is the starting address, but there's cases that the patch location is not at the start.
Quoted 7 lines
Not to mention, you can still use the kallsyms address. If you did the work to find it, then use it (it may not be as efficient as I mentioned before). That's basically what is done today (so I am told), and this patch was to create a file where you don't need to look up kallsyms when you know which function to use. The functions are sorted by address, so if you know of a unique function near the duplicate, you just find the duplicate that's near the unique function name.
Thanks for explanation. IIUC, typically with endbr enabled, the patch address typically is the entry_addr + 4. So it is indeed easy to correlate the entry in available_filter_functions_addrs and in kallsyms.
I am actually interested in how available_filter_functions_addrs will be used. For example, bpf_program__attach_kprobe_multi_opts() can already take addresses from kallsyms. How to use available_filter_functions_addrs to facilitate kprobe_multi? Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
Quoted 2 lines
-- Steve
Steven Rostedt
maintainerQuoted 6 lines · On Mon, 12 Jun 2023 07:49:53 -0700 Yonghong Song <[email protected]> wrote:
I am actually interested in how
available_filter_functions_addrswill be used. For example,bpf_program__attach_kprobe_multi_opts()can already take addresses from kallsyms. How to useavailable_filter_functions_addrsto facilitatekprobe_multi? Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
I agree, having that information would also be useful to me.
Jiri? Andrii?
-- Steve
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Mon, 12 Jun 2023 07:49:53 -0700 Yonghong Song <[email protected]> wrote:
I am actually interested in how
available_filter_functions_addrswill be used. For example,bpf_program__attach_kprobe_multi_opts()can already take addresses from kallsyms. How to useavailable_filter_functions_addrsto facilitatekprobe_multi?
the problem is that we need to do 2 passes:
- through available_filter_functions and find out if the function is traceable
- through /proc/kallsyms to get the address for traceable function
having available_filter_functions symbols together with addresses allow us to skip the kallsyms step
and we are ok with the address in available_filter_functions_addr not being the function entry, because kprobe_multi uses fprobe and that handles both entry and patch-site address properly
Quoted 6 lines
Quoted 2 lines
Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
I agree, having that information would also be useful to me.
Jiri? Andrii?
so we have 2 interfaces how to create kprobe_multi link:
a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through available_filter_functions file
2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
-> ftrace_set_filter_ips
b) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through available_filter_functions file
2) user takes those symbols and translates them to addresses through /proc/kallsyms
3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
-> ftrace_set_filter_ips
The new available_filter_functions_addrs file helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.
I tested the new available_filter_functions_addrs changes with some hacked selftest changes, you can check it in here [1].
I assume Jackie Liu will send new version of her patchset [2] based on this new available_filter_functions_addrs file.
I think we should have these changes coming together and add some perf measurements from before and after to make the benefit apparent.
jirka
[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=bpf/avail_addrs&id=fecaeeaf40bae034715ab2e9a46ca1dc16371e8e
[2] https://lore.kernel.org/bpf/[email protected]/#r
Quoted 62 lines · Jiri Olsa
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Mon, 12 Jun 2023 07:49:53 -0700 Yonghong Song <[email protected]> wrote:
I am actually interested in how
available_filter_functions_addrswill be used. For example,bpf_program__attach_kprobe_multi_opts()can already take addresses from kallsyms. How to useavailable_filter_functions_addrsto facilitatekprobe_multi?the problem is that we need to do 2 passes:
- through
available_filter_functionsand find out if the function is traceable
- through/proc/kallsymsto get the address for traceable functionhaving
available_filter_functionssymbols together with addresses allow us to skip the kallsyms stepand we are ok with the address in
available_filter_functions_addrnot being the function entry, becausekprobe_multiuses fprobe and that handles both entry and patch-site address properlyQuoted 6 lines
Quoted 2 lines
Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
I agree, having that information would also be useful to me.
Jiri? Andrii?
so we have 2 interfaces how to create
kprobe_multilink:a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through
available_filter_functionsfile2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
->ftrace_set_filter_ipsb) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through
available_filter_functionsfile2) user takes those symbols and translates them to addresses through
/proc/kallsyms3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
->ftrace_set_filter_ipsThe new
available_filter_functions_addrsfile helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.
Quoted 9 lines
I tested the new
available_filter_functions_addrschanges with some hacked selftest changes, you can check it in here [1].I assume Jackie Liu will send new version of her patchset [2] based on this new
available_filter_functions_addrsfile.I think we should have these changes coming together and add some perf measurements from before and after to make the benefit apparent.
If Steven would be ok with it, can we land this change through the bpf-next tree? Then we can have BPF selftest added in the same patch set that parses a new file and uses bpf_program__attach_kprobe_multi() to attach using explicit addresses.
This should make it clear to everyone how this is meant to be used and will be a good test that everything works end-to-end.
Steven Rostedt
maintainerQuoted 8 lines · On Mon, 12 Jun 2023 16:28:49 -0700 Andrii Nakryiko <[email protected]>…
If Steven would be ok with it, can we land this change through the bpf-next tree? Then we can have BPF selftest added in the same patch set that parses a new file and uses
bpf_program__attach_kprobe_multi()to attach using explicit addresses.This should make it clear to everyone how this is meant to be used and will be a good test that everything works end-to-end.
This touches some of the code I'm working with, so I rather have it be in my tree.
-- Steve
Quoted 61 lines · Jiri Olsa
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Mon, 12 Jun 2023 07:49:53 -0700 Yonghong Song <[email protected]> wrote:
I am actually interested in how
available_filter_functions_addrswill be used. For example,bpf_program__attach_kprobe_multi_opts()can already take addresses from kallsyms. How to useavailable_filter_functions_addrsto facilitatekprobe_multi?the problem is that we need to do 2 passes:
- through
available_filter_functionsand find out if the function is traceable
- through/proc/kallsymsto get the address for traceable functionhaving
available_filter_functionssymbols together with addresses allow us to skip the kallsyms stepand we are ok with the address in
available_filter_functions_addrnot being the function entry, becausekprobe_multiuses fprobe and that handles both entry and patch-site address properlyQuoted 6 lines
Quoted 2 lines
Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
I agree, having that information would also be useful to me.
Jiri? Andrii?
so we have 2 interfaces how to create
kprobe_multilink:a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through
available_filter_functionsfile2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
->ftrace_set_filter_ipsb) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through
available_filter_functionsfile2) user takes those symbols and translates them to addresses through
/proc/kallsyms3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
->ftrace_set_filter_ipsThe new
available_filter_functions_addrsfile helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.
Quoted 5 lines
I tested the new
available_filter_functions_addrschanges with some hacked selftest changes, you can check it in here [1].I assume Jackie Liu will send new version of her patchset [2] based on this new
available_filter_functions_addrsfile.
Yes, once the new interface is released, I will release a v2 version patch based on it.
Signature · 1 line
Jackie Liu
Quoted 9 lines
I think we should have these changes coming together and add some perf measurements from before and after to make the benefit apparent.
jirka
[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=bpf/avail_addrs&id=fecaeeaf40bae034715ab2e9a46ca1dc16371e8e
[2] https://lore.kernel.org/bpf/[email protected]/#r
Quoted 14 lines · Jiri Olsa
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Mon, 12 Jun 2023 07:49:53 -0700 Yonghong Song <[email protected]> wrote:
I am actually interested in how
available_filter_functions_addrswill be used. For example,bpf_program__attach_kprobe_multi_opts()can already take addresses from kallsyms. How to useavailable_filter_functions_addrsto facilitatekprobe_multi?the problem is that we need to do 2 passes:
- through
available_filter_functionsand find out if the function is traceable
- through/proc/kallsymsto get the address for traceable function
Quoted 6 lines
having
available_filter_functionssymbols together with addresses allow us to skip the kallsyms stepand we are ok with the address in
available_filter_functions_addrnot being the function entry, becausekprobe_multiuses fprobe and that handles both entry and patch-site address properly
Thanks for explanation! It would be great if we can put more details in this email into the commit message!
Quoted 55 lines
Quoted 7 lines
Quoted 3 lines
Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
I agree, having that information would also be useful to me.
Jiri? Andrii?
so we have 2 interfaces how to create
kprobe_multilink:a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through
available_filter_functionsfile2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
->ftrace_set_filter_ipsb) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through
available_filter_functionsfile2) user takes those symbols and translates them to addresses through
/proc/kallsyms3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
->ftrace_set_filter_ipsThe new
available_filter_functions_addrsfile helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.I tested the new
available_filter_functions_addrschanges with some hacked selftest changes, you can check it in here [1].I assume Jackie Liu will send new version of her patchset [2] based on this new
available_filter_functions_addrsfile.I think we should have these changes coming together and add some perf measurements from before and after to make the benefit apparent.
jirka
[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=bpf/avail_addrs&id=fecaeeaf40bae034715ab2e9a46ca1dc16371e8e
[2] https://lore.kernel.org/bpf/[email protected]/#r
Steven Rostedt
maintainerQuoted 2 lines · On Mon, 12 Jun 2023 22:04:28 -0700 Yonghong Song <[email protected]> wrote:
Thanks for explanation! It would be great if we can put more details in this email into the commit message!
I agree.
This is the patch I just pulled into my queue:
From: Jiri Olsa <[email protected]>
Date: Sun, 11 Jun 2023 15:00:29 +0200
Subject: [PATCH] ftrace: Show all functions with addresses in
available_filter_functions_addrs
Adding new available_filter_functions_addrs file that shows all available functions (same as available_filter_functions) together with addresses, like:
# cat available_filter_functions_addrs | head ffffffff81000770 __traceiter_initcall_level ffffffff810007c0 __traceiter_initcall_start ffffffff81000810 __traceiter_initcall_finish ffffffff81000860 trace_initcall_finish_cb ...
Note displayed address is the patch-site address and can differ from /proc/kallsyms address.
It's useful to have address avilable for traceable symbols, so we don't need to allways cross check kallsyms with available_filter_functions (or the other way around) and have all the data in single file.
For backwards compatibility reasons we can't change the existing available_filter_functions file output, but we need to add new file.
The problem is that we need to do 2 passes:
- through available_filter_functions and find out if the function is traceable
- through /proc/kallsyms to get the address for traceable function
Having available_filter_functions symbols together with addresses allow us to skip the kallsyms step and we are ok with the address in available_filter_functions_addr not being the function entry, because kprobe_multi uses fprobe and that handles both entry and patch-site address properly.
We have 2 interfaces how to create kprobe_multi link:
a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through available_filter_functions file
2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
-> ftrace_set_filter_ips
b) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through available_filter_functions file
2) user takes those symbols and translates them to addresses through /proc/kallsyms
3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
-> ftrace_set_filter_ips
The new available_filter_functions_addrs file helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.
--- Documentation/trace/ftrace.rst | 6 ++++++ include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 37 ++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+)
Documentation/trace/ftrace.rst +6 -0
@@ -324,6 +324,12 @@ of ftrace. Here is a list of some of the key files: "set_graph_function", or "set_graph_notrace". (See the section "dynamic ftrace" below for more details.) + available_filter_functions_addrs: + + Similar to available_filter_functions, but with address displayed + for each function. The displayed address is the patch-site address + and can differ from /proc/kallsyms address. + dyn_ftrace_total_info: This file is for debugging purposes. The number of functions that
include/linux/ftrace.h +1 -0
@@ -633,6 +633,7 @@ enum {FTRACE_ITER_MOD = (1 <<5 ),FTRACE_ITER_ENABLED = (1 <<6 ),FTRACE_ITER_TOUCHED = (1 <<7 ), + FTRACE_ITER_ADDRS = (1 << 8), };void arch_ftrace_update_code (int command);
kernel/trace/ftrace.c +37 -0
@@ -3861,6 +3861,9 @@ static int t_show(struct seq_file *m, void *v)if (!rec)return 0 ; +if (iter->flags &FTRACE_ITER_ADDRS ) +seq_printf (m,"%lx " , rec->ip); +if (print_rec (m, rec->ip)) {/* This should only happen when a rec is disabled */ WARN_ON_ONCE (!(rec->flags &FTRACE_FL_DISABLED ));
@@ -3996,6 +3999,30 @@ ftrace_touched_open(struct inode *inode, struct file *file)return 0 ; } +static int +ftrace_avail_addrs_open (struct inode *inode,struct file *file) +{ +struct ftrace_iterator *iter; +int ret; + + ret =security_locked_down (LOCKDOWN_TRACEFS ); +if (ret) +return ret; + +if (unlikely (ftrace_disabled)) +return -ENODEV ; + + iter =__seq_open_private (file, &show_ftrace_seq_ops,sizeof (*iter)); +if (!iter) +return -ENOMEM ; + + iter->pg = ftrace_pages_start; + iter->flags =FTRACE_ITER_ADDRS ; + iter->ops = &global_ops; + +return 0 ; +} +/** * ftrace_regex_open - initialize function tracer filter files * @ops: The ftrace_ops that hold the hash filters
@@ -5916,6 +5943,13 @@ static const struct file_operations ftrace_touched_fops = { .release = seq_release_private, }; +static const struct file_operations ftrace_avail_addrs_fops = { + .open = ftrace_avail_addrs_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release_private, +}; +static const struct file_operations ftrace_filter_fops = { .open = ftrace_filter_open, .read = seq_read,
@@ -6377,6 +6411,9 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)trace_create_file ("available_filter_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_avail_fops); +trace_create_file ("available_filter_functions_addrs" ,TRACE_MODE_READ , + d_tracer,NULL , &ftrace_avail_addrs_fops); +trace_create_file ("enabled_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_enabled_fops);
Signature · 1 line
2.39.2
Quoted 9 lines · Steven Rostedt
Quoted 2 lines · On Mon, 12 Jun 2023 22:04:28 -0700 Yonghong Song <[email protected]> wrote:
Thanks for explanation! It would be great if we can put more details in this email into the commit message!
I agree.
This is the patch I just pulled into my queue:
great, thanks
jirka
Quoted 187 lines
From: Jiri Olsa <[email protected]>
Date: Sun, 11 Jun 2023 15:00:29 +0200
Subject: [PATCH] ftrace: Show all functions with addresses in
available_filter_functions_addrsAdding new
available_filter_functions_addrsfile that shows all available functions (same asavailable_filter_functions) together with addresses, like:# cat available_filter_functions_addrs | head ffffffff81000770 __traceiter_initcall_level ffffffff810007c0 __traceiter_initcall_start ffffffff81000810 __traceiter_initcall_finish ffffffff81000860 trace_initcall_finish_cb ...Note displayed address is the patch-site address and can differ from
/proc/kallsymsaddress.It's useful to have address avilable for traceable symbols, so we don't need to allways cross check kallsyms with
available_filter_functions(or the other way around) and have all the data in single file.For backwards compatibility reasons we can't change the existing
available_filter_functionsfile output, but we need to add new file.The problem is that we need to do 2 passes:
- through
available_filter_functionsand find out if the function is traceable
- through/proc/kallsymsto get the address for traceable functionHaving
available_filter_functionssymbols together with addresses allow us to skip the kallsyms step and we are ok with the address inavailable_filter_functions_addrnot being the function entry, becausekprobe_multiuses fprobe and that handles both entry and patch-site address properly.We have 2 interfaces how to create
kprobe_multilink:a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through
available_filter_functionsfile2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
->ftrace_set_filter_ipsb) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through
available_filter_functionsfile2) user takes those symbols and translates them to addresses through
/proc/kallsyms3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
->ftrace_set_filter_ipsThe new
available_filter_functions_addrsfile helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.Cc Masami Hiramatsu <[email protected]>Cc Mark Rutland <[email protected]>Cc Andrii Nakryiko <[email protected]>Cc Jackie Liu <[email protected]>Suggested-by Steven Rostedt (Google) <[email protected]>Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Jiri Olsa <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>--- Documentation/trace/ftrace.rst | 6 ++++++ include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 37 ++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+)Documentation/trace/ftrace.rst +6
-0@@ -324,6 +324,12 @@ of ftrace. Here is a list of some of the key files: "set_graph_function", or "set_graph_notrace". (See the section "dynamic ftrace" below for more details.) + available_filter_functions_addrs: + + Similar to available_filter_functions, but with address displayed + for each function. The displayed address is the patch-site address + and can differ from /proc/kallsyms address. + dyn_ftrace_total_info: This file is for debugging purposes. The number of functions thatinclude/linux/ftrace.h +1
-0@@ -633,6 +633,7 @@ enum {FTRACE_ITER_MOD = (1 <<5 ),FTRACE_ITER_ENABLED = (1 <<6 ),FTRACE_ITER_TOUCHED = (1 <<7 ), + FTRACE_ITER_ADDRS = (1 << 8), };void arch_ftrace_update_code (int command);kernel/trace/ftrace.c +37
-0@@ -3861,6 +3861,9 @@ static int t_show(struct seq_file *m, void *v)if (!rec)return 0 ; +if (iter->flags &FTRACE_ITER_ADDRS ) +seq_printf (m,"%lx " , rec->ip); +if (print_rec (m, rec->ip)) {/* This should only happen when a rec is disabled */ WARN_ON_ONCE (!(rec->flags &FTRACE_FL_DISABLED ));@@ -3996,6 +3999,30 @@ ftrace_touched_open(struct inode *inode, struct file *file)return 0 ; } +static int +ftrace_avail_addrs_open (struct inode *inode,struct file *file) +{ +struct ftrace_iterator *iter; +int ret; + + ret =security_locked_down (LOCKDOWN_TRACEFS ); +if (ret) +return ret; + +if (unlikely (ftrace_disabled)) +return -ENODEV ; + + iter =__seq_open_private (file, &show_ftrace_seq_ops,sizeof (*iter)); +if (!iter) +return -ENOMEM ; + + iter->pg = ftrace_pages_start; + iter->flags =FTRACE_ITER_ADDRS ; + iter->ops = &global_ops; + +return 0 ; +} +/** * ftrace_regex_open - initialize function tracer filter files * @ops: The ftrace_ops that hold the hash filters @@ -5916,6 +5943,13 @@ static const struct file_operations ftrace_touched_fops = { .release = seq_release_private, }; +static const struct file_operations ftrace_avail_addrs_fops = { + .open = ftrace_avail_addrs_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release_private, +}; +static const struct file_operations ftrace_filter_fops = { .open = ftrace_filter_open, .read = seq_read,@@ -6377,6 +6411,9 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)trace_create_file ("available_filter_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_avail_fops); +trace_create_file ("available_filter_functions_addrs" ,TRACE_MODE_READ , + d_tracer,NULL , &ftrace_avail_addrs_fops); +trace_create_file ("enabled_functions" ,TRACE_MODE_READ , d_tracer,NULL , &ftrace_enabled_fops);Signature · 1 line
2.39.2
Quoted 61 lines · Jiri Olsa
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Mon, 12 Jun 2023 07:49:53 -0700 Yonghong Song <[email protected]> wrote:
I am actually interested in how
available_filter_functions_addrswill be used. For example,bpf_program__attach_kprobe_multi_opts()can already take addresses from kallsyms. How to useavailable_filter_functions_addrsto facilitatekprobe_multi?the problem is that we need to do 2 passes:
- through
available_filter_functionsand find out if the function is traceable
- through/proc/kallsymsto get the address for traceable functionhaving
available_filter_functionssymbols together with addresses allow us to skip the kallsyms stepand we are ok with the address in
available_filter_functions_addrnot being the function entry, becausekprobe_multiuses fprobe and that handles both entry and patch-site address properlyQuoted 6 lines
Quoted 2 lines
Do we need to change kernel APIs? It would be great at least we got a RFC patch to answer these questions.
I agree, having that information would also be useful to me.
Jiri? Andrii?
so we have 2 interfaces how to create
kprobe_multilink:a) passing symbols to kernel
1) user gathers symbols and need to ensure that they are trace-able -> pass through
available_filter_functionsfile2) kernel takes those symbols and translates them to addresses through kallsyms api
3) addresses are passed to fprobe/ftrace through:
register_fprobe_ips
->ftrace_set_filter_ipsb) passing addresses to kernel
1) user gathers symbols and needs to ensure that they are trace-able -> pass through
available_filter_functionsfile2) user takes those symbols and translates them to addresses through
/proc/kallsyms3) addresses are passed to the kernel and kernel calls:
register_fprobe_ips
->ftrace_set_filter_ipsThe new
available_filter_functions_addrsfile helps us with option b), because we can make 'b 1' and 'b 2' in one step - while filtering traceable functions, we get the address directly.
Quoted 8 lines
I tested the new
available_filter_functions_addrschanges with some hacked selftest changes, you can check it in here [1].I assume Jackie Liu will send new version of her patchset [2] based on this new
available_filter_functions_addrsfile.I think we should have these changes coming together and add some perf measurements from before and after to make the benefit apparent.
FYI I did some perf meassurements and the speedup is not substantial :-\
looks like the symbols resolving to addresses we do in kernel for kprobe_multi link is more faster/cheaper than I thought
but still there is 'some' speedup and we will get rid of the extra /proc/kallsyms parsing, so I think it's still worth it to have the new file
base:
Performance counter stats for './test_progs -n 103/1':
103,423,103 cycles:u
79,279,231,029 cycles:k
79,382,694,663 cycles 18.627593589 seconds time elapsed
0.025999000 seconds user
18.323855000 seconds syswith fix:
Performance counter stats for './test_progs -n 103/1':
126,659,572 cycles:u
77,951,768,179 cycles:k
78,078,467,451 cycles 18.651464273 seconds time elapsed
0.025001000 seconds user
18.243828000 seconds sysjirka
Steven Rostedt
maintainerQuoted 4 lines · On Wed, 14 Jun 2023 16:14:19 +0200 Jiri Olsa <[email protected]> wrote:
FYI I did some perf meassurements and the speedup is not substantial :-\
looks like the symbols resolving to addresses we do in kernel for
kprobe_multilink is more faster/cheaper than I thought
The symbol lookup is supposed to be fast, but it's not "free", whereas this is "free". I didn't expect a big speedup.
-- Steve
Quoted 4 lines
but still there is 'some' speedup and we will get rid of the extra
/proc/kallsymsparsing, so I think it's still worth it to have the new file