Question: fentry on kernel func optimized by compiler
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940
Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in kallsyms_lookup_name. Therefore, the two do not match.
func_proto = btf_type_by_id(desc_btf, func->type);
if (!func_proto || !btf_type_is_func_proto(func_proto)) {
verbose(env, "kernel function btf_id %u does not have a
valid func_proto\n",
func_id);
return -EINVAL;
}
func_name = btf_name_by_offset(desc_btf, func->name_off);
addr = kallsyms_lookup_name(func_name);
if (!addr) {
verbose(env, "cannot find address for kernel function
%s\n",
func_name);
return -EINVAL;
}I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324
So can we add a judgment from libbpf. If it is an optimized function, pass the suffix of the optimized function from the user space to the kernel, and then perform a function name concatenation, like:
func_name = btf_name_by_offset(desc_btf, func->name_off);
if (optimize) {
func_name = func_name + ".isra.0"
}
addr = kallsyms_lookup_name(func_name);Signature · 2 lines
Best Regards Tao Chen
Song Liu
reviewerHi Tao,
Compiler optimizations can cause issues for tracing kernel functions. Please refer to Yonghong and Alan's presentation [1] for various cases.
Thanks,
Song
[1] https://lpc.events/event/18/contributions/1945/
Quoted 49 lines · Tao Chen
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function, pass the suffix of the optimized function from the user space to the kernel, and then perform a function name concatenation, like:
func_name = btf_name_by_offset(desc_btf, func->name_off); if (optimize) { func_name = func_name + ".isra.0" } addr = kallsyms_lookup_name(func_name);--
Best Regards
Tao Chen
Quoted 11 lines · Song Liu
Hi Tao,
Compiler optimizations can cause issues for tracing kernel functions. Please refer to Yonghong and Alan's presentation [1] for various cases.
Thanks,
Song
Hi Song,
Thank you for the materials you provided. I'll take a look at them.
Quoted 50 lines · Tao Chen
Quoted 49 lines
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function, pass the suffix of the optimized function from the user space to the kernel, and then perform a function name concatenation, like:
func_name = btf_name_by_offset(desc_btf, func->name_off); if (optimize) { func_name = func_name + ".isra.0" } addr = kallsyms_lookup_name(func_name);--
Best Regards
Tao Chen
Signature · 2 lines
Best Regards Tao Chen
Quoted 29 lines · Tao Chen
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }
Quoted 7 lines
I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function,
No, we cannot. It's a different function at that point and libbpf isn't going to be in the business of guessing on behalf of the user whether it's ok to do or not.
But the user can use multi-kprobe with prefix* naming, if they encountered (or are anticipating) this situation and think it's fine for them.
As for fentry/fexit, you need to have the correct BTF ID associated with that function anyways, so I'm not sure that currently you can attach fentry/fexit to such compiler-optimized functions at all (pahole won't produce BTF for such functions, right?).
Quoted 14 lines
pass the suffix of the optimized function from the user space to the kernel, and then perform a function name concatenation, like:
func_name = btf_name_by_offset(desc_btf, func->name_off); if (optimize) { func_name = func_name + ".isra.0" } addr = kallsyms_lookup_name(func_name);--
Best Regards
Tao Chen
在 2025/3/29 01:21, Andrii Nakryiko 写道:
Hi Andrri,
Quoted 38 lines · Tao Chen
Quoted 29 lines
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }Quoted 7 lines
I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function,
Quoted 8 lines
No, we cannot. It's a different function at that point and libbpf isn't going to be in the business of guessing on behalf of the user whether it's ok to do or not.
But the user can use multi-kprobe with
prefix*naming, if they encountered (or are anticipating) this situation and think it's fine for them.
I will try multi-kprobe feature, and briefly checked and found that the multi-kprobe is implemented based on fprobe. Is its performance similar to that of fentry? Thanks.
Quoted 5 lines
As for fentry/fexit, you need to have the correct BTF ID associated with that function anyways, so I'm not sure that currently you can attach fentry/fexit to such compiler-optimized functions at all (pahole won't produce BTF for such functions, right?).
Yes, it is.
Quoted 14 lines
Quoted 14 lines
pass the suffix of the optimized function from the user space to the kernel, and then perform a function name concatenation, like:
func_name = btf_name_by_offset(desc_btf, func->name_off); if (optimize) { func_name = func_name + ".isra.0" } addr = kallsyms_lookup_name(func_name);--
Best Regards
Tao Chen
Signature · 2 lines
Best Regards Tao Chen
Quoted 42 lines · Andrii Nakryiko
Quoted 29 lines · Tao Chen
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }Quoted 7 lines
I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function,
No, we cannot. It's a different function at that point and libbpf isn't going to be in the business of guessing on behalf of the user whether it's ok to do or not.
Quoted 9 lines
But the user can use multi-kprobe with
prefix*naming, if they encountered (or are anticipating) this situation and think it's fine for them.As for fentry/fexit, you need to have the correct BTF ID associated with that function anyways, so I'm not sure that currently you can attach fentry/fexit to such compiler-optimized functions at all (pahole won't produce BTF for such functions, right?).
Yep, BTF will not be there for all cases, but ever since we've had the "optimized_func" BTF feature, we've have encoded BTF for suffixed functions as long as their parameters are not optimized away and as long as we don't have multiple inconsistent representations associated with a function (say two differing function signatures for the same name). Optimization away of parameters happens quite frequently, but not always for .isra.0 functions so they are potentially sometimes safe for fentry.
The complication here is that - by design - the function name in BTF will be the prefix; i.e. "foo" not "foo.isra.0". So how we match up the BTF with the right suffixed function is an issue; a single function prefix can have ".isra.0" and ".cold.0" suffixes associated for example. The latter isn't really a function entry point (in the C code at least); it's just a split of the function into common path and less common path for better code locality for the more commonly-executed code.
Yonghong and I talked about this a bit last year in Plumbers, but it did occur to me that there are conditions where we could match up the prefix from BTF with a guaranteed fentry point for the function using info we have today.
/sys/kernel/tracing/available_filter_functions_addr has similar info to /proc/kallysyms but as far as I understand it we are also guaranteed that the associated addresses correspond to real function entry points. So because the BTF representation currently ensures consistency _and_ available function parameters, I think we could use available_filter_functions_addr to carry out the match and provide the right function address for the BTF representation.
In the future, the hope is we can handle inconsistent representations too in BTF, but the above represents a possible approach we could implement today I think, though I may be missing something. Thanks!
Alan
Quoted 74 lines · Alan Maguire
Quoted 42 lines · Andrii Nakryiko
Quoted 29 lines · Tao Chen
Hi,
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }Quoted 7 lines
I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function,
No, we cannot. It's a different function at that point and libbpf isn't going to be in the business of guessing on behalf of the user whether it's ok to do or not.
Quoted 9 lines
But the user can use multi-kprobe with
prefix*naming, if they encountered (or are anticipating) this situation and think it's fine for them.As for fentry/fexit, you need to have the correct BTF ID associated with that function anyways, so I'm not sure that currently you can attach fentry/fexit to such compiler-optimized functions at all (pahole won't produce BTF for such functions, right?).
Yep, BTF will not be there for all cases, but ever since we've had the "
optimized_func" BTF feature, we've have encoded BTF for suffixed functions as long as their parameters are not optimized away and as long as we don't have multiple inconsistent representations associated with a function (say two differing function signatures for the same name). Optimization away of parameters happens quite frequently, but not always for .isra.0 functions so they are potentially sometimes safe for fentry.The complication here is that - by design - the function name in BTF will be the prefix; i.e. "foo" not "foo.isra.0". So how we match up the BTF with the right suffixed function is an issue; a single function prefix can have ".isra.0" and ".cold.0" suffixes associated for example. The latter isn't really a function entry point (in the C code at least); it's just a split of the function into common path and less common path for better code locality for the more commonly-executed code.
Yonghong and I talked about this a bit last year in Plumbers, but it did occur to me that there are conditions where we could match up the prefix from BTF with a guaranteed fentry point for the function using info we have today.
Quoted 8 lines
/sys/kernel/tracing/available_filter_functions_addrhas similar info to/proc/kallysymsbut as far as I understand it we are also guaranteed that the associated addresses correspond to real function entry points. So because the BTF representation currently ensures consistency _and_ available function parameters, I think we could useavailable_filter_functions_addrto carry out the match and provide the right function address for the BTF representation.
Hi, Alan
Sorry for not replying in time. As you said,it seems much simpler when use the func addr from available_filter_functions_addr. It seems to be a bit similar to the way of passing function addresses in kprobe_multi. @Andrii @Jiri what do you think?
Quoted 5 lines
In the future, the hope is we can handle inconsistent representations too in BTF, but the above represents a possible approach we could implement today I think, though I may be missing something. Thanks!
Alan
Signature · 2 lines
Best Regards Tao Chen
Quoted 5 lines · Tao Chen
Quoted 4 lines · Alan Maguire
Quoted 3 lines · Andrii Nakryiko
Quoted 2 lines · Tao Chen
Hi,
hi, sorry for late reply
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.
hum, would you have example BTF/kernel/config with such case? I'd think if function is in BTF you should find it through kallsyms, the other way around is not garanteed
Quoted 56 lines
Quoted 56 lines
Quoted 30 lines
Quoted 18 lines
func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }Quoted 7 lines
I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function,
No, we cannot. It's a different function at that point and libbpf isn't going to be in the business of guessing on behalf of the user whether it's ok to do or not.
Quoted 9 lines
But the user can use multi-kprobe with
prefix*naming, if they encountered (or are anticipating) this situation and think it's fine for them.As for fentry/fexit, you need to have the correct BTF ID associated with that function anyways, so I'm not sure that currently you can attach fentry/fexit to such compiler-optimized functions at all (pahole won't produce BTF for such functions, right?).
Yep, BTF will not be there for all cases, but ever since we've had the "
optimized_func" BTF feature, we've have encoded BTF for suffixed functions as long as their parameters are not optimized away and as long as we don't have multiple inconsistent representations associated with a function (say two differing function signatures for the same name). Optimization away of parameters happens quite frequently, but not always for .isra.0 functions so they are potentially sometimes safe for fentry.The complication here is that - by design - the function name in BTF will be the prefix; i.e. "foo" not "foo.isra.0". So how we match up the BTF with the right suffixed function is an issue; a single function prefix can have ".isra.0" and ".cold.0" suffixes associated for example. The latter isn't really a function entry point (in the C code at least); it's just a split of the function into common path and less common path for better code locality for the more commonly-executed code.
Quoted 8 lines
Quoted 8 lines
Yonghong and I talked about this a bit last year in Plumbers, but it did occur to me that there are conditions where we could match up the prefix from BTF with a guaranteed fentry point for the function using info we have today.
/sys/kernel/tracing/available_filter_functions_addrhas similar info to/proc/kallysymsbut as far as I understand it we are also guaranteed that the associated addresses correspond to real function entry points.
available_filter_functions_addr contains exactly the same functions as available_filter_functions (all functions managed by ftrace), it just adds address value for each function as a hint to user space
it's used in kprobe_multi bench test to get all traceable addresses, and passed to kprobe_multi link, which uses following fprobe interface to attach:
err = register_fprobe_ips(&link->fp, addrs, cnt);
Quoted 4 lines
Quoted 4 lines
So because the BTF representation currently ensures consistency _and_ available function parameters, I think we could use
available_filter_functions_addrto carry out the match and provide the right function address for the BTF representation.
where would you do the match to available_filter_functions_addr?
also not sure how it's connected to the original issue, that seems to be related to pahole eliminating unsafe functions in BTF?
jirka
Quoted 19 lines
Quoted 1 line
Hi, Alan
Sorry for not replying in time. As you said,it seems much simpler when use the func addr fromavailable_filter_functions_addr. It seems to be a bit similar to the way of passing function addresses inkprobe_multi. @Andrii @Jiri what do you think?Quoted 5 lines
In the future, the hope is we can handle inconsistent representations too in BTF, but the above represents a possible approach we could implement today I think, though I may be missing something. Thanks!
Alan
Signature · 2 lines
Best Regards Tao Chen
Quoted 14 lines · Jiri Olsa
Quoted 5 lines · Tao Chen
Quoted 4 lines · Alan Maguire
Quoted 3 lines · Andrii Nakryiko
Quoted 2 lines · Tao Chen
Hi,
hi, sorry for late reply
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I recently encountered a problem when using fentry to trace kernel functions optimized by compiler, the specific situation is as follows:
https://github.com/bpftrace/bpftrace/issues/3940
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Simply put, some functions have been optimized by the compiler. The original function names are found through BTF, but the optimized functions are the ones that exist in
kallsyms_lookup_name. Therefore, the two do not match.hum, would you have example BTF/kernel/config with such case? I'd think if function is in BTF you should find it through kallsyms, the other way around is not garanteed
It seems that there is no special configuration. I am using the official Ubuntu 24.04. The issue link is: https://github.com/bpftrace/bpftrace/issues/3940
Quoted 73 lines
Quoted 56 lines
Quoted 56 lines
Quoted 30 lines
Quoted 18 lines
func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", func_name); return -EINVAL; }Quoted 7 lines
I have made a simple statistics and there are approximately more than 2,000 functions in Ubuntu 24.04.
dylane@2404:~$ cat /proc/kallsyms | grep isra | wc -l 2324So can we add a judgment from libbpf. If it is an optimized function,
No, we cannot. It's a different function at that point and libbpf isn't going to be in the business of guessing on behalf of the user whether it's ok to do or not.
Quoted 9 lines
But the user can use multi-kprobe with
prefix*naming, if they encountered (or are anticipating) this situation and think it's fine for them.As for fentry/fexit, you need to have the correct BTF ID associated with that function anyways, so I'm not sure that currently you can attach fentry/fexit to such compiler-optimized functions at all (pahole won't produce BTF for such functions, right?).
Yep, BTF will not be there for all cases, but ever since we've had the "
optimized_func" BTF feature, we've have encoded BTF for suffixed functions as long as their parameters are not optimized away and as long as we don't have multiple inconsistent representations associated with a function (say two differing function signatures for the same name). Optimization away of parameters happens quite frequently, but not always for .isra.0 functions so they are potentially sometimes safe for fentry.The complication here is that - by design - the function name in BTF will be the prefix; i.e. "foo" not "foo.isra.0". So how we match up the BTF with the right suffixed function is an issue; a single function prefix can have ".isra.0" and ".cold.0" suffixes associated for example. The latter isn't really a function entry point (in the C code at least); it's just a split of the function into common path and less common path for better code locality for the more commonly-executed code.
Quoted 8 lines
Quoted 8 lines
Yonghong and I talked about this a bit last year in Plumbers, but it did occur to me that there are conditions where we could match up the prefix from BTF with a guaranteed fentry point for the function using info we have today.
/sys/kernel/tracing/available_filter_functions_addrhas similar info to/proc/kallysymsbut as far as I understand it we are also guaranteed that the associated addresses correspond to real function entry points.
available_filter_functions_addrcontains exactly the same functions asavailable_filter_functions(all functions managed by ftrace), it just adds address value for each function as a hint to user spaceit's used in
kprobe_multibench test to get all traceable addresses, and passed tokprobe_multilink, which uses following fprobe interface to attach:
Quoted 8 lines
err = register_fprobe_ips(&link->fp, addrs, cnt);Quoted 4 lines
Quoted 4 lines
So because the BTF representation currently ensures consistency _and_ available function parameters, I think we could use
available_filter_functions_addrto carry out the match and provide the right function address for the BTF representation.where would you do the match to
available_filter_functions_addr?
maybe in libbpf? If it is the situation encountered above, use the passed-in address from user; otherwise, still use the original logic.
func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name);
Quoted 26 lines
also not sure how it's connected to the original issue, that seems to be related to pahole eliminating unsafe functions in BTF?
jirka
Quoted 19 lines
Quoted 1 line
Hi, Alan
Sorry for not replying in time. As you said,it seems much simpler when use the func addr fromavailable_filter_functions_addr. It seems to be a bit similar to the way of passing function addresses inkprobe_multi. @Andrii @Jiri what do you think?Quoted 5 lines
In the future, the hope is we can handle inconsistent representations too in BTF, but the above represents a possible approach we could implement today I think, though I may be missing something. Thanks!
Alan
Signature · 2 lines
Best Regards Tao Chen
Signature · 2 lines
Best Regards Tao Chen