BTF is not generated for gcc-built kernel with the latest pahole
PATCH RESEND 1/1 BTF is not generated for gcc-built kernel with the latest pahole
Hello,
(I resend this because kconfig was too big)
I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log
[ 21.595724] trace_fprobe: error: BTF is not available or not supported
Command: f kfree $arg*
^
[ 21.596032] trace_fprobe: error: Invalid $-valiable specified
Command: f kfree $arg*
^/ # strings /sys/kernel/btf/vmlinux | grep kfree kfree_on_onlinemaybe_kfree_parameter trace_event_raw_rcu_invoke_kfree_bulk_callback trace_event_data_offsets_rcu_invoke_kfree_bulk_callback btf_trace_rcu_invoke_kfree_bulk_callback early_boot_kfree_rcu __bpf_trace_rcu_invoke_kfree_bulk_callback perf_trace_rcu_invoke_kfree_bulk_callback trace_event_raw_event_rcu_invoke_kfree_bulk_callback trace_raw_output_rcu_invoke_kfree_bulk_callback __probestub_rcu_invoke_kfree_bulk_callback __traceiter_rcu_invoke_kfree_bulk_callback kfree_rcu_cpu_workkfree_rcu_cpukfree_rcu_batch_init kfree_rcu_scheduler_running kfree_rcu_shrink_scankfree_rcu_shrink_countkfree_rcu_monitorkfree_rcu_work
Here is the gcc version which I'm using.
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I also attached the kernel config file.
What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 5 lines · Masami Hiramatsu
Hello,
(I resend this because kconfig was too big)I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
hi,
I can't reproduce on my setup with your .config
does 'bpftool btf dump file ./vmlinux' show any error?
is there any error in the kernel build output?
Quoted 16 lines
When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events sh: write error: Invalid argument# cat /sys/kernel/tracing/error_log [ 21.595724] trace_fprobe: error: BTF is not available or not supported Command: f kfree $arg* ^ [ 21.596032] trace_fprobe: error: Invalid $-valiable specified Command: f kfree $arg* ^
Quoted 1 line
/ # strings
/sys/kernel/btf/vmlinux| grep kfree
hm, if you have this file present, you have BTF in
Quoted 22 lines
kfree_on_onlinemaybe_kfree_parametertrace_event_raw_rcu_invoke_kfree_bulk_callbacktrace_event_data_offsets_rcu_invoke_kfree_bulk_callbackbtf_trace_rcu_invoke_kfree_bulk_callbackearly_boot_kfree_rcu__bpf_trace_rcu_invoke_kfree_bulk_callbackperf_trace_rcu_invoke_kfree_bulk_callbacktrace_event_raw_event_rcu_invoke_kfree_bulk_callbacktrace_raw_output_rcu_invoke_kfree_bulk_callback__probestub_rcu_invoke_kfree_bulk_callback__traceiter_rcu_invoke_kfree_bulk_callbackkfree_rcu_cpu_workkfree_rcu_cpukfree_rcu_batch_initkfree_rcu_scheduler_runningkfree_rcu_shrink_scankfree_rcu_shrink_countkfree_rcu_monitorkfree_rcu_work
Quoted 3 lines
Here is the gcc version which I'm using.
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch)
Quoted 5 lines
I also attached the kernel config file.
What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?
should work fine with both gcc and clang
jirka
Quoted 5 lines
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Hi Jiri,
Quoted 7 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu
Hello,
(I resend this because kconfig was too big)I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
Quoted 6 lines
hi,
I can't reproduce on my setup with your .configdoes 'bpftool btf dump file ./vmlinux' show any error?
is there any error in the kernel build output?
Yes, here it is. I saw these 2 lines.
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
Quoted 17 lines
Quoted 17 lines
When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events sh: write error: Invalid argument# cat /sys/kernel/tracing/error_log [ 21.595724] trace_fprobe: error: BTF is not available or not supported Command: f kfree $arg* ^ [ 21.596032] trace_fprobe: error: Invalid $-valiable specified Command: f kfree $arg* ^
Quoted 3 lines
Quoted 1 line
/ # strings
/sys/kernel/btf/vmlinux| grep kfreehm, if you have this file present, you have BTF in
Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. kfree_rcu_batch_init) it works.
Quoted 23 lines
Quoted 23 lines
kfree_on_onlinemaybe_kfree_parametertrace_event_raw_rcu_invoke_kfree_bulk_callbacktrace_event_data_offsets_rcu_invoke_kfree_bulk_callbackbtf_trace_rcu_invoke_kfree_bulk_callbackearly_boot_kfree_rcu__bpf_trace_rcu_invoke_kfree_bulk_callbackperf_trace_rcu_invoke_kfree_bulk_callbacktrace_event_raw_event_rcu_invoke_kfree_bulk_callbacktrace_raw_output_rcu_invoke_kfree_bulk_callback__probestub_rcu_invoke_kfree_bulk_callback__traceiter_rcu_invoke_kfree_bulk_callbackkfree_rcu_cpu_workkfree_rcu_cpukfree_rcu_batch_initkfree_rcu_scheduler_runningkfree_rcu_shrink_scankfree_rcu_shrink_countkfree_rcu_monitorkfree_rcu_work
Quoted 6 lines
Quoted 3 lines
Here is the gcc version which I'm using.
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch)
Curiously, with Clang 16.0.0, it works (but many different errors are shown, see below *).
So the combination of gcc/clang and pahole version can affect it.
Quoted 8 lines
Quoted 6 lines
I also attached the kernel config file.
What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?should work fine with both gcc and clang
And I guess it depends on compiler version, doesn't it?
Thank you,
(*)
BTF .btf.vmlinux.bin.odie__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled!die__process_unit: tag not supported 0xa (label)!die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled!die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled!
LD .tmp_vmlinux.kallsyms1
Quoted 10 lines
jirka
Quoted 5 lines
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 15 lines · Masami Hiramatsu (Google)
Hi Jiri,
Quoted 10 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu
Hello,
(I resend this because kconfig was too big)I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
hi,
I can't reproduce on my setup with your .config
Quoted 8 lines
Quoted 3 lines
does 'bpftool btf dump file ./vmlinux' show any error?
is there any error in the kernel build output?
Yes, here it is. I saw these 2 lines.
die__process:DW_TAG_compile_unit,DW_TAG_type_unit,DW_TAG_partial_unitorDW_TAG_skeleton_unitexpected got INVALID (0x0)!die__process:DW_TAG_compile_unit,DW_TAG_type_unit,DW_TAG_partial_unitorDW_TAG_skeleton_unitexpected got INVALID (0x0)!
This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in die__process() and happens if the dwarf_tag() associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it..
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events sh: write error: Invalid argument# cat /sys/kernel/tracing/error_log [ 21.595724] trace_fprobe: error: BTF is not available or not supported Command: f kfree $arg* ^ [ 21.596032] trace_fprobe: error: Invalid $-valiable specified Command: f kfree $arg* ^
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
/ # strings
/sys/kernel/btf/vmlinux| grep kfreehm, if you have this file present, you have BTF in
Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g.
kfree_rcu_batch_init) it works.
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter
FWIW I see most of the below in my BTF output on bpf-next, though I am missing a few, possibly due to differing config options since they don't appear in kallsyms either. I don't see the DWARF tag label not handled messages so it's possible that's a symptom of something
I suspect however some form of corruption in the DWARF representation may be the reason a lot of these are missing. Would be worth trying to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect.
Quoted 263 lines
Quoted 23 lines
Quoted 23 lines
kfree_on_onlinemaybe_kfree_parametertrace_event_raw_rcu_invoke_kfree_bulk_callbacktrace_event_data_offsets_rcu_invoke_kfree_bulk_callbackbtf_trace_rcu_invoke_kfree_bulk_callbackearly_boot_kfree_rcu__bpf_trace_rcu_invoke_kfree_bulk_callbackperf_trace_rcu_invoke_kfree_bulk_callbacktrace_event_raw_event_rcu_invoke_kfree_bulk_callbacktrace_raw_output_rcu_invoke_kfree_bulk_callback__probestub_rcu_invoke_kfree_bulk_callback__traceiter_rcu_invoke_kfree_bulk_callbackkfree_rcu_cpu_workkfree_rcu_cpukfree_rcu_batch_initkfree_rcu_scheduler_runningkfree_rcu_shrink_scankfree_rcu_shrink_countkfree_rcu_monitorkfree_rcu_workQuoted 6 lines
Quoted 3 lines
Here is the gcc version which I'm using.
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch)
Curiously, with Clang 16.0.0, it works (but many different errors are shown, see below *).
So the combination of gcc/clang and pahole version can affect it.Quoted 8 lines
Quoted 6 lines
I also attached the kernel config file.
What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?should work fine with both gcc and clang
And I guess it depends on compiler version, doesn't it?
Thank you,
(*)
BTF .btf.vmlinux.bin.odie__process_unit:DW_TAG_label(0xa) @ <0x4b138> not handled!die__process_unit: tag not supported0xa(label)!die__process_unit:DW_TAG_label(0xa) @ <0x4b241> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b263> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b290> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b2c0> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b2eb> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b317> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b33a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b35a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b37d> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b39e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b3c4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b3e7> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b40f> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b435> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b457> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b477> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b4a5> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b4d5> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b505> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b530> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b560> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b585> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b5b2> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b5d9> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b605> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b62e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b652> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b670> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b694> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b6b3> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b6d9> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b705> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b72b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b753> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b77f> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b7b3> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b7e4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b811> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b83c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b869> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b88c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b8bd> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b8e7> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b90b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b930> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b960> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b997> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4b9ce> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4ba00> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4ba24> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4ba4d> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4ba89> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4babc> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bae4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bb0b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bb2e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bb4e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bb6d> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bb8a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bba8> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bbc5> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bbe1> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bc01> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bc1c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bc38> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bc58> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bc79> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bc95> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4bcb2> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7efd6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7effe> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f11c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f143> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f178> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f1a4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f1ca> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f1fb> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f22f> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f25a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f28d> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x7f2b7> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eea81> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eea9d> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eeac3> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eeaf3> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eeb0f> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eeb30> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x1eeb59> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x340734> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6b4a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6b67> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6b8a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6ba5> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6bc4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6bea> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6c07> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6c2a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6c4e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6c79> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6c9b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6cc3> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6ceb> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x4c6d15> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x30626ac> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x30626cd> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x3062814> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x3062833> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x3062b8b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f623f6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62416> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62437> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62458> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6280b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62b09> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62b37> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62c45> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62c60> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62d69> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f62e8d> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6305a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63ec2> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63edf> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63efc> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63f19> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63f36> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63f5b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63f80> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63fa5> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f63fca> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6fcae> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6fcc7> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6fed7> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6fef0> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f6fdcb> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70352> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7036f> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70394> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f703b1> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f703d6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f703f3> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70418> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70435> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7045a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7057b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f705a6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f705cf> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f705f8> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70621> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7064a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70673> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7069c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f706c5> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f706ee> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70716> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7073e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70767> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70790> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f707b9> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f707e2> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7080b> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70834> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70864> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70892> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f708c0> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f708ee> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7091c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f7094a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70978> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f709a6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f709d4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70a01> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70a2e> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70a5c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70a8a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70ab8> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70ae6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70b14> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70b42> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70b72> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70ba0> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70bce> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70bfc> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70c2a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70c58> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70c86> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70cb4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70ce2> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70d0f> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70d3c> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70d6a> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70d98> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70dc6> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70df4> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70e22> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70e50> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70e71> not handled!die__process_unit:DW_TAG_label(0xa) @ <0x5f70e94> not handled!
LD .tmp_vmlinux.kallsyms1Quoted 10 lines
jirka
Quoted 5 lines
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 25 lines · On Wed, 26 Jul 2023 15:46:03 +0100 Alan Maguire <[email protected]> wrote:
Quoted 15 lines · Masami Hiramatsu (Google)
Hi Jiri,
Quoted 10 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu
Hello,
(I resend this because kconfig was too big)I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
hi,
I can't reproduce on my setup with your .configQuoted 8 lines
Quoted 3 lines
does 'bpftool btf dump file ./vmlinux' show any error?
is there any error in the kernel build output?
Yes, here it is. I saw these 2 lines.
die__process:DW_TAG_compile_unit,DW_TAG_type_unit,DW_TAG_partial_unitorDW_TAG_skeleton_unitexpected got INVALID (0x0)!die__process:DW_TAG_compile_unit,DW_TAG_type_unit,DW_TAG_partial_unitorDW_TAG_skeleton_unitexpected got INVALID (0x0)!
Quoted 6 lines
This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in
die__process()and happens if thedwarf_tag()associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it..
And this warning message is not good for debugging. It would better dump the message with DIE index so that we can analyze the DWARF with other tools.
Quoted 31 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)
e.g.
# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events sh: write error: Invalid argument# cat /sys/kernel/tracing/error_log [ 21.595724] trace_fprobe: error: BTF is not available or not supported Command: f kfree $arg* ^ [ 21.596032] trace_fprobe: error: Invalid $-valiable specified Command: f kfree $arg* ^Quoted 8 lines
Quoted 3 lines
Quoted 1 line
/ # strings
/sys/kernel/btf/vmlinux| grep kfreehm, if you have this file present, you have BTF in
Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g.
kfree_rcu_batch_init) it works.Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
Quoted 4 lines
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
So this will generate BTF from vmlinux DWARF info.
Quoted 8 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter
Ah, that's nice. Let me try.
Quoted 9 lines
FWIW I see most of the below in my BTF output on bpf-next, though I am missing a few, possibly due to differing config options since they don't appear in kallsyms either. I don't see the DWARF tag label not handled messages so it's possible that's a symptom of something
I suspect however some form of corruption in the DWARF representation may be the reason a lot of these are missing. Would be worth trying to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect.
OK, let me dump the dwarf and check what information it has for "kfree".
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 16 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.outSo this will generate BTF from vmlinux DWARF info.
Quoted 3 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
Quoted 7 lines
Quoted 5 lines
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameterAh, that's nice. Let me try.
$ pahole --version v1.23
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
OK, so something failed.
$ grep skipping /tmp/pahole.out | wc -l 0
Nothing to be skipped.
$ grep -w kfree /tmp/pahole.out | wc -l 0 $ grep -w vfs_read /tmp/pahole.out | wc -l 0
But both kfree and vfs_read are not found.
$ perf probe -k ./vmlinux -V kfree
Available variables at kfree
@<kfree+0>
(unknown_type) object
$ perf probe -k ./vmlinux -V vfs_read
Available variables at vfs_read
@<vfs_read+0>
char* buf
loff_t* pos
size_t count
struct file* fileHowever, perf probe can find both in the DWARF info.
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.outSo this will generate BTF from vmlinux DWARF info.
Quoted 8 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter
Quoted 5 lines
Quoted 1 line
Ah, that's nice. Let me try.
$ pahole --version v1.23
shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?
Quoted 29 lines
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!OK, so something failed.
$ grep skipping /tmp/pahole.out | wc -l 0Nothing to be skipped.
$ grep -w kfree /tmp/pahole.out | wc -l 0 $ grep -w vfs_read /tmp/pahole.out | wc -l 0But both kfree and
vfs_readare not found.$ perf probe -k ./vmlinux -V kfree Available variables at kfree @<kfree+0> (unknown_type) object $ perf probe -k ./vmlinux -V vfs_read Available variables at vfs_read @<vfs_read+0> char* buf loff_t* pos size_t count struct file* file
Quoted 4 lines
However, perf probe can find both in the DWARF info.
Thank you,
Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).
One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Alan
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.outSo this will generate BTF from vmlinux DWARF info.
Quoted 8 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Ah, that's nice. Let me try.
$ pahole --version v1.23shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?
Here it is.
$ ldd pahole linux-vdso.so.1 (0x00007ffd6b1e2000) libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)
Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.
I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
gcc-11.3 | clang-16.0
---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)
So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.
Quoted 32 lines
Quoted 32 lines
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!OK, so something failed.
$ grep skipping /tmp/pahole.out | wc -l 0Nothing to be skipped.
$ grep -w kfree /tmp/pahole.out | wc -l 0 $ grep -w vfs_read /tmp/pahole.out | wc -l 0But both kfree and
vfs_readare not found.$ perf probe -k ./vmlinux -V kfree Available variables at kfree @<kfree+0> (unknown_type) object $ perf probe -k ./vmlinux -V vfs_read Available variables at vfs_read @<vfs_read+0> char* buf loff_t* pos size_t count struct file* fileHowever, perf probe can find both in the DWARF info.
Quoted 7 lines
Quoted 2 lines
Thank you,
Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).
Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
Quoted 3 lines
One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.
Thank you,
Quoted 2 lines
Alan
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 73 lines · Masami Hiramatsu (Google)
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.outSo this will generate BTF from vmlinux DWARF info.
Quoted 8 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameterQuoted 8 lines
Quoted 5 lines
Quoted 1 line
Ah, that's nice. Let me try.
$ pahole --version v1.23shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?
Here it is.
$ ldd pahole linux-vdso.so.1 (0x00007ffd6b1e2000) libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.
I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
gcc-11.3 | clang-16.0---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)
Quoted 3 lines
So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.
One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference.
You can check DWARF version associated with CUs with
readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'
Quoted 43 lines
Quoted 32 lines
Quoted 32 lines
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!OK, so something failed.
$ grep skipping /tmp/pahole.out | wc -l 0Nothing to be skipped.
$ grep -w kfree /tmp/pahole.out | wc -l 0 $ grep -w vfs_read /tmp/pahole.out | wc -l 0But both kfree and
vfs_readare not found.$ perf probe -k ./vmlinux -V kfree Available variables at kfree @<kfree+0> (unknown_type) object $ perf probe -k ./vmlinux -V vfs_read Available variables at vfs_read @<vfs_read+0> char* buf loff_t* pos size_t count struct file* fileHowever, perf probe can find both in the DWARF info.
Quoted 7 lines
Quoted 2 lines
Thank you,
Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).
Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
Quoted 6 lines
Quoted 3 lines
One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.
That might help. Thanks!
Alan
Quoted 6 lines
Thank you,
Quoted 2 lines
Alan
A message that is not in the archive (sent off-list or not yet received): CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com
PATCH RESEND 1/1 BTF is not generated for gcc-built kernel with the latest pahole
Sorry, replied only to Alan :-\
- Arnaldo
---------- Forwarded message ---------
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Tue, Aug 1, 2023 at 3:26 PM
Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
To: Alan Maguire <[email protected]>
Quoted 10 lines · Alan Maguire
Quoted 9 lines · Masami Hiramatsu (Google)
Quoted 4 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).
Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
See below, but this one is interesting, gcc-13 works with CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.
That might help. Thanks!
Probably he needs to tweak these CONFIG_ entries:
⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools]$
And make it use CONFIG_DEBUG_INFO_DWARF4=y,CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n
For DWARF5 I need to forward port what I have in:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit
- Arnaldo
Quoted 22 lines · On Tue, 1 Aug 2023 15:28:38 -0300 Arnaldo Carvalho de Melo…
Sorry, replied only to Alan :-\
- Arnaldo
---------- Forwarded message ---------
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Tue, Aug 1, 2023 at 3:26 PM
Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
To: Alan Maguire <[email protected]>Quoted 9 lines · Alan Maguire
Quoted 8 lines · Masami Hiramatsu (Google)
Quoted 4 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).
See below, but this one is interesting, gcc-13 works with
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y?
I'm using gcc-11.3, and it seems to work.
Quoted 12 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!
Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.
That might help. Thanks!
Probably he needs to tweak these
CONFIG_entries:
Quoted 8 lines
⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
#CONFIG_DEBUG_INFO_DWARF4is not set
#CONFIG_DEBUG_INFO_DWARF5is not set
⬢[acme@toolbox perf-tools]$And make it use
CONFIG_DEBUG_INFO_DWARF4=y,CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n
OK, interesting. Let me check again with DWARF4.
Thank you!
Quoted 6 lines
For DWARF5 I need to forward port what I have in:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit
- Arnaldo
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu:
Quoted 74 lines · Masami Hiramatsu (Google)
Quoted 73 lines
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.outSo this will generate BTF from vmlinux DWARF info.
Quoted 8 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameterQuoted 8 lines
Quoted 5 lines
Quoted 1 line
Ah, that's nice. Let me try.
$ pahole --version v1.23shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?
Here it is.
$ ldd pahole linux-vdso.so.1 (0x00007ffd6b1e2000) libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.
I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
gcc-11.3 | clang-16.0---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)
Quoted 8 lines
Quoted 3 lines
So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.
One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that
To check that please tweak:
⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools-next]$
i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable CONFIG_DEBUG_INFO_DWARF4.
- Arnaldo
Quoted 7 lines
pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference.
You can check DWARF version associated with CUs with
readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'
Quoted 86 lines · On Tue, 1 Aug 2023 18:45:49 -0300 Arnaldo Carvalho de Melo <[email protected]>…
Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu:
Quoted 74 lines · Masami Hiramatsu (Google)
Quoted 73 lines
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines
Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):
In a built kernel directory (where unstripped vmlinux is present):
$ PAHOLE_FLAGS=$(./scripts/pahole_flags) $ PAHOLE=/usr/local/bin/pahole $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.outSo this will generate BTF from vmlinux DWARF info.
Quoted 8 lines
If you want to investigate why a function has been left out, look for "skipping" verbose output like this:
skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameterQuoted 8 lines
Quoted 5 lines
Quoted 1 line
Ah, that's nice. Let me try.
$ pahole --version v1.23shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?
Here it is.
$ ldd pahole linux-vdso.so.1 (0x00007ffd6b1e2000) libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.
I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)
gcc-11.3 | clang-16.0---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)Quoted 8 lines
Quoted 3 lines
So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.
One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that
To check that please tweak:
Quoted 8 lines
⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
#CONFIG_DEBUG_INFO_DWARF4is not set
#CONFIG_DEBUG_INFO_DWARF5is not set
⬢[acme@toolbox perf-tools-next]$i.e. disable
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULTand enableCONFIG_DEBUG_INFO_DWARF4.
Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
LD .tmp_vmlinux.btf BTF .btf.vmlinux.bin.o LD .tmp_vmlinux.kallsyms1
And
/ # strings /sys/kernel/btf/vmlinux | wc -l 89921
/ # strings /sys/kernel/btf/vmlinux | grep -w kfree kfree
It seems the BTF is correctly generated. (with DWARF5, the number of symbols are about 30000.)
Maybe we should check the pahole version and choose the DWARF4 if BTF is selected.
Thank you!
Quoted 10 lines
- Arnaldo
Quoted 7 lines
pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference.
You can check DWARF version associated with CUs with
readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Em Thu, Jul 27, 2023 at 09:38:14AM +0900, Masami Hiramatsu escreveu:
Quoted 35 lines
Quoted 25 lines · On Wed, 26 Jul 2023 15:46:03 +0100 Alan Maguire <[email protected]> wrote:
Quoted 15 lines · Masami Hiramatsu (Google)
Hi Jiri,
Quoted 10 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu
Hello,
(I resend this because kconfig was too big)I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
hi,
I can't reproduce on my setup with your .configQuoted 8 lines
Quoted 3 lines
does 'bpftool btf dump file ./vmlinux' show any error?
is there any error in the kernel build output?
Yes, here it is. I saw these 2 lines.
die__process:DW_TAG_compile_unit,DW_TAG_type_unit,DW_TAG_partial_unitorDW_TAG_skeleton_unitexpected got INVALID (0x0)!die__process:DW_TAG_compile_unit,DW_TAG_type_unit,DW_TAG_partial_unitorDW_TAG_skeleton_unitexpected got INVALID (0x0)!Quoted 6 lines
This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in
die__process()and happens if thedwarf_tag()associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it..
Quoted 3 lines
And this warning message is not good for debugging. It would better dump the message with DIE index so that we can analyze the DWARF with other tools.
Good idea, I'll add that.
- Arnaldo