Skip to contentSkip to conversation

BTF is not generated for gcc-built kernel with the latest pahole

PATCH RESEND In review Masami Hiramatsu (Google) 5 participants 14 messages Jul 26, 2023 – Aug 2, 2023 1 message not in the archive BPF

lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 BTF is not generated for gcc-built kernel with the latest pahole

Hello,
(I resend this because kconfig was too big)

I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).
When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)

e.g.

# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log 
[   21.595724] trace_fprobe: error: BTF is not available or not supported
  Command: f kfree $arg*
                   ^
[   21.596032] trace_fprobe: error: Invalid $-valiable specified
  Command: f kfree $arg*
                   ^

/ # strings /sys/kernel/btf/vmlinux | grep kfree kfree_on_online
maybe_kfree_parameter trace_event_raw_rcu_invoke_kfree_bulk_callback trace_event_data_offsets_rcu_invoke_kfree_bulk_callback btf_trace_rcu_invoke_kfree_bulk_callback early_boot_kfree_rcu __bpf_trace_rcu_invoke_kfree_bulk_callback perf_trace_rcu_invoke_kfree_bulk_callback trace_event_raw_event_rcu_invoke_kfree_bulk_callback trace_raw_output_rcu_invoke_kfree_bulk_callback __probestub_rcu_invoke_kfree_bulk_callback __traceiter_rcu_invoke_kfree_bulk_callback kfree_rcu_cpu_work
kfree_rcu_cpu
kfree_rcu_batch_init kfree_rcu_scheduler_running kfree_rcu_shrink_scan
kfree_rcu_shrink_count
kfree_rcu_monitor
kfree_rcu_work

Here is the gcc version which I'm using.

gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)

I also attached the kernel config file.

What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#1 lore raw
Quoted 5 lines · Masami Hiramatsu

Hello,
(I resend this because kconfig was too big)

I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).

hi,
I can't reproduce on my setup with your .config

does 'bpftool btf dump file ./vmlinux' show any error?

is there any error in the kernel build output?

Quoted 16 lines

When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)

e.g.

# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log 
[   21.595724] trace_fprobe: error: BTF is not available or not supported
  Command: f kfree $arg*
                   ^
[   21.596032] trace_fprobe: error: Invalid $-valiable specified
  Command: f kfree $arg*
                   ^
Quoted 1 line

/ # strings /sys/kernel/btf/vmlinux | grep kfree

hm, if you have this file present, you have BTF in

Quoted 22 lines

kfree_on_online
maybe_kfree_parameter trace_event_raw_rcu_invoke_kfree_bulk_callback trace_event_data_offsets_rcu_invoke_kfree_bulk_callback btf_trace_rcu_invoke_kfree_bulk_callback early_boot_kfree_rcu __bpf_trace_rcu_invoke_kfree_bulk_callback perf_trace_rcu_invoke_kfree_bulk_callback trace_event_raw_event_rcu_invoke_kfree_bulk_callback trace_raw_output_rcu_invoke_kfree_bulk_callback __probestub_rcu_invoke_kfree_bulk_callback __traceiter_rcu_invoke_kfree_bulk_callback kfree_rcu_cpu_work
kfree_rcu_cpu
kfree_rcu_batch_init kfree_rcu_scheduler_running kfree_rcu_shrink_scan
kfree_rcu_shrink_count
kfree_rcu_monitor
kfree_rcu_work

Quoted 3 lines

Here is the gcc version which I'm using.

gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)

I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch)

Quoted 5 lines

I also attached the kernel config file.

What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?

should work fine with both gcc and clang

jirka

Quoted 5 lines

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#2 lore raw

Hi Jiri,

Quoted 7 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu

Hello,
(I resend this because kconfig was too big)

I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).

Quoted 6 lines

hi,
I can't reproduce on my setup with your .config

does 'bpftool btf dump file ./vmlinux' show any error?

is there any error in the kernel build output?

Yes, here it is. I saw these 2 lines.

die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

Quoted 17 lines
Quoted 17 lines

When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)

e.g.

# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log 
[   21.595724] trace_fprobe: error: BTF is not available or not supported
  Command: f kfree $arg*
                   ^
[   21.596032] trace_fprobe: error: Invalid $-valiable specified
  Command: f kfree $arg*
                   ^
Quoted 3 lines
Quoted 1 line

/ # strings /sys/kernel/btf/vmlinux | grep kfree

hm, if you have this file present, you have BTF in

Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. kfree_rcu_batch_init) it works.

Quoted 23 lines
Quoted 23 lines

kfree_on_online
maybe_kfree_parameter trace_event_raw_rcu_invoke_kfree_bulk_callback trace_event_data_offsets_rcu_invoke_kfree_bulk_callback btf_trace_rcu_invoke_kfree_bulk_callback early_boot_kfree_rcu __bpf_trace_rcu_invoke_kfree_bulk_callback perf_trace_rcu_invoke_kfree_bulk_callback trace_event_raw_event_rcu_invoke_kfree_bulk_callback trace_raw_output_rcu_invoke_kfree_bulk_callback __probestub_rcu_invoke_kfree_bulk_callback __traceiter_rcu_invoke_kfree_bulk_callback kfree_rcu_cpu_work
kfree_rcu_cpu
kfree_rcu_batch_init kfree_rcu_scheduler_running kfree_rcu_shrink_scan
kfree_rcu_shrink_count
kfree_rcu_monitor
kfree_rcu_work

Quoted 6 lines
Quoted 3 lines

Here is the gcc version which I'm using.

gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)

I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch)

Curiously, with Clang 16.0.0, it works (but many different errors are shown, see below *).
So the combination of gcc/clang and pahole version can affect it.

Quoted 8 lines
Quoted 6 lines

I also attached the kernel config file.

What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?

should work fine with both gcc and clang

And I guess it depends on compiler version, doesn't it?

Thank you,

(*)
BTF .btf.vmlinux.bin.o
die__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled!
die__process_unit: tag not supported 0xa (label)!
die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled!
LD .tmp_vmlinux.kallsyms1

Quoted 10 lines

jirka

Quoted 5 lines

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#3 lore raw
Quoted 15 lines · Masami Hiramatsu (Google)

Hi Jiri,

Quoted 10 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu

Hello,
(I resend this because kconfig was too big)

I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).

hi,
I can't reproduce on my setup with your .config

Quoted 8 lines
Quoted 3 lines

does 'bpftool btf dump file ./vmlinux' show any error?

is there any error in the kernel build output?

Yes, here it is. I saw these 2 lines.

die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in die__process() and happens if the dwarf_tag() associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it..

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)

e.g.

# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log 
[   21.595724] trace_fprobe: error: BTF is not available or not supported
  Command: f kfree $arg*
                   ^
[   21.596032] trace_fprobe: error: Invalid $-valiable specified
  Command: f kfree $arg*
                   ^
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

/ # strings /sys/kernel/btf/vmlinux | grep kfree

hm, if you have this file present, you have BTF in

Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. kfree_rcu_batch_init) it works.

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

FWIW I see most of the below in my BTF output on bpf-next, though I am missing a few, possibly due to differing config options since they don't appear in kallsyms either. I don't see the DWARF tag label not handled messages so it's possible that's a symptom of something

I suspect however some form of corruption in the DWARF representation may be the reason a lot of these are missing. Would be worth trying to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect.

Quoted 263 lines
Quoted 23 lines
Quoted 23 lines

kfree_on_online
maybe_kfree_parameter trace_event_raw_rcu_invoke_kfree_bulk_callback trace_event_data_offsets_rcu_invoke_kfree_bulk_callback btf_trace_rcu_invoke_kfree_bulk_callback early_boot_kfree_rcu __bpf_trace_rcu_invoke_kfree_bulk_callback perf_trace_rcu_invoke_kfree_bulk_callback trace_event_raw_event_rcu_invoke_kfree_bulk_callback trace_raw_output_rcu_invoke_kfree_bulk_callback __probestub_rcu_invoke_kfree_bulk_callback __traceiter_rcu_invoke_kfree_bulk_callback kfree_rcu_cpu_work
kfree_rcu_cpu
kfree_rcu_batch_init kfree_rcu_scheduler_running kfree_rcu_shrink_scan
kfree_rcu_shrink_count
kfree_rcu_monitor
kfree_rcu_work

Quoted 6 lines
Quoted 3 lines

Here is the gcc version which I'm using.

gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)

I tried with gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4) and latest pahole (master branch)

Curiously, with Clang 16.0.0, it works (but many different errors are shown, see below *).
So the combination of gcc/clang and pahole version can affect it.

Quoted 8 lines
Quoted 6 lines

I also attached the kernel config file.

What is the recommended combination of the tools?
Should I use Clang to build the kernel for BTF?

should work fine with both gcc and clang

And I guess it depends on compiler version, doesn't it?

Thank you,

(*)
BTF .btf.vmlinux.bin.o
die__process_unit: DW_TAG_label (0xa) @ <0x4b138> not handled!
die__process_unit: tag not supported 0xa (label)!
die__process_unit: DW_TAG_label (0xa) @ <0x4b241> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b263> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b290> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b2c0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b2eb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b317> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b33a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b35a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b37d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b39e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b3c4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b3e7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b40f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b435> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b457> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b477> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b4a5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b4d5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b505> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b530> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b560> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b585> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b5b2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b5d9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b605> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b62e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b652> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b670> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b694> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b6b3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b6d9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b705> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b72b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b753> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b77f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b7b3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b7e4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b811> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b83c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b869> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b88c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b8bd> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b8e7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b90b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b930> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b960> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b997> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4b9ce> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba00> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba24> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba4d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4ba89> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4babc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bae4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb0b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb2e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb4e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb6d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bb8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bba8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bbc5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bbe1> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc01> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc1c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc38> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc58> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc79> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bc95> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4bcb2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7efd6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7effe> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f11c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f143> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f178> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1a4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1ca> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f1fb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f22f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f25a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f28d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x7f2b7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eea81> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eea9d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeac3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeaf3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb0f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb30> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x1eeb59> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x340734> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b4a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b67> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6b8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6ba5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6bc4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6bea> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c07> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c2a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c4e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c79> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6c9b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6cc3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6ceb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x4c6d15> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x30626ac> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x30626cd> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062814> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062833> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x3062b8b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f623f6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62416> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62437> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62458> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6280b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62b09> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62b37> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62c45> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62c60> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62d69> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f62e8d> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6305a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63ec2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63edf> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63efc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f19> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f36> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f5b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63f80> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63fa5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f63fca> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcae> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fcc7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fed7> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fef0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f6fdcb> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70352> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7036f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70394> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703b1> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703d6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f703f3> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70418> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70435> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7045a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7057b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705a6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705cf> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f705f8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70621> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7064a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70673> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7069c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f706c5> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f706ee> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70716> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7073e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70767> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70790> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f707b9> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f707e2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7080b> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70834> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70864> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70892> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f708c0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f708ee> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7091c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f7094a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70978> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f709a6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f709d4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a01> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a2e> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a5c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70a8a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ab8> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ae6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b14> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b42> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70b72> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ba0> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70bce> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70bfc> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c2a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c58> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70c86> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70cb4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70ce2> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d0f> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d3c> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d6a> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70d98> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70dc6> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70df4> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e22> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e50> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e71> not handled!
die__process_unit: DW_TAG_label (0xa) @ <0x5f70e94> not handled!
LD .tmp_vmlinux.kallsyms1

Quoted 10 lines

jirka

Quoted 5 lines

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#4 lore raw
Quoted 25 lines · On Wed, 26 Jul 2023 15:46:03 +0100 Alan Maguire <[email protected]> wrote:
Quoted 15 lines · Masami Hiramatsu (Google)

Hi Jiri,

Quoted 10 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu

Hello,
(I resend this because kconfig was too big)

I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).

hi,
I can't reproduce on my setup with your .config

Quoted 8 lines
Quoted 3 lines

does 'bpftool btf dump file ./vmlinux' show any error?

is there any error in the kernel build output?

Yes, here it is. I saw these 2 lines.

die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

Quoted 6 lines

This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in die__process() and happens if the dwarf_tag() associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it..

And this warning message is not good for debugging. It would better dump the message with DIE index so that we can analyze the DWARF with other tools.

Quoted 31 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

When I'm using the distro origin pahole (v1.22) it works. (I also checked v1.23 and v1.24, both partially generated BTF)

e.g.

# echo 'f kfree $arg*' >> /sys/kernel/tracing/dynamic_events
sh: write error: Invalid argument
# cat /sys/kernel/tracing/error_log 
[   21.595724] trace_fprobe: error: BTF is not available or not supported
  Command: f kfree $arg*
                   ^
[   21.596032] trace_fprobe: error: Invalid $-valiable specified
  Command: f kfree $arg*
                   ^
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

/ # strings /sys/kernel/btf/vmlinux | grep kfree

hm, if you have this file present, you have BTF in

Yes, it seems that the BTF itself is generated, but many entries seems dropped compared with pahole v1.22. So, if a given symbol has BTF, (e.g. kfree_rcu_batch_init) it works.

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

Quoted 4 lines

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 8 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Ah, that's nice. Let me try.

Quoted 9 lines

FWIW I see most of the below in my BTF output on bpf-next, though I am missing a few, possibly due to differing config options since they don't appear in kallsyms either. I don't see the DWARF tag label not handled messages so it's possible that's a symptom of something

I suspect however some form of corruption in the DWARF representation may be the reason a lot of these are missing. Would be worth trying to "objdump -g vmlinux >vmlinux.dwarf" (file will be huge tho) I suspect.

OK, let me dump the dwarf and check what information it has for "kfree".

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#5 lore raw
Quoted 16 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 3 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

Quoted 7 lines
Quoted 5 lines

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Ah, that's nice. Let me try.

$ pahole --version 
v1.23
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

OK, so something failed.

$ grep skipping /tmp/pahole.out  | wc -l
0

Nothing to be skipped.

$ grep -w kfree /tmp/pahole.out | wc -l
0
$ grep -w vfs_read /tmp/pahole.out | wc -l
0

But both kfree and vfs_read are not found.

$ perf probe -k ./vmlinux -V kfree
Available variables at kfree
        @<kfree+0>
                (unknown_type)  object
$ perf probe -k ./vmlinux -V vfs_read
Available variables at vfs_read
        @<vfs_read+0>
                char*   buf
                loff_t* pos
                size_t  count
                struct file*    file

However, perf probe can find both in the DWARF info.

Thank you,

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#6 lore raw
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 8 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Quoted 5 lines
Quoted 1 line

Ah, that's nice. Let me try.

$ pahole --version 
v1.23

shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?

Quoted 29 lines
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

OK, so something failed.

$ grep skipping /tmp/pahole.out  | wc -l
0

Nothing to be skipped.

$ grep -w kfree /tmp/pahole.out | wc -l
0
$ grep -w vfs_read /tmp/pahole.out | wc -l
0

But both kfree and vfs_read are not found.

$ perf probe -k ./vmlinux -V kfree
Available variables at kfree
        @<kfree+0>
                (unknown_type)  object
$ perf probe -k ./vmlinux -V vfs_read
Available variables at vfs_read
        @<vfs_read+0>
                char*   buf
                loff_t* pos
                size_t  count
                struct file*    file
Quoted 4 lines

However, perf probe can find both in the DWARF info.

Thank you,

Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).

One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!

Alan

#7 lore raw
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 8 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Ah, that's nice. Let me try.

$ pahole --version 
v1.23

shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?

Here it is.

$ ldd pahole 
	linux-vdso.so.1 (0x00007ffd6b1e2000)
	libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
	libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
	libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
	libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)

Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.

I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)

            gcc-11.3 | clang-16.0

---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)

So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.

Quoted 32 lines
Quoted 32 lines
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

OK, so something failed.

$ grep skipping /tmp/pahole.out  | wc -l
0

Nothing to be skipped.

$ grep -w kfree /tmp/pahole.out | wc -l
0
$ grep -w vfs_read /tmp/pahole.out | wc -l
0

But both kfree and vfs_read are not found.

$ perf probe -k ./vmlinux -V kfree
Available variables at kfree
        @<kfree+0>
                (unknown_type)  object
$ perf probe -k ./vmlinux -V vfs_read
Available variables at vfs_read
        @<vfs_read+0>
                char*   buf
                loff_t* pos
                size_t  count
                struct file*    file

However, perf probe can find both in the DWARF info.

Quoted 7 lines
Quoted 2 lines

Thank you,

Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).

Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).

Quoted 3 lines

One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!

Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.

Thank you,

Quoted 2 lines

Alan

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#8 lore raw
Quoted 73 lines · Masami Hiramatsu (Google)
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 8 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Ah, that's nice. Let me try.

$ pahole --version 
v1.23

shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?

Here it is.

$ ldd pahole 
	linux-vdso.so.1 (0x00007ffd6b1e2000)
	libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
	libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
	libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
	libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)

Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.

I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)

            gcc-11.3 | clang-16.0

---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)

Quoted 3 lines

So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.

One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference.

You can check DWARF version associated with CUs with

readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'

Quoted 43 lines
Quoted 32 lines
Quoted 32 lines
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

OK, so something failed.

$ grep skipping /tmp/pahole.out  | wc -l
0

Nothing to be skipped.

$ grep -w kfree /tmp/pahole.out | wc -l
0
$ grep -w vfs_read /tmp/pahole.out | wc -l
0

But both kfree and vfs_read are not found.

$ perf probe -k ./vmlinux -V kfree
Available variables at kfree
        @<kfree+0>
                (unknown_type)  object
$ perf probe -k ./vmlinux -V vfs_read
Available variables at vfs_read
        @<vfs_read+0>
                char*   buf
                loff_t* pos
                size_t  count
                struct file*    file

However, perf probe can find both in the DWARF info.

Quoted 7 lines
Quoted 2 lines

Thank you,

Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).

Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).

Quoted 6 lines
Quoted 3 lines

One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!

Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.

That might help. Thanks!

Alan

Quoted 6 lines

Thank you,

Quoted 2 lines

Alan

A message that is not in the archive (sent off-list or not yet received): CA+JHD93Liq95RvfChifmnE7E9mKR42_W7rtpqgY9KAgYyGTZwQ@mail.gmail.com

#10 lore raw

PATCH RESEND 1/1 BTF is not generated for gcc-built kernel with the latest pahole

Sorry, replied only to Alan :-\

- Arnaldo

---------- Forwarded message ---------
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Tue, Aug 1, 2023 at 3:26 PM
Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
To: Alan Maguire <[email protected]>

Quoted 10 lines · Alan Maguire
Quoted 9 lines · Masami Hiramatsu (Google)
Quoted 4 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:

Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).

Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).

See below, but this one is interesting, gcc-13 works with CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!

Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.

That might help. Thanks!

Probably he needs to tweak these CONFIG_ entries:

⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools]$

And make it use CONFIG_DEBUG_INFO_DWARF4=y,
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n

For DWARF5 I need to forward port what I have in:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit

- Arnaldo

#11 lore raw
Quoted 22 lines · On Tue, 1 Aug 2023 15:28:38 -0300 Arnaldo Carvalho de Melo…

Sorry, replied only to Alan :-\

- Arnaldo

---------- Forwarded message ---------
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Tue, Aug 1, 2023 at 3:26 PM
Subject: Re: [RESEND] BTF is not generated for gcc-built kernel with the latest pahole
To: Alan Maguire <[email protected]>

Quoted 9 lines · Alan Maguire
Quoted 8 lines · Masami Hiramatsu (Google)
Quoted 4 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:

Unfortunately (or fortunately?) I haven't been able to reproduce so far I'm afraid. I used your config and built gcc 13 from source; everything worked as expected, with no warnings or missing functions (aside from the ones skipped due to inconsistent prototypes etc).

Quoted 4 lines
Quoted 1 line
Quoted 1 line

Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13).

See below, but this one is interesting, gcc-13 works with CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y?

I'm using gcc-11.3, and it seems to work.

Quoted 12 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

One other thing I can think of - is it possible libdw[arf]/libelf versions might be having an effect here? I'm using libdwarf.so.1.2, libdw-0.188, libelf-0.188. I can try and match yours. Thanks!

Both libdw/libelf are 0.181. I didn't install libdwarf. Hmm, I should update the libdw (elfutils) too.

That might help. Thanks!

Probably he needs to tweak these CONFIG_ entries:

Quoted 8 lines

⬢[acme@toolbox perf-tools]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools]$

And make it use CONFIG_DEBUG_INFO_DWARF4=y,
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=n

OK, interesting. Let me check again with DWARF4.

Thank you!

Quoted 6 lines

For DWARF5 I need to forward port what I have in:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=WIP-imported-unit

- Arnaldo

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#12 lore raw
In reply to Alan Maguire

Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu:

Quoted 74 lines · Masami Hiramatsu (Google)
Quoted 73 lines
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 8 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Ah, that's nice. Let me try.

$ pahole --version 
v1.23

shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?

Here it is.

$ ldd pahole 
	linux-vdso.so.1 (0x00007ffd6b1e2000)
	libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
	libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
	libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
	libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)

Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.

I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)

            gcc-11.3 | clang-16.0

---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)

Quoted 8 lines
Quoted 3 lines

So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.

One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that

To check that please tweak:

⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools-next]$

i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable CONFIG_DEBUG_INFO_DWARF4.

- Arnaldo

Quoted 7 lines

pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference.

You can check DWARF version associated with CUs with

readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'

#13 lore raw
Quoted 86 lines · On Tue, 1 Aug 2023 18:45:49 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Tue, Aug 01, 2023 at 06:36:49PM +0100, Alan Maguire escreveu:

Quoted 74 lines · Masami Hiramatsu (Google)
Quoted 73 lines
Quoted 26 lines · On Mon, 31 Jul 2023 16:45:24 +0100 Alan Maguire <[email protected]> wrote:
Quoted 25 lines · Masami Hiramatsu (Google)
Quoted 22 lines · On Thu, 27 Jul 2023 09:38:14 +0900 Masami Hiramatsu (Google)…
Quoted 10 lines

Yep, BPF generation is more selective about what it emits in 1.25 to avoid cases where a kernel function signature is ambiguous (multiple functions of the same name with different signatures) or where it has unexpected register use. You can observe this via pahole's --verbose option (a lot of outut is emitted):

In a built kernel directory (where unstripped vmlinux is present):

$ PAHOLE_FLAGS=$(./scripts/pahole_flags)
$ PAHOLE=/usr/local/bin/pahole
$ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out

So this will generate BTF from vmlinux DWARF info.

Quoted 8 lines

If you want to investigate why a function has been left out, look for "skipping" verbose output like this:

skipping addition of 'access_error'(access_error) due to multiple inconsistent function prototypes
skipping addition of 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) due to unexpected register used for parameter

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Ah, that's nice. Let me try.

$ pahole --version 
v1.23

shouldn't this be v1.25? Is it possible pahole is picking up the wrong libdwarves? what does "ldd pahole" say?

Here it is.

$ ldd pahole 
	linux-vdso.so.1 (0x00007ffd6b1e2000)
	libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000)
	libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000)
	libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000)
	libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000)

Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without sync the submodule.

I also confirmed the same issue on Ubuntu 22.04's combination, which update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3)

            gcc-11.3 | clang-16.0

---------------------------+----------------
v1.22 OK OK
(ubuntu)
v1.22 OK OK
(local)
v1.23 NG OK
(local)
v1.24 NG -
(local)
v1.25 NG -
(ubuntu)
v1.25 NG OK
(local)

Quoted 8 lines
Quoted 3 lines

So, as far as I checked, there is something wrong between v1.22 and v1.23 which is also related to gcc-11.3.

One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that

To check that please tweak:

Quoted 8 lines

⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
⬢[acme@toolbox perf-tools-next]$

i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable CONFIG_DEBUG_INFO_DWARF4.

Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.

  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1

And

/ # strings /sys/kernel/btf/vmlinux | wc -l 89921
/ # strings /sys/kernel/btf/vmlinux | grep -w kfree kfree

It seems the BTF is correctly generated. (with DWARF5, the number of symbols are about 30000.)
Maybe we should check the pahole version and choose the DWARF4 if BTF is selected.

Thank you!

Quoted 10 lines

- Arnaldo

Quoted 7 lines

pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference.

You can check DWARF version associated with CUs with

readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit'

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#14 lore raw
In reply to Masami Hiramatsu (Google)

Em Thu, Jul 27, 2023 at 09:38:14AM +0900, Masami Hiramatsu escreveu:

Quoted 35 lines
Quoted 25 lines · On Wed, 26 Jul 2023 15:46:03 +0100 Alan Maguire <[email protected]> wrote:
Quoted 15 lines · Masami Hiramatsu (Google)

Hi Jiri,

Quoted 10 lines · On Wed, 26 Jul 2023 12:04:08 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 5 lines · Masami Hiramatsu

Hello,
(I resend this because kconfig was too big)

I found that BTF is not generated for gcc-built kernel with that latest pahole (v1.25).

hi,
I can't reproduce on my setup with your .config

Quoted 8 lines
Quoted 3 lines

does 'bpftool btf dump file ./vmlinux' show any error?

is there any error in the kernel build output?

Yes, here it is. I saw these 2 lines.

die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)!

Quoted 6 lines

This is strange, looks like some CUs were encoded incorrectly or we are parsing incorrectly. The error originates in die__process() and happens if the dwarf_tag() associated with the DIE isn't an expected unit; it's not even a valid tag value (0) it looks like. I've not built with gcc 13 yet so it's possible that's the reason you're seeing this, I'll try to reproduce it..

Quoted 3 lines

And this warning message is not good for debugging. It would better dump the message with DIE index so that we can analyze the DWARF with other tools.

Good idea, I'll add that.

- Arnaldo