Skip to content

RFC BTF archive with unmodified pahole+toolchain

Arnaldo Carvalho de Melo bpf dwarves 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest4 patches 16 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4libbpf: Simplify error handling removing needless repeated err checkstools/lib/bpf/btf.c+1 -30 replies
2/4libbpf: Check if there is extra data at the end of a BTFtools/lib/bpf/btf.c+32 -100 replies
3/4libbpf: Add support for detecting and dedup'ing a BTF archivetools/lib/bpf/btf.c tools/lib/bpf/btf.h+41 -00 replies
4/4libbpf: Check if an ELF .BTF section is an archive and combine/deduptools/lib/bpf/btf.c+7 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC 0/4] BTF archive with unmodified pahole+toolchain

From: Arnaldo Carvalho de Melo <[email protected]>

Hi,

I've finally managed to act on some idea I shared with a few
folks while in Montreal, namely using unmodified pahole to generate BTF for each .o right after it is produced, i.e. with this patch:

  acme@number:~/git/linux$ git diff
  diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
  index 1d581ba5df66f2b5..ad9e788910636715 100644
  --- a/scripts/Makefile.lib
  +++ b/scripts/Makefile.lib
  @@ -240,7 +240,7 @@ cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -
   endif
   quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
  -      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \
  +      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< && ${PAHOLE} --btf_encode ${PAHOLE_FLAGS} $@ \
                  $(cmd_ld_single) \
                  $(cmd_objtool)

acme@number:~/git/linux$

A kernel built with this ends up with a vmlinux with a .BTF section that has all the .o .BTF sections concatenated.

This (the series of .BTF concatenated by the unmodified linker) somehow survives the pre-existing pahole call to generate BTF from DWARF and we end up with this "BTF archive".

With the minimal set of changes in this series:

 tools/lib/bpf/btf.c | 91 ++++++++++++++++++++++++++++++++++++++-------
 tools/lib/bpf/btf.h |  3 ++
 2 files changed, 81 insertions(+), 13 deletions(-)

With the first patch being just a trivial error handling simplification, we end up being able to get the same vmlinux.h result from bpftool built with this libbpf:

  acme@number:~/git/bpf-next$ tools/bpf/bpftool/bpftool btf dump file ~/vmlinux-v6.16.0+.btf_archive format c >
  +from_archive_combined+dedup_in_libbpf
  acme@number:~/git/bpf-next$ tools/bpf/bpftool/bpftool btf dump file ../build/v6.16.0+/vmlinux format c >
  +from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf
  acme@number:~/git/bpf-next$ diff -u from_archive_combined+dedup_in_libbpf from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf | head
  acme@number:~/git/bpf-next$ wc -l from_archive_combined+dedup_in_libbpf from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf
   161588 from_archive_combined+dedup_in_libbpf
   161588 from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf
   323176 total
  acme@number:~/git/bpf-next$

If we use completely unmodified libbpf, bpftool, etc, the "BTF archive" in the resulting vmlinux .BTF ELF section is still consumable, but just the first "CU" (the first .o .BTF ELF section) is visible, the one for init/main.o:

acme@number:~/git/linux$ bpftool version
bpftool v7.5.0
using libbpf v1.5
features: llvm, skeletons
acme@number:~/git/linux$
acme@number:~/git/bpf-next$ bpftool btf dump file ~/vmlinux-v6.16.0+.btf_archive format c | wc -l
11361
acme@number:~/git/linux$ bpftool btf dump file ../build/v6.16.0+/init/main.o format c | wc -l
11361
acme@number:~/git/linux$

Furthermore:

acme@number:~/git/linux$ bpftool btf dump file ../build/v6.16.0+/init/main.o format c > a
acme@number:~/git/linux$ bpftool btf dump file ~/vmlinux-v6.16.0+.btf_archive format c > b
acme@number:~/git/linux$

Each patch has extra explanations of the process.

This is complementary to today's series from Alan Maguire, as we can use the one liner for the kernel build process to test his series without requiring installing a toolchain that generates BTF for each .o file that will result in vmlinux.

Next steps on my side are to:

1. change pahole for when it receives --format_path=btf check if btf__is_archive(btf) is true, then just replace the current vmlinux .BTF contents with the raw data in this just loaded BTF, short circuiting the whole process.

2. the kernel build process should be changed to allow one to ask for just BTF, not DWARF, and if so, using the above method, strip the DWARF info after using it to generate BTF.

Then when compilers are producing BTF, we switch to that, falling back to the above method when a compiler is known to generate buggy BTF.

And also to use in CIs, to compare the output generated by the various methods in the various components.

3. In 2 we can even use the same scheme we use for parallelizing DWARF loading when loading all the BTF archive members concatenated in vmlinux to dedup them.

BTW, this is the size of a vmlinux ELF .BTF section with an BTF archive:

acme@number:~/git/linux$ readelf -SW ../build/v6.16.0+/vmlinux | grep BTF
  [15] .BTF        PROGBITS   ffffffff82ff2000  21f2000 16db5976 00   A  0   0  1
  [16] .BTF_ids    PROGBITS   ffffffff99da8000 18fa8000   001238 00   A  0   0  1
acme@number:~/git/linux$

~365 MiB

While the DWARF for that file is at:

  [44] .debug_aranges    PROGBITS  0000000000000000 1aa00000   03bfb0 00      0   0 16
  [45] .debug_info       PROGBITS  0000000000000000 1aa3bfb0 1154b512 00      0   0  1
  [46] .debug_abbrev     PROGBITS  0000000000000000 2bf874c2   81492f 00      0   0  1
  [47] .debug_line       PROGBITS  0000000000000000 2c79bdf1  1ec4abd 00      0   0  1
  [48] .debug_frame      PROGBITS  0000000000000000 2e6608b0   3fd470 00      0   0  8
  [49] .debug_str        PROGBITS  0000000000000000 2ea5dd20   59bbe8 01  MS  0   0  1
  [50] .debug_line_str   PROGBITS  0000000000000000 2eff9908   02de43 01  MS  0   0  1
  [51] .debug_loclists   PROGBITS  0000000000000000 2f02774b  2683cbb 00      0   0  1
  [52] .debug_rnglists   PROGBITS  0000000000000000 316ab406   4f875b 00      0   0  1
Quoted 1 line
Quoted 1 line
Quoted 1 line

0x1154b512 + 0x81492f + 0x1ec4abd + 0x3fd470 + 0x59bbe8

341563734

~325 MiB

But then BTF, when dedup'ed gets down to:

acme@number:~/git/linux$ readelf -SW ../build/v6.16.0+.no-btf_archive/vmlinux | grep BTF
  [15] .BTF              PROGBITS        ffffffff82fef000 21ef000 64fb32 00   A  0   0  1
  [16] .BTF_ids          PROGBITS        ffffffff8363f000 283f000 001238 00   A  0   0  1
acme@number:~/git/linux$ 

~6.3 MiB

And also BTF has some info generated from other sources besides DWARF, like kfuncs, per cpu, etc.

Also an observation: for distros the optimal way to produce BTF _and_ DWARF seems to be is the one we have now, don't bother generating .BTF for all .o, just generate DWARF and at the end generate BTF from it 8-)

For developers not needing DWARF and not caring about reproducible builds then there are other clever tricks to use like go on adding each generated BTF using the technique in this patchset, i.e. using btf__add_btf() and trowing away the just generated BTF to then at the end do the btf__archive_dedup() (also introduced in this patchset) to have the end result dropped to disk. But I'm getting carried away, sry.

There are many other details that need to be double checked but I think the current status is good enough for experimentation.

Cheers,

- Arnaldo

Arnaldo Carvalho de Melo (4):
libbpf: Simplify error handling removing needless repeated err checks
libbpf: Check if there is extra data at the end of a BTF
libbpf: Add support for detecting and dedup'ing a BTF archive
libbpf: Check if an ELF .BTF section is an archive and combine/dedup

 tools/lib/bpf/btf.c | 91 ++++++++++++++++++++++++++++++++++++++-------
 tools/lib/bpf/btf.h |  3 ++
 2 files changed, 81 insertions(+), 13 deletions(-)
Signature · 1 line
2.50.1