Skip to content

pahole: Reproducible parallel DWARF loading/serial BTF encoding

Arnaldo Carvalho de Melo bpf dwarves 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest12 patches 23 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12core: Allow asking for a reproducible builddwarves.h pahole.c+9 -00 replies
2/12pahole: Disable BTF multithreaded encoded when doing reproducible buildspahole.c+22 -34 replies
3/12dwarf_loader: Separate creating the cu/dcu pair from processing itdwarf_loader.c+19 -51 reply
4/12dwarf_loader: Introduce dwarf_cus__process_cu()dwarf_loader.c+11 -70 replies
5/12dwarf_loader: Create the cu/dcu pair in dwarf_cus__nextcu()dwarf_loader.c+15 -40 replies
6/12dwarf_loader: Remove unused 'thr_data' arg from dwarf_cus__create_and_process_cu()dwarf_loader.c+3 -50 replies
7/12core: Add unlocked cus__add() variantdwarves.c dwarves.h+7 -30 replies
8/12core: Add cus__remove(), counterpart of cus__add()dwarves.c dwarves.h+17 -00 replies
9/12dwarf_loader: Add the cu to the cus list early, remove on LSK_DELETEdwarf_loader.c+11 -50 replies
10/12core/dwarf_loader: Add functions to set state of CU processingdwarf_loader.c dwarves.c dwarves.h+56 -30 replies
11/12pahole: Encode BTF serially in a reproducible buildpahole.c+50 -10 replies
12/12tests: Add a BTF reproducible generation testtests/reproducible_build.sh+56 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC/PATCHES 00/12] pahole: Reproducible parallel DWARF loading/serial BTF encoding

Hi,

This allows us to have reproducible builds while keeping the
DWARF loading phase in parallel, achieving a noticeable speedup as showed in the commit log messages:

On a:

model name : Intel(R) Core(TM) i7-14700K

8 performance cores (16 threads), 12 efficiency cores.

Serial encoding:

  $ perf stat -e cycles -r5 pahole --btf_encode_detached=vmlinux.btf.serial vmlinux
             5.18276 +- 0.00952 seconds time elapsed  ( +-  0.18% )

Parallel, but non-reproducible:

  $ perf stat -e cycles -r5 pahole -j --btf_encode_detached=vmlinux.btf.parallel vmlinux
              1.8529 +- 0.0159 seconds time elapsed  ( +-  0.86% )

reproducible build done using parallel DWARF loading + CUs-ordered-as-in-vmlinux serial BTF encoding:

  $ perf stat -e cycles -r5 pahole -j --reproducible_build --btf_encode_detached=vmlinux.btf.parallel.reproducible_build vmlinux
              2.3632 +- 0.0164 seconds time elapsed  ( +-  0.69% )

Please take a look, its in the 'next' branch at:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=next

There is a new tool to do regression testing on this feature:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=c751214c19bf8591bf8e4abdc677cbadee08f630

And here a more detailed set of tests using it:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=4451467ca16a6e31834f6f98661c63587ce556f7

Working on libbpf to allow for parallel reproducible BTF encoding is the next step.

Thanks a lot,

- Arnaldo

Arnaldo Carvalho de Melo (12):
core: Allow asking for a reproducible build
pahole: Disable BTF multithreaded encoded when doing reproducible builds
dwarf_loader: Separate creating the cu/dcu pair from processing it
dwarf_loader: Introduce dwarf_cus__process_cu()
dwarf_loader: Create the cu/dcu pair in dwarf_cus__nextcu()
dwarf_loader: Remove unused 'thr_data' arg from dwarf_cus__create_and_process_cu()
core: Add unlocked cus__add() variant
core: Add cus__remove(), counterpart of cus__add()
dwarf_loader: Add the cu to the cus list early, remove on LSK_DELETE core/dwarf_loader: Add functions to set state of CU processing
pahole: Encode BTF serially in a reproducible build
tests: Add a BTF reproducible generation test

 dwarf_loader.c              | 73 +++++++++++++++++++++++---------
 dwarves.c                   | 58 ++++++++++++++++++++++++-
 dwarves.h                   | 17 ++++++++
 pahole.c                    | 84 +++++++++++++++++++++++++++++++++++--
 tests/reproducible_build.sh | 56 +++++++++++++++++++++++++
 5 files changed, 264 insertions(+), 24 deletions(-)
 create mode 100755 tests/reproducible_build.sh
Signature · 1 line
2.44.0