pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set
PATCH v3 1/1 pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set
The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set, printing a debug message in verbose mode.
The impact on build time appears to be minimal enough even if we incorrectly make this decision (on a Ryzen 7 7735HS):
$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4 Benchmark 1: sh pahole-test -j --reproducible_build Time (mean ± σ): 3.991 s ± 0.045 s [User: 7.223 s, System: 3.741 s] Range (min … max): 3.950 s … 4.042 s 4 runs
Benchmark 2: sh pahole-test -j1
Time (mean ± σ): 7.083 s ± 0.095 s [User: 4.805 s, System: 2.242 s]
Range (min … max): 6.964 s … 7.191 s 4 runs
Benchmark 3: sh pahole-test -j
Time (mean ± σ): 3.858 s ± 0.089 s [User: 13.447 s, System: 7.078 s]
Range (min … max): 3.763 s … 3.978 s 4 runs
Summary
sh pahole-test -j ran
1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build 1.84 ± 0.05 times faster than sh pahole-test -j1
Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"
And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group
---
v2 -> v3:
- added message about the env var to man page.
- fixed typo / style of comment in code
v1 -> v2:
- added message in verbose mode if setting reproducible build through this.
man-pages/pahole.1 | 3 +++ pahole.c | 11 +++++++++++ 2 files changed, 14 insertions(+)
man-pages/pahole.1 +3 -0
@@ -327,6 +327,9 @@ If one wants to add an extra feature to the set of standard ones, the '+' prefix \-\-btf_features=+reproducible_build will add all default features plus the 'reproducible_build' extra feature. +Alternatively, reproducible_build will also be enabled if the SOURCE_DATE_EPOCH environment variable +is set, as that is commonly set in build systems that aim for reproducible builds. + .TP .B \-\-btf_features_strict Identical to \-\-btf_features above, but pahole will exit if it encounters an unrecognized feature.
pahole.c +11 -0
@@ -3705,6 +3705,17 @@ int main(int argc, char *argv[]) goto out; } +/* +* SOURCE_DATE_EPOCH being set means whoever called us likely tries +* to do a reproducible build. The setting only impacts btf_encode code +* so only check in this case to avoid needless messages. +*/ +if (btf_encode &&getenv ("SOURCE_DATE_EPOCH" )) { +if (!conf_load.reproducible_build && global_verbose) +printf ("pahole: auto-enabling reproducible build (SOURCE_DATE_EPOCH set)\n" ); + conf_load.reproducible_build = true; + } +if (languages.str &&parse_languages ())return rc;
Signature · 1 line
2.45.1
Thanks for the reply!
This can wait until you're back to work, no worry about latency.
Arnaldo Carvalho de Melo wrote on Sun, Jul 07, 2024 at 11:12:06AM -0300:
Quoted 6 lines
I'm in vacation, but it occurred to me that a
--set_from_env SOURCE_DATE_EPOCH=reproducible_build
Meaning: if variable
SOURCE_DATE_EPOCH
Is set, set the boolean command linereproducible_build.
(I understood this as a new command line option to set any btf feature from env, if the above option is passed to pahole)
From my position there are two faces to this:
- for new kernels, a change in Kbuild to add som flag would definitely work.
The patch to set --btf_features=reproducible_build to pahole from Kbuild if KBUILD_BUILD_TIMESTAMP is set got shot down by Masahiro Yamada, but hopefully something will be worked out eventually...
- for nixos/old kernels, the current pahole version produces reproducible output if SOURCE_DATE_EPOCH is set (because someone patched it to default to -j1 in this case in our tree), and now the reproducible build option exists I'd like to upstream something more appropriate that'll "just work" without worrying about old kernel trees and adding new explicit pahole arguments there.
So adding a new switch to set a flag from env probably won't help me much, but a new env var like PAHOLE_BTF_FEATURES (PAHOLE_FLAGS is used/overwritten by Kbuild...) that would just call into the --btf-feature parsing code, e.g. exporting PAHOLE_BTF_FEATURES=+reproducible_build would toggle the option) would probably work for me.
(Have to be careful that this is not the same as --reproducible-build, so perhaps something else might be better... but regardless it'll probably work out for linux kernel builds)
It's less convenient than having SOURCE_DATE_EPOCH just "make it work", but at least there'll be a single place to edit that'll work on all older kernels for a given distro.
(and the "just work" side of things might come for newer kernels at some point for distros that care less...)
What do you think?
Signature · 1 line
Dominique Martinet | Asmadeus