Skip to contentSkip to conversation

pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

PATCH In review Dominique Martinet 1 participant 2 messages Jul 7, 2024 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set, printing a debug message in verbose mode.

The impact on build time appears to be minimal enough even if we incorrectly make this decision (on a Ryzen 7 7735HS):

$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4
Benchmark 1: sh pahole-test -j --reproducible_build
  Time (mean ± σ):      3.991 s ±  0.045 s    [User: 7.223 s, System: 3.741 s]
  Range (min … max):    3.950 s …  4.042 s    4 runs

Benchmark 2: sh pahole-test -j1
Time (mean ± σ): 7.083 s ± 0.095 s [User: 4.805 s, System: 2.242 s]
Range (min … max): 6.964 s … 7.191 s 4 runs

Benchmark 3: sh pahole-test -j
Time (mean ± σ): 3.858 s ± 0.089 s [User: 13.447 s, System: 7.078 s]
Range (min … max): 3.763 s … 3.978 s 4 runs

Summary
sh pahole-test -j ran
1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build 1.84 ± 0.05 times faster than sh pahole-test -j1

Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"

And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group

---
v2 -> v3:
- added message about the env var to man page.
- fixed typo / style of comment in code

v1 -> v2:
- added message in verbose mode if setting reproducible build through this.

 man-pages/pahole.1 |  3 +++
 pahole.c           | 11 +++++++++++
 2 files changed, 14 insertions(+)
man-pages/pahole.1 +3 -0
index f0605935a9f1..d9c2c1ac6912 100644
@@ -327,6 +327,9 @@ If one wants to add an extra feature to the set of standard ones, the '+' prefix
 \-\-btf_features=+reproducible_build will add all default features plus the 'reproducible_build' extra
 feature.
 
+Alternatively, reproducible_build will also be enabled if the SOURCE_DATE_EPOCH environment variable
+is set, as that is commonly set in build systems that aim for reproducible builds.
+
 .TP
 .B \-\-btf_features_strict
 Identical to \-\-btf_features above, but pahole will exit if it encounters an unrecognized feature.
pahole.c +11 -0
index c866095c8be6..7724980ff106 100644
@@ -3705,6 +3705,17 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	/*
+	 * SOURCE_DATE_EPOCH being set means whoever called us likely tries
+	 * to do a reproducible build. The setting only impacts btf_encode code
+	 * so only check in this case to avoid needless messages.
+	 */
+	if (btf_encode && getenv("SOURCE_DATE_EPOCH")) {
+		if (!conf_load.reproducible_build && global_verbose)
+			printf("pahole: auto-enabling reproducible build (SOURCE_DATE_EPOCH set)\n");
+		conf_load.reproducible_build = true;
+	}
+
 	if (languages.str && parse_languages())
 		return rc;
 
Signature · 1 line
2.45.1
#2 lore raw
In reply to a message not in the archive

Thanks for the reply!

This can wait until you're back to work, no worry about latency.

Arnaldo Carvalho de Melo wrote on Sun, Jul 07, 2024 at 11:12:06AM -0300:

Quoted 6 lines

I'm in vacation, but it occurred to me that a

--set_from_env SOURCE_DATE_EPOCH=reproducible_build

Meaning: if variable SOURCE_DATE_EPOCH
Is set, set the boolean command line reproducible_build.

(I understood this as a new command line option to set any btf feature from env, if the above option is passed to pahole)

From my position there are two faces to this:
- for new kernels, a change in Kbuild to add som flag would definitely work.
The patch to set --btf_features=reproducible_build to pahole from Kbuild if KBUILD_BUILD_TIMESTAMP is set got shot down by Masahiro Yamada, but hopefully something will be worked out eventually...

- for nixos/old kernels, the current pahole version produces reproducible output if SOURCE_DATE_EPOCH is set (because someone patched it to default to -j1 in this case in our tree), and now the reproducible build option exists I'd like to upstream something more appropriate that'll "just work" without worrying about old kernel trees and adding new explicit pahole arguments there.

So adding a new switch to set a flag from env probably won't help me much, but a new env var like PAHOLE_BTF_FEATURES (PAHOLE_FLAGS is used/overwritten by Kbuild...) that would just call into the --btf-feature parsing code, e.g. exporting PAHOLE_BTF_FEATURES=+reproducible_build would toggle the option) would probably work for me.
(Have to be careful that this is not the same as --reproducible-build, so perhaps something else might be better... but regardless it'll probably work out for linux kernel builds)

It's less convenient than having SOURCE_DATE_EPOCH just "make it work", but at least there'll be a single place to edit that'll work on all older kernels for a given distro.
(and the "just work" side of things might come for newer kernels at some point for distros that care less...)

What do you think?

Signature · 1 line
Dominique Martinet | Asmadeus