Skip to contentSkip to conversation

pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

PATCH Superseded Dominique Martinet 2 participants 2 messages Jul 5, 2024 – Jul 6, 2024

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set, printing a debug message in verbose mode.

The impact on build time appears to be minimal enough even if we incorrectly make this decision (on a Ryzen 7 7735HS):

$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4
Benchmark 1: sh pahole-test -j --reproducible_build
  Time (mean ± σ):      3.991 s ±  0.045 s    [User: 7.223 s, System: 3.741 s]
  Range (min … max):    3.950 s …  4.042 s    4 runs

Benchmark 2: sh pahole-test -j1
Time (mean ± σ): 7.083 s ± 0.095 s [User: 4.805 s, System: 2.242 s]
Range (min … max): 6.964 s … 7.191 s 4 runs

Benchmark 3: sh pahole-test -j
Time (mean ± σ): 3.858 s ± 0.089 s [User: 13.447 s, System: 7.078 s]
Range (min … max): 3.763 s … 3.978 s 4 runs

Summary
sh pahole-test -j ran
1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build 1.84 ± 0.05 times faster than sh pahole-test -j1

Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"

And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group

---
v1 -> v2:
- added message in verbose mode if setting reproducible build through this.
I wasn't quite sure what is least intrusive message-wise -- most verbose messages just go to stdout, but using pahole in a script to extract some c structs would see this mangled in output which is obviously not great. Since --reproducible_build only impacts BTF generation at this time, I'm only checking when generating BTF to avoid such needless messages -- btf generation always goes through a file so there is no risk of mangling the output in this case.

 pahole.c | 9 +++++++++
 1 file changed, 9 insertions(+)
pahole.c +9 -0
index c866095c8be6..2792e11265d6 100644
@@ -3705,6 +3705,15 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	// SOURCE_DATE_EPOCH being set means whoever called us likeely tries
+	// to do a reproducible build. The setting only impacts btf_encode code
+	// so only check in this case to avoid needless messages.
+	if (btf_encode && getenv("SOURCE_DATE_EPOCH")) {
+		if (!conf_load.reproducible_build && global_verbose)
+			printf("pahole: auto-enabling reproducible build (SOURCE_DATE_EPOCH set)\n");
+		conf_load.reproducible_build = true;
+	}
+
 	if (languages.str && parse_languages())
 		return rc;
 
Signature · 1 line
2.45.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 27 lines · Dominique Martinet

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set, printing a debug message in verbose mode.

The impact on build time appears to be minimal enough even if we incorrectly make this decision (on a Ryzen 7 7735HS):

$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4
Benchmark 1: sh pahole-test -j --reproducible_build
  Time (mean ± σ):      3.991 s ±  0.045 s    [User: 7.223 s, System: 3.741 s]
  Range (min … max):    3.950 s …  4.042 s    4 runs

Benchmark 2: sh pahole-test -j1
Time (mean ± σ): 7.083 s ± 0.095 s [User: 4.805 s, System: 2.242 s]
Range (min … max): 6.964 s … 7.191 s 4 runs

Benchmark 3: sh pahole-test -j
Time (mean ± σ): 3.858 s ± 0.089 s [User: 13.447 s, System: 7.078 s]
Range (min … max): 3.763 s … 3.978 s 4 runs

Summary
sh pahole-test -j ran
1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build 1.84 ± 0.05 times faster than sh pahole-test -j1

Quoted 9 lines

Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"

And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group

One small thing below, and asuggestion; it'd be good to mention this behaviour in man-pages/pahole.1 where we discuss the reproducible_build feature.

Something like the commit message would do, i.e.

"The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds so the "reproducible_build" BTF feature is automatically set when this environment variable is set. This ensures that kernel builds will generated reproducible BTF in such cases."

Thanks!

Alan

Quoted 12 lines

---
v1 -> v2:
- added message in verbose mode if setting reproducible build through this.
I wasn't quite sure what is least intrusive message-wise -- most verbose messages just go to stdout, but using pahole in a script to extract some c structs would see this mangled in output which is obviously not great. Since --reproducible_build only impacts BTF generation at this time, I'm only checking when generating BTF to avoid such needless messages -- btf generation always goes through a file so there is no risk of mangling the output in this case.

yep, that seems like a reasonable approach to me.

pahole.c int main(int argc, char *argv[]) · line 3708

Quoted 12 lines
 pahole.c | 9 +++++++++
 1 file changed, 9 insertions(+)
pahole.c +1 -0
index c866095c8be6..2792e11265d6 100644
@@ -3705,6 +3705,15 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	// SOURCE_DATE_EPOCH being set means whoever called us likeely tries

nit: typo on likely. also (tho it's not always 100% consistent I admit I'd use a multi-line comment style

/*
* SOURCE_DATE_EPOCH being set means whoever called us likely tries
...
*/

Quoted 11 lines
pahole.c +8 -0
+	// to do a reproducible build. The setting only impacts btf_encode code
+	// so only check in this case to avoid needless messages.
+	if (btf_encode && getenv("SOURCE_DATE_EPOCH")) {
+		if (!conf_load.reproducible_build && global_verbose)
+			printf("pahole: auto-enabling reproducible build (SOURCE_DATE_EPOCH set)\n");
+		conf_load.reproducible_build = true;
+	}
+
 	if (languages.str && parse_languages())
 		return rc;