Skip to contentSkip to conversation

pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

PATCH Superseded Dominique Martinet 3 participants 9 messages Jun 26, 2024 – Jul 1, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pahole: enable --reproducible_build when SOURCE_DATE_EPOCH is set

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set.

The impact on build time appears to be minimal enough even if we incorrectly make this decision (on a Ryzen 7 7735HS):

$ hyperfine 'sh pahole-test -j --reproducible_build' 'sh pahole-test -j1' 'sh pahole-test -j' -p 'cp .tmp_vmlinux.orig .tmp_vmlinux.btf' -m 4
Benchmark 1: sh pahole-test -j --reproducible_build
  Time (mean ± σ):      3.991 s ±  0.045 s    [User: 7.223 s, System: 3.741 s]
  Range (min … max):    3.950 s …  4.042 s    4 runs

Benchmark 2: sh pahole-test -j1
Time (mean ± σ): 7.083 s ± 0.095 s [User: 4.805 s, System: 2.242 s]
Range (min … max): 6.964 s … 7.191 s 4 runs

Benchmark 3: sh pahole-test -j
Time (mean ± σ): 3.858 s ± 0.089 s [User: 13.447 s, System: 7.078 s]
Range (min … max): 3.763 s … 3.978 s 4 runs

Summary
sh pahole-test -j ran
1.03 ± 0.03 times faster than sh pahole-test -j --reproducible_build 1.84 ± 0.05 times faster than sh pahole-test -j1

Where pahole-test is the pahole command of the linux build (minus -j):
LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats --lang_exclude=rust --skip_encoding_btf_inconsistent_proto --btf_gen_optimized .tmp_vmlinux.btf "$@"

And .tmp_vmlinux.orig was the .tmp_vmlinux.btf binary before being
processed:
ld -m elf_x86_64 -z noexecstack --no-warn-rwx-segments --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn --script=./arch/x86/kernel/vmlinux.lds -o .tmp_vmlinux.btf --whole-archive vmlinux.a .vmlinux.export.o init/version-timestamp.o --no-whole-archive --start-group --end-group
---
I opened this as https://github.com/acmel/dwarves/pull/51 a couple of
weeks ago but I guess sending by e-mail is probably more appropriate.
(I didn't find a MAINTAINERS list or similar so sending to recipients of
a random patch I found, please tell me if I missed someone)

(Note that because --btf_feature init is lazy setting something like --btf_feature=default could override this if we set it too early, so I'm only checking the env var after argp parsing)

 pahole.c | 4 ++++
 1 file changed, 4 insertions(+)
pahole.c +4 -0
index 954498d2ad4f..2b010658330c 100644
@@ -3705,6 +3705,10 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	/* This being set means whoever called us tries to do a reproducible build */
+	if (getenv("SOURCE_DATE_EPOCH"))
+		conf_load.reproducible_build = true;
+
 	if (languages.str && parse_languages())
 		return rc;
 
Signature · 1 line
2.44.0
#1 lore raw
Quoted 6 lines · Dominique Martinet

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set.

From the kernel perspective it looks like as a result of [1] the setting of SOURCE_DATE_EPOCH will be converted into KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set? Are you planning for use-cases outside of kernel BTF generation?

[1]
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/

#2 lore raw

Thanks for the reply!

Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100:

Quoted 7 lines · Dominique Martinet
Quoted 5 lines

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set.

Quoted 5 lines

From the kernel perspective it looks like as a result of [1]
[1] https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
the setting of SOURCE_DATE_EPOCH will be converted into
KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the
BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set?

Hmm, that patch (2015) never got in apparently?...
And I don't see anythiing that'd make pahole output reproducible if either variables are set in the kernel either right now; we still build plenty of older kernels so even if such a patch gets in I'd rather this is checked in pahole than kbuild.

For context, the patch I'm replacing right now is this one in nixpkgs (2023):
https://github.com/NixOS/nixpkgs/commit/e1daacf987d6064bd7dc65a30e7715990bc1c011

(Where we used to force -j1 if SOURCE_DATE_EPOCH is set, and the new feature makes it more appropriate for upstreaming so I decided to send the patch)

Quoted 1 line

Are you planning for use-cases outside of kernel BTF generation?

From a quick check the only users of pahole in nixpkgs and alpine are the linux kernel, so as far as I'm concerned if the linux build is reproducible I'm happy.
OTOH I'm not sure we're actually properly setting KBUILD_BUILD_TIMESTAMP properly everywhere (for example external modules), whereas SOURCE_DATE_EPOCH is now universally set by the upper build system -- back in 2015 when that patch was sent the reproducible build efforts just started and SOURCE_DATE_EPOCH wasn't well supported, but now I can't imagine it not being set.

So I'd rather keep checking on SOURCE_DATE_EPOCH, but if you're sure there won't ever be any user other than the kernel for pahole's btf generation then I can change the patch to KBUILD_BUILD_TIMESTAMP and we can go hunt users not setting it explicitly our our side.

Thanks,

Signature · 1 line
Dominique Martinet | Asmadeus
#3 lore raw
Quoted 16 lines · Dominique Martinet

Thanks for the reply!

Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100:

Quoted 7 lines · Dominique Martinet
Quoted 5 lines

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set.

Quoted 5 lines

From the kernel perspective it looks like as a result of [1]
[1] https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
the setting of SOURCE_DATE_EPOCH will be converted into
KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the
BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set?

Quoted 3 lines

Hmm, that patch (2015) never got in apparently?...
And I don't see anythiing that'd make pahole output reproducible if either variables are set in the kernel either right now; we still build

I can send a patch for this to bpf-next if it's needed. But it'd be good to figure out the right approach first, more below...

Quoted 3 lines

plenty of older kernels so even if such a patch gets in I'd rather this is checked in pahole than kbuild.

I'm not sure that's the right answer though - there's a bunch of existing infrastructure in the kernel tree that relies on KBUILD_BUILD_TIMESTAMP, so to integrate properly with the existing kbuild infrastructure around reproducible builds feels like the right direction to me. It's a bit more work, but not too much. Arnaldo may disagree of course, and he's the maintainer so it's his decision - this is just my personal take.

We've tried to explicitly enable flags in pahole Makefile.btf so that it's obvious from pahole invocations which features are wanted, and to get that support on older kernels it would just be a matter of backporting the

if > 1.26

	use --btf_features

...patch, and a patch for adding the reproducible build option to stable kernels. Similar has been done in the past for other pahole features.

All that's needed here for bpf-next tree is I think is the below 3-line addition:

scripts/Makefile.btf +4 -2
index b75f09f3f424..40bb72662967 100644
@@ -21,6 +21,10 @@ else
 # Switch to using --btf_features for v1.26 and later.
 pahole-flags-$(call test-ge, $(pahole-ver), 126)  = -j
--btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
 
+ifneq ($(KBUILD_BUILD_TIMESTAMP),)
+pahole-flags-$(call test-ge, $(pahole-ver), 126) +=
--btf_features=reproducible_build
+endif
+
 ifneq ($(KBUILD_EXTMOD),)

module-pahole-flags-$(call test-ge, $(pahole-ver), 126) +=
--btf_features=distilled_base
endif

Can you test at your end if that does the right thing? Thanks!

Alan

#4 lore raw
Quoted 70 lines · Alan Maguire
Quoted 16 lines · Dominique Martinet

Thanks for the reply!

Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100:

Quoted 7 lines · Dominique Martinet
Quoted 5 lines

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set.

Quoted 5 lines

From the kernel perspective it looks like as a result of [1]
[1] https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
the setting of SOURCE_DATE_EPOCH will be converted into
KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the
BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set?

Quoted 3 lines

Hmm, that patch (2015) never got in apparently?...
And I don't see anythiing that'd make pahole output reproducible if either variables are set in the kernel either right now; we still build

I can send a patch for this to bpf-next if it's needed. But it'd be good to figure out the right approach first, more below...

Quoted 3 lines

plenty of older kernels so even if such a patch gets in I'd rather this is checked in pahole than kbuild.

I'm not sure that's the right answer though - there's a bunch of existing infrastructure in the kernel tree that relies on KBUILD_BUILD_TIMESTAMP, so to integrate properly with the existing kbuild infrastructure around reproducible builds feels like the right direction to me. It's a bit more work, but not too much. Arnaldo may disagree of course, and he's the maintainer so it's his decision - this is just my personal take.

We've tried to explicitly enable flags in pahole Makefile.btf so that it's obvious from pahole invocations which features are wanted, and to get that support on older kernels it would just be a matter of backporting the

if > 1.26

	use --btf_features

...patch, and a patch for adding the reproducible build option to stable kernels. Similar has been done in the past for other pahole features.

All that's needed here for bpf-next tree is I think is the below 3-line addition:

scripts/Makefile.btf +4 -3
index b75f09f3f424..40bb72662967 100644
@@ -21,6 +21,10 @@ else
 # Switch to using --btf_features for v1.26 and later.
 pahole-flags-$(call test-ge, $(pahole-ver), 126)  = -j
--btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
 
+ifneq ($(KBUILD_BUILD_TIMESTAMP),)
+pahole-flags-$(call test-ge, $(pahole-ver), 126) +=
--btf_features=reproducible_build
+endif
+
 ifneq ($(KBUILD_EXTMOD),)
 module-pahole-flags-$(call test-ge, $(pahole-ver), 126) +=
--btf_features=distilled_base
 endif
Quoted 4 lines

Can you test at your end if that does the right thing? Thanks!

Alan

FWIW I verified that with the above change, and

export KBUILD_BUILD_TIMESTAMP=$(date -I)

...the feature flag is added to pahole and as a result we get reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across multiple kernel builds.

#5 lore raw
Quoted 72 lines · Alan Maguire
Quoted 67 lines · Alan Maguire
Quoted 14 lines · Dominique Martinet

Alan Maguire wrote on Wed, Jun 26, 2024 at 03:29:37PM +0100:

Quoted 7 lines · Dominique Martinet
Quoted 5 lines

The SOURCE_DATE_EPOCH environment variable is used in reproducible builds to set build timestamps and is generally synonymous with reproducible builds: instead of making users figure out how to pass pahole flags (e.g. explicitly setting PAHOLE_FLAGS in linux) just assume that the user wants a reproducible build if this variable is set.

Quoted 5 lines

From the kernel perspective it looks like as a result of [1]
[1] https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
the setting of SOURCE_DATE_EPOCH will be converted into
KBUILD_BUILD_TIMESTAMP; I thought the plan was to add the
BTF feature flag explicitly if KBUILD_BUILD_TIMESTAMP was set?

Quoted 3 lines

Hmm, that patch (2015) never got in apparently?...
And I don't see anythiing that'd make pahole output reproducible if either variables are set in the kernel either right now; we still build

I can send a patch for this to bpf-next if it's needed. But it'd be good to figure out the right approach first, more below...

Quoted 2 lines

plenty of older kernels so even if such a patch gets in I'd rather this is checked in pahole than kbuild.

I'm not sure that's the right answer though - there's a bunch of existing infrastructure in the kernel tree that relies on KBUILD_BUILD_TIMESTAMP, so to integrate properly with the existing kbuild infrastructure around reproducible builds feels like the right direction to me. It's a bit more work, but not too much. Arnaldo may disagree of course, and he's the maintainer so it's his decision - this is just my personal take.

We've tried to explicitly enable flags in pahole Makefile.btf so that it's obvious from pahole invocations which features are wanted, and to get that support on older kernels it would just be a matter of backporting the

if > 1.26

	use --btf_features

...patch, and a patch for adding the reproducible build option to stable kernels. Similar has been done in the past for other pahole features.

All that's needed here for bpf-next tree is I think is the below 3-line addition:

scripts/Makefile.btf +4 -3
index b75f09f3f424..40bb72662967 100644
@@ -21,6 +21,10 @@ else
 # Switch to using --btf_features for v1.26 and later.
 pahole-flags-$(call test-ge, $(pahole-ver), 126)  = -j
--btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
 
+ifneq ($(KBUILD_BUILD_TIMESTAMP),)
+pahole-flags-$(call test-ge, $(pahole-ver), 126) +=
--btf_features=reproducible_build
+endif
+
 ifneq ($(KBUILD_EXTMOD),)
 module-pahole-flags-$(call test-ge, $(pahole-ver), 126) +=
--btf_features=distilled_base
 endif
Quoted 1 line

Can you test at your end if that does the right thing? Thanks!

FWIW I verified that with the above change, and

Quoted 5 lines

export KBUILD_BUILD_TIMESTAMP=$(date -I)

...the feature flag is added to pahole and as a result we get reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across multiple kernel builds.

This sounds reasonable, pahole doesn't have to look at something so specific for a particular use case (the kernel) while using what pahole would look (KBUILD_BUILD_TIMESTAMP) but in the kernel build scripts, that in turn uses the prefered, self-documenting way of asking for some specific BTF feature (--btf_features).

- Arnaldo

#6 lore raw
In reply to Alan Maguire

Alan Maguire wrote on Fri, Jun 28, 2024 at 09:31:17AM +0100:

Quoted 7 lines

FWIW I verified that with the above change, and

export KBUILD_BUILD_TIMESTAMP=$(date -I)

...the feature flag is added to pahole and as a result we get reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across multiple kernel builds.

Sorry for the lack of reply, I didn't have time to test until now, and I have no idea how to override the kernel config in nixos so the checking build through nix build took all night.
I've now also confirmed this works including for modules BTF & linux "package" build on nixos on master.

However from there I built a random kernel module and have also
confirmed they don't set KBUILD_BUILD_TIMESTAMP for external modules --
from a quick look it's not part of the default flags defined as
buildFlags here[1] because each modules' SOURCE_DATE_EPOCH isn't
available at this level (and I have no idea how to call a command at
this level), the configurePhase where it's set only applies to the
kernel itself and not the gazillion of modules.
[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/manual-config.nix

Add to that that nixos is supporting a dozen of kernels -- all mainlines stable versions but also a dozen of variants (rt, whatever xanmod and zen are, and if you look for things like mobile-nixos you'll also get much older and not update sunxi and other embedded trees) I definitely won't put the effort to backport equivalent patches to all these kernels

I wanted the patch for alpine as well but I honestly don't have the time to do the same confirmation work there; if they have any external modules I don't see the flag being set either, at least they don't have so many kernels.
(but unlike nixos the kernel build "framework" code isn't shared and I know of at least one out of tree embedded kernel that doesn't set KBUILD_BUILD_TIMESTAMP and only has SOURCE_DATE_EPOCH...)

So:
- in theory, sure; this patch works. I say we can do this independantly
of any other effort. The propere way of setting reproducible build in the kernel is not SOURCE_DATE_EPOCH but KBUILD_BUILD_TIMESTAMP and handling it at kernel makefile level is sound.
- in practice, (nixos hat) some modules won't set it but will only have
SOURCE_DATE_EPOCH available, and there are trees that won't be updated for years (and that's being optimistic), so our local patch in pahole is here to stay for a while and the benefits far outweight the rebasing work.
- (alpine hat) alpine doesn't seem to care as much about
reproducibility (the nixos patch wasn't in and nobody complained so far), undecided at this point. Makefile.btf patch is probably good enough, but since I'm maintaining the patch anyway it's no extra work to add it there, undecided at this point.

Thanks,

Signature · 1 line
Dominique Martinet | Asmadeus
#7 lore raw
Quoted 40 lines · Dominique Martinet

Alan Maguire wrote on Fri, Jun 28, 2024 at 09:31:17AM +0100:

Quoted 7 lines

FWIW I verified that with the above change, and

export KBUILD_BUILD_TIMESTAMP=$(date -I)

...the feature flag is added to pahole and as a result we get reproducible BTF in vmlinux; .btf.vmlinux.bin.o is identical across multiple kernel builds.

Sorry for the lack of reply, I didn't have time to test until now, and I have no idea how to override the kernel config in nixos so the checking build through nix build took all night.
I've now also confirmed this works including for modules BTF & linux "package" build on nixos on master.

However from there I built a random kernel module and have also
confirmed they don't set KBUILD_BUILD_TIMESTAMP for external modules --
from a quick look it's not part of the default flags defined as
buildFlags here[1] because each modules' SOURCE_DATE_EPOCH isn't
available at this level (and I have no idea how to call a command at
this level), the configurePhase where it's set only applies to the
kernel itself and not the gazillion of modules.
[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/manual-config.nix

Add to that that nixos is supporting a dozen of kernels -- all mainlines stable versions but also a dozen of variants (rt, whatever xanmod and zen are, and if you look for things like mobile-nixos you'll also get much older and not update sunxi and other embedded trees) I definitely won't put the effort to backport equivalent patches to all these kernels

I wanted the patch for alpine as well but I honestly don't have the time to do the same confirmation work there; if they have any external modules I don't see the flag being set either, at least they don't have so many kernels.
(but unlike nixos the kernel build "framework" code isn't shared and I know of at least one out of tree embedded kernel that doesn't set KBUILD_BUILD_TIMESTAMP and only has SOURCE_DATE_EPOCH...)

Quoted 5 lines

So:
- in theory, sure; this patch works. I say we can do this independantly
of any other effort. The propere way of setting reproducible build in the kernel is not SOURCE_DATE_EPOCH but KBUILD_BUILD_TIMESTAMP and handling it at kernel makefile level is sound.

Okay, so I propose we try and get the KBUILD_BUILD_TIMESTAMP-based change upstream. I'll send that later today. That doesn't mean we can't add the envvar-based method too, it just means we have something upstream that handles reproducible builds more explicitly.

Quoted 12 lines

- in practice, (nixos hat) some modules won't set it but will only have
SOURCE_DATE_EPOCH available, and there are trees that won't be updated for years (and that's being optimistic), so our local patch in pahole is here to stay for a while and the benefits far outweight the rebasing work.
- (alpine hat) alpine doesn't seem to care as much about
reproducibility (the nixos patch wasn't in and nobody complained so far), undecided at this point. Makefile.btf patch is probably good enough, but since I'm maintaining the patch anyway it's no extra work to add it there, undecided at this point.

One thing I think would be worth doing in your patch would be adding debug logging (if in verbose mode) that we are enabling reproducible builds due to the presence of the SOURCE_DATE_EPOCH envvar. I don't think we can do this in the non-debug case because build logs would just be flooded with a few thousand of these messages for each module we add BTF to using pahole. With that small addition I'd be happy with the change. Can you send a v2 with that added? Thanks!

Alan

Quoted 1 line

Thanks,

#8 lore raw

Alan Maguire wrote on Mon, Jul 01, 2024 at 05:44:15PM +0100:

Quoted 9 lines
Quoted 4 lines

- in theory, sure; this patch works. I say we can do this independantly
of any other effort. The propere way of setting reproducible build in the kernel is not SOURCE_DATE_EPOCH but KBUILD_BUILD_TIMESTAMP and handling it at kernel makefile level is sound.

Okay, so I propose we try and get the KBUILD_BUILD_TIMESTAMP-based change upstream. I'll send that later today. That doesn't mean we can't add the envvar-based method too, it just means we have something upstream that handles reproducible builds more explicitly.

Thanks.

Quoted 13 lines
Quoted 12 lines

- in practice, (nixos hat) some modules won't set it but will only have
SOURCE_DATE_EPOCH available, and there are trees that won't be updated for years (and that's being optimistic), so our local patch in pahole is here to stay for a while and the benefits far outweight the rebasing work.
- (alpine hat) alpine doesn't seem to care as much about
reproducibility (the nixos patch wasn't in and nobody complained so far), undecided at this point. Makefile.btf patch is probably good enough, but since I'm maintaining the patch anyway it's no extra work to add it there, undecided at this point.

Quoted 7 lines

One thing I think would be worth doing in your patch would be adding debug logging (if in verbose mode) that we are enabling reproducible builds due to the presence of the SOURCE_DATE_EPOCH envvar. I don't think we can do this in the non-debug case because build logs would just be flooded with a few thousand of these messages for each module we add BTF to using pahole. With that small addition I'd be happy with the change. Can you send a v2 with that added? Thanks!

That makes sense, I'll send a v2 with a debug log message tonight/tomorrow

Signature · 1 line
Dominique Martinet | Asmadeus