Skip to contentSkip to conversation

Improve reproducible builds

PATCH In review Frédéric Pierret (fepitre) 4 participants 9 messages Oct 30, 2020

lore
Share & tools
lore
#0 lore raw

cover Improve reproducible builds

This two fixes improve reproducibility of resulting Xen binaries

Frédéric Pierret (fepitre) (2):
No insert of the build timestamp into the x86 xen efi binary xen/common/makefile: remove gzip timestamp

 xen/arch/x86/Makefile | 1 +
 xen/common/Makefile   | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/2 No insert of the build timestamp into the x86 xen efi binary

This is for improving reproducible builds.

---
 xen/arch/x86/Makefile | 1 +
 1 file changed, 1 insertion(+)
xen/arch/x86/Makefile +1 -0
index b388861679..f5a529afd5 100644
@@ -170,6 +170,7 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+EFI_LDFLAGS += --no-insert-timestamp
 
 # Check if the compiler supports the MS ABI.
 export XEN_BUILD_EFI := $(shell $(CC) $(XEN_CFLAGS) -c efi/check.c -o efi/check.o 2>/dev/null && echo y)
Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH v1 1/2] No insert of the build timestamp into the x86 xen efi binary

xen/arch/x86/Makefile EFI_LDFLAGS += --major-image-version=$(XEN_VERSION) · line 173

Quoted 7 lines · Frédéric Pierret (fepitre)
xen/arch/x86/Makefile +1 -0
@@ -170,6 +170,7 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+EFI_LDFLAGS += --no-insert-timestamp

Generally I prefer binaries to carry timestamps, when they are intended to do so (i.e. when they have a respective field). So I think if no timestamp is wanted, it should be as an option (not sure about the default).

This said, I didn't think time stamps got meaningfully in the way of reproducible builds - ignoring the minor differences cause by them, especially when they sit at well known offsets in the binaries, shouldn't be a big deal.

Jan

#3 lore raw

Re: [PATCH v1 1/2] No insert of the build timestamp into the x86 xen efi binary

Quoted 14 lines · Jan Beulich

xen/arch/x86/Makefile EFI_LDFLAGS += --major-image-version=$(XEN_VERSION) · line 173

Quoted 7 lines · Frédéric Pierret (fepitre)
xen/arch/x86/Makefile +1 -0
@@ -170,6 +170,7 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+EFI_LDFLAGS += --no-insert-timestamp

Generally I prefer binaries to carry timestamps, when they are intended to do so (i.e. when they have a respective field). So I think if no timestamp is wanted, it should be as an option (not sure about the default).

What about setting it to the SOURCE_DATE_EPOCH[1] variable value, if present? Of course if there is an option to set explicit timestamp value.

[1] https://reproducible-builds.org/docs/source-date-epoch/

Quoted 4 lines

This said, I didn't think time stamps got meaningfully in the way of reproducible builds - ignoring the minor differences cause by them, especially when they sit at well known offsets in the binaries, shouldn't be a big deal.

It is a big deal. There is a huge difference between running sha256sum (or your other favorite hash) on two build artifacts, and using a specialized tool/script to compare each file separately. Note the xen.efi may be buried very deep in the thing you compare, for example inside deb/rpm and then ISO image (installation image), at which point it's far from "they sit at well known offsets".

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
#4 lore raw

Re: [PATCH v1 1/2] No insert of the build timestamp into the x86 xen efi binary

Quoted 16 lines · Marek Marczykowski-Górecki
Quoted 14 lines · Jan Beulich

xen/arch/x86/Makefile EFI_LDFLAGS += --major-image-version=$(XEN_VERSION) · line 173

Quoted 7 lines · Frédéric Pierret (fepitre)
xen/arch/x86/Makefile +1 -0
@@ -170,6 +170,7 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+EFI_LDFLAGS += --no-insert-timestamp

Generally I prefer binaries to carry timestamps, when they are intended to do so (i.e. when they have a respective field). So I think if no timestamp is wanted, it should be as an option (not sure about the default).

Quoted 5 lines

What about setting it to the SOURCE_DATE_EPOCH[1] variable value, if present? Of course if there is an option to set explicit timestamp value.

[1] https://reproducible-builds.org/docs/source-date-epoch/

Why not.

Quoted 11 lines
Quoted 4 lines

This said, I didn't think time stamps got meaningfully in the way of reproducible builds - ignoring the minor differences cause by them, especially when they sit at well known offsets in the binaries, shouldn't be a big deal.

It is a big deal. There is a huge difference between running sha256sum (or your other favorite hash) on two build artifacts, and using a specialized tool/script to compare each file separately. Note the xen.efi may be buried very deep in the thing you compare, for example inside deb/rpm and then ISO image (installation image), at which point it's far from "they sit at well known offsets".

If you care about checking images / blobs where binaries with time stamps are merely constituent parts, why not strip the time stamps at the time of creation of those images / blobs (or as a minor intermediate step, in case you want to e.g. record the hashes for later reference)?

Jan

#5 lore raw

Re: [PATCH v1 1/2] No insert of the build timestamp into the x86 xen efi binary

Quoted 15 lines · Jan Beulich
Quoted 14 lines · Marek Marczykowski-Górecki
Quoted 13 lines · Jan Beulich

xen/arch/x86/Makefile EFI_LDFLAGS += --major-image-version=$(XEN_VERSION) · line 173

Quoted 7 lines · Frédéric Pierret (fepitre)
xen/arch/x86/Makefile +1 -0
@@ -170,6 +170,7 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+EFI_LDFLAGS += --no-insert-timestamp

Generally I prefer binaries to carry timestamps, when they are intended to do so (i.e. when they have a respective field). So I think if no timestamp is wanted, it should be as an option (not sure about the default).

Quoted 6 lines
Quoted 5 lines

What about setting it to the SOURCE_DATE_EPOCH[1] variable value, if present? Of course if there is an option to set explicit timestamp value.

[1] https://reproducible-builds.org/docs/source-date-epoch/

Why not.

SOURCE_DATE_EPOCH is the right way to fix this.

It probably wants to default to something sane in the root Makefile, so it covers tools as well.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

This said, I didn't think time stamps got meaningfully in the way of reproducible builds - ignoring the minor differences cause by them, especially when they sit at well known offsets in the binaries, shouldn't be a big deal.

It is a big deal. There is a huge difference between running sha256sum (or your other favorite hash) on two build artifacts, and using a specialized tool/script to compare each file separately. Note the xen.efi may be buried very deep in the thing you compare, for example inside deb/rpm and then ISO image (installation image), at which point it's far from "they sit at well known offsets".

Quoted 5 lines

If you care about checking images / blobs where binaries with time stamps are merely constituent parts, why not strip the time stamps at the time of creation of those images / blobs (or as a minor intermediate step, in case you want to e.g. record the hashes for later reference)?

Because that is a disaster to maintain.  A critical part of reproducible builds is not needing custom comparison logic for every binary artefact.

~Andrew

#6 lore raw

Re: [PATCH v1 1/2] No insert of the build timestamp into the x86 xen efi binary

Quoted 16 lines · Andrew Cooper
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Marek Marczykowski-Górecki
Quoted 13 lines · Jan Beulich

xen/arch/x86/Makefile EFI_LDFLAGS += --major-image-version=$(XEN_VERSION) · line 173

Quoted 7 lines · Frédéric Pierret (fepitre)
xen/arch/x86/Makefile +1 -0
@@ -170,6 +170,7 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+EFI_LDFLAGS += --no-insert-timestamp

Generally I prefer binaries to carry timestamps, when they are intended to do so (i.e. when they have a respective field). So I think if no timestamp is wanted, it should be as an option (not sure about the default).

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

What about setting it to the SOURCE_DATE_EPOCH[1] variable value, if present? Of course if there is an option to set explicit timestamp value.

[1] https://reproducible-builds.org/docs/source-date-epoch/

Why not.

SOURCE_DATE_EPOCH is the right way to fix this.

Hmm, reading 'ld' man page, I don't see an option to set explicit value, on --insert-timestamp / --no-insert-timestamp.

Quoted 2 lines

It probably wants to default to something sane in the root Makefile, so it covers tools as well.

In practice, the package build system (deb, rpm, etc) do set it based on last package changelog entry, so _for package build case_ it isn't needed. But probably nice addition.

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
#7 lore raw

PATCH 2/2 xen/common/makefile: remove gzip timestamp

In reply to Frédéric Pierret (fepitre)

This is for improving reproducible builds.

---
 xen/common/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/Makefile +1 -1
index 06881d023c..32cd650ba8 100644
@@ -77,7 +77,7 @@ obj-$(CONFIG_HAS_DEVICE_TREE) += libfdt/
 
 CONF_FILE := $(if $(patsubst /%,,$(KCONFIG_CONFIG)),$(XEN_ROOT)/xen/)$(KCONFIG_CONFIG)
 config.gz: $(CONF_FILE)
-	gzip -c $< >$@
+	gzip -n -c $< >$@
 
 config_data.o: config.gz
 
Signature · 1 line
2.26.2