Skip to contentSkip to conversation

Improvements for reproducible builds

PATCH In review Maximilian Engelhardt 3 participants 8 messages Dec 18, 2020 – Dec 23, 2020

lore
Share & tools
lore
#0 lore raw

cover Improvements for reproducible builds

While working on a reproducible build of xen in Debian I came up with the following three patches which are needed to build xen reproducibly in Debian. Reproducible builds are useful to verify the binary has actually been built from the corresponding source.

The first patch is an extension of [1] which only omits the timestamp if SOURCE_DATE_EPOCH is defined. Patch two fixes an ordering issue in the generated documentation and the last patch uses the date from SOURCE_DATE_EPOCH if available in the documentation.

[1] https://lists.xenproject.org/archives/html/xen-devel/2020-10/msg02161.html

Maximilian Engelhardt (3):
xen/arch/x86: don't insert timestamp when SOURCE_DATE_EPOCH is defined
docs: use predictable ordering in generated documentation
docs: set date to SOURCE_DATE_EPOCH if available

 docs/Makefile         | 8 +++++++-
 docs/xen-headers      | 2 +-
 xen/arch/x86/Makefile | 6 ++++++
 3 files changed, 14 insertions(+), 2 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/3 xen/arch/x86: don't insert timestamp when SOURCE_DATE_EPOCH is defined

By default a timestamp gets added to the xen efi binary. Unfortunately ld doesn't seem to provide a way to set a custom date, like from SOURCE_DATE_EPOCH, so set a zero value for the timestamp (option --no-insert-timestamp) if SOURCE_DATE_EPOCH is defined. This makes reproducible builds possible.

This is an alternative to the patch suggested in [1]. This patch only omits the timestamp when SOURCE_DATE_EPOCH is defined.

[1] https://lists.xenproject.org/archives/html/xen-devel/2020-10/msg02161.html

---
 xen/arch/x86/Makefile | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/Makefile +6 -0
index 8f2180485b..863aed043f 100644
@@ -184,6 +184,12 @@ EFI_LDFLAGS += --major-image-version=$(XEN_VERSION)
 EFI_LDFLAGS += --minor-image-version=$(XEN_SUBVERSION)
 EFI_LDFLAGS += --major-os-version=2 --minor-os-version=0
 EFI_LDFLAGS += --major-subsystem-version=2 --minor-subsystem-version=0
+# It seems ld unfortunately can't set a custom timestamp, so add a zero value
+# for the timestamp (option --no-insert-timestamp) if SOURCE_DATE_EPOCH is
+# defined to make reproducible builds possible.
+ifdef SOURCE_DATE_EPOCH
+EFI_LDFLAGS += --no-insert-timestamp
+endif
 
 $(TARGET).efi: VIRT_BASE = 0x$(shell $(NM) efi/relocs-dummy.o | sed -n 's, A VIRT_START$$,,p')
 $(TARGET).efi: ALT_BASE = 0x$(shell $(NM) efi/relocs-dummy.o | sed -n 's, A ALT_START$$,,p')
Signature · 1 line
2.20.1
Acked-by on v1 1/3
#2 lore raw

Re: [XEN PATCH 1/3] xen/arch/x86: don't insert timestamp when SOURCE_DATE_EPOCH is defined

Quoted 6 lines · Maximilian Engelhardt

By default a timestamp gets added to the xen efi binary. Unfortunately ld doesn't seem to provide a way to set a custom date, like from SOURCE_DATE_EPOCH, so set a zero value for the timestamp (option --no-insert-timestamp) if SOURCE_DATE_EPOCH is defined. This makes reproducible builds possible.

Quoted 6 lines

This is an alternative to the patch suggested in [1]. This patch only omits the timestamp when SOURCE_DATE_EPOCH is defined.

[1] https://lists.xenproject.org/archives/html/xen-devel/2020-10/msg02161.html

#3 lore raw

PATCH 2/3 docs: use predictable ordering in generated documentation

In reply to Maximilian Engelhardt

When the seq number is equal, sort by the title to get predictable output ordering. This is useful for reproducible builds.

---
 docs/xen-headers | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
docs/xen-headers +1 -1
index 54155632c4..8c434d77e2 100755
@@ -331,7 +331,7 @@ sub output_index () {
 <h2>Starting points</h2>
 <ul>
 END
-    foreach my $ic (sort { $a->{Seq} <=> $b->{Seq} } @incontents) {
+    foreach my $ic (sort { $a->{Seq} <=> $b->{Seq} or $a->{Title} cmp $b->{Title} } @incontents) {
         $o .= "<li><a href=\"$ic->{Href}\">$ic->{Title}</a></li>\n";
     }
     $o .= "</ul>\n";
Signature · 1 line
2.20.1
#5 lore raw

PATCH 3/3 docs: set date to SOURCE_DATE_EPOCH if available

In reply to Maximilian Engelhardt

Use the solution described in [1] to replace the call to the 'date' command with a version that uses SOURCE_DATE_EPOCH if available. This is needed for reproducible builds.

[1] https://reproducible-builds.org/docs/source-date-epoch/

---
 docs/Makefile | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
docs/Makefile +7 -1
index 8de1efb6f5..ac6792ff7c 100644
@@ -3,7 +3,13 @@ include $(XEN_ROOT)/Config.mk
 -include $(XEN_ROOT)/config/Docs.mk
 
 VERSION		:= $(shell $(MAKE) -C $(XEN_ROOT)/xen --no-print-directory xenversion)
-DATE		:= $(shell date +%Y-%m-%d)
+
+DATE_FMT	:= +%Y-%m-%d
+ifdef SOURCE_DATE_EPOCH
+DATE		:= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)" 2>/dev/null || date -u -r "$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)" 2>/dev/null || date -u "$(DATE_FMT)")
+else
+DATE		:= $(shell date "$(DATE_FMT)")
+endif
 
 DOC_ARCHES      := arm x86_32 x86_64
 MAN_SECTIONS    := 1 5 7 8
Signature · 1 line
2.20.1
#6 lore raw

Re: [XEN PATCH 3/3] docs: set date to SOURCE_DATE_EPOCH if available

docs/Makefile include $(XEN_ROOT)/Config.mk · line 9

Quoted 11 lines · Maximilian Engelhardt
docs/Makefile +4 -1
@@ -3,7 +3,13 @@ include $(XEN_ROOT)/Config.mk
 -include $(XEN_ROOT)/config/Docs.mk
 
 VERSION		:= $(shell $(MAKE) -C $(XEN_ROOT)/xen --no-print-directory xenversion)
-DATE		:= $(shell date +%Y-%m-%d)
+
+DATE_FMT	:= +%Y-%m-%d
+ifdef SOURCE_DATE_EPOCH
+DATE		:= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)" 2>/dev/null || date -u -r "$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)" 2>/dev/null || date -u "$(DATE_FMT)")

Looking at the doc for a (deliberately) old "date", I can't find any mention of the -d "@..." syntax. I take it the command would fail on that system. It would then go on to try the -r variant, which has entirely different meaning on GNU (Linux) systems.

docs/ being subject to configuring, why don't you determine the capabilities of "date" there and invoke just the one command that was found suitable for the system?

Jan

#7 lore raw

Re: [XEN PATCH 3/3] docs: set date to SOURCE_DATE_EPOCH if available

Quoted 21 lines · On Montag, 21. Dezember 2020 10:01:14 CET Jan Beulich wrote:
Quoted 13 lines · Maximilian Engelhardt
docs/Makefile +4 -1
@@ -3,7 +3,13 @@ include $(XEN_ROOT)/Config.mk
 
 -include $(XEN_ROOT)/config/Docs.mk
 
 VERSION		:= $(shell $(MAKE) -C $(XEN_ROOT)/xen --no-print-directory
 xenversion)> 
-DATE		:= $(shell date +%Y-%m-%d)
+
+DATE_FMT	:= +%Y-%m-%d
+ifdef SOURCE_DATE_EPOCH
+DATE		:= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)"
Quoted 2 lines
2>/dev/null || date -u -r "$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)"
2>/dev/null || date -u "$(DATE_FMT)")

Looking at the doc for a (deliberately) old "date", I can't find any mention of the -d "@..." syntax. I take it the command would fail on that system. It would then go on to try the -r variant, which has entirely different meaning on GNU (Linux) systems.

Quoted 5 lines

docs/ being subject to configuring, why don't you determine the capabilities of "date" there and invoke just the one command that was found suitable for the system?

Jan

Hi Jan,

I did some research. The -d "@..." syntax was introduced about 2005. Testing a live CD from 2006 (KNOPPIX_V5.0.1CD-2006-06-01-EN.iso) it was supported there. The documentation about this syntax has only been added in 2011 to the date command. I'm wondering if anybody running such an old system wants to use SOURCE_DATE_EPOCH.

However, I came up with a patch to determine which suitable date version is available and only call that, as you suggested. I will post the new patch soon.

Maxi