Skip to contentSkip to conversation

Reproducibility: use of SOURCE_DATE_EPOCH

PATCH In review Frédéric Pierret (fepitre) 3 participants 13 messages Oct 31, 2020 – Nov 3, 2020

lore
Share & tools
lore
#0 lore raw

cover Reproducibility: use of SOURCE_DATE_EPOCH

It intends to take into account feedback on previous patch removing timestamp from xen efi binary. If defined, we can set build dates and time variables with respect to SOURCE_DATE_EPOCH. A default value is provided if sources come from git repository else, that's up to the builder to handle it.

Frédéric Pierret (fepitre) (2):
Define build dates/time based on SOURCE_DATE_EPOCH Define SOURCE_DATE_EPOCH based on git log

 tools/firmware/hvmloader/Makefile | 4 ++++
 tools/firmware/vgabios/Makefile   | 4 ++++
 xen/Makefile                      | 7 +++++++
 3 files changed, 15 insertions(+)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/2 Define build dates/time based on SOURCE_DATE_EPOCH

It improves reproducibility if SOURCE_DATE_EPOCH is defined while building xen binary

---
 tools/firmware/hvmloader/Makefile | 4 ++++
 tools/firmware/vgabios/Makefile   | 4 ++++
 xen/Makefile                      | 5 +++++
 3 files changed, 13 insertions(+)
tools/firmware/hvmloader/Makefile +4 -0
index e980ce7c5f..923e3c8b9a 100644
@@ -21,7 +21,11 @@ XEN_ROOT = $(CURDIR)/../../..
 include $(XEN_ROOT)/tools/firmware/Rules.mk
 
 # SMBIOS spec requires format mm/dd/yyyy
+ifneq ($(SOURCE_DATE_EPOCH),)
+SMBIOS_REL_DATE ?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "+%m/%d/%Y" 2>/dev/null)
+else
 SMBIOS_REL_DATE ?= $(shell date +%m/%d/%Y)
+endif
 
 CFLAGS += $(CFLAGS_xeninclude)
 
tools/firmware/vgabios/Makefile +4 -0
index 3284812fde..9b8b687a73 100644
@@ -5,7 +5,11 @@ BCC = bcc
 AS86 = as86
 
 RELEASE = `pwd | sed "s-.*/--"`
+ifneq ($(SOURCE_DATE_EPOCH),)
+VGABIOS_REL_DATE ?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "+%d %b %Y" 2>/dev/null)
+else
 VGABIOS_REL_DATE ?= `date '+%d %b %Y'`
+endif
 RELVERS = `pwd | sed "s-.*/--" | sed "s/vgabios//" | sed "s/-//"`
 
 VGABIOS_DATE = "-DVGABIOS_DATE=\"$(VGABIOS_REL_DATE)\""
xen/Makefile +5 -0
index bf0c804d43..30b1847515 100644
@@ -8,8 +8,13 @@ export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
 
 export XEN_WHOAMI	?= $(USER)
 export XEN_DOMAIN	?= $(shell ([ -x /bin/dnsdomainname ] && /bin/dnsdomainname) || ([ -x /bin/domainname ] && /bin/domainname || echo [unknown]))
+ifneq ($(SOURCE_DATE_EPOCH),)
+export XEN_BUILD_DATE	?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" 2>/dev/null)
+export XEN_BUILD_TIME	?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" +%T 2>/dev/null)
+else
 export XEN_BUILD_DATE	?= $(shell LC_ALL=C date)
 export XEN_BUILD_TIME	?= $(shell LC_ALL=C date +%T)
+endif
 export XEN_BUILD_HOST	?= $(shell hostname)
 
 # Best effort attempt to find a python interpreter, defaulting to Python 3 if
Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH v1 1/2] Define build dates/time based on SOURCE_DATE_EPOCH

tools/firmware/hvmloader/Makefile XEN_ROOT = $(CURDIR)/../../.. · line 28

Quoted 11 lines · Frédéric Pierret (fepitre)
tools/firmware/hvmloader/Makefile +4 -0
@@ -21,7 +21,11 @@ XEN_ROOT = $(CURDIR)/../../..
 include $(XEN_ROOT)/tools/firmware/Rules.mk
 
 # SMBIOS spec requires format mm/dd/yyyy
+ifneq ($(SOURCE_DATE_EPOCH),)
+SMBIOS_REL_DATE ?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "+%m/%d/%Y" 2>/dev/null)
+else
 SMBIOS_REL_DATE ?= $(shell date +%m/%d/%Y)
+endif

As this pattern recurs, how about abstracting it away via a definition (perhaps to be placed in ./Config.mk) along the lines of (variable name subject to improvement)

DATE_EPOCH_OPTS := $(if $(SOURCE_DATE_EPOCH),-u -d "@$(SOURCE_DATE_EPOCH)")

and then here simply

SMBIOS_REL_DATE ?= $(shell date $(DATE_EPOCH_OPTS) +%m/%d/%Y)

(i.e. in particular also without any "ifneq()")?

Jan

#3 lore raw

Re: [PATCH v1 1/2] Define build dates/time based on SOURCE_DATE_EPOCH

Le 11/3/20 à 10:10 AM, Jan Beulich a écrit :

Quoted 19 lines · Frédéric Pierret (fepitre)

tools/firmware/hvmloader/Makefile XEN_ROOT = $(CURDIR)/../../.. · line 28

Quoted 11 lines
tools/firmware/hvmloader/Makefile +4 -0
@@ -21,7 +21,11 @@ XEN_ROOT = $(CURDIR)/../../..
  include $(XEN_ROOT)/tools/firmware/Rules.mk
 
  # SMBIOS spec requires format mm/dd/yyyy
+ifneq ($(SOURCE_DATE_EPOCH),)
+SMBIOS_REL_DATE ?= $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "+%m/%d/%Y" 2>/dev/null)
+else
  SMBIOS_REL_DATE ?= $(shell date +%m/%d/%Y)
+endif

As this pattern recurs, how about abstracting it away via a definition (perhaps to be placed in ./Config.mk) along the lines of (variable name subject to improvement)

DATE_EPOCH_OPTS := $(if $(SOURCE_DATE_EPOCH),-u -d "@$(SOURCE_DATE_EPOCH)")

Quoted 8 lines

and then here simply

SMBIOS_REL_DATE ?= $(shell date $(DATE_EPOCH_OPTS) +%m/%d/%Y)

(i.e. in particular also without any "ifneq()")?

Jan

Hi Jan,

Yes it makes sense. I'll prepare another version with your comments. Thank you.

Frédéric

#4 lore raw

PATCH 2/2 Define SOURCE_DATE_EPOCH based on git log

In reply to Frédéric Pierret (fepitre)
---
 xen/Makefile | 2 ++
 1 file changed, 2 insertions(+)
xen/Makefile +2 -0
index 30b1847515..4cc35556ef 100644
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
 export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
 -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)
+
 export XEN_WHOAMI	?= $(USER)
 export XEN_DOMAIN	?= $(shell ([ -x /bin/dnsdomainname ] && /bin/dnsdomainname) || ([ -x /bin/domainname ] && /bin/domainname || echo [unknown]))
 ifneq ($(SOURCE_DATE_EPOCH),)
Signature · 1 line
2.26.2
#5 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 7 lines · Frédéric Pierret (fepitre)
xen/Makefile +1 -0
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
 export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
 -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

In patch 1 you also use the variable under tools/. Why do you place this here rather than in the top level Makefile?

This said I'm not convinced anyway that we want this to be the default. I'd rather see this as something to get set by the package build process of distros, outside of any of the source files.

Jan

#6 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

Le 11/3/20 à 10:15 AM, Jan Beulich a écrit :

Quoted 12 lines · Frédéric Pierret (fepitre)

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 7 lines
xen/Makefile +1 -0
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
  export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
  -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

In patch 1 you also use the variable under tools/. Why do you place this here rather than in the top level Makefile?

Quoted 7 lines

This said I'm not convinced anyway that we want this to be the default. I'd rather see this as something to get set by the package build process of distros, outside of any of the source files.

Jan

In fact this was intended to provide a default/example value. Indeed, each package manager should handle this with changelog or such.

This is not mandatory and if not wanted by default, maybe add this example value into the INSTALL documentation?

Frédéric

#7 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

Quoted 20 lines · Frédéric Pierret

Le 11/3/20 à 10:15 AM, Jan Beulich a écrit :

Quoted 12 lines · Frédéric Pierret (fepitre)

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 7 lines
xen/Makefile +1 -0
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
  export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
  -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

In patch 1 you also use the variable under tools/. Why do you place this here rather than in the top level Makefile?

Quoted 4 lines

This said I'm not convinced anyway that we want this to be the default. I'd rather see this as something to get set by the package build process of distros, outside of any of the source files.

Quoted 4 lines

In fact this was intended to provide a default/example value. Indeed, each package manager should handle this with changelog or such.

This is not mandatory and if not wanted by default, maybe add this example value into the INSTALL documentation?

I'm certainly fine with putting this in the docs. (Whether INSTALL is the right place I can#t tell.)

Jan

#8 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

In reply to Frédéric Pierret (fepitre)

Hi Frédéric,

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 13 lines · Frédéric Pierret (fepitre)
---
  xen/Makefile | 2 ++
  1 file changed, 2 insertions(+)
xen/Makefile +1 -0
index 30b1847515..4cc35556ef 100644
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
  export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
  -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

It is possible to download a tarball for Xen release (see [1]). They don't contain the .git directory and therefore this command would fail.

Should we fallback to "date" in this case?

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 13

Quoted 5 lines
xen/Makefile +1 -0
+
  export XEN_WHOAMI	?= $(USER)
  export XEN_DOMAIN	?= $(shell ([ -x /bin/dnsdomainname ] && /bin/dnsdomainname) || ([ -x /bin/domainname ] && /bin/domainname || echo [unknown]))
  ifneq ($(SOURCE_DATE_EPOCH),)

Cheers,

[1] https://xenproject.org/downloads/

Signature · 1 line
Julien Grall
#9 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

Quoted 15 lines · Julien Grall

Hi Frédéric,

Quoted 12 lines · Frédéric Pierret (fepitre)
---
  xen/Makefile | 2 ++
  1 file changed, 2 insertions(+)
xen/Makefile +0 -0
index 30b1847515..4cc35556ef 100644
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
  export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
  -include xen-version
 
Quoted 6 lines

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 1 line
xen/Makefile +1 -0
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

It is possible to download a tarball for Xen release (see [1]). They don't contain the .git directory and therefore this command would fail.

Should we fallback to "date" in this case?

Isn't this what already happens? The variable would be assigned an empty value in this case, wouldn't it?

Jan

#10 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

Le 11/3/20 à 11:05 AM, Jan Beulich a écrit :

Quoted 3 lines · Julien Grall
Quoted 2 lines

Hi Frédéric,

Hi Julien,

Quoted 18 lines
Quoted 18 lines · Frédéric Pierret (fepitre)

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 13 lines
---
   xen/Makefile | 2 ++
   1 file changed, 2 insertions(+)
xen/Makefile +1 -0
index 30b1847515..4cc35556ef 100644
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
   export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
   -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

It is possible to download a tarball for Xen release (see [1]). They don't contain the .git directory and therefore this command would fail.

Quoted 4 lines
Quoted 1 line

Should we fallback to "date" in this case?

Isn't this what already happens? The variable would be assigned an empty value in this case, wouldn't it?

Julien, Jan, yes it already fallback to "date" if the variable is empty (it's the reason of "2>/dev/null") in the other test of check if SOURCE_DATE_EPOCH is defined. Maybe there is more elegant way for this. Depending on the wanted here for providing or not a default value in case of git sources, this could be documented instead as suggested previously.

Quoted 2 lines

Jan

Regards,
Frédéric

#11 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

In reply to Jan Beulich
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Julien Grall

Hi Frédéric,

xen/Makefile export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION) · line 9

Quoted 13 lines · Frédéric Pierret (fepitre)
---
   xen/Makefile | 2 ++
   1 file changed, 2 insertions(+)
xen/Makefile +1 -0
index 30b1847515..4cc35556ef 100644
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
   export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
   -include xen-version
 
+export SOURCE_DATE_EPOCH	?= $(shell git log -1 --format=%ct 2>/dev/null)

It is possible to download a tarball for Xen release (see [1]). They don't contain the .git directory and therefore this command would fail.

Quoted 4 lines
Quoted 1 line

Should we fallback to "date" in this case?

Isn't this what already happens? The variable would be assigned an empty value in this case, wouldn't it?

My question was whether empty SOURCE_DATE_EPOCH is acceptable?

Looking at patch #1, the users of the variable will use "date" if it is empty. Why can't this behavior be common?

Cheers,

Signature · 1 line
Julien Grall
#12 lore raw

Re: [PATCH v1 2/2] Define SOURCE_DATE_EPOCH based on git log

Le 11/3/20 à 11:11 AM, Julien Grall a écrit :

Quoted 25 lines · Jan Beulich
Quoted 22 lines · Julien Grall
Quoted 20 lines

Hi Frédéric,

Quoted 10 lines · Frédéric Pierret (fepitre)

---
   xen/Makefile | 2 ++

   1 file changed, 2 insertions(+)
xen/Makefile +0 -0
index 30b1847515..4cc35556ef 100644
@@ -6,6 +6,8 @@ export XEN_EXTRAVERSION ?= -unstable$(XEN_VENDORVERSION)
    export XEN_FULLVERSION   = $(XEN_VERSION).$(XEN_SUBVERSION)$(XEN_EXTRAVERSION)
Quoted 2 lines

   -include xen-version
+export SOURCE_DATE_EPOCH    ?= $(shell git log -1 --format=%ct 2>/dev/null)

It is possible to download a tarball for Xen release (see [1]). They don't contain the .git directory and therefore this command would fail.

Should we fallback to "date" in this case?

Quoted 9 lines
Quoted 2 lines

Isn't this what already happens? The variable would be assigned an empty value in this case, wouldn't it?

My question was whether empty SOURCE_DATE_EPOCH is acceptable?

Looking at patch #1, the users of the variable will use "date" if it is empty. Why can't this behavior be common?

Cheers,

In fact, we could fallback to date in SOURCE_DATE_EPOCH definition and in this case this would always be defined. Now, I'm wondering how misleading that could be with respect to its definition (see [1]): "The value MUST be reproducible (deterministic) across different executions of the build, depending only on the source code.". In this case, if someone looks to the code and interpret the build time etc, defined with respect to SOURCE_DATE_EPOCH, that would be odd?

Regards,
Frédéric

[1]: https://reproducible-builds.org/specs/source-date-epoch/