Skip to contentSkip to conversation

Reproducible builds

PATCH New Jérôme Pouiller 2 participants 21 messages Nov 17, 2016 – Nov 18, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Reproducible builds

This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

I dropped some patchs from original series because either:
- I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH

    in gcc, ...)

- I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
- They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)

This version focuses on timestamps. It provide good enough results as soon as OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files. Only patch called "remove full path from .pyc" try to solve this issue. Another big step could be done by removing rpaths from ELF generated with libtool.

Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
- since gcc versions supporting SOURCE_DATE_EPOCH are not widely available, external toolchains probably won't work.
- /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION, ccache is incompatible with reproducible
- debug symbols are not reproducible

Since this feature is experimental I did not (yet) reported these incompatibilities in menuconfig.

Gilles Chanteperdrix (3):
reproducibility: generate SOURCE_DATE_EPOCH reproducibility/linux: override build timestamp reproducibility/busybox: disable build timestamps

J?r?me Pouiller (12):
gcc6: honor SOURCE_DATE_EPOCH
gcc5: honor SOURCE_DATE_EPOCH
reproducible: add '-n' to gzip invocations fs/tar: make results reproducible reproducibility/linux: inhibit build-id
reproducible: lock modification times in $TARGET_DIR
fakedate: new package
reproducible: enable fakedate
python2: generate reproducible .pyc
python3: generate reproducible .pyc
python2: remove full path from .pyc
python3: remove full path from .pyc

 Makefile                                           |   5 +
 fs/common.mk                                       |   3 +
 fs/tar/tar.mk                                      |   2 +-
 linux/linux.mk                                     |  15 ++
 package/busybox/busybox.mk                         |   6 +
 package/fakedate/fakedate                          |  28 +++
 package/fakedate/fakedate.mk                       |  14 ++
 .../gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch    | 213 +++++++++++++++++++
 .../gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch    | 234 +++++++++++++++++++++
 package/python/python.mk                           |  36 ++--
 package/python3/python3.mk                         |  35 +--
 support/dependencies/check-host-tar.sh             |   5 +-
 toolchain/toolchain/toolchain.mk                   |   4 +
 13 files changed, 570 insertions(+), 30 deletions(-)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk
 create mode 100644 package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch
 create mode 100644 package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/15 gcc6: honor SOURCE_DATE_EPOCH

Support $SOURCE_DATE_EPOCH is necessary to create reproducible build. Unfortunately, it is not yet widely available. This patch add support for SOURCE_DATE_EPOCH to Buildroot internal toolchains based on gcc 6.

This work was sponsored by `BA Robotic Systems'.

---
 .../gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch    | 234 +++++++++++++++++++++
 1 file changed, 234 insertions(+)
 create mode 100644 package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch
package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch new file +234 -0
new file mode 100644
index 0000000..3ed2e25
@@ -0,0 +1,234 @@
+From e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934 Mon Sep 17 00:00:00 2001
+From: doko <doko@138bc75d-0d04-0410-961f-82ee72b054a4>
+Date: Thu, 28 Apr 2016 09:12:05 +0000
+Subject: [PATCH] gcc/c-family/ChangeLog:
+
+Upstream status:
+  Accepted.
+  See http://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934
+
+2016-04-28  Eduard Sanou  <[email protected]>
+	    Matthias Klose  <[email protected]>
+
+	* c-common.c (get_source_date_epoch): New function, gets the environment
+	variable SOURCE_DATE_EPOCH and parses it as long long with error
+	handling.
+	* c-common.h (get_source_date_epoch): Prototype.
+	* c-lex.c (c_lex_with_flags): set parse_in->source_date_epoch.
+
+gcc/ChangeLog:
+
+2016-04-28  Eduard Sanou  <[email protected]>
+	    Matthias Klose  <[email protected]>
+
+	* doc/cppenv.texi: Document SOURCE_DATE_EPOCH environment variable.
+
+libcpp/ChangeLog:
+
+2016-04-28  Eduard Sanou  <[email protected]>
+	    Matthias Klose  <[email protected]>
+
+	* include/cpplib.h (cpp_init_source_date_epoch): Prototype.
+	* init.c (cpp_init_source_date_epoch): New function.
+	* internal.h: Added source_date_epoch variable to struct
+	cpp_reader to store a reproducible date.
+	* macro.c (_cpp_builtin_macro_text): Set pfile->date timestamp from
+	pfile->source_date_epoch instead of localtime if source_date_epoch is
+	set, to be used for __DATE__ and __TIME__ macros to help reproducible
+	builds.
+
+Signed-off-by: Eduard Sanou <[email protected]>
+Signed-off-by: Matthias Klose <[email protected]>
+
+git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk at 235550 138bc75d-0d04-0410-961f-82ee72b054a4
+---
+ gcc/c-family/c-common.c |   33 +++++++++++++++++++++++++++++++++
+ gcc/c-family/c-common.h |    5 +++++
+ gcc/c-family/c-lex.c    |    3 +++
+ gcc/doc/cppenv.texi     |   17 +++++++++++++++++
+ libcpp/include/cpplib.h |    3 +++
+ libcpp/init.c           |    9 ++++++++-
+ libcpp/internal.h       |    4 ++++
+ libcpp/macro.c          |   21 ++++++++++++++-------
+
+diff --git a/gcc/c-family/c-common.c b/gcc/c-family/c-common.c
+index 1f0d76a..c086dee 100644
+--- a/gcc/c-family/c-common.c
++++ b/gcc/c-family/c-common.c
+@@ -12784,4 +12784,37 @@ valid_array_size_p (location_t loc, tree type, tree name)
+   return true;
+ }
+ 
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++time_t
++get_source_date_epoch ()
++{
++  char *source_date_epoch;
++  long long epoch;
++  char *endptr;
++
++  source_date_epoch = getenv ("SOURCE_DATE_EPOCH");
++  if (!source_date_epoch)
++    return (time_t) -1;
++
++  errno = 0;
++  epoch = strtoll (source_date_epoch, &endptr, 10);
++  if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
++      || (errno != 0 && epoch == 0))
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "strtoll: %s\n", xstrerror(errno));
++  if (endptr == source_date_epoch)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "no digits were found: %s\n", endptr);
++  if (*endptr != '\0')
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "trailing garbage: %s\n", endptr);
++  if (epoch < 0)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "value must be nonnegative: %lld \n", epoch);
++
++  return (time_t) epoch;
++}
++
+ #include "gt-c-family-c-common.h"
+diff --git a/gcc/c-family/c-common.h b/gcc/c-family/c-common.h
+index 1309549..3a7805f 100644
+--- a/gcc/c-family/c-common.h
++++ b/gcc/c-family/c-common.h
+@@ -1471,4 +1471,9 @@ extern bool valid_array_size_p (location_t, tree, tree);
+ extern bool cilk_ignorable_spawn_rhs_op (tree);
+ extern bool cilk_recognize_spawn (tree, tree *);
+ 
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++extern time_t get_source_date_epoch (void);
++
+ #endif /* ! GCC_C_COMMON_H */
+diff --git a/gcc/c-family/c-lex.c b/gcc/c-family/c-lex.c
+index 6b020a4..ff7eb25 100644
+--- a/gcc/c-family/c-lex.c
++++ b/gcc/c-family/c-lex.c
+@@ -388,6 +388,9 @@ c_lex_with_flags (tree *value, location_t *loc, unsigned char *cpp_flags,
+   enum cpp_ttype type;
+   unsigned char add_flags = 0;
+   enum overflow_type overflow = OT_NONE;
++  time_t source_date_epoch = get_source_date_epoch ();
++
++  cpp_init_source_date_epoch (parse_in, source_date_epoch);
+ 
+   timevar_push (TV_CPP);
+  retry:
+diff --git a/gcc/doc/cppenv.texi b/gcc/doc/cppenv.texi
+index 22c8cb3..e958e93 100644
+--- a/gcc/doc/cppenv.texi
++++ b/gcc/doc/cppenv.texi
+@@ -79,4 +79,21 @@ main input file is omitted.
+ @ifclear cppmanual
+ @xref{Preprocessor Options}.
+ @end ifclear
++
++ at item SOURCE_DATE_EPOCH
++
++If this variable is set, its value specifies a UNIX timestamp to be
++used in replacement of the current date and time in the @code{__DATE__}
++and @code{__TIME__} macros, so that the embedded timestamps become
++reproducible.
++
++The value of @env{SOURCE_DATE_EPOCH} must be a UNIX timestamp,
++defined as the number of seconds (excluding leap seconds) since
++01 Jan 1970 00:00:00 represented in ASCII, identical to the output of
++@samp{@command{date +%s}}.
++
++The value should be a known timestamp such as the last modification
++time of the source or package and it should be set by the build
++process.
++
+ @end vtable
+diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h
+index 35b0375..4998b3a 100644
+--- a/libcpp/include/cpplib.h
++++ b/libcpp/include/cpplib.h
+@@ -784,6 +784,9 @@ extern void cpp_init_special_builtins (cpp_reader *);
+ /* Set up built-ins like __FILE__.  */
+ extern void cpp_init_builtins (cpp_reader *, int);
+ 
++/* Initialize the source_date_epoch value.  */
++extern void cpp_init_source_date_epoch (cpp_reader *, time_t);
++
+ /* This is called after options have been parsed, and partially
+    processed.  */
+ extern void cpp_post_options (cpp_reader *);
+diff --git a/libcpp/init.c b/libcpp/init.c
+index 4343075..f5ff85b 100644
+--- a/libcpp/init.c
++++ b/libcpp/init.c
+@@ -533,8 +533,15 @@ cpp_init_builtins (cpp_reader *pfile, int hosted)
+     _cpp_define_builtin (pfile, "__OBJC__ 1");
+ }
+ 
++/* Initialize the source_date_epoch value.  */
++void
++cpp_init_source_date_epoch (cpp_reader *pfile, time_t source_date_epoch)
++{
++  pfile->source_date_epoch = source_date_epoch; 
++}
++
+ /* Sanity-checks are dependent on command-line options, so it is
+-   called as a subroutine of cpp_read_main_file ().  */
++   called as a subroutine of cpp_read_main_file.  */
+ #if CHECKING_P
+ static void sanity_checks (cpp_reader *);
+ static void sanity_checks (cpp_reader *pfile)
+diff --git a/libcpp/internal.h b/libcpp/internal.h
+index 9ce8707..e3eb26b 100644
+--- a/libcpp/internal.h
++++ b/libcpp/internal.h
+@@ -502,6 +502,10 @@ struct cpp_reader
+   const unsigned char *date;
+   const unsigned char *time;
+ 
++  /* Externally set timestamp to replace current date and time useful for
++     reproducibility.  */
++  time_t source_date_epoch;
++
+   /* EOF token, and a token forcing paste avoidance.  */
+   cpp_token avoid_paste;
+   cpp_token eof;
+diff --git a/libcpp/macro.c b/libcpp/macro.c
+index c251553..c2a8376 100644
+--- a/libcpp/macro.c
++++ b/libcpp/macro.c
+@@ -357,13 +357,20 @@ _cpp_builtin_macro_text (cpp_reader *pfile, cpp_hashnode *node,
+ 	  time_t tt;
+ 	  struct tm *tb = NULL;
+ 
+-	  /* (time_t) -1 is a legitimate value for "number of seconds
+-	     since the Epoch", so we have to do a little dance to
+-	     distinguish that from a genuine error.  */
+-	  errno = 0;
+-	  tt = time(NULL);
+-	  if (tt != (time_t)-1 || errno == 0)
+-	    tb = localtime (&tt);
++	  /* Set a reproducible timestamp for __DATE__ and __TIME__ macro
++	     usage if SOURCE_DATE_EPOCH is defined.  */
++	  if (pfile->source_date_epoch != (time_t) -1)
++	     tb = gmtime (&pfile->source_date_epoch);
++	  else
++	    {
++	      /* (time_t) -1 is a legitimate value for "number of seconds
++		 since the Epoch", so we have to do a little dance to
++		 distinguish that from a genuine error.  */
++	      errno = 0;
++	      tt = time (NULL);
++	      if (tt != (time_t)-1 || errno == 0)
++		tb = localtime (&tt);
++	    }
+ 
+ 	  if (tb)
+ 	    {
+-- 
+1.7.1
+
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/15 gcc5: honor SOURCE_DATE_EPOCH

In reply to Jérôme Pouiller

Support $SOURCE_DATE_EPOCH is necessary to create reproducible build. Unfortunately, it is not yet widely available. This patch add support for SOURCE_DATE_EPOCH to Buildroot internal toolchains based on gcc 5.

This work was sponsored by `BA Robotic Systems'.

---
 .../gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch    | 213 +++++++++++++++++++++
 1 file changed, 213 insertions(+)
 create mode 100644 package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch
package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch new file +213 -0
new file mode 100644
index 0000000..0b644a7
@@ -0,0 +1,213 @@
+From: J?r?me Pouiller <[email protected]>
+Date: Fri, 04 Nov 2016 11:45:07 +0100
+Subject: [PATCH] Honor SOURCE_DATE_EPOCH
+
+Upstream status:
+  Accepted.
+  See http://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934
+
+Current patch is backported from equivalent written for gcc 6.1
+
+2016-04-28  Eduard Sanou  <[email protected]>
+	    Matthias Klose  <[email protected]>
+
+	* c-common.c (get_source_date_epoch): New function, gets the environment
+	variable SOURCE_DATE_EPOCH and parses it as long long with error
+	handling.
+	* c-common.h (get_source_date_epoch): Prototype.
+	* c-lex.c (c_lex_with_flags): set parse_in->source_date_epoch.
+
+gcc/ChangeLog:
+
+2016-04-28  Eduard Sanou  <[email protected]>
+	    Matthias Klose  <[email protected]>
+
+	* doc/cppenv.texi: Document SOURCE_DATE_EPOCH environment variable.
+
+libcpp/ChangeLog:
+
+2016-04-28  Eduard Sanou  <[email protected]>
+	    Matthias Klose  <[email protected]>
+
+	* include/cpplib.h (cpp_init_source_date_epoch): Prototype.
+	* init.c (cpp_init_source_date_epoch): New function.
+	* internal.h: Added source_date_epoch variable to struct
+	cpp_reader to store a reproducible date.
+	* macro.c (_cpp_builtin_macro_text): Set pfile->date timestamp from
+	pfile->source_date_epoch instead of localtime if source_date_epoch is
+	set, to be used for __DATE__ and __TIME__ macros to help reproducible
+	builds.
+
+Signed-off-by: J?r?me Pouiller <[email protected]>
+
+---
+ gcc/c-family/c-common.c |   33 +++++++++++++++++++++++++++++++++
+ gcc/c-family/c-common.h |    6 ++++++
+ gcc/c-family/c-lex.c    |    3 +++
+ gcc/doc/cppenv.texi     |   17 +++++++++++++++++
+ libcpp/include/cpplib.h |    3 +++
+ libcpp/init.c           |    7 +++++++
+ libcpp/internal.h       |    4 ++++
+ libcpp/macro.c          |   21 ++++++++++++++-------
+ 8 files changed, 87 insertions(+), 7 deletions(-)
+
+--- a/gcc/c-family/c-common.c
++++ b/gcc/c-family/c-common.c
+@@ -12318,4 +12318,37 @@ pointer_to_zero_sized_aggr_p (tree t)
+   return (TYPE_SIZE (t) && integer_zerop (TYPE_SIZE (t)));
+ }
+ 
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++time_t
++get_source_date_epoch ()
++{
++  char *source_date_epoch;
++  long long epoch;
++  char *endptr;
++
++  source_date_epoch = getenv ("SOURCE_DATE_EPOCH");
++  if (!source_date_epoch)
++    return (time_t) -1;
++
++  errno = 0;
++  epoch = strtoll (source_date_epoch, &endptr, 10);
++  if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
++      || (errno != 0 && epoch == 0))
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "strtoll: %s\n", xstrerror(errno));
++  if (endptr == source_date_epoch)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "no digits were found: %s\n", endptr);
++  if (*endptr != '\0')
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "trailing garbage: %s\n", endptr);
++  if (epoch < 0)
++    fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: "
++		 "value must be nonnegative: %lld \n", epoch);
++
++  return (time_t) epoch;
++}
++
+ #include "gt-c-family-c-common.h"
+--- a/gcc/c-family/c-common.h
++++ b/gcc/c-family/c-common.h
+@@ -1437,4 +1437,10 @@ extern bool contains_cilk_spawn_stmt (tr
+ extern tree cilk_for_number_of_iterations (tree);
+ extern bool check_no_cilk (tree, const char *, const char *,
+ 		           location_t loc = UNKNOWN_LOCATION);
++
++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
++   timestamp to replace embedded current dates to get reproducible
++   results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.  */
++extern time_t get_source_date_epoch (void);
++
+ #endif /* ! GCC_C_COMMON_H */
+--- a/gcc/c-family/c-lex.c
++++ b/gcc/c-family/c-lex.c
+@@ -402,6 +402,9 @@ c_lex_with_flags (tree *value, location_
+   enum cpp_ttype type;
+   unsigned char add_flags = 0;
+   enum overflow_type overflow = OT_NONE;
++  time_t source_date_epoch = get_source_date_epoch ();
++
++  cpp_init_source_date_epoch (parse_in, source_date_epoch);
+ 
+   timevar_push (TV_CPP);
+  retry:
+--- a/gcc/doc/cppenv.texi
++++ b/gcc/doc/cppenv.texi
+@@ -79,4 +79,21 @@ main input file is omitted.
+ @ifclear cppmanual
+ @xref{Preprocessor Options}.
+ @end ifclear
++
++ at item SOURCE_DATE_EPOCH
++
++If this variable is set, its value specifies a UNIX timestamp to be
++used in replacement of the current date and time in the @code{__DATE__}
++and @code{__TIME__} macros, so that the embedded timestamps become
++reproducible.
++
++The value of @env{SOURCE_DATE_EPOCH} must be a UNIX timestamp,
++defined as the number of seconds (excluding leap seconds) since
++01 Jan 1970 00:00:00 represented in ASCII, identical to the output of
++@samp{@command{date +%s}}.
++
++The value should be a known timestamp such as the last modification
++time of the source or package and it should be set by the build
++process.
++
+ @end vtable
+--- a/libcpp/include/cpplib.h
++++ b/libcpp/include/cpplib.h
+@@ -775,6 +775,9 @@ extern void cpp_init_special_builtins (c
+ /* Set up built-ins like __FILE__.  */
+ extern void cpp_init_builtins (cpp_reader *, int);
+ 
++/* Initialize the source_date_epoch value.  */
++extern void cpp_init_source_date_epoch (cpp_reader *, time_t);
++
+ /* This is called after options have been parsed, and partially
+    processed.  */
+ extern void cpp_post_options (cpp_reader *);
+--- a/libcpp/init.c
++++ b/libcpp/init.c
+@@ -530,6 +530,13 @@ cpp_init_builtins (cpp_reader *pfile, in
+     _cpp_define_builtin (pfile, "__OBJC__ 1");
+ }
+ 
++/* Initialize the source_date_epoch value.  */
++void
++cpp_init_source_date_epoch (cpp_reader *pfile, time_t source_date_epoch)
++{
++  pfile->source_date_epoch = source_date_epoch; 
++}
++
+ /* Sanity-checks are dependent on command-line options, so it is
+    called as a subroutine of cpp_read_main_file ().  */
+ #if ENABLE_CHECKING
+--- a/libcpp/internal.h
++++ b/libcpp/internal.h
+@@ -502,6 +502,10 @@ struct cpp_reader
+   const unsigned char *date;
+   const unsigned char *time;
+ 
++  /* Externally set timestamp to replace current date and time useful for
++     reproducibility.  */
++  time_t source_date_epoch;
++
+   /* EOF token, and a token forcing paste avoidance.  */
+   cpp_token avoid_paste;
+   cpp_token eof;
+--- a/libcpp/macro.c
++++ b/libcpp/macro.c
+@@ -350,13 +350,20 @@ _cpp_builtin_macro_text (cpp_reader *pfi
+ 	  time_t tt;
+ 	  struct tm *tb = NULL;
+ 
+-	  /* (time_t) -1 is a legitimate value for "number of seconds
+-	     since the Epoch", so we have to do a little dance to
+-	     distinguish that from a genuine error.  */
+-	  errno = 0;
+-	  tt = time(NULL);
+-	  if (tt != (time_t)-1 || errno == 0)
+-	    tb = localtime (&tt);
++	  /* Set a reproducible timestamp for __DATE__ and __TIME__ macro
++	     usage if SOURCE_DATE_EPOCH is defined.  */
++	  if (pfile->source_date_epoch != (time_t) -1)
++	     tb = gmtime (&pfile->source_date_epoch);
++	  else
++	    {
++	      /* (time_t) -1 is a legitimate value for "number of seconds
++		 since the Epoch", so we have to do a little dance to
++		 distinguish that from a genuine error.  */
++	      errno = 0;
++	      tt = time (NULL);
++	      if (tt != (time_t)-1 || errno == 0)
++		tb = localtime (&tt);
++	    }
+ 
+ 	  if (tb)
+ 	    {
Signature · 1 line
1.9.1
#3 lore raw

PATCH 3/15 reproducibility: generate SOURCE_DATE_EPOCH

In reply to Jérôme Pouiller

From: Gilles Chanteperdrix <[email protected]>

When reproducibility is requested, generate a global SOURCE_DATE_EPOCH environment variable which contains either the date of Buildroot last commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to have the same build date, so in case of new commit or release, all packages will appear to have been change, even though some of them may not have changed in fact.

The meaning of SOURCE_DATE_EPOCH is specified by the following specification:
https://reproducible-builds.org/specs/source-date-epoch/

---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)
Makefile +4 -0
index eff814b..17e49e4 100644
@@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
+SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES))
 endif
 
 # To put more focus on warnings, be less verbose as default
Signature · 1 line
1.9.1
#4 lore raw

PATCH 4/15 reproducible: add '-n' to gzip invocations

In reply to Jérôme Pouiller

Default invocation to gzip include timestamp in output file. This feature is incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.

The environment variable GZIP can hold a set of default options for gzip. So instead to find all gzip invocation in build process, we just export 'GZIP=-n'.

Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop include timestamp and does not provide any way to disable it.

This work was sponsored by `BA Robotic Systems'.

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)
Makefile +1 -0
index 17e49e4..c21ba4a 100644
@@ -249,6 +249,7 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export GZIP=-n
 export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)
 SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
Signature · 1 line
1.9.1
#5 lore raw

PATCH 5/15 fs/tar: make results reproducible

In reply to Jérôme Pouiller

In order to make tar images reproducible, we use --sort flag. However, this flags is available only from tar 1.28. So we also bump necessary host-tar version.

This work was sponsored by `BA Robotic Systems'.

---
 fs/tar/tar.mk                          | 2 +-
 support/dependencies/check-host-tar.sh | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)
fs/tar/tar.mk +1 -1
index 11c69c5..5a1b263 100644
@@ -7,7 +7,7 @@
 TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 define ROOTFS_TAR_CMD
-	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
+	tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) .
 endef
 
 $(eval $(call ROOTFS_TARGET,tar))
support/dependencies/check-host-tar.sh +2 -3
index 932d3c4..cef6d82 100755
@@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then
   minor=0
 fi
 
-# Minimal version = 1.17 (previous versions do not correctly unpack archives
-# containing hard-links if the --strip-components option is used).
+# Minimal version = 1.28 (previous versions do not does not support --sort=name)
 major_min=1
-minor_min=17
+minor_min=28
 if [ $major -gt $major_min ]; then
 	echo $tar
 else
Signature · 1 line
1.9.1
#6 lore raw

PATCH 6/15 reproducibility/linux: override build timestamp

In reply to Jérôme Pouiller

From: Gilles Chanteperdrix <[email protected]>

Linux kernel include a few information about build environment in its binary. This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build information when BR2_REPRODUCIBLE is enabled.

---
 linux/linux.mk | 8 ++++++++
 1 file changed, 8 insertions(+)
linux/linux.mk +8 -0
index 988427c..7e826cc 100644
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	BR_BINARIES_DIR=$(BINARIES_DIR)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+	KBUILD_BUILD_VERSION=1 \
+	KBUILD_BUILD_USER=buildroot \
+	KBUILD_BUILD_HOST=buildroot \
+	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
 # Get the real Linux version, which tells us where kernel modules are
 # going to be installed in the target filesystem.
 LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
Signature · 1 line
1.9.1
#7 lore raw

PATCH 7/15 reproducibility/linux: inhibit build-id

In reply to Jérôme Pouiller

By default, Linux kernel enable 'build-id'. 'build-id' tends to add random bytes in section .notes of kernel image[1]:

  $ readelf -Wn .../vmlinux
  Displaying notes found at file offset 0x00008000 with length 0x00000024:
    Owner                 Data size       Description
    GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2

Therefore, we patch kernel Makefile to disable 'build-id'.

[1] https://kernelnewbies.org/BuildId

This work was sponsored by `BA Robotic Systems'.

---
 linux/linux.mk | 7 +++++++
 1 file changed, 7 insertions(+)
linux/linux.mk +7 -0
index 7e826cc..a63d1f3 100644
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST
 endef
 LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define LINUX_REMOVE_BUILD_ID
+	sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile
+endef
+LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID
+endif
+
 ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
 LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
 else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
Signature · 1 line
1.9.1
#8 lore raw

PATCH 8/15 reproducibility/busybox: disable build timestamps

In reply to Jérôme Pouiller

From: Gilles Chanteperdrix <[email protected]>

Busybox include a few information about build environment in its binary. This feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build information when BR2_REPRODUCIBLE is enabled.

---
 package/busybox/busybox.mk | 6 ++++++
 1 file changed, 6 insertions(+)
package/busybox/busybox.mk +6 -0
index fc23a90..f4a241d 100644
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	CFLAGS="$(BUSYBOX_CFLAGS)" \
 	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+	KCONFIG_NOTIMESTAMP=1
+endif
+
 BUSYBOX_MAKE_OPTS = \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \
Signature · 1 line
1.9.1
#9 lore raw

PATCH 9/15 reproducible: lock modification times in $TARGET_DIR

In reply to Jérôme Pouiller

Make sure all files in $TARGET_DIR has a defined modification time before to generate filesystems.

This work was sponsored by `BA Robotic Systems'.

---
 fs/common.mk | 3 +++
 1 file changed, 3 insertions(+)
fs/common.mk +3 -0
index 2dbef4d..981dcb1 100644
@@ -95,6 +95,9 @@ endif
 	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
 		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+ifeq ($$(BR2_REPRODUCIBLE),y)
+	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
+endif
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
 	chmod a+x $$(FAKEROOT_SCRIPT)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
Signature · 1 line
1.9.1
#10 lore raw

PATCH 10/15 fakedate: new package

In reply to Jérôme Pouiller

`date' is widely used by packages to include build information in their binaries. Unfortunately, this is incompatible with BR2_REPRODUCIBLE.

Instead to find all `date' invocation in build process, we add small tool allowing to alway return same date.

This work was sponsored by `BA Robotic Systems'.

---
 package/fakedate/fakedate    | 28 ++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk | 14 ++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk
package/fakedate/fakedate new file +28 -0
new file mode 100755
index 0000000..2eded22
@@ -0,0 +1,28 @@
+#!/bin/sh
+# vim: set sw=4 expandtab:
+#
+# Licence: GPL
+# Created: 2016-11-04 16:31:18+01:00
+# Main authors:
+#     - J?r?me Pouiller <[email protected]>
+#
+
+PATH=/bin:/usr/bin
+LOG=/dev/null
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+    INHIBIT=0
+    for i in "$@"; do
+        case $i in
+        -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*)
+            INHIBIT=1
+            ;;
+        esac
+    done
+    if [ $INHIBIT -eq 0 ]; then
+        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
+        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
+        exec date -d "@$SOURCE_DATE_EPOCH" "$@"
+    fi
+fi
+
+exec date "$@"
package/fakedate/fakedate.mk new file +14 -0
new file mode 100644
index 0000000..e81ce5d
@@ -0,0 +1,14 @@
+################################################################################
+#
+# fakedate
+#
+################################################################################
+
+# source included in buildroot
+HOST_FAKEDATE_LICENSE = GPLv2+
+
+define HOST_FAKEDATE_INSTALL_CMDS
+	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/date
+endef
+
+$(eval $(host-generic-package))
Signature · 1 line
1.9.1
#11 lore raw

PATCH 11/15 reproducible: enable fakedate

In reply to Jérôme Pouiller

Enable fakedate for whole build process.

This work was sponsored by `BA Robotic Systems'.

---
 toolchain/toolchain/toolchain.mk | 4 ++++
 1 file changed, 4 insertions(+)
toolchain/toolchain/toolchain.mk +4 -0
index d317e91..b88dd94 100644
@@ -10,6 +10,10 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TOOLCHAIN_DEPENDENCIES += host-fakedate
+endif
+
 TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 
 # Apply a hack that Rick Felker suggested[1] to avoid conflicts between libc
Signature · 1 line
1.9.1
#12 lore raw

PATCH 12/15 python2: generate reproducible .pyc

In reply to Jérôme Pouiller

.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.

In order to guarantee .pyc are regenerated regardless their modification time, we remove .pyc before to compile. However, I wonder if it wouldn't be simpler to always call compile_all with 'force' flag.

This work was sponsored by `BA Robotic Systems'.

---
 package/python/python.mk | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
package/python/python.mk +20 -10
index cc65376..b0ff1fd 100644
@@ -226,6 +226,26 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+# Normally, *.pyc files should not have been compiled, but just in
+# case, we make sure we remove all of them.
+# However, do not remove .pyc if source .py is not present.
+ifneq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_REPRODUCIBLE)),)
+define PYTHON_REMOVE_PYC_FILES
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		sed -z -e s/py$$/pyc/ | \
+		xargs -0 --no-run-if-empty rm -f
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
+endif
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON_FIX_TIME
+find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
+endif
+
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
@@ -245,16 +265,6 @@ endef
 PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
 endif
 
-# Normally, *.pyc files should not have been compiled, but just in
-# case, we make sure we remove all of them.
-ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
-define PYTHON_REMOVE_PYC_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | \
-		xargs -0 --no-run-if-empty rm -f
-endef
-PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES
-endif
-
 # In all cases, we don't want to keep the optimized .pyo files
 define PYTHON_REMOVE_PYO_FILES
 	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyo' -print0 | \
Signature · 1 line
1.9.1
#13 lore raw

PATCH 13/15 python3: generate reproducible .pyc

In reply to Jérôme Pouiller

.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.

In order to guarantee .pyc are regenerated regardless their modification time, we remove .pyc before to compile. However, I wonder if it wouldn't be simpler to always call compile_all with 'force' flag.

This work was sponsored by `BA Robotic Systems'.

---
 package/python3/python3.mk | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)
package/python3/python3.mk +19 -10
index b3f31c0..158c29c 100644
@@ -219,6 +219,25 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+# Normally, *.pyc files should not have been compiled, but just in
+# case, we make sure we remove all of them.
+ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
+define PYTHON3_REMOVE_PYC_FILES
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		sed -z -e s/py$$/pyc/ | \
+		xargs -0 --no-run-if-empty rm -f
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES
+endif
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON3_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
+endif
+
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
@@ -238,16 +257,6 @@ endef
 PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES
 endif
 
-# Normally, *.pyc files should not have been compiled, but just in
-# case, we make sure we remove all of them.
-ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
-define PYTHON3_REMOVE_PYC_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.pyc' -print0 | \
-		xargs -0 --no-run-if-empty rm -f
-endef
-PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES
-endif
-
 # In all cases, we don't want to keep the optimized .opt-1.pyc and
 # .opt-2.pyc files, since they can't work without their non-optimized
 # variant.
Signature · 1 line
1.9.1
#14 lore raw

PATCH 14/15 python2: remove full path from .pyc

In reply to Jérôme Pouiller

.pyc files include path to source .py file. This patch change the way `pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

---
 package/python/python.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
package/python/python.mk +3 -3
index b0ff1fd..35f971e 100644
@@ -248,9 +248,9 @@ endif
 
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
Signature · 1 line
1.9.1
#15 lore raw

PATCH 15/15 python3: remove full path from .pyc

In reply to Jérôme Pouiller

.pyc files include path to source .py file. This patch change the way `pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

---
 package/python3/python3.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
package/python3/python3.mk +3 -3
index 158c29c..3b2dd31 100644
@@ -240,9 +240,9 @@ endif
 
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
Signature · 1 line
1.9.1
#16 lore raw

cover Reproducible builds

In reply to Jérôme Pouiller

Hello,

Quoted 3 lines · J?r?me Pouiller

This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

Thanks for taking over this effort. This is definitely interesting.

Quoted 2 lines

Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)

I guess we can patch lzop to avoid this issue, right?

Quoted 2 lines

- since gcc versions supporting SOURCE_DATE_EPOCH are not widely available, external toolchains probably won't work.

Instead of patching gcc, can we solve the problem in the toolchain wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__ by passing -D__DATE__=... -D__TIME__=... to gcc ?

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#17 lore raw

cover Reproducible builds

Quoted 1 line · Thomas Petazzoni

On Thu, 17 Nov 2016 11:00:24 +0100, J?r?me Pouiller wrote:

[...]

Quoted 4 lines
Quoted 2 lines

Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)

I guess we can patch lzop to avoid this issue, right?

I guess also.

Quoted 7 lines
Quoted 3 lines

- since gcc versions supporting SOURCE_DATE_EPOCH are not widely
available,
external toolchains probably won't work.

Instead of patching gcc, can we solve the problem in the toolchain wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__ by passing -D__DATE__=... -D__TIME__=... to gcc ?

Yes, I can switch to this solution.

Signature · 1 line
J?r?me Pouiller
#18 lore raw

cover Reproducible builds

Hello,

Quoted 9 lines · J?r?me Pouiller
Quoted 7 lines
Quoted 3 lines

- since gcc versions supporting SOURCE_DATE_EPOCH are not widely
available,
external toolchains probably won't work.

Instead of patching gcc, can we solve the problem in the toolchain wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__ by passing -D__DATE__=... -D__TIME__=... to gcc ?

Yes, I can switch to this solution.

I think it's a reasonable use of the wrapper, and makes the thing work fine for external toolchain as well, which is nice.

Another (unrelated) question: how can we test this stuff? You're providing the initial steps for it, but at some point, we will want to validate which packages behave properly and which packages don't behave properly. Have you already thought of automated testing we can do (in the autobuilders perhaps?) to exercise this "reproducible builds" functionality? Should we from time to time do a given builds twice, each time in a different environment (different date, different time, different user, different path, different timezone, different locale, etc.) ?

To what extent do we try to be reproducible? Do we try to have something that produces the same result when executed multiple times, but on the same machine, in the same environment (i.e only date/time changes) ? Or do we try to go the point where a given Buildroot .config gives a byte identical output regardless of the host machine on which you do the build? It would be great to know what our definition of "reproducible is", and document it in the help text of the BR2_REPRODUCIBLE option.

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#19 lore raw

cover Reproducible builds

On 2016-11-17 15:08, Thomas Petazzoni wrote:
[...]

Quoted 9 lines

Another (unrelated) question: how can we test this stuff? You're providing the initial steps for it, but at some point, we will want to validate which packages behave properly and which packages don't behave properly. Have you already thought of automated testing we can do (in the autobuilders perhaps?) to exercise this "reproducible builds" functionality? Should we from time to time do a given builds twice, each time in a different environment (different date, different time, different user, different path, different timezone, different locale, etc.) ?

In a first time, compiling each configuration twice, diffing $(TARGET_DIR)
and find guilty packages using packages-file-list.txt should be enough to find
a many errors. We may also grep $TARGET_DIR for most common error patterns
(hostname, user, ...)

In a second time, I think reproducible build is an excellent tool to validate
top level parallelization.

Finally, for more accurate checks, we need to compile same configuration on
different host. I didn't check autobuilder sources, but it seems it would
involve a new autobuilder design.

Quoted 8 lines

To what extent do we try to be reproducible? Do we try to have something that produces the same result when executed multiple times, but on the same machine, in the same environment (i.e only date/time changes) ? Or do we try to go the point where a given Buildroot .config gives a byte identical output regardless of the host machine on which you do the build? It would be great to know what our definition of "reproducible is", and document it in the help text of the BR2_REPRODUCIBLE option.

Ideally, "reproducible" mean that a binary result only depends on Buildroot
version and user configuration (sure, if a custom script have a non-reproducible
behavior, result won't be reproducible).
Currently, there are few known (and doubtless many unknown) bugs:
- build paths have to be the sames
- lzop shouldn't be used

BR,

Signature · 1 line
J?r?me Pouiller
#20 lore raw

cover Reproducible builds

Hello,

Quoted 7 lines · J?r?me Pouiller

In a first time, compiling each configuration twice, diffing $(TARGET_DIR)
and find guilty packages using packages-file-list.txt should be enough to find
a many errors. We may also grep $TARGET_DIR for most common error patterns
(hostname, user, ...)

OK. So for now we concentrate on timestamp-related variations.

Quoted 2 lines

In a second time, I think reproducible build is an excellent tool to validate top level parallelization.

Yes, right.

Quoted 3 lines

Finally, for more accurate checks, we need to compile same configuration on different host. I didn't check autobuilder sources, but it seems it would involve a new autobuilder design.

This is clearly not possible with the current autobuilder design, indeed. It will require a fairly major rework. However, we are already thinking of changing the autobuilder architecture to have the configuration generated not by the build slaves themselves, but instead by the server. This could be a first step towards having the server generate a given configuration, and give it to two separate autobuilder slaves. But it's a lot of work to do.

Quoted 3 lines

Ideally, "reproducible" mean that a binary result only depends on Buildroot version and user configuration (sure, if a custom script have a non-reproducible behavior, result won't be reproducible).

Right. I believe we should document in the BR2_REPRODUCIBLE help text what is our current definition of reproducible. I.e that for the moment we only make builds reproducibles if they are built on the same machine, in the same path.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com