Reproducible builds
cover Reproducible builds
This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905
I dropped some patchs from original series because either:
- I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
in gcc, ...)
- I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
- They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)
This version focuses on timestamps. It provide good enough results as soon as OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files. Only patch called "remove full path from .pyc" try to solve this issue. Another big step could be done by removing rpaths from ELF generated with libtool.
Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
- since gcc versions supporting SOURCE_DATE_EPOCH are not widely available, external toolchains probably won't work.
- /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION, ccache is incompatible with reproducible
- debug symbols are not reproducible
Since this feature is experimental I did not (yet) reported these incompatibilities in menuconfig.
Gilles Chanteperdrix (3):
reproducibility: generate SOURCE_DATE_EPOCH reproducibility/linux: override build timestamp reproducibility/busybox: disable build timestamps
J?r?me Pouiller (12):
gcc6: honor SOURCE_DATE_EPOCH
gcc5: honor SOURCE_DATE_EPOCH
reproducible: add '-n' to gzip invocations fs/tar: make results reproducible reproducibility/linux: inhibit build-id
reproducible: lock modification times in $TARGET_DIR
fakedate: new package
reproducible: enable fakedate
python2: generate reproducible .pyc
python3: generate reproducible .pyc
python2: remove full path from .pyc
python3: remove full path from .pyc
Makefile | 5 + fs/common.mk | 3 + fs/tar/tar.mk | 2 +- linux/linux.mk | 15 ++ package/busybox/busybox.mk | 6 + package/fakedate/fakedate | 28 +++ package/fakedate/fakedate.mk | 14 ++ .../gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch | 213 +++++++++++++++++++ .../gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch | 234 +++++++++++++++++++++ package/python/python.mk | 36 ++-- package/python3/python3.mk | 35 +-- support/dependencies/check-host-tar.sh | 5 +- toolchain/toolchain/toolchain.mk | 4 + 13 files changed, 570 insertions(+), 30 deletions(-) create mode 100755 package/fakedate/fakedate create mode 100644 package/fakedate/fakedate.mk create mode 100644 package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch create mode 100644 package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch
Signature · 1 line
1.9.1
PATCH 1/15 gcc6: honor SOURCE_DATE_EPOCH
Support $SOURCE_DATE_EPOCH is necessary to create reproducible build. Unfortunately, it is not yet widely available. This patch add support for SOURCE_DATE_EPOCH to Buildroot internal toolchains based on gcc 6.
This work was sponsored by `BA Robotic Systems'.
--- .../gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch | 234 +++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100644 package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch
package/gcc/6.1.0/950-honor-SOURCE_DATE_EPOCH.patch new file +234 -0
@@ -0,0 +1,234 @@ +From e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934 Mon Sep 17 00:00:00 2001 +From: doko <doko@138bc75d-0d04-0410-961f-82ee72b054a4> +Date: Thu, 28 Apr 2016 09:12:05 +0000 +Subject: [PATCH] gcc/c-family/ChangeLog: + +Upstream status: + Accepted. + See http://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934 + +2016-04-28 Eduard Sanou <[email protected]> + Matthias Klose <[email protected]> + + * c-common.c (get_source_date_epoch): New function, gets the environment + variable SOURCE_DATE_EPOCH and parses it as long long with error + handling. + * c-common.h (get_source_date_epoch): Prototype. + * c-lex.c (c_lex_with_flags): set parse_in->source_date_epoch. + +gcc/ChangeLog: + +2016-04-28 Eduard Sanou <[email protected]> + Matthias Klose <[email protected]> + + * doc/cppenv.texi: Document SOURCE_DATE_EPOCH environment variable. + +libcpp/ChangeLog: + +2016-04-28 Eduard Sanou <[email protected]> + Matthias Klose <[email protected]> + + * include/cpplib.h (cpp_init_source_date_epoch): Prototype. + * init.c (cpp_init_source_date_epoch): New function. + * internal.h: Added source_date_epoch variable to struct + cpp_reader to store a reproducible date. + * macro.c (_cpp_builtin_macro_text): Set pfile->date timestamp from + pfile->source_date_epoch instead of localtime if source_date_epoch is + set, to be used for __DATE__ and __TIME__ macros to help reproducible + builds. + +Signed-off-by: Eduard Sanou <[email protected]> +Signed-off-by: Matthias Klose <[email protected]> + +git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk at 235550 138bc75d-0d04-0410-961f-82ee72b054a4 +--- + gcc/c-family/c-common.c | 33 +++++++++++++++++++++++++++++++++ + gcc/c-family/c-common.h | 5 +++++ + gcc/c-family/c-lex.c | 3 +++ + gcc/doc/cppenv.texi | 17 +++++++++++++++++ + libcpp/include/cpplib.h | 3 +++ + libcpp/init.c | 9 ++++++++- + libcpp/internal.h | 4 ++++ + libcpp/macro.c | 21 ++++++++++++++------- + +diff --git a/gcc/c-family/c-common.c b/gcc/c-family/c-common.c +index 1f0d76a..c086dee 100644 +--- a/gcc/c-family/c-common.c ++++ b/gcc/c-family/c-common.c +@@ -12784,4 +12784,37 @@ valid_array_size_p (location_t loc, tree type, tree name) + return true; + } + ++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic ++ timestamp to replace embedded current dates to get reproducible ++ results. Returns -1 if SOURCE_DATE_EPOCH is not defined. */ ++time_t ++get_source_date_epoch () ++{ ++ char *source_date_epoch; ++ long long epoch; ++ char *endptr; ++ ++ source_date_epoch = getenv ("SOURCE_DATE_EPOCH"); ++ if (!source_date_epoch) ++ return (time_t) -1; ++ ++ errno = 0; ++ epoch = strtoll (source_date_epoch, &endptr, 10); ++ if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN)) ++ || (errno != 0 && epoch == 0)) ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "strtoll: %s\n", xstrerror(errno)); ++ if (endptr == source_date_epoch) ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "no digits were found: %s\n", endptr); ++ if (*endptr != '\0') ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "trailing garbage: %s\n", endptr); ++ if (epoch < 0) ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "value must be nonnegative: %lld \n", epoch); ++ ++ return (time_t) epoch; ++} ++ + #include "gt-c-family-c-common.h" +diff --git a/gcc/c-family/c-common.h b/gcc/c-family/c-common.h +index 1309549..3a7805f 100644 +--- a/gcc/c-family/c-common.h ++++ b/gcc/c-family/c-common.h +@@ -1471,4 +1471,9 @@ extern bool valid_array_size_p (location_t, tree, tree); + extern bool cilk_ignorable_spawn_rhs_op (tree); + extern bool cilk_recognize_spawn (tree, tree *); + ++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic ++ timestamp to replace embedded current dates to get reproducible ++ results. Returns -1 if SOURCE_DATE_EPOCH is not defined. */ ++extern time_t get_source_date_epoch (void); ++ + #endif /* ! GCC_C_COMMON_H */ +diff --git a/gcc/c-family/c-lex.c b/gcc/c-family/c-lex.c +index 6b020a4..ff7eb25 100644 +--- a/gcc/c-family/c-lex.c ++++ b/gcc/c-family/c-lex.c +@@ -388,6 +388,9 @@ c_lex_with_flags (tree *value, location_t *loc, unsigned char *cpp_flags, + enum cpp_ttype type; + unsigned char add_flags = 0; + enum overflow_type overflow = OT_NONE; ++ time_t source_date_epoch = get_source_date_epoch (); ++ ++ cpp_init_source_date_epoch (parse_in, source_date_epoch); + + timevar_push (TV_CPP); + retry: +diff --git a/gcc/doc/cppenv.texi b/gcc/doc/cppenv.texi +index 22c8cb3..e958e93 100644 +--- a/gcc/doc/cppenv.texi ++++ b/gcc/doc/cppenv.texi +@@ -79,4 +79,21 @@ main input file is omitted. + @ifclear cppmanual + @xref{Preprocessor Options}. + @end ifclear ++ ++ at item SOURCE_DATE_EPOCH ++ ++If this variable is set, its value specifies a UNIX timestamp to be ++used in replacement of the current date and time in the @code{__DATE__} ++and @code{__TIME__} macros, so that the embedded timestamps become ++reproducible. ++ ++The value of @env{SOURCE_DATE_EPOCH} must be a UNIX timestamp, ++defined as the number of seconds (excluding leap seconds) since ++01 Jan 1970 00:00:00 represented in ASCII, identical to the output of ++@samp{@command{date +%s}}. ++ ++The value should be a known timestamp such as the last modification ++time of the source or package and it should be set by the build ++process. ++ + @end vtable +diff --git a/libcpp/include/cpplib.h b/libcpp/include/cpplib.h +index 35b0375..4998b3a 100644 +--- a/libcpp/include/cpplib.h ++++ b/libcpp/include/cpplib.h +@@ -784,6 +784,9 @@ extern void cpp_init_special_builtins (cpp_reader *); + /* Set up built-ins like __FILE__. */ + extern void cpp_init_builtins (cpp_reader *, int); + ++/* Initialize the source_date_epoch value. */ ++extern void cpp_init_source_date_epoch (cpp_reader *, time_t); ++ + /* This is called after options have been parsed, and partially + processed. */ + extern void cpp_post_options (cpp_reader *); +diff --git a/libcpp/init.c b/libcpp/init.c +index 4343075..f5ff85b 100644 +--- a/libcpp/init.c ++++ b/libcpp/init.c +@@ -533,8 +533,15 @@ cpp_init_builtins (cpp_reader *pfile, int hosted) + _cpp_define_builtin (pfile, "__OBJC__ 1"); + } + ++/* Initialize the source_date_epoch value. */ ++void ++cpp_init_source_date_epoch (cpp_reader *pfile, time_t source_date_epoch) ++{ ++ pfile->source_date_epoch = source_date_epoch; ++} ++ + /* Sanity-checks are dependent on command-line options, so it is +- called as a subroutine of cpp_read_main_file (). */ ++ called as a subroutine of cpp_read_main_file. */ + #if CHECKING_P + static void sanity_checks (cpp_reader *); + static void sanity_checks (cpp_reader *pfile) +diff --git a/libcpp/internal.h b/libcpp/internal.h +index 9ce8707..e3eb26b 100644 +--- a/libcpp/internal.h ++++ b/libcpp/internal.h +@@ -502,6 +502,10 @@ struct cpp_reader + const unsigned char *date; + const unsigned char *time; + ++ /* Externally set timestamp to replace current date and time useful for ++ reproducibility. */ ++ time_t source_date_epoch; ++ + /* EOF token, and a token forcing paste avoidance. */ + cpp_token avoid_paste; + cpp_token eof; +diff --git a/libcpp/macro.c b/libcpp/macro.c +index c251553..c2a8376 100644 +--- a/libcpp/macro.c ++++ b/libcpp/macro.c +@@ -357,13 +357,20 @@ _cpp_builtin_macro_text (cpp_reader *pfile, cpp_hashnode *node, + time_t tt; + struct tm *tb = NULL; + +- /* (time_t) -1 is a legitimate value for "number of seconds +- since the Epoch", so we have to do a little dance to +- distinguish that from a genuine error. */ +- errno = 0; +- tt = time(NULL); +- if (tt != (time_t)-1 || errno == 0) +- tb = localtime (&tt); ++ /* Set a reproducible timestamp for __DATE__ and __TIME__ macro ++ usage if SOURCE_DATE_EPOCH is defined. */ ++ if (pfile->source_date_epoch != (time_t) -1) ++ tb = gmtime (&pfile->source_date_epoch); ++ else ++ { ++ /* (time_t) -1 is a legitimate value for "number of seconds ++ since the Epoch", so we have to do a little dance to ++ distinguish that from a genuine error. */ ++ errno = 0; ++ tt = time (NULL); ++ if (tt != (time_t)-1 || errno == 0) ++ tb = localtime (&tt); ++ } + + if (tb) + { +-- +1.7.1 +
Signature · 1 line
1.9.1
PATCH 2/15 gcc5: honor SOURCE_DATE_EPOCH
In reply to Jérôme PouillerSupport $SOURCE_DATE_EPOCH is necessary to create reproducible build. Unfortunately, it is not yet widely available. This patch add support for SOURCE_DATE_EPOCH to Buildroot internal toolchains based on gcc 5.
This work was sponsored by `BA Robotic Systems'.
--- .../gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch | 213 +++++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch
package/gcc/5.4.0/950-honor-SOURCE_DATE_EPOCH.patch new file +213 -0
@@ -0,0 +1,213 @@ +From: J?r?me Pouiller <[email protected]> +Date: Fri, 04 Nov 2016 11:45:07 +0100 +Subject: [PATCH] Honor SOURCE_DATE_EPOCH + +Upstream status: + Accepted. + See http://gcc.gnu.org/git/?p=gcc.git;a=patch;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934 + +Current patch is backported from equivalent written for gcc 6.1 + +2016-04-28 Eduard Sanou <[email protected]> + Matthias Klose <[email protected]> + + * c-common.c (get_source_date_epoch): New function, gets the environment + variable SOURCE_DATE_EPOCH and parses it as long long with error + handling. + * c-common.h (get_source_date_epoch): Prototype. + * c-lex.c (c_lex_with_flags): set parse_in->source_date_epoch. + +gcc/ChangeLog: + +2016-04-28 Eduard Sanou <[email protected]> + Matthias Klose <[email protected]> + + * doc/cppenv.texi: Document SOURCE_DATE_EPOCH environment variable. + +libcpp/ChangeLog: + +2016-04-28 Eduard Sanou <[email protected]> + Matthias Klose <[email protected]> + + * include/cpplib.h (cpp_init_source_date_epoch): Prototype. + * init.c (cpp_init_source_date_epoch): New function. + * internal.h: Added source_date_epoch variable to struct + cpp_reader to store a reproducible date. + * macro.c (_cpp_builtin_macro_text): Set pfile->date timestamp from + pfile->source_date_epoch instead of localtime if source_date_epoch is + set, to be used for __DATE__ and __TIME__ macros to help reproducible + builds. + +Signed-off-by: J?r?me Pouiller <[email protected]> + +--- + gcc/c-family/c-common.c | 33 +++++++++++++++++++++++++++++++++ + gcc/c-family/c-common.h | 6 ++++++ + gcc/c-family/c-lex.c | 3 +++ + gcc/doc/cppenv.texi | 17 +++++++++++++++++ + libcpp/include/cpplib.h | 3 +++ + libcpp/init.c | 7 +++++++ + libcpp/internal.h | 4 ++++ + libcpp/macro.c | 21 ++++++++++++++------- + 8 files changed, 87 insertions(+), 7 deletions(-) + +--- a/gcc/c-family/c-common.c ++++ b/gcc/c-family/c-common.c +@@ -12318,4 +12318,37 @@ pointer_to_zero_sized_aggr_p (tree t) + return (TYPE_SIZE (t) && integer_zerop (TYPE_SIZE (t))); + } + ++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic ++ timestamp to replace embedded current dates to get reproducible ++ results. Returns -1 if SOURCE_DATE_EPOCH is not defined. */ ++time_t ++get_source_date_epoch () ++{ ++ char *source_date_epoch; ++ long long epoch; ++ char *endptr; ++ ++ source_date_epoch = getenv ("SOURCE_DATE_EPOCH"); ++ if (!source_date_epoch) ++ return (time_t) -1; ++ ++ errno = 0; ++ epoch = strtoll (source_date_epoch, &endptr, 10); ++ if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN)) ++ || (errno != 0 && epoch == 0)) ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "strtoll: %s\n", xstrerror(errno)); ++ if (endptr == source_date_epoch) ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "no digits were found: %s\n", endptr); ++ if (*endptr != '\0') ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "trailing garbage: %s\n", endptr); ++ if (epoch < 0) ++ fatal_error (UNKNOWN_LOCATION, "environment variable $SOURCE_DATE_EPOCH: " ++ "value must be nonnegative: %lld \n", epoch); ++ ++ return (time_t) epoch; ++} ++ + #include "gt-c-family-c-common.h" +--- a/gcc/c-family/c-common.h ++++ b/gcc/c-family/c-common.h +@@ -1437,4 +1437,10 @@ extern bool contains_cilk_spawn_stmt (tr + extern tree cilk_for_number_of_iterations (tree); + extern bool check_no_cilk (tree, const char *, const char *, + location_t loc = UNKNOWN_LOCATION); ++ ++/* Read SOURCE_DATE_EPOCH from environment to have a deterministic ++ timestamp to replace embedded current dates to get reproducible ++ results. Returns -1 if SOURCE_DATE_EPOCH is not defined. */ ++extern time_t get_source_date_epoch (void); ++ + #endif /* ! GCC_C_COMMON_H */ +--- a/gcc/c-family/c-lex.c ++++ b/gcc/c-family/c-lex.c +@@ -402,6 +402,9 @@ c_lex_with_flags (tree *value, location_ + enum cpp_ttype type; + unsigned char add_flags = 0; + enum overflow_type overflow = OT_NONE; ++ time_t source_date_epoch = get_source_date_epoch (); ++ ++ cpp_init_source_date_epoch (parse_in, source_date_epoch); + + timevar_push (TV_CPP); + retry: +--- a/gcc/doc/cppenv.texi ++++ b/gcc/doc/cppenv.texi +@@ -79,4 +79,21 @@ main input file is omitted. + @ifclear cppmanual + @xref{Preprocessor Options}. + @end ifclear ++ ++ at item SOURCE_DATE_EPOCH ++ ++If this variable is set, its value specifies a UNIX timestamp to be ++used in replacement of the current date and time in the @code{__DATE__} ++and @code{__TIME__} macros, so that the embedded timestamps become ++reproducible. ++ ++The value of @env{SOURCE_DATE_EPOCH} must be a UNIX timestamp, ++defined as the number of seconds (excluding leap seconds) since ++01 Jan 1970 00:00:00 represented in ASCII, identical to the output of ++@samp{@command{date +%s}}. ++ ++The value should be a known timestamp such as the last modification ++time of the source or package and it should be set by the build ++process. ++ + @end vtable +--- a/libcpp/include/cpplib.h ++++ b/libcpp/include/cpplib.h +@@ -775,6 +775,9 @@ extern void cpp_init_special_builtins (c + /* Set up built-ins like __FILE__. */ + extern void cpp_init_builtins (cpp_reader *, int); + ++/* Initialize the source_date_epoch value. */ ++extern void cpp_init_source_date_epoch (cpp_reader *, time_t); ++ + /* This is called after options have been parsed, and partially + processed. */ + extern void cpp_post_options (cpp_reader *); +--- a/libcpp/init.c ++++ b/libcpp/init.c +@@ -530,6 +530,13 @@ cpp_init_builtins (cpp_reader *pfile, in + _cpp_define_builtin (pfile, "__OBJC__ 1"); + } + ++/* Initialize the source_date_epoch value. */ ++void ++cpp_init_source_date_epoch (cpp_reader *pfile, time_t source_date_epoch) ++{ ++ pfile->source_date_epoch = source_date_epoch; ++} ++ + /* Sanity-checks are dependent on command-line options, so it is + called as a subroutine of cpp_read_main_file (). */ + #if ENABLE_CHECKING +--- a/libcpp/internal.h ++++ b/libcpp/internal.h +@@ -502,6 +502,10 @@ struct cpp_reader + const unsigned char *date; + const unsigned char *time; + ++ /* Externally set timestamp to replace current date and time useful for ++ reproducibility. */ ++ time_t source_date_epoch; ++ + /* EOF token, and a token forcing paste avoidance. */ + cpp_token avoid_paste; + cpp_token eof; +--- a/libcpp/macro.c ++++ b/libcpp/macro.c +@@ -350,13 +350,20 @@ _cpp_builtin_macro_text (cpp_reader *pfi + time_t tt; + struct tm *tb = NULL; + +- /* (time_t) -1 is a legitimate value for "number of seconds +- since the Epoch", so we have to do a little dance to +- distinguish that from a genuine error. */ +- errno = 0; +- tt = time(NULL); +- if (tt != (time_t)-1 || errno == 0) +- tb = localtime (&tt); ++ /* Set a reproducible timestamp for __DATE__ and __TIME__ macro ++ usage if SOURCE_DATE_EPOCH is defined. */ ++ if (pfile->source_date_epoch != (time_t) -1) ++ tb = gmtime (&pfile->source_date_epoch); ++ else ++ { ++ /* (time_t) -1 is a legitimate value for "number of seconds ++ since the Epoch", so we have to do a little dance to ++ distinguish that from a genuine error. */ ++ errno = 0; ++ tt = time (NULL); ++ if (tt != (time_t)-1 || errno == 0) ++ tb = localtime (&tt); ++ } + + if (tb) + {
Signature · 1 line
1.9.1
PATCH 3/15 reproducibility: generate SOURCE_DATE_EPOCH
In reply to Jérôme PouillerFrom: Gilles Chanteperdrix <[email protected]>
When reproducibility is requested, generate a global SOURCE_DATE_EPOCH environment variable which contains either the date of Buildroot last commit if running from a git repository, or the latest release date.
This means that all packages embedding build dates will appear to have the same build date, so in case of new commit or release, all packages will appear to have been change, even though some of them may not have changed in fact.
The meaning of SOURCE_DATE_EPOCH is specified by the following specification:
https://reproducible-builds.org/specs/source-date-epoch/
--- Makefile | 4 ++++ 1 file changed, 4 insertions(+)
Makefile +4 -0
@@ -249,6 +249,10 @@ ifeq ($(BR2_REPRODUCIBLE),y) export TZ=UTC export LANG=C export LC_ALL=C +export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH) +SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) +SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s) +SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(SOURCE_DATE_CHANGES)) endif # To put more focus on warnings, be less verbose as default
Signature · 1 line
1.9.1
PATCH 4/15 reproducible: add '-n' to gzip invocations
In reply to Jérôme PouillerDefault invocation to gzip include timestamp in output file. This feature is incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.
The environment variable GZIP can hold a set of default options for gzip. So instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
Notice bzip2, lzma and xz are impacted by this problem. On the other hand, lzop include timestamp and does not provide any way to disable it.
This work was sponsored by `BA Robotic Systems'.
--- Makefile | 1 + 1 file changed, 1 insertion(+)
Makefile +1 -0
@@ -249,6 +249,7 @@ ifeq ($(BR2_REPRODUCIBLE),y) export TZ=UTC export LANG=C export LC_ALL=C +export GZIP=-n export SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH) SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) SOURCE_DATE_CHANGES = $(shell date -d `echo $(BR2_VERSION) | sed 's/^\(....\)\.\(..\).*/\1-\2-01T23:59:59/'` +%s)
Signature · 1 line
1.9.1
PATCH 5/15 fs/tar: make results reproducible
In reply to Jérôme PouillerIn order to make tar images reproducible, we use --sort flag. However, this flags is available only from tar 1.28. So we also bump necessary host-tar version.
This work was sponsored by `BA Robotic Systems'.
--- fs/tar/tar.mk | 2 +- support/dependencies/check-host-tar.sh | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-)
fs/tar/tar.mk +1 -1
@@ -7,7 +7,7 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) define ROOTFS_TAR_CMD - tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) . + tar $(TAR_OPTS) -cf $@ --sort=name --numeric-owner -C $(TARGET_DIR) . endef $(eval $(call ROOTFS_TARGET,tar))
support/dependencies/check-host-tar.sh +2 -3
@@ -26,10 +26,9 @@ if [ ! -z "${version_bsd}" ] ; then minor=0 fi -# Minimal version = 1.17 (previous versions do not correctly unpack archives -# containing hard-links if the --strip-components option is used). +# Minimal version = 1.28 (previous versions do not does not support --sort=name) major_min=1 -minor_min=17 +minor_min=28 if [ $major -gt $major_min ]; then echo $tar else
Signature · 1 line
1.9.1
PATCH 6/15 reproducibility/linux: override build timestamp
In reply to Jérôme PouillerFrom: Gilles Chanteperdrix <[email protected]>
Linux kernel include a few information about build environment in its binary. This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build information when BR2_REPRODUCIBLE is enabled.
--- linux/linux.mk | 8 ++++++++ 1 file changed, 8 insertions(+)
linux/linux.mk +8 -0
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \ $(TARGET_MAKE_ENV) \ BR_BINARIES_DIR=$(BINARIES_DIR) +ifeq ($(BR2_REPRODUCIBLE),y) +LINUX_MAKE_ENV += \ + KBUILD_BUILD_VERSION=1 \ + KBUILD_BUILD_USER=buildroot \ + KBUILD_BUILD_HOST=buildroot \ + KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))" +endif + # Get the real Linux version, which tells us where kernel modules are # going to be installed in the target filesystem. LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
Signature · 1 line
1.9.1
PATCH 7/15 reproducibility/linux: inhibit build-id
In reply to Jérôme PouillerBy default, Linux kernel enable 'build-id'. 'build-id' tends to add random bytes in section .notes of kernel image[1]:
$ readelf -Wn .../vmlinux
Displaying notes found at file offset 0x00008000 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2Therefore, we patch kernel Makefile to disable 'build-id'.
[1] https://kernelnewbies.org/BuildId
This work was sponsored by `BA Robotic Systems'.
--- linux/linux.mk | 7 +++++++ 1 file changed, 7 insertions(+)
linux/linux.mk +7 -0
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST endef LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST +ifeq ($(BR2_REPRODUCIBLE),y) +define LINUX_REMOVE_BUILD_ID + sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile +endef +LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID +endif + ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y) LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
Signature · 1 line
1.9.1
PATCH 8/15 reproducibility/busybox: disable build timestamps
In reply to Jérôme PouillerFrom: Gilles Chanteperdrix <[email protected]>
Busybox include a few information about build environment in its binary. This feature is incompatible with BR2_REPRODUCIBLE feature. This patch overload build information when BR2_REPRODUCIBLE is enabled.
--- package/busybox/busybox.mk | 6 ++++++ 1 file changed, 6 insertions(+)
package/busybox/busybox.mk +6 -0
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \ $(TARGET_MAKE_ENV) \ CFLAGS="$(BUSYBOX_CFLAGS)" \ CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)" + +ifeq ($(BR2_REPRODUCIBLE),y) +BUSYBOX_MAKE_ENV += \ + KCONFIG_NOTIMESTAMP=1 +endif + BUSYBOX_MAKE_OPTS = \ CC="$(TARGET_CC)" \ ARCH=$(KERNEL_ARCH) \
Signature · 1 line
1.9.1
PATCH 9/15 reproducible: lock modification times in $TARGET_DIR
In reply to Jérôme PouillerMake sure all files in $TARGET_DIR has a defined modification time before to generate filesystems.
This work was sponsored by `BA Robotic Systems'.
--- fs/common.mk | 3 +++ 1 file changed, 3 insertions(+)
fs/common.mk +3 -0
@@ -95,6 +95,9 @@ endif $$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\ echo "echo '$$(TERM_BOLD)>>> Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \ echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep)) +ifeq ($$(BR2_REPRODUCIBLE),y) + echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT) +endif $$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT) chmod a+x $$(FAKEROOT_SCRIPT) PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
Signature · 1 line
1.9.1
PATCH 10/15 fakedate: new package
In reply to Jérôme Pouiller`date' is widely used by packages to include build information in their binaries. Unfortunately, this is incompatible with BR2_REPRODUCIBLE.
Instead to find all `date' invocation in build process, we add small tool allowing to alway return same date.
This work was sponsored by `BA Robotic Systems'.
--- package/fakedate/fakedate | 28 ++++++++++++++++++++++++++++ package/fakedate/fakedate.mk | 14 ++++++++++++++ 2 files changed, 42 insertions(+) create mode 100755 package/fakedate/fakedate create mode 100644 package/fakedate/fakedate.mk
package/fakedate/fakedate new file +28 -0
@@ -0,0 +1,28 @@ +#!/bin/sh +# vim: set sw=4 expandtab: +# +# Licence: GPL +# Created: 2016-11-04 16:31:18+01:00 +# Main authors: +# - J?r?me Pouiller <[email protected]> +# + +PATH=/bin:/usr/bin +LOG=/dev/null +if [ -n "$SOURCE_DATE_EPOCH" ]; then + INHIBIT=0 + for i in "$@"; do + case $i in + -d|-[!-]*d|--date=*|-f|-[!-]*f|--file=*) + INHIBIT=1 + ;; + esac + done + if [ $INHIBIT -eq 0 ]; then + echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2 + echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG + exec date -d "@$SOURCE_DATE_EPOCH" "$@" + fi +fi + +exec date "$@"
package/fakedate/fakedate.mk new file +14 -0
@@ -0,0 +1,14 @@ +################################################################################ +# +# fakedate +# +################################################################################ + +# source included in buildroot +HOST_FAKEDATE_LICENSE = GPLv2+ + +define HOST_FAKEDATE_INSTALL_CMDS + $(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/date +endef + +$(eval $(host-generic-package))
Signature · 1 line
1.9.1
PATCH 11/15 reproducible: enable fakedate
In reply to Jérôme PouillerEnable fakedate for whole build process.
This work was sponsored by `BA Robotic Systems'.
--- toolchain/toolchain/toolchain.mk | 4 ++++ 1 file changed, 4 insertions(+)
toolchain/toolchain/toolchain.mk +4 -0
@@ -10,6 +10,10 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y) TOOLCHAIN_DEPENDENCIES += toolchain-external endif +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_DEPENDENCIES += host-fakedate +endif + TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO # Apply a hack that Rick Felker suggested[1] to avoid conflicts between libc
Signature · 1 line
1.9.1
PATCH 12/15 python2: generate reproducible .pyc
In reply to Jérôme Pouiller.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
In order to guarantee .pyc are regenerated regardless their modification time, we remove .pyc before to compile. However, I wonder if it wouldn't be simpler to always call compile_all with 'force' flag.
This work was sponsored by `BA Robotic Systems'.
--- package/python/python.mk | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-)
package/python/python.mk +20 -10
@@ -226,6 +226,26 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ $(eval $(autotools-package)) $(eval $(host-autotools-package)) +# Normally, *.pyc files should not have been compiled, but just in +# case, we make sure we remove all of them. +# However, do not remove .pyc if source .py is not present. +ifneq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_REPRODUCIBLE)),) +define PYTHON_REMOVE_PYC_FILES + find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + sed -z -e s/py$$/pyc/ | \ + xargs -0 --no-run-if-empty rm -f +endef +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES +endif + +ifeq ($(BR2_REPRODUCIBLE),y) +define PYTHON_FIX_TIME +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH) +endef +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME +endif + define PYTHON_CREATE_PYC_FILES PYTHONPATH="$(PYTHON_PATH)" \ $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
@@ -245,16 +265,6 @@ endef PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES endif -# Normally, *.pyc files should not have been compiled, but just in -# case, we make sure we remove all of them. -ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y) -define PYTHON_REMOVE_PYC_FILES - find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | \ - xargs -0 --no-run-if-empty rm -f -endef -PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PYC_FILES -endif - # In all cases, we don't want to keep the optimized .pyo files define PYTHON_REMOVE_PYO_FILES find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyo' -print0 | \
Signature · 1 line
1.9.1
PATCH 13/15 python3: generate reproducible .pyc
In reply to Jérôme Pouiller.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
In order to guarantee .pyc are regenerated regardless their modification time, we remove .pyc before to compile. However, I wonder if it wouldn't be simpler to always call compile_all with 'force' flag.
This work was sponsored by `BA Robotic Systems'.
--- package/python3/python3.mk | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
package/python3/python3.mk +19 -10
@@ -219,6 +219,25 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat $(eval $(autotools-package)) $(eval $(host-autotools-package)) +# Normally, *.pyc files should not have been compiled, but just in +# case, we make sure we remove all of them. +ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y) +define PYTHON3_REMOVE_PYC_FILES + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \ + sed -z -e s/py$$/pyc/ | \ + xargs -0 --no-run-if-empty rm -f +endef +PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES +endif + +ifeq ($(BR2_REPRODUCIBLE),y) +define PYTHON3_FIX_TIME + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \ + xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH) +endef +PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME +endif + define PYTHON3_CREATE_PYC_FILES PYTHONPATH="$(PYTHON3_PATH)" \ $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
@@ -238,16 +257,6 @@ endef PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES endif -# Normally, *.pyc files should not have been compiled, but just in -# case, we make sure we remove all of them. -ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y) -define PYTHON3_REMOVE_PYC_FILES - find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.pyc' -print0 | \ - xargs -0 --no-run-if-empty rm -f -endef -PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PYC_FILES -endif - # In all cases, we don't want to keep the optimized .opt-1.pyc and # .opt-2.pyc files, since they can't work without their non-optimized # variant.
Signature · 1 line
1.9.1
PATCH 14/15 python2: remove full path from .pyc
In reply to Jérôme Pouiller.pyc files include path to source .py file. This patch change the way `pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.
This work was sponsored by `BA Robotic Systems'.
--- package/python/python.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
package/python/python.mk +3 -3
@@ -248,9 +248,9 @@ endif define PYTHON_CREATE_PYC_FILES PYTHONPATH="$(PYTHON_PATH)" \ - $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ - support/scripts/pycompile.py \ - $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) + cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ + $(TOPDIR)/support/scripts/pycompile.py \ + usr/lib/python$(PYTHON_VERSION_MAJOR) endef ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
Signature · 1 line
1.9.1
PATCH 15/15 python3: remove full path from .pyc
In reply to Jérôme Pouiller.pyc files include path to source .py file. This patch change the way `pycompile.py' is launched in order to only keep part relative to $TARGET_DIR.
This work was sponsored by `BA Robotic Systems'.
--- package/python3/python3.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
package/python3/python3.mk +3 -3
@@ -240,9 +240,9 @@ endif define PYTHON3_CREATE_PYC_FILES PYTHONPATH="$(PYTHON3_PATH)" \ - $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \ - support/scripts/pycompile.py \ - $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) + cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \ + $(TOPDIR)/support/scripts/pycompile.py \ + usr/lib/python$(PYTHON3_VERSION_MAJOR) endef ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
Signature · 1 line
1.9.1
cover Reproducible builds
In reply to Jérôme PouillerHello,
Quoted 3 lines · J?r?me Pouiller
This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905
Thanks for taking over this effort. This is definitely interesting.
Quoted 2 lines
Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
I guess we can patch lzop to avoid this issue, right?
Quoted 2 lines
- since gcc versions supporting
SOURCE_DATE_EPOCHare not widely available, external toolchains probably won't work.
Instead of patching gcc, can we solve the problem in the toolchain wrapper? I.e, maybe the toolchain wrapper can set __DATE__ and __TIME__ by passing -D__DATE__=... -D__TIME__=... to gcc ?
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
cover Reproducible builds
Quoted 1 line · Thomas Petazzoni
On Thu, 17 Nov 2016 11:00:24 +0100, J?r?me Pouiller wrote:
[...]
Quoted 4 lines
Quoted 2 lines
Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)I guess we can patch lzop to avoid this issue, right?
I guess also.
Quoted 7 lines
Quoted 3 lines
- since gcc versions supporting
SOURCE_DATE_EPOCHare not widely
available,
external toolchains probably won't work.Instead of patching gcc, can we solve the problem in the toolchain wrapper? I.e, maybe the toolchain wrapper can set
__DATE__and__TIME__by passing -D__DATE__=... -D__TIME__=... to gcc ?
Yes, I can switch to this solution.
Signature · 1 line
J?r?me Pouiller
cover Reproducible builds
Hello,
Quoted 9 lines · J?r?me Pouiller
Quoted 7 lines
Quoted 3 lines
- since gcc versions supporting
SOURCE_DATE_EPOCHare not widely
available,
external toolchains probably won't work.Instead of patching gcc, can we solve the problem in the toolchain wrapper? I.e, maybe the toolchain wrapper can set
__DATE__and__TIME__by passing -D__DATE__=... -D__TIME__=... to gcc ?Yes, I can switch to this solution.
I think it's a reasonable use of the wrapper, and makes the thing work fine for external toolchain as well, which is nice.
Another (unrelated) question: how can we test this stuff? You're providing the initial steps for it, but at some point, we will want to validate which packages behave properly and which packages don't behave properly. Have you already thought of automated testing we can do (in the autobuilders perhaps?) to exercise this "reproducible builds" functionality? Should we from time to time do a given builds twice, each time in a different environment (different date, different time, different user, different path, different timezone, different locale, etc.) ?
To what extent do we try to be reproducible? Do we try to have something that produces the same result when executed multiple times, but on the same machine, in the same environment (i.e only date/time changes) ? Or do we try to go the point where a given Buildroot .config gives a byte identical output regardless of the host machine on which you do the build? It would be great to know what our definition of "reproducible is", and document it in the help text of the BR2_REPRODUCIBLE option.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
cover Reproducible builds
On 2016-11-17 15:08, Thomas Petazzoni wrote:
[...]
Quoted 9 lines
Another (unrelated) question: how can we test this stuff? You're providing the initial steps for it, but at some point, we will want to validate which packages behave properly and which packages don't behave properly. Have you already thought of automated testing we can do (in the autobuilders perhaps?) to exercise this "reproducible builds" functionality? Should we from time to time do a given builds twice, each time in a different environment (different date, different time, different user, different path, different timezone, different locale, etc.) ?
In a first time, compiling each configuration twice, diffing $(TARGET_DIR)
and find guilty packages using packages-file-list.txt should be enough to find
a many errors. We may also grep $TARGET_DIR for most common error patterns
(hostname, user, ...)
In a second time, I think reproducible build is an excellent tool to validate
top level parallelization.
Finally, for more accurate checks, we need to compile same configuration on
different host. I didn't check autobuilder sources, but it seems it would
involve a new autobuilder design.
Quoted 8 lines
To what extent do we try to be reproducible? Do we try to have something that produces the same result when executed multiple times, but on the same machine, in the same environment (i.e only date/time changes) ? Or do we try to go the point where a given Buildroot .config gives a byte identical output regardless of the host machine on which you do the build? It would be great to know what our definition of "reproducible is", and document it in the help text of the
BR2_REPRODUCIBLEoption.
Ideally, "reproducible" mean that a binary result only depends on Buildroot
version and user configuration (sure, if a custom script have a non-reproducible
behavior, result won't be reproducible).
Currently, there are few known (and doubtless many unknown) bugs:
- build paths have to be the sames
- lzop shouldn't be used
BR,
Signature · 1 line
J?r?me Pouiller
cover Reproducible builds
Hello,
Quoted 7 lines · J?r?me Pouiller
In a first time, compiling each configuration twice, diffing $(TARGET_DIR)
and find guilty packages using packages-file-list.txt should be enough to find
a many errors. We may also grep $TARGET_DIR for most common error patterns
(hostname, user, ...)
OK. So for now we concentrate on timestamp-related variations.
Quoted 2 lines
In a second time, I think reproducible build is an excellent tool to validate top level parallelization.
Yes, right.
Quoted 3 lines
Finally, for more accurate checks, we need to compile same configuration on different host. I didn't check autobuilder sources, but it seems it would involve a new autobuilder design.
This is clearly not possible with the current autobuilder design, indeed. It will require a fairly major rework. However, we are already thinking of changing the autobuilder architecture to have the configuration generated not by the build slaves themselves, but instead by the server. This could be a first step towards having the server generate a given configuration, and give it to two separate autobuilder slaves. But it's a lot of work to do.
Quoted 3 lines
Ideally, "reproducible" mean that a binary result only depends on Buildroot version and user configuration (sure, if a custom script have a non-reproducible behavior, result won't be reproducible).
Right. I believe we should document in the BR2_REPRODUCIBLE help text what is our current definition of reproducible. I.e that for the moment we only make builds reproducibles if they are built on the same machine, in the same path.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com