Reproducible builds

PATCH Superseded Jérôme Pouiller 2 participants 22 messages Nov 23, 2016

Subsystems (2)

VFS Kbuild

lore

Share & tools

lore

Jérôme Pouiller

author

2016-11-23 11:08 UTC #0 lore raw

v3 cover Reproducible builds

This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

I dropped some patchs from original series because either:
- I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH

    in gcc, ...)

- I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
- They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)

This version focuses on timestamps. It provide good enough results as soon as OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files. Only patch called "remove full path from .pyc" try to solve this issue. Another big step could be done by removing rpaths from ELF generated with libtool.

Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
- /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION, ccache is incompatible with reproducible

Since this feature is experimental I did not (yet) reported these incompatibilities in menuconfig.

Gilles Chanteperdrix (3):
reproducibility: generate SOURCE_DATE_EPOCH reproducibility/linux: override build timestamp reproducibility/busybox: disable build timestamps

J?r?me Pouiller (15):
reproducible: fix DATE/TIME macros in toolchain-wrapper
reproducible: add '-n' to gzip invocations
fs/tar: make results reproducible reproducibility/linux: inhibit build-id
reproducible: lock modification times in $TARGET_DIR
fakedate: new package
core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk
reproducible: enable fakedate
pycompile: allow to force compilation
python2: generate reproducible .pyc
python3: generate reproducible .pyc
python2: remove full path from .pyc
python3: remove full path from .pyc
reproducible: improve help text
reproducible: fix coding style

 Config.in                            |  6 +++
 Makefile                             | 12 ++++--
 fs/common.mk                         |  3 ++
 fs/tar/tar.mk                        |  3 +-
 linux/linux.mk                       | 15 ++++++++
 package/busybox/busybox.mk           |  6 +++
 package/fakedate/fakedate            | 59 ++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk         | 15 ++++++++
 package/python/python.mk             | 15 ++++++--
 package/python3/python3.mk           | 15 ++++++--
 support/dependencies/dependencies.mk |  2 -
 support/scripts/pycompile.py         | 11 +++++-
 toolchain/toolchain-wrapper.c        | 74 +++++++++++++++++++++++++++++++++++-
 13 files changed, 222 insertions(+), 14 deletions(-)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #1 lore raw

PATCH v3 1/18 reproducibility: generate SOURCE_DATE_EPOCH

From: Gilles Chanteperdrix <[email protected]>

When reproducibility is requested, generate a global SOURCE_DATE_EPOCH environment variable which contains either the date of Buildroot last commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to have the same build date, so in case of new commit or release, all packages will appear to have been changed, even though some of them may not have changed in fact.

The meaning of SOURCE_DATE_EPOCH is specified by the following specification:
https://reproducible-builds.org/specs/source-date-epoch/

Signed-off-by Gilles Chanteperdrix <[email protected]>

Signed-off-by J?r?me Pouiller <[email protected]>

---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

Makefile +4 -0

index eff814b..fcb80ec 100644
@@ -87,6 +87,8 @@ all:
 
 # Set and export the version string
 export BR2_VERSION := 2016.11-rc1
+# Actual time the release is cut (for reproducible builds)
+BR2_VERSION_EPOCH = 1478206447
 
 # Save running make version since it's clobbered by the make package
 RUNNING_MAKE_VERSION := $(MAKE_VERSION)

@@ -249,6 +251,8 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(BR2_VERSION_EPOCH))
 endif
 
 # To put more focus on warnings, be less verbose as default

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:15 UTC #2 lore raw

PATCH v3 1/18 reproducibility: generate SOURCE_DATE_EPOCH

Hello,

On Wednesday 23 November 2016 12:08:01 J?r?me Pouiller wrote:
[...]

Makefile ifeq ($(BR2_REPRODUCIBLE),y) · line 256

Quoted 3 lines

Makefile +2 -0

+SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(BR2_VERSION_EPOCH))
 endif

Arf, I have forgotten a fixup that do s/SOURCE_DATE_GIT/BR2_VERSION_GIT_EPOCH/.

Signature · 3 lines

J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

Jérôme Pouiller

author

2016-11-23 11:08 UTC #3 lore raw

PATCH v3 2/18 reproducible: fix DATE/TIME macros in toolchain-wrapper

In reply to Jérôme Pouiller

The use __DATE__ and __TIME__ are one of most common sources of non-reproducible binaries. In order to fix that, gcc begin to support SOURCE_DATE_EPOCH variable. This patch take advantage of toolchain-wrapper to provide support of SOURCE_DATE_EPOCH to older gcc versions.

Function get_source_date_epoch() come directly from gcc git.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 toolchain/toolchain-wrapper.c | 74 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 73 insertions(+), 1 deletion(-)

toolchain/toolchain-wrapper.c +73 -1

index 925d013..26d01b6 100644
@@ -22,12 +22,17 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
+#include <time.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
+// strlen("-D__TIME__=\"HH:MM:SS\"") + 1 = 22
+static char source_time[22];
+// strlen("-D__DATE__=\"MMM DD YYYY\"") + 1 = 25
+static char source_date[25];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are

@@ -39,8 +44,11 @@ static char sysroot[PATH_MAX];
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
+ * 	-D__TIME__=
+ * 	-D__DATE__=
+ * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	3
+#define EXCLUSIVE_ARGS	6
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE

@@ -136,6 +144,47 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
+/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
+ * timestamp to replace embedded current dates to get reproducible
+ * results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.
+ */
+time_t get_source_date_epoch()
+{
+	char *source_date_epoch;
+	long long epoch;
+	char *endptr;
+
+	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
+	if (!source_date_epoch)
+		return (time_t) -1;
+
+	errno = 0;
+	epoch = strtoll (source_date_epoch, &endptr, 10);
+	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
+			|| (errno != 0 && epoch == 0)) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"strtoll: %s\n", strerror(errno));
+		exit(2);
+	}
+	if (endptr == source_date_epoch) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"no digits were found: %s\n", endptr);
+		exit(2);
+	}
+	if (*endptr != '\0') {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"trailing garbage: %s\n", endptr);
+		exit(2);
+	}
+	if (epoch < 0) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"value must be nonnegative: %lld \n", epoch);
+		exit(2);
+	}
+
+	return (time_t) epoch;
+}
+
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;

@@ -146,6 +195,7 @@ int main(int argc, char **argv)
 	char *paranoid_wrapper;
 	int paranoid;
 	int ret, i, count = 0, debug;
+	time_t source_date_epoch;
 
 	/* Calculate the relative paths */
 	basename = strrchr(progpath, '/');

@@ -251,6 +301,28 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
+	source_date_epoch = get_source_date_epoch();
+	if (source_date_epoch != -1) {
+		struct tm *tm = localtime(&source_date_epoch);
+		if (!tm) {
+			perror("__FILE__: localtime");
+			return 3;
+		}
+		ret = strftime(source_time, sizeof(source_time), "-D__TIME__=\"%T\"", tm);
+		if (!ret) {
+			perror("__FILE__: overflow");
+			return 3;
+		}
+		*cur++ = source_time;
+		ret = strftime(source_date, sizeof(source_date), "-D__DATE__=\"%b %e %Y\"", tm);
+		if (!ret) {
+			perror("__FILE__: overflow");
+			return 3;
+		}
+		*cur++ = source_date;
+		*cur++ = "-Wno-builtin-macro-redefined";
+	}
+
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #4 lore raw

PATCH v3 3/18 reproducible: add '-n' to gzip invocations

In reply to Jérôme Pouiller

Default invocation to gzip include timestamp in output file. This feature is incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.

The environment variable GZIP can hold a set of default options for gzip. So instead to find all gzip invocation in build process, we just export 'GZIP=-n'.

Notice bzip2, lzma and xz are not impacted by this problem. On the other hand, lzop does include timestamp and does not provide any way to disable it.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

Makefile +1 -0

index fcb80ec..92271d9 100644
@@ -251,6 +251,7 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export GZIP = -n
 SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(BR2_VERSION_EPOCH))
 endif

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #5 lore raw

PATCH v3 4/18 fs/tar: make results reproducible

In reply to Jérôme Pouiller

In order to make tar images reproducible, force files order in tarball.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 fs/tar/tar.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

fs/tar/tar.mk +2 -1

index 11c69c5..70dc454 100644
@@ -7,7 +7,8 @@
 TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 define ROOTFS_TAR_CMD
-	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
+	( cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \
+		tar $(TAR_OPTS) -cf $@ --null -T - --no-recursion --numeric-owner )
 endef
 
 $(eval $(call ROOTFS_TARGET,tar))

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #6 lore raw

PATCH v3 5/18 reproducibility/linux: override build timestamp

In reply to Jérôme Pouiller

From: Gilles Chanteperdrix <[email protected]>

Linux kernel include a few information about build environment in its binary. This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build information when BR2_REPRODUCIBLE is enabled.

Note that usage of KBUILD_BUILD_TIMESTAMP is not mandatory since Buildroot use `fakedate'. However, native solution is prefered when upstream provide one.

Signed-off-by Gilles Chanteperdrix <[email protected]>

Reviewed-by "Yann E. MORIN" <[email protected]>

Signed-off-by J?r?me Pouiller <[email protected]>

---
 linux/linux.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

linux/linux.mk +8 -0

index 988427c..7e826cc 100644
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	BR_BINARIES_DIR=$(BINARIES_DIR)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+	KBUILD_BUILD_VERSION=1 \
+	KBUILD_BUILD_USER=buildroot \
+	KBUILD_BUILD_HOST=buildroot \
+	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
 # Get the real Linux version, which tells us where kernel modules are
 # going to be installed in the target filesystem.
 LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #7 lore raw

PATCH v3 6/18 reproducibility/linux: inhibit build-id

In reply to Jérôme Pouiller

Linux kernel include build path in debug sections. These sections are stripped and do not impact build reproducibility directly. However, 'build-id' depends on content of all sections, including debug sections. So, it add random bytes in section .notes of kernel image[1]:

  $ readelf -Wn .../vmlinux
  Displaying notes found at file offset 0x00008000 with length 0x00000024:
    Owner                 Data size       Description
    GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2

In order to not depend on build path, patch kernel Makefile to disable 'build-id'.

[1] https://kernelnewbies.org/BuildId

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 linux/linux.mk | 7 +++++++
 1 file changed, 7 insertions(+)

linux/linux.mk +7 -0

index 7e826cc..a63d1f3 100644
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST
 endef
 LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define LINUX_REMOVE_BUILD_ID
+	sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile
+endef
+LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID
+endif
+
 ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
 LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
 else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #8 lore raw

PATCH v3 7/18 reproducibility/busybox: disable build timestamps

In reply to Jérôme Pouiller

From: Gilles Chanteperdrix <[email protected]>

Busybox includes some information about the build environment in its binary. For BR2_REPRODUCIBLE, remove that information.

Signed-off-by Gilles Chanteperdrix <[email protected]>

Signed-off-by J?r?me Pouiller <[email protected]>

Reviewed-by "Yann E. MORIN" <[email protected]>

Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 package/busybox/busybox.mk | 6 ++++++
 1 file changed, 6 insertions(+)

package/busybox/busybox.mk +6 -0

index fc23a90..f4a241d 100644
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	CFLAGS="$(BUSYBOX_CFLAGS)" \
 	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+	KCONFIG_NOTIMESTAMP=1
+endif
+
 BUSYBOX_MAKE_OPTS = \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #9 lore raw

PATCH v3 8/18 reproducible: lock modification times in $TARGET_DIR

In reply to Jérôme Pouiller

Make sure all files in $TARGET_DIR have a defined modification time before to generate filesystems.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 fs/common.mk | 3 +++
 1 file changed, 3 insertions(+)

fs/common.mk +3 -0

index 2dbef4d..981dcb1 100644
@@ -95,6 +95,9 @@ endif
 	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
 		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+ifeq ($$(BR2_REPRODUCIBLE),y)
+	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
+endif
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
 	chmod a+x $$(FAKEROOT_SCRIPT)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #10 lore raw

PATCH v3 9/18 fakedate: new package

In reply to Jérôme Pouiller

`date' is widely used by packages to include build information in their binaries. Unfortunately, this is incompatible with BR2_REPRODUCIBLE.

Instead of having to identify all `date' invocations in the different packages, this commit adds a small tool that allows to always return the same date.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 package/fakedate/fakedate    | 59 ++++++++++++++++++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk | 15 +++++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

package/fakedate/fakedate new file +59 -0

new file mode 100755
index 0000000..074c517
@@ -0,0 +1,59 @@
+#!/bin/sh
+# vim: set sw=4 expandtab:
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+# Copyright (C) 2016 J?r?me Pouiller <[email protected]>
+#
+LOG=/dev/null
+
+# Sanity check
+if ! readlink -f "$0" | grep -q fakedate; then
+    echo "fakedate: Please name this script \`fakedate'"
+    exit 1
+fi
+
+DATE_BIN=false
+# Do not call `date'd directly since it will produce an infinite recursion.
+# Instead, find path of true `date' binary.
+for P in `echo $PATH | tr ':' ' '`; do
+    if [ -x "$P/date" ]; then
+        if readlink -f "$P/date" | grep -qv fakedate; then
+            DATE_BIN="$P/date"
+            break;
+        fi
+    fi
+done
+
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+    FORCE_EPOCH=1
+    for i in "$@"; do
+        # Use of --date, --file and --reference (and their short option counter
+        # parts) is incompatible with SOURCE_DATE_EPOCH.
+        # -u and -R are the only short options without argument. So they could
+        # appear between '-' and option we want to match.
+        if echo "$i" | grep -qE '^-([uR]*d|-date|[uR]*f|-file|[uR]*r|--reference)'; then
+            FORCE_EPOCH=0
+            break;
+        fi
+    done
+    if [ $FORCE_EPOCH -eq 1 ]; then
+        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
+        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
+        exec $DATE_BIN -d "@$SOURCE_DATE_EPOCH" "$@"
+    fi
+fi
+
+exec $DATE_BIN "$@"

package/fakedate/fakedate.mk new file +15 -0

new file mode 100644
index 0000000..61d4bd7
@@ -0,0 +1,15 @@
+################################################################################
+#
+# fakedate
+#
+################################################################################
+
+# source included in buildroot
+HOST_FAKEDATE_LICENSE = GPLv2+
+
+define HOST_FAKEDATE_INSTALL_CMDS
+	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/fakedate
+	ln -sfn fakedate $(HOST_DIR)/usr/bin/date
+endef
+
+$(eval $(host-generic-package))

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #11 lore raw

PATCH v3 10/18 core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk

In reply to Jérôme Pouiller

Usually, Buildroot does never initialize variables with empty content. DEPENDENCIES_HOST_PREREQ was an unjustified exception.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 support/dependencies/dependencies.mk | 2 --
 1 file changed, 2 deletions(-)

support/dependencies/dependencies.mk +0 -2

index 4334dac..d4b0409 100644
@@ -5,8 +5,6 @@
 #
 ################################################################################
 
-DEPENDENCIES_HOST_PREREQ :=
-
 # suitable-host-pkg: calls check-host-$(1).sh shell script. Parameter (2)
 # can be the candidate to be checked. If not present, the check-host-$(1).sh
 # script should use 'which' to find a candidate. The script should return

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #12 lore raw

PATCH v3 11/18 reproducible: enable fakedate

In reply to Jérôme Pouiller

Enable fakedate for whole build process.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

Makefile +1 -0

index 92271d9..c8bfed4 100644
@@ -254,6 +254,7 @@ export LC_ALL=C
 export GZIP = -n
 SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(BR2_VERSION_EPOCH))
+DEPENDENCIES_HOST_PREREQ += host-fakedate
 endif
 
 # To put more focus on warnings, be less verbose as default

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #13 lore raw

PATCH v3 12/18 pycompile: allow to force compilation

In reply to Jérôme Pouiller

Signed-off-by J?r?me Pouiller <[email protected]>

---
 support/scripts/pycompile.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

support/scripts/pycompile.py +10 -1

index fde711a..5598f8a 100644
@@ -10,6 +10,7 @@ from __future__ import print_function
 import sys
 import py_compile
 import compileall
+import argparse
 
 class ReportProblem:
     def __nonzero__(self):

@@ -21,4 +22,12 @@ class ReportProblem:
 
 report_problem = ReportProblem()
 
-compileall.compile_dir(sys.argv[1], quiet=report_problem)
+parser = argparse.ArgumentParser(description='Compile Python source files in a directory tree.')
+parser.add_argument("target", metavar='DIRECTORY',
+        help='Directory to scan')
+parser.add_argument("--force", action='store_true',
+        help="Force compilation even if alread compiled")
+
+args = parser.parse_args()
+
+compileall.compile_dir(args.target, force=args.force, quiet=report_problem)

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #14 lore raw

PATCH v3 13/18 python2: generate reproducible .pyc

In reply to Jérôme Pouiller

.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 package/python/python.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

package/python/python.mk +9 -1

index cc65376..c17b267 100644
@@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON_FIX_TIME
+find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
+endif
+
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
+		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
 		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #15 lore raw

PATCH v3 14/18 python3: generate reproducible .pyc

In reply to Jérôme Pouiller

.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

---
 package/python3/python3.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

package/python3/python3.mk +9 -1

index b3f31c0..a526c0f 100644
@@ -219,10 +219,18 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON3_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
+endif
+
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
+		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
 		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #16 lore raw

PATCH v3 15/18 python2: remove full path from .pyc

In reply to Jérôme Pouiller

.pyc files include path to source .py file. This patch changes the way `pycompile.py' is launched in order to only keep the part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 package/python/python.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

package/python/python.mk +4 -3

index c17b267..ffe4e14 100644
@@ -236,9 +236,10 @@ endif
 
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		$(if $(BR2_REPRODUCIBLE),--force) \
+		usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #17 lore raw

PATCH v3 16/18 python3: remove full path from .pyc

In reply to Jérôme Pouiller

.pyc files include path to source .py file. This patch changes the way `pycompile.py' is launched in order to only keep the part relative to $TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by J?r?me Pouiller <[email protected]>

Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 package/python3/python3.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

package/python3/python3.mk +4 -3

index a526c0f..4f0ae5b 100644
@@ -229,9 +229,10 @@ endif
 
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		$(if $(BR2_REPRODUCIBLE),--force) \
+		usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #18 lore raw

PATCH v3 17/18 reproducible: improve help text

In reply to Jérôme Pouiller

Signed-off-by J?r?me Pouiller <[email protected]>

---
 Config.in | 6 ++++++
 1 file changed, 6 insertions(+)

Config.in +6 -0

index 5cf0c4d..e382cf6 100644
@@ -707,6 +707,12 @@ config BR2_REPRODUCIBLE
 	  this allows to generate exactly identical binaries from one
 	  build to the other, including on different machines.
 
+	  The current implementation is restricted to builds with the same
+	  output directory. Many (absolute) paths are recorded in intermediary
+	  files, and it is very likely that some of these paths leak into the
+	  target rootfs. If you build with the same O=... path, however, the
+	  result is identical.
+
 	  This is labeled as an experimental feature, as not all
 	  packages behave properly to ensure reproducibility.

Signature · 1 line

1.9.1

Jérôme Pouiller

author

2016-11-23 11:08 UTC #19 lore raw

PATCH v3 18/18 reproducible: fix coding style

In reply to Jérôme Pouiller

Signed-off-by J?r?me Pouiller <[email protected]>

---
 Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Makefile +3 -3

index c8bfed4..51335c9 100644
@@ -248,9 +248,9 @@ endif
 
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
-export TZ=UTC
-export LANG=C
-export LC_ALL=C
+export TZ = UTC
+export LANG = C
+export LC_ALL = C
 export GZIP = -n
 SOURCE_DATE_GIT = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(SOURCE_DATE_GIT),$(BR2_VERSION_EPOCH))

Signature · 1 line

1.9.1

Thomas Petazzoni

2016-11-23 12:24 UTC #20 lore raw

v3 cover Reproducible builds

In reply to Jérôme Pouiller

Hello,

Quoted 15 lines · J?r?me Pouiller

This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905
I dropped some patchs from original series because either:
- I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
    in gcc, ...)
- I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
- They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)
This version focuses on timestamps. It provide good enough results as soon as OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files. Only patch called "remove full path from .pyc" try to solve this issue. Another big step could be done by removing rpaths from ELF generated with libtool.

Quoted 7 lines

Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
- /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION, ccache is incompatible with reproducible
Since this feature is experimental I did not (yet) reported these incompatibilities in menuconfig.

What are the changes between v2 and v3? There is no changelog, either in the cover letter nor in the patches themselves. It would be useful to have a changelog.

Thanks,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Jérôme Pouiller

author

2016-11-23 12:54 UTC #21 lore raw

v3 cover Reproducible builds

Hello Thomas,

Quoted 23 lines · On Wednesday 23 November 2016 13:24:14 Thomas Petazzoni wrote:

Hello,
Quoted 20 lines · J?r?me Pouiller
This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905
I dropped some patchs from original series because either:
- I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
    in gcc, ...)
- I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
- They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)
This version focuses on timestamps. It provide good enough results as soon as OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files. Only patch called "remove full path from .pyc" try to solve this issue. Another big step could be done by removing rpaths from ELF generated with libtool.
Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
- /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION, ccache is incompatible with reproducible

Quoted 6 lines

Quoted 2 lines
Since this feature is experimental I did not (yet) reported these incompatibilities in menuconfig.
What are the changes between v2 and v3? There is no changelog, either in the cover letter nor in the patches themselves. It would be useful to have a changelog.

Oops, I have forgotten --notes. I resend soon.

Signature · 3 lines

J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr