Reproducible builds
v4 cover Reproducible builds
This series try to continue work initiated by Gilles Chanteperdrix:
http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905
I dropped some patchs from original series because either:
- I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
in gcc, ...)
- I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
- They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)
This version focuses on timestamps. It provide good enough results as soon as OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files. Only patch called "remove full path from .pyc" try to solve this issue. Another big step could be done by removing rpaths from ELF generated with libtool.
Other thing known to break reproducibility:
- use of lzop (it unconditionally include timestamps in result)
- /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION, ccache is incompatible with reproducible
Since this feature is experimental I did not (yet) reported these incompatibilities in menuconfig.
Gilles Chanteperdrix (3):
reproducibility: generate SOURCE_DATE_EPOCH reproducibility/linux: override build timestamp reproducibility/busybox: disable build timestamps
J?r?me Pouiller (15):
reproducible: fix DATE/TIME macros in toolchain-wrapper
reproducible: add '-n' to gzip invocations
fs/tar: make results reproducible reproducibility/linux: inhibit build-id
reproducible: lock modification times in $TARGET_DIR
fakedate: new package
core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk
reproducible: enable fakedate
pycompile: allow to force compilation
python2: generate reproducible .pyc
python3: generate reproducible .pyc
python2: remove full path from .pyc
python3: remove full path from .pyc
reproducible: improve help text
reproducible: fix coding style
Config.in | 6 +++ Makefile | 12 ++++-- fs/common.mk | 3 ++ fs/tar/tar.mk | 3 +- linux/linux.mk | 15 ++++++++ package/busybox/busybox.mk | 6 +++ package/fakedate/fakedate | 59 ++++++++++++++++++++++++++++ package/fakedate/fakedate.mk | 15 ++++++++ package/python/python.mk | 15 ++++++-- package/python3/python3.mk | 15 ++++++-- support/dependencies/dependencies.mk | 2 - support/scripts/pycompile.py | 11 +++++- toolchain/toolchain-wrapper.c | 74 +++++++++++++++++++++++++++++++++++- 13 files changed, 222 insertions(+), 14 deletions(-) create mode 100755 package/fakedate/fakedate create mode 100644 package/fakedate/fakedate.mk
Signature · 1 line
1.9.1
PATCH v4 1/18 reproducibility: generate SOURCE_DATE_EPOCH
From: Gilles Chanteperdrix <[email protected]>
When reproducibility is requested, generate a global SOURCE_DATE_EPOCH environment variable which contains either the date of Buildroot last commit if running from a git repository, or the latest release date.
This means that all packages embedding build dates will appear to have the same build date, so in case of new commit or release, all packages will appear to have been changed, even though some of them may not have changed in fact.
The meaning of SOURCE_DATE_EPOCH is specified by the following specification:
https://reproducible-builds.org/specs/source-date-epoch/
---
Notes:
v4:
- s/SOURCE_DATE_GIT/BR2_VERSION_GIT_EPOCH/
v3:
- Fix typo in commit message (Arnout)
- Introduce BR2_VERSION_EPOCH (Thomas and Arnout)
- Set and export SOURCE_DATE_EPOCH together (Thomas)
Makefile | 4 ++++ 1 file changed, 4 insertions(+)
Makefile +4 -0
@@ -87,6 +87,8 @@ all: # Set and export the version string export BR2_VERSION := 2016.11-rc1 +# Actual time the release is cut (for reproducible builds) +BR2_VERSION_EPOCH = 1478206447 # Save running make version since it's clobbered by the make package RUNNING_MAKE_VERSION := $(MAKE_VERSION)
@@ -249,6 +251,8 @@ ifeq ($(BR2_REPRODUCIBLE),y) export TZ=UTC export LANG=C export LC_ALL=C +BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) +export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) endif # To put more focus on warnings, be less verbose as default
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 1/18 reproducibility: generate SOURCE_DATE_EPOCH
Hello,
Quoted 11 lines · J?r?me Pouiller
From: Gilles Chanteperdrix <[email protected]>
When reproducibility is requested, generate a global
SOURCE_DATE_EPOCHenvironment variable which contains either the date of Buildroot last commit if running from a git repository, or the latest release date.This means that all packages embedding build dates will appear to have the same build date, so in case of new commit or release, all packages will appear to have been changed, even though some of them may not have changed in fact.
Quoted 7 lines
The meaning of
SOURCE_DATE_EPOCHis specified by the following specification:
https://reproducible-builds.org/specs/source-date-epoch/Signed-off-by Gilles Chanteperdrix <[email protected]>Signed-off-by J?r?me Pouiller <[email protected]>---
Applied to next, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v4 2/18 reproducible: fix DATE/TIME macros in toolchain-wrapper
In reply to Jérôme PouillerThe use __DATE__ and __TIME__ are one of most common sources of non-reproducible binaries. In order to fix that, gcc begin to support SOURCE_DATE_EPOCH variable. This patch take advantage of toolchain-wrapper to provide support of SOURCE_DATE_EPOCH to older gcc versions.
Function get_source_date_epoch() come directly from gcc git.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Handle $SOURCE_DATE_EPOCH at runtime (Thomas)
v2:
- Overload __TIME__ and __DATE__ instead of patching gcc (Thomas)
toolchain/toolchain-wrapper.c | 74 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-)
toolchain/toolchain-wrapper.c +73 -1
@@ -22,12 +22,17 @@#include <unistd.h> #include <stdlib.h> #include <errno.h> +#include <time.h> #ifdef BR_CCACHE static char ccache_path[PATH_MAX ];#endif static char path[PATH_MAX ];static char sysroot[PATH_MAX ]; +// strlen("-D__TIME__=\"HH:MM:SS\"") + 1 = 22 +static char source_time[22 ]; +// strlen("-D__DATE__=\"MMM DD YYYY\"") + 1 = 25 +static char source_date[25 ];/** * GCC errors out with certain combinations of arguments (examples are
@@ -39,8 +44,11 @@ static char sysroot[PATH_MAX];* -mfloat-abi= * -march= * -mcpu= +* -D__TIME__= +* -D__DATE__= +* -Wno-builtin-macro-redefined */ -#define EXCLUSIVE_ARGS 3 +#define EXCLUSIVE_ARGS 6 static char *predef_args[] = {#ifdef BR_CCACHE
@@ -136,6 +144,47 @@ static void check_unsafe_path(const char *arg, } } +/* Read SOURCE_DATE_EPOCH from environment to have a deterministic +* timestamp to replace embedded current dates to get reproducible +* results. Returns -1 if SOURCE_DATE_EPOCH is not defined. +*/ +time_t get_source_date_epoch () +{ +char *source_date_epoch; +long long epoch; +char *endptr; + + source_date_epoch =getenv ("SOURCE_DATE_EPOCH" ); +if (!source_date_epoch) +return (time_t) -1 ; + + errno =0 ; + epoch =strtoll (source_date_epoch, &endptr,10 ); +if ((errno ==ERANGE && (epoch ==LLONG_MAX || epoch ==LLONG_MIN )) + || (errno !=0 && epoch ==0 )) { +fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " +"strtoll: %s\n" ,strerror (errno)); +exit (2 ); + } +if (endptr == source_date_epoch) { +fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " +"no digits were found: %s\n" , endptr); +exit (2 ); + } +if (*endptr !='\0' ) { +fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " +"trailing garbage: %s\n" , endptr); +exit (2 ); + } +if (epoch <0 ) { +fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " +"value must be nonnegative: %lld \n" , epoch); +exit (2 ); + } + +return (time_t ) epoch; +} +int main (int argc,char **argv) {char **args, **cur, **exec_args;
@@ -146,6 +195,7 @@ int main(int argc, char **argv)char *paranoid_wrapper;int paranoid;int ret, i, count =0 , debug; +time_t source_date_epoch;/* Calculate the relative paths */ basename =strrchr (progpath,'/' );
@@ -251,6 +301,28 @@ int main(int argc, char **argv) }#endif /* ARCH || CPU */ + source_date_epoch =get_source_date_epoch (); +if (source_date_epoch !=-1 ) { +struct tm *tm =localtime (&source_date_epoch); +if (!tm) { +perror ("__FILE__: localtime" ); +return 3 ; + } + ret =strftime (source_time,sizeof (source_time),"-D__TIME__=\"%T\"" , tm); +if (!ret) { +perror ("__FILE__: overflow" ); +return 3 ; + } + *cur++ = source_time; + ret =strftime (source_date,sizeof (source_date),"-D__DATE__=\"%b %e %Y\"" , tm); +if (!ret) { +perror ("__FILE__: overflow" ); +return 3 ; + } + *cur++ = source_date; + *cur++ ="-Wno-builtin-macro-redefined" ; + } + paranoid_wrapper =getenv ("BR_COMPILER_PARANOID_UNSAFE_PATH" );if (paranoid_wrapper &&strlen (paranoid_wrapper) >0 ) paranoid =1 ;
Signature · 1 line
1.9.1
PATCH v4 3/18 reproducible: add '-n' to gzip invocations
In reply to Jérôme PouillerDefault invocation to gzip include timestamp in output file. This feature is incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.
The environment variable GZIP can hold a set of default options for gzip. So instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
Notice bzip2, lzma and xz are not impacted by this problem. On the other hand, lzop does include timestamp and does not provide any way to disable it.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Coding style (Arnout)
- Fix typos in commit message (Arnout)
Makefile | 1 + 1 file changed, 1 insertion(+)
Makefile +1 -0
@@ -251,6 +251,7 @@ ifeq ($(BR2_REPRODUCIBLE),y) export TZ=UTC export LANG=C export LC_ALL=C +export GZIP = -n BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) endif
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 3/18 reproducible: add '-n' to gzip invocations
Hello,
Quoted 9 lines · J?r?me Pouiller
Default invocation to gzip include timestamp in output file. This feature is incompatible with
BR2_REPRODUCIBLE. It is possible to disable it with '-n'.The environment variable GZIP can hold a set of default options for gzip. So instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
Notice bzip2, lzma and xz are not impacted by this problem. On the other hand, lzop does include timestamp and does not provide any way to disable it.
Quoted 6 lines
This work was sponsored by `BA Robotic Systems'.
Signed-off-by J?r?me Pouiller <[email protected]>Reviewed-by Thomas Petazzoni <[email protected]>---
Applied to next, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v4 4/18 fs/tar: make results reproducible
In reply to Jérôme PouillerIn order to make tar images reproducible, force files order in tarball.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Use `find | sort |' instead of --sort (Arnout)
fs/tar/tar.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/tar/tar.mk +2 -1
@@ -7,7 +7,8 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) define ROOTFS_TAR_CMD - tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) . + ( cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \ + tar $(TAR_OPTS) -cf $@ --null -T - --no-recursion --numeric-owner ) endef $(eval $(call ROOTFS_TARGET,tar))
Signature · 1 line
1.9.1
PATCH v4 4/18 fs/tar: make results reproducible
Hello,
fs/tar/tar.mk line 11
Quoted 4 lines · J?r?me Pouiller
define
ROOTFS_TAR_CMDfs/tar/tar.mk +2
-1- tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) . + ( cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \ + tar $(TAR_OPTS) -cf $@ --null -T - --no-recursion --numeric-owner )
I've remove the space after ( and before ).
Applied with this changed. Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH v4 5/18 reproducibility/linux: override build timestamp
In reply to Jérôme PouillerFrom: Gilles Chanteperdrix <[email protected]>
Linux kernel include a few information about build environment in its binary. This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build information when BR2_REPRODUCIBLE is enabled.
Note that usage of KBUILD_BUILD_TIMESTAMP is not mandatory since Buildroot use `fakedate'. However, native solution is prefered when upstream provide one.
---
Notes:
v3:
- Justify use of KBUILD_BUILD_TIMESTAMP (Arnout)
linux/linux.mk | 8 ++++++++ 1 file changed, 8 insertions(+)
linux/linux.mk +8 -0
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \ $(TARGET_MAKE_ENV) \ BR_BINARIES_DIR=$(BINARIES_DIR) +ifeq ($(BR2_REPRODUCIBLE),y) +LINUX_MAKE_ENV += \ + KBUILD_BUILD_VERSION=1 \ + KBUILD_BUILD_USER=buildroot \ + KBUILD_BUILD_HOST=buildroot \ + KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))" +endif + # Get the real Linux version, which tells us where kernel modules are # going to be installed in the target filesystem. LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 5/18 reproducibility/linux: override build timestamp
Hello,
Quoted 6 lines · J?r?me Pouiller
From: Gilles Chanteperdrix <[email protected]>
Linux kernel include a few information about build environment in its binary. This feature is incompatible with
BR2_REPRODUCIBLE. This patch overload build information whenBR2_REPRODUCIBLEis enabled.
Quoted 8 lines
Note that usage of
KBUILD_BUILD_TIMESTAMPis not mandatory since Buildroot use `fakedate'. However, native solution is prefered when upstream provide one.Signed-off-by Gilles Chanteperdrix <[email protected]>Reviewed-by "Yann E. MORIN" <[email protected]>Signed-off-by J?r?me Pouiller <[email protected]>---
Applied to next, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v4 6/18 reproducibility/linux: inhibit build-id
In reply to Jérôme PouillerLinux kernel include build path in debug sections. These sections are stripped and do not impact build reproducibility directly. However, 'build-id' depends on content of all sections, including debug sections. So, it add random bytes in section .notes of kernel image[1]:
$ readelf -Wn .../vmlinux
Displaying notes found at file offset 0x00008000 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2In order to not depend on build path, patch kernel Makefile to disable 'build-id'.
[1] https://kernelnewbies.org/BuildId
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Better explain why disabling build-id is necessary (in fact, it not necessary if build paths are preserved between builds)
linux/linux.mk | 7 +++++++ 1 file changed, 7 insertions(+)
linux/linux.mk +7 -0
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST endef LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST +ifeq ($(BR2_REPRODUCIBLE),y) +define LINUX_REMOVE_BUILD_ID + sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile +endef +LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID +endif + ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y) LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
Signature · 1 line
1.9.1
PATCH v4 7/18 reproducibility/busybox: disable build timestamps
In reply to Jérôme PouillerFrom: Gilles Chanteperdrix <[email protected]>
Busybox includes some information about the build environment in its binary. For BR2_REPRODUCIBLE, remove that information.
---
Notes:
v3:
- Reword commit message (Arnout)
package/busybox/busybox.mk | 6 ++++++ 1 file changed, 6 insertions(+)
package/busybox/busybox.mk +6 -0
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \ $(TARGET_MAKE_ENV) \ CFLAGS="$(BUSYBOX_CFLAGS)" \ CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)" + +ifeq ($(BR2_REPRODUCIBLE),y) +BUSYBOX_MAKE_ENV += \ + KCONFIG_NOTIMESTAMP=1 +endif + BUSYBOX_MAKE_OPTS = \ CC="$(TARGET_CC)" \ ARCH=$(KERNEL_ARCH) \
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 7/18 reproducibility/busybox: disable build timestamps
Hello,
Quoted 11 lines · J?r?me Pouiller
From: Gilles Chanteperdrix <[email protected]>
Busybox includes some information about the build environment in its binary. For
BR2_REPRODUCIBLE, remove that information.Signed-off-by Gilles Chanteperdrix <[email protected]>Signed-off-by J?r?me Pouiller <[email protected]>Reviewed-by "Yann E. MORIN" <[email protected]>Reviewed-by Thomas Petazzoni <[email protected]>---
Applied to next, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v4 8/18 reproducible: lock modification times in $TARGET_DIR
In reply to Jérôme PouillerMake sure all files in $TARGET_DIR have a defined modification time before to generate filesystems.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Typo in commit log (Thomas)
fs/common.mk | 3 +++ 1 file changed, 3 insertions(+)
fs/common.mk +3 -0
@@ -95,6 +95,9 @@ endif $$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\ echo "echo '$$(TERM_BOLD)>>> Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \ echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep)) +ifeq ($$(BR2_REPRODUCIBLE),y) + echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT) +endif $$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT) chmod a+x $$(FAKEROOT_SCRIPT) PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 8/18 reproducible: lock modification times in $TARGET_DIR
Hello,
Quoted 9 lines · J?r?me Pouiller
Make sure all files in $TARGET_DIR have a defined modification time before to generate filesystems.
This work was sponsored by `BA Robotic Systems'.
Signed-off-by J?r?me Pouiller <[email protected]>Reviewed-by Thomas Petazzoni <[email protected]>---
Applied to next, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v4 9/18 fakedate: new package
In reply to Jérôme Pouiller`date' is widely used by packages to include build information in their binaries. Unfortunately, this is incompatible with BR2_REPRODUCIBLE.
Instead of having to identify all `date' invocations in the different packages, this commit adds a small tool that allows to always return the same date.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Do not force $PATH. Find true `date' binary instead (Thomas)
- Add explanations (Thomas)
- Fix options detection (Arnout)
- Rename INHIBIT in FORCE_EPOCH (Thomas)
- Break after FORCE_EPOCH=0 (Arnout)
- Add support for '--reference'
- Add full copyright blurb (Arnout)
package/fakedate/fakedate | 59 ++++++++++++++++++++++++++++++++++++++++++++ package/fakedate/fakedate.mk | 15 +++++++++++ 2 files changed, 74 insertions(+) create mode 100755 package/fakedate/fakedate create mode 100644 package/fakedate/fakedate.mk
package/fakedate/fakedate new file +59 -0
@@ -0,0 +1,59 @@ +#!/bin/sh +# vim: set sw=4 expandtab: +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +# +# Copyright (C) 2016 J?r?me Pouiller <[email protected]> +# +LOG=/dev/null + +# Sanity check +if ! readlink -f "$0" | grep -q fakedate; then + echo "fakedate: Please name this script \`fakedate'" + exit 1 +fi + +DATE_BIN=false +# Do not call `date'd directly since it will produce an infinite recursion. +# Instead, find path of true `date' binary. +for P in `echo $PATH | tr ':' ' '`; do + if [ -x "$P/date" ]; then + if readlink -f "$P/date" | grep -qv fakedate; then + DATE_BIN="$P/date" + break; + fi + fi +done + +if [ -n "$SOURCE_DATE_EPOCH" ]; then + FORCE_EPOCH=1 + for i in "$@"; do + # Use of --date, --file and --reference (and their short option counter + # parts) is incompatible with SOURCE_DATE_EPOCH. + # -u and -R are the only short options without argument. So they could + # appear between '-' and option we want to match. + if echo "$i" | grep -qE '^-([uR]*d|-date|[uR]*f|-file|[uR]*r|--reference)'; then + FORCE_EPOCH=0 + break; + fi + done + if [ $FORCE_EPOCH -eq 1 ]; then + echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2 + echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG + exec $DATE_BIN -d "@$SOURCE_DATE_EPOCH" "$@" + fi +fi + +exec $DATE_BIN "$@"
package/fakedate/fakedate.mk new file +15 -0
@@ -0,0 +1,15 @@ +################################################################################ +# +# fakedate +# +################################################################################ + +# source included in buildroot +HOST_FAKEDATE_LICENSE = GPLv2+ + +define HOST_FAKEDATE_INSTALL_CMDS + $(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/fakedate + ln -sfn fakedate $(HOST_DIR)/usr/bin/date +endef + +$(eval $(host-generic-package))
Signature · 1 line
1.9.1
PATCH v4 10/18 core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk
In reply to Jérôme PouillerUsually, Buildroot does never initialize variables with empty content. DEPENDENCIES_HOST_PREREQ was an unjustified exception.
---
Notes:
v3:
- New patch
support/dependencies/dependencies.mk | 2 -- 1 file changed, 2 deletions(-)
support/dependencies/dependencies.mk +0 -2
@@ -5,8 +5,6 @@ # ################################################################################ -DEPENDENCIES_HOST_PREREQ := - # suitable-host-pkg: calls check-host-$(1).sh shell script. Parameter (2) # can be the candidate to be checked. If not present, the check-host-$(1).sh # script should use 'which' to find a candidate. The script should return
Signature · 1 line
1.9.1
PATCH v4 11/18 reproducible: enable fakedate
In reply to Jérôme PouillerEnable fakedate for whole build process.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Build host-fakedate before toolchain (Thomas)
Makefile | 1 + 1 file changed, 1 insertion(+)
Makefile +1 -0
@@ -254,6 +254,7 @@ export LC_ALL=C export GZIP = -n BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH)) +DEPENDENCIES_HOST_PREREQ += host-fakedate endif # To put more focus on warnings, be less verbose as default
Signature · 1 line
1.9.1
PATCH v4 12/18 pycompile: allow to force compilation
In reply to Jérôme Pouiller---
Notes:
v3:
- new patch
support/scripts/pycompile.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
support/scripts/pycompile.py +10 -1
@@ -10,6 +10,7 @@ from __future__ import print_function import sys import py_compile import compileall +import argparse class ReportProblem: def __nonzero__(self):
@@ -21,4 +22,12 @@ class ReportProblem: report_problem = ReportProblem() -compileall.compile_dir(sys.argv[1], quiet=report_problem) +parser = argparse.ArgumentParser(description='Compile Python source files in a directory tree.') +parser.add_argument("target", metavar='DIRECTORY', + help='Directory to scan') +parser.add_argument("--force", action='store_true', + help="Force compilation even if alread compiled") + +args = parser.parse_args() + +compileall.compile_dir(args.target, force=args.force, quiet=report_problem)
Signature · 1 line
1.9.1
PATCH v4 12/18 pycompile: allow to force compilation
Hello,
Quoted 1 line · J?r?me Pouiller
Signed-off-by J?r?me Pouiller <[email protected]>
Empty commit log. Why do we want to do this?
Remember: for any non-trivial patch, an empty commit log is always wrong.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH v4 13/18 python2: generate reproducible .pyc
In reply to Jérôme Pouiller.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Force compilation instead of removing .pyc
package/python/python.mk | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
package/python/python.mk +9 -1
@@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ $(eval $(autotools-package)) $(eval $(host-autotools-package)) +ifeq ($(BR2_REPRODUCIBLE),y) +define PYTHON_FIX_TIME +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH) +endef +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME +endif + define PYTHON_CREATE_PYC_FILES PYTHONPATH="$(PYTHON_PATH)" \ $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ - support/scripts/pycompile.py \ + support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \ $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) endef
Signature · 1 line
1.9.1
PATCH v4 13/18 python2: generate reproducible .pyc
Hello,
Quoted 3 lines · J?r?me Pouiller
.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
"before to compile" -> "before compiling".
Quoted 8 lines
package/python/python.mk +0
-0@@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ $(eval $(autotools-package)) $(eval $(host-autotools-package))
package/python/python.mk PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ · line 238
Quoted 10 lines
package/python/python.mk +8
-0+ifeq ($(BR2_REPRODUCIBLE),y) +define PYTHON_FIX_TIME +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH) +endef +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME +endif + define PYTHON_CREATE_PYC_FILES PYTHONPATH="$(PYTHON_PATH)" \
It would make more sense to just do:
$(PYTHON_FIX_TIME)
here, rather than registering it as a PYTHON_TARGET_FINALIZE_HOOKS. Also, maybe it should be named PYTHON_SET_PY_FILES_TIME.
package/python/python.mk PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ · line 240
Quoted 3 lines
package/python/python.mk +1
-1$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ - support/scripts/pycompile.py \ + support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
Why do we need to force?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH v4 13/18 python2: generate reproducible .pyc
Quoted 30 lines · Thomas Petazzoni
Hello,
Quoted 3 lines · J?r?me Pouiller
.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
"before to compile" -> "before compiling".
Quoted 8 lines
package/python/python.mk +0
-0@@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ $(eval $(autotools-package)) $(eval $(host-autotools-package))package/python/python.mk PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ · line 238
Quoted 10 lines
package/python/python.mk +8
-0+ifeq ($(BR2_REPRODUCIBLE),y) +define PYTHON_FIX_TIME +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH) +endef +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME +endif + define PYTHON_CREATE_PYC_FILES PYTHONPATH="$(PYTHON_PATH)" \It would make more sense to just do:
Quoted 3 lines
$(PYTHON_FIX_TIME)here, rather than registering it as a
PYTHON_TARGET_FINALIZE_HOOKS.
No. It iterates over all python files in $(TARGET_DIR), not just the ones
installed by this package, so it really should be a TARGET_FINALIZE_HOOK.
Regards,
Arnout
Quoted 10 lines
Also, maybe it should be named
PYTHON_SET_PY_FILES_TIME.package/python/python.mk PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ · line 240
Quoted 3 lines
package/python/python.mk +1
-1$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ - support/scripts/pycompile.py \ + support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \Why do we need to force?
Thomas
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 13/18 python2: generate reproducible .pyc
Hello Thomas,
On Thursday 24 November 2016 20:06:55 Arnout Vandecappelle wrote:
[...]
Quoted 5 lines
Quoted 5 lines
package/python/python.mk PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ · line 240
Quoted 3 lines
package/python/python.mk +1
-1$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ - support/scripts/pycompile.py \ + support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \Why do we need to force?
Pycompile rely on .py modification times to know if a file need to be recompiled. Since reproducible builds tweak modification time, it is safer to force recompilation of all source files.
(I will add this explanation to commit log)
Signature · 3 lines
J?r?me Pouiller, Sysmic Embedded Linux specialist http://www.sysmic.fr
PATCH v4 14/18 python3: generate reproducible .pyc
In reply to Jérôme Pouiller.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Force compilation instead of removing .pyc
package/python3/python3.mk | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
package/python3/python3.mk +9 -1
@@ -219,10 +219,18 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat $(eval $(autotools-package)) $(eval $(host-autotools-package)) +ifeq ($(BR2_REPRODUCIBLE),y) +define PYTHON3_FIX_TIME + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \ + xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH) +endef +PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME +endif + define PYTHON3_CREATE_PYC_FILES PYTHONPATH="$(PYTHON3_PATH)" \ $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \ - support/scripts/pycompile.py \ + support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \ $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) endef
Signature · 1 line
1.9.1
PATCH v4 14/18 python3: generate reproducible .pyc
Hello,
Quoted 3 lines · J?r?me Pouiller
.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
Is there a way of avoiding the modification time contained inside the .pyc files? According to
http://wiki.baserock.org/projects/deterministic-builds/: "Debian solve this by creating the .pyc and .pyo files at package install time", which is essentially what we do. How do they handle this exactly?
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH v4 14/18 python3: generate reproducible .pyc
Hello Thomas,
Quoted 7 lines · On Wednesday 23 November 2016 23:09:39 Thomas Petazzoni wrote:
Hello,
Quoted 3 lines · J?r?me Pouiller
.pyc files contain modification time of .py source. In order to make build reproducible, we fix modification time of all .py before to compile .pyc files.
Quoted 5 lines
Is there a way of avoiding the modification time contained inside the .pyc files? According to
http://wiki.baserock.org/projects/deterministic-builds/: "Debian solve this by creating the .pyc and .pyo files at package install time", which is essentially what we do. How do they handle this exactly?
Debian compile .pyc and .pyo when user install package. deb files does not contains compiled files. Therefore, .pyc reproducibility does not matter.
In Buildroot, .pyc are included in filesystem. .pyc reproducibility does matter.
Signature · 3 lines
J?r?me Pouiller, Sysmic Embedded Linux specialist http://www.sysmic.fr
PATCH v4 15/18 python2: remove full path from .pyc
In reply to Jérôme Pouiller.pyc files include path to source .py file. This patch changes the way `pycompile.py' is launched in order to only keep the part relative to $TARGET_DIR.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Typo in commit log (Arnout)
package/python/python.mk | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
package/python/python.mk +4 -3
@@ -236,9 +236,10 @@ endif define PYTHON_CREATE_PYC_FILES PYTHONPATH="$(PYTHON_PATH)" \ - $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ - support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \ - $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) + cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \ + $(TOPDIR)/support/scripts/pycompile.py \ + $(if $(BR2_REPRODUCIBLE),--force) \ + usr/lib/python$(PYTHON_VERSION_MAJOR) endef ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
Signature · 1 line
1.9.1
PATCH v4 16/18 python3: remove full path from .pyc
In reply to Jérôme Pouiller.pyc files include path to source .py file. This patch changes the way `pycompile.py' is launched in order to only keep the part relative to $TARGET_DIR.
This work was sponsored by `BA Robotic Systems'.
---
Notes:
v3:
- Typo in commit log (Arnout)
package/python3/python3.mk | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
package/python3/python3.mk +4 -3
@@ -229,9 +229,10 @@ endif define PYTHON3_CREATE_PYC_FILES PYTHONPATH="$(PYTHON3_PATH)" \ - $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \ - support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \ - $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) + cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \ + $(TOPDIR)/support/scripts/pycompile.py \ + $(if $(BR2_REPRODUCIBLE),--force) \ + usr/lib/python$(PYTHON3_VERSION_MAJOR) endef ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
Signature · 1 line
1.9.1
PATCH v4 17/18 reproducible: improve help text
In reply to Jérôme Pouiller---
Notes:
v3:
- Reword help message (Arnout)
v2:
- New patch (Thomas)
Config.in | 6 ++++++ 1 file changed, 6 insertions(+)
Config.in +6 -0
@@ -707,6 +707,12 @@ config BR2_REPRODUCIBLE this allows to generate exactly identical binaries from one build to the other, including on different machines. + The current implementation is restricted to builds with the same + output directory. Many (absolute) paths are recorded in intermediary + files, and it is very likely that some of these paths leak into the + target rootfs. If you build with the same O=... path, however, the + result is identical. + This is labeled as an experimental feature, as not all packages behave properly to ensure reproducibility.
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 17/18 reproducible: improve help text
Hello,
Quoted 2 lines · J?r?me Pouiller
Signed-off-by J?r?me Pouiller <[email protected]>---
Applied to next, after tweaking the commit title. Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v4 18/18 reproducible: fix coding style
In reply to Jérôme Pouiller---
Notes:
v3:
- New patch (Arnout)
Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Makefile +3 -3
@@ -248,9 +248,9 @@ endif # timezone and locale may affect build output ifeq ($(BR2_REPRODUCIBLE),y) -export TZ=UTC -export LANG=C -export LC_ALL=C +export TZ = UTC +export LANG = C +export LC_ALL = C export GZIP = -n BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at) export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH v4 18/18 reproducible: fix coding style
Hello,
Quoted 5 lines · J?r?me Pouiller
Signed-off-by J?r?me Pouiller <[email protected]>---
Notes:
v3:
- New patch (Arnout)
Applied to next, after tweaking the commit title and adding some details in the commit log. Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com