toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
PATCH v2 1/2 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
---
Changes v1 -> v2:
- Handle inside toolchain-wrapper.mk instead of both .mk and .c
--- toolchain/toolchain-wrapper.mk | 4 ++++ 1 file changed, 4 insertions(+)
toolchain/toolchain-wrapper.mk +4 -0
@@ -21,6 +21,10 @@ TOOLCHAIN_WRAPPER_OPTS = \ $(call qstrip,$(BR2_SSP_OPTION)) \ $(call qstrip,$(BR2_TARGET_OPTIMIZATION)) +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_OPTS += -Wl,--build-id=none +endif + # We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a # separate argument when used in execv() by the toolchain wrapper. TOOLCHAIN_WRAPPER_ARGS += \
Signature · 1 line
2.22.0
PATCH v2 2/2 toolchain/toolchain-wrapper: handle __FILE__ macro for reproducibility
Many tools use __FILE__ for debugging and __FILE__ captures the build path. This results in non-reproducible images when building in different directories.
If the config uses GCC 8 or above, we use -ffile-prefix-map=old=new and let gcc take care of the path remapping in __FILE__. Since GCC versions before v8 did not have this feature, we use a dummy string in that case.
---
Changes v2:
- New patch
--- toolchain/toolchain-wrapper.mk | 5 +++++ 1 file changed, 5 insertions(+)
toolchain/toolchain-wrapper.mk +5 -0
@@ -23,6 +23,11 @@ TOOLCHAIN_WRAPPER_OPTS = \ ifeq ($(BR2_REPRODUCIBLE),y) TOOLCHAIN_WRAPPER_OPTS += -Wl,--build-id=none +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_8),y) +TOOLCHAIN_WRAPPER_OPTS += -ffile-prefix-map=$(BASE_DIR)=buildroot +else +TOOLCHAIN_WRAPPER_OPTS += -D__FILE__=\"file-path-replaced-by-buildroot-for-reproducibility\" +endif endif # We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a
Signature · 1 line
2.22.0
PATCH v2 2/2 toolchain/toolchain-wrapper: handle __FILE__ macro for reproducibility
Hi Atharva,
Le 31/08/2019 ? 20:01, Atharva Lele a ?crit?:
Quoted 2 lines
Many tools use
__FILE__for debugging and__FILE__captures the build path. This results in non-reproducible images when building in different directories.
Based on the commit log in gcc adding this option [1], -ffile-prefix-map also remap file names in __BASE_FILE__ and __builtin_FILE().
Specifying -ffile-prefix-map option is equivalent to specifying all the individual options -f*-prefix-map (-fmacro-prefix-map and -fdebug-prefix-map).
[1]
https://github.com/gcc-mirror/gcc/commit/859b51f83662d01e4f158ea9327db9ca37fe70b3
Quoted 18 lines
If the config uses GCC 8 or above, we use -ffile-prefix-map=old=new and let gcc take care of the path remapping in
__FILE__. Since GCC versions before v8 did not have this feature, we use a dummy string in that case.Signed-off-by Atharva Lele <[email protected]>---
Changes v2:
- New patch--- toolchain/toolchain-wrapper.mk | 5 +++++ 1 file changed, 5 insertions(+)toolchain/toolchain-wrapper.mk +0
-0
toolchain/toolchain-wrapper.mk TOOLCHAIN_WRAPPER_OPTS = \ · line 29
Quoted 7 lines
toolchain/toolchain-wrapper.mk +4
-0@@ -23,6 +23,11 @@ TOOLCHAIN_WRAPPER_OPTS = \ ifeq ($(BR2_REPRODUCIBLE),y) TOOLCHAIN_WRAPPER_OPTS += -Wl,--build-id=none +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_8),y) +TOOLCHAIN_WRAPPER_OPTS += -ffile-prefix-map=$(BASE_DIR)=buildroot +else +TOOLCHAIN_WRAPPER_OPTS += -D__FILE__=\"file-path-replaced-by-buildroot-for-reproducibility\"
Maybe __BASE_FILE__ should be handle here too in order to have the save behaviour as -ffile-prefix-map.
I'm not sure about __builtin_FILE().
Best regards,
Romain
Quoted 5 lines
toolchain/toolchain-wrapper.mk +1
-0+endif endif # We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a
PATCH v2 2/2 toolchain/toolchain-wrapper: handle __FILE__ macro for reproducibility
Quoted 35 lines · Romain Naour
Hi Atharva,
Le 31/08/2019 ? 20:01, Atharva Lele a ?crit :
Quoted 2 lines
Many tools use
__FILE__for debugging and__FILE__captures the build path. This results in non-reproducible images when building in different directories.Based on the commit log in gcc adding this option [1], -ffile-prefix-map also remap file names in
__BASE_FILE__and__builtin_FILE().Specifying -ffile-prefix-map option is equivalent to specifying all the individual options -f*-prefix-map (-fmacro-prefix-map and -fdebug-prefix-map).
[1]
https://github.com/gcc-mirror/gcc/commit/859b51f83662d01e4f158ea9327db9ca37fe70b3Quoted 19 lines
If the config uses GCC 8 or above, we use -ffile-prefix-map=old=new and let gcc take care of the path remapping in
__FILE__. Since GCC versions before v8 did not have this feature, we use a dummy string in that case.Signed-off-by Atharva Lele <[email protected]>---
Changes v2:
- New patch--- toolchain/toolchain-wrapper.mk | 5 +++++ 1 file changed, 5 insertions(+)toolchain/toolchain-wrapper.mk +0
-0@@ -23,6 +23,11 @@ TOOLCHAIN_WRAPPER_OPTS = \
Quoted 9 lines
toolchain/toolchain-wrapper.mk TOOLCHAIN_WRAPPER_OPTS = \ · line 29
Quoted 6 lines
toolchain/toolchain-wrapper.mk +4
-0ifeq ($(BR2_REPRODUCIBLE),y) TOOLCHAIN_WRAPPER_OPTS += -Wl,--build-id=none +ifeq ($(BR2_TOOLCHAIN_GCC_AT_LEAST_8),y) +TOOLCHAIN_WRAPPER_OPTS += -ffile-prefix-map=$(BASE_DIR)=buildroot +else +TOOLCHAIN_WRAPPER_OPTS += -D__FILE__=\"file-path-replaced-by-buildroot-for-reproducibility\"Maybe
__BASE_FILE__should be handle here too in order to have the save behaviour as -ffile-prefix-map.
You're right. I will update the patch to handle __BASE__FILE__ as well.
Quoted 1 line
I'm not sure about
__builtin_FILE().
I'm not sure either.. maybe we should keep it as is for now. Arnout, any thoughts on this?
Quoted 10 lines
Best regards,
RomainQuoted 5 lines
toolchain/toolchain-wrapper.mk +1
-0+endif endif # We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a
Signature · 2 lines
Regards, Atharva Lele
PATCH v2 2/2 toolchain/toolchain-wrapper: handle __FILE__ macro for reproducibility
On 16/09/2019 13:54, Atharva Lele wrote:
[snip]
Quoted 4 lines
Quoted 1 line
I'm not sure about
__builtin_FILE().I'm not sure either.. maybe we should keep it as is for now. Arnout, any thoughts on this?
There's no way to handle __builtin_FILE() correctly, because it's defined as a
function, not a macro. However, we already don't handle __FILE__ correctly (since it should contain the actual file name, and not just a constant), so I guess doing something like -D__builtin_FILE()=\"...\" should work.
BTW, now I think a bit more about this... The __FILE__ macro (and related) are
used in 99% of the cases for debugging and logging output. Putting 'file-path-replaced-by-buildroot-for-reproducibility' there is hardly helpful. Maybe it would be better to just put the empty string. Romain, what do you think?
Regards,
Arnout
PATCH v2 2/2 toolchain/toolchain-wrapper: handle __FILE__ macro for reproducibility
In reply to Atharva LeleHello,
Quoted 8 lines · On Sat, 31 Aug 2019 23:31:13 +0530 Atharva Lele <[email protected]> wrote:
Many tools use
__FILE__for debugging and__FILE__captures the build path. This results in non-reproducible images when building in different directories.If the config uses GCC 8 or above, we use -ffile-prefix-map=old=new and let gcc take care of the path remapping in
__FILE__. Since GCC versions before v8 did not have this feature, we use a dummy string in that case.Signed-off-by Atharva Lele <[email protected]>
I have applied this patch, with the following changes:
[Thomas:
- as suggested by Arnout, use the empty string for the __FILE__ and __BASE_FILE__ value
- as suggested by Romain, also handle __BASE_FILE__ in addition to __FILE__
- pass -Wno-builtin-macro-redefined to avoid build errors when -Werror is passed]
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 1/2 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
In reply to Atharva LeleHi Atharva,
Le 31/08/2019 ? 20:01, Atharva Lele a ?crit?:
Quoted 10 lines
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
--build-id option is available in binutils since 2.18 release (from 2007). (Add a new ELF linker option, --build-id, to generate a unique per-binary identifier embedded in a note section.) [1]
I don't think there is still toolchain using such old version with an up-to-date Buildroot tree :)
[1] https://lwn.net/Articles/273637/
Best regards,
Romain
Quoted 25 lines
Signed-off-by Atharva Lele <[email protected]>---
Changes v1 -> v2:
- Handle inside toolchain-wrapper.mk instead of both .mk and .c--- toolchain/toolchain-wrapper.mk | 4 ++++ 1 file changed, 4 insertions(+)toolchain/toolchain-wrapper.mk +4
-0@@ -21,6 +21,10 @@ TOOLCHAIN_WRAPPER_OPTS = \ $(call qstrip,$(BR2_SSP_OPTION)) \ $(call qstrip,$(BR2_TARGET_OPTIMIZATION)) +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_OPTS += -Wl,--build-id=none +endif + # We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a # separate argument when used in execv() by the toolchain wrapper. TOOLCHAIN_WRAPPER_ARGS += \
Thomas Petazzoni
AppliedPATCH v2 1/2 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
In reply to Atharva LeleQuoted 11 lines · On Sat, 31 Aug 2019 23:31:12 +0530 Atharva Lele <[email protected]> wrote:
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
Quoted 7 lines
Signed-off-by Atharva Lele <[email protected]>---
Changes v1 -> v2:
- Handle inside toolchain-wrapper.mk instead of both .mk and .c--- toolchain/toolchain-wrapper.mk | 4 ++++ 1 file changed, 4 insertions(+)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com