toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
PATCH 1/1 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
--- toolchain/toolchain-wrapper.c | 3 +++ toolchain/toolchain-wrapper.mk | 4 ++++ 2 files changed, 7 insertions(+)
toolchain/toolchain-wrapper.c +3 -0
@@ -98,6 +98,9 @@ static char *predef_args[] = {#if defined(BR_MIPS_TARGET_BIG_ENDIAN ) || defined(BR_ARC_TARGET_BIG_ENDIAN ) "-EB",#endif +#ifdef BR2_REPRODUCIBLE + "-Wl,--build-id=none", +#endif #ifdef BR_ADDITIONAL_CFLAGS BR_ADDITIONAL_CFLAGS#endif
toolchain/toolchain-wrapper.mk +4 -0
@@ -59,6 +59,10 @@ else ifeq ($(BR2_RELRO_FULL),y) TOOLCHAIN_WRAPPER_ARGS += -DBR2_RELRO_FULL endif +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR2_REPRODUCIBLE +endif + define TOOLCHAIN_WRAPPER_BUILD $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \ -s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \
Signature · 1 line
2.22.0
PATCH 1/1 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
Atharva, All,
On 2019-08-16 22:33 +0530, Atharva Lele spake thusly:
Quoted 22 lines
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
Signed-off-by Atharva Lele <[email protected]>--- toolchain/toolchain-wrapper.c | 3 +++ toolchain/toolchain-wrapper.mk | 4 ++++ 2 files changed, 7 insertions(+)toolchain/toolchain-wrapper.c +0
-0
toolchain/toolchain-wrapper.c static char *predef_args[] = { · line 102
Quoted 5 lines
toolchain/toolchain-wrapper.c +2
-0@@ -98,6 +98,9 @@ static char *predef_args[] = {#if defined(BR_MIPS_TARGET_BIG_ENDIAN ) || defined(BR_ARC_TARGET_BIG_ENDIAN ) "-EB",#endif +#ifdef BR2_REPRODUCIBLE + "-Wl,--build-id=none",
Actually, I would have preferred that we do pass a reproducible value instead of none, probably something based on SOURCE_DATE_EPOCH for eaxmple:
#ifdef BR2_BUILD_ID
"-Wl,--build-id=" BR2_BUILD_ID,
#endifAnd then, below....
Quoted 9 lines
toolchain/toolchain-wrapper.c +1
-0+#endif #ifdef BR_ADDITIONAL_CFLAGS BR_ADDITIONAL_CFLAGS#endif toolchain/toolchain-wrapper.mk +0
-0
toolchain/toolchain-wrapper.mk else ifeq ($(BR2_RELRO_FULL),y) · line 63
Quoted 5 lines
toolchain/toolchain-wrapper.mk +2
-0@@ -59,6 +59,10 @@ else ifeq ($(BR2_RELRO_FULL),y) TOOLCHAIN_WRAPPER_ARGS += -DBR2_RELRO_FULL endif +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR2_REPRODUCIBLE
... here, you'd set something like:
ifeq ($(BR2_REPRODUCIBLE),y)
TOOLCHAIN_WRAPPER_ARGS += -DBR2_BUILD_ID="$(BR2_BUILD_ID)"
endifand then in the main Makefile, in the BR2_REPRODUCIBLE condition (lines 518 and following), you'd need something like:
ifeq ($(BR2_REPRODUCIBLE),y)
[...]
BR2_BUILD_ID = $(shell echo $(SOURCE_DATE_EPOCH) |sha256sum |cut -d ' ' -f 1)
endifNote: as per the docs, build-id can be any hex-string.
But before re-sending, please wait for feedback from others.
Regards,
Yann E. MORIN.
Quoted 12 lines
toolchain/toolchain-wrapper.mk +2
-0+endif + define TOOLCHAIN_WRAPPER_BUILD $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \ -s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \Signature · 6 lines
2.22.0 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
Quoted 35 lines · Yann E. MORIN
Atharva, All,
On 2019-08-16 22:33 +0530, Atharva Lele spake thusly:
Quoted 22 lines
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
Signed-off-by Atharva Lele <[email protected]>--- toolchain/toolchain-wrapper.c | 3 +++ toolchain/toolchain-wrapper.mk | 4 ++++ 2 files changed, 7 insertions(+)toolchain/toolchain-wrapper.c +0
-0toolchain/toolchain-wrapper.c static char *predef_args[] = { · line 102
Quoted 5 lines
toolchain/toolchain-wrapper.c +2
-0@@ -98,6 +98,9 @@ static char *predef_args[] = {#if defined(BR_MIPS_TARGET_BIG_ENDIAN ) || defined(BR_ARC_TARGET_BIG_ENDIAN ) "-EB",#endif +#ifdef BR2_REPRODUCIBLE + "-Wl,--build-id=none",Actually, I would have preferred that we do pass a reproducible value instead of none, probably something based on
SOURCE_DATE_EPOCHfor eaxmple:
Quoted 3 lines
#ifdef BR2_BUILD_ID
"-Wl,--build-id=" BR2_BUILD_ID,
#endif That would break the meaning of build-id. build-id uniquely identifies that
specific file. It is expected that if build-ids are identical, the relevant parts of the ELF file (i.e. excluding debug info, notes, whatnot) are the same. Setting all build-ids to the same value would completely defeat its purpose, so then it's better to just remove it.
Regards,
Arnout
Quoted 52 lines
And then, below....
Quoted 9 lines
toolchain/toolchain-wrapper.c +1
-0+#endif #ifdef BR_ADDITIONAL_CFLAGS BR_ADDITIONAL_CFLAGS#endif toolchain/toolchain-wrapper.mk +0
-0toolchain/toolchain-wrapper.mk else ifeq ($(BR2_RELRO_FULL),y) · line 63
Quoted 5 lines
toolchain/toolchain-wrapper.mk +2
-0@@ -59,6 +59,10 @@ else ifeq ($(BR2_RELRO_FULL),y) TOOLCHAIN_WRAPPER_ARGS += -DBR2_RELRO_FULL endif +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR2_REPRODUCIBLE... here, you'd set something like:
ifeq ($(BR2_REPRODUCIBLE),y) TOOLCHAIN_WRAPPER_ARGS += -DBR2_BUILD_ID="$(BR2_BUILD_ID)" endifand then in the main
Makefile, in theBR2_REPRODUCIBLEcondition (lines 518 and following), you'd need something like:ifeq ($(BR2_REPRODUCIBLE),y) [...] BR2_BUILD_ID = $(shell echo $(SOURCE_DATE_EPOCH) |sha256sum |cut -d ' ' -f 1) endifNote: as per the docs, build-id can be any hex-string.
But before re-sending, please wait for feedback from others.
Regards,
Yann E. MORIN.Quoted 12 lines
toolchain/toolchain-wrapper.mk +2
-0+endif + define TOOLCHAIN_WRAPPER_BUILD $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \ -s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \Signature · 6 lines
2.22.0 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
PATCH 1/1 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
In reply to Atharva LeleAtharva, All,
On 2019-08-16 22:33 +0530, Atharva Lele spake thusly:
Quoted 22 lines
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
Signed-off-by Atharva Lele <[email protected]>--- toolchain/toolchain-wrapper.c | 3 +++ toolchain/toolchain-wrapper.mk | 4 ++++ 2 files changed, 7 insertions(+)toolchain/toolchain-wrapper.c +0
-0
toolchain/toolchain-wrapper.c static char *predef_args[] = { · line 102
Quoted 5 lines
toolchain/toolchain-wrapper.c +2
-0@@ -98,6 +98,9 @@ static char *predef_args[] = {#if defined(BR_MIPS_TARGET_BIG_ENDIAN ) || defined(BR_ARC_TARGET_BIG_ENDIAN ) "-EB",#endif +#ifdef BR2_REPRODUCIBLE + "-Wl,--build-id=none",
One thing I forgot in my previous review: --build-id has been supported only since binutils 2.18, released 12 years ago now (20017-08-28).
I'm not sure if some oldish toolchains are still using this version, and if so, whether we want to support those or not...
Regards,
Yann E. MORIN.
Quoted 26 lines
toolchain/toolchain-wrapper.c +1
-0+#endif #ifdef BR_ADDITIONAL_CFLAGS BR_ADDITIONAL_CFLAGS#endif toolchain/toolchain-wrapper.mk +4
-0@@ -59,6 +59,10 @@ else ifeq ($(BR2_RELRO_FULL),y) TOOLCHAIN_WRAPPER_ARGS += -DBR2_RELRO_FULL endif +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR2_REPRODUCIBLE +endif + define TOOLCHAIN_WRAPPER_BUILD $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \ -s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \Signature · 6 lines
2.22.0 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 toolchain/toolchain-wrapper: explicitly set Build ID to none if BR2_REPRODUCIBLE
Quoted 34 lines · Yann E. MORIN
Atharva, All,
On 2019-08-16 22:33 +0530, Atharva Lele spake thusly:
Quoted 22 lines
Build ID is added to binaries at link time. Building in different output directories causes some packages to have different Build IDs, thus resulting in non-reproducibility.
Adding "-Wl,--build-id=none" fixes this issue by disabling setting of Build ID.
Diffoscope output for Build ID issue: https://gitlab.com/snippets/1886180/raw
After this patch, build is reproducible - i.e. diffoscope does not produce any output.
Signed-off-by Atharva Lele <[email protected]>--- toolchain/toolchain-wrapper.c | 3 +++ toolchain/toolchain-wrapper.mk | 4 ++++ 2 files changed, 7 insertions(+)toolchain/toolchain-wrapper.c +0
-0toolchain/toolchain-wrapper.c static char *predef_args[] = { · line 102
Quoted 5 lines
toolchain/toolchain-wrapper.c +2
-0@@ -98,6 +98,9 @@ static char *predef_args[] = {#if defined(BR_MIPS_TARGET_BIG_ENDIAN ) || defined(BR_ARC_TARGET_BIG_ENDIAN ) "-EB",#endif +#ifdef BR2_REPRODUCIBLE + "-Wl,--build-id=none",One thing I forgot in my previous review: --build-id has been supported only since binutils 2.18, released 12 years ago now (20017-08-28).
Quoted 2 lines
I'm not sure if some oldish toolchains are still using this version, and if so, whether we want to support those or not...
We don't really have anything to deal with old binutils. Similar issues have
popped up in the past, and we always considered it long enough ago to swipe it under the carpet.
The oldest binutils we have in our external toolchains is 2.24.x, so I think
we're good.
Regards,
Arnout
Quoted 31 lines
Regards,
Yann E. MORIN.Quoted 26 lines
toolchain/toolchain-wrapper.c +1
-0+#endif #ifdef BR_ADDITIONAL_CFLAGS BR_ADDITIONAL_CFLAGS#endif toolchain/toolchain-wrapper.mk +4
-0@@ -59,6 +59,10 @@ else ifeq ($(BR2_RELRO_FULL),y) TOOLCHAIN_WRAPPER_ARGS += -DBR2_RELRO_FULL endif +ifeq ($(BR2_REPRODUCIBLE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR2_REPRODUCIBLE +endif + define TOOLCHAIN_WRAPPER_BUILD $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \ -s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \Signature · 6 lines
2.22.0 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot