Skip to contentSkip to conversation

reproducible builds: strip embedded file paths

PATCH New Vagrant Cascadian 5 participants 11 messages Nov 19, 2021 – Nov 27, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

A recent commit 12753d22563f7d2d01f2c6644c7b66b06eb5c90f introduced a use of __FILE__ to display the file in some debugging output.

Unfortunately, the use of __FILE__ can embed the absolute build path in the resulting binary, resulting in different binaries depending on which path the build was performed in.

All previous versions tested in Debian built reproducibly:

https://tests.reproducible-builds.org/debian/history/amd64/opensbi.html

The attached patch passes -ffile-prefix-map to CFLAGS in the top-level Makefile, which should strip the top-level build path from the resulting build. This should be supported in both gcc 8 and clang 10.

This patch was also submitted as a pull request:

https://github.com/riscv-software-src/opensbi/pull/229

Please CC me with any concerns or questions.

Thanks!

live well,
vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Pass-ffile-prefix-map-to-ensure-reproducible-builds-.patch
Type: text/x-diff
Size: 948 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/opensbi/attachments/20211118/b4f783cc/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/opensbi/attachments/20211118/b4f783cc/attachment.sig>

#1 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

? 2021-11-18???? 17:05 -0800?Vagrant Cascadian???

Quoted 6 lines

Unfortunately, the use of __FILE__ can embed the absolute build path in
the resulting binary, resulting in different binaries depending on which
path the build was performed in.

Under what scenarios will absolute paths be embed.

Regards,
Xiang W

#2 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

Quoted 10 lines · Xiang W

? 2021-11-18???? 17:05 -0800?Vagrant Cascadian???

Quoted 6 lines

Unfortunately, the use of __FILE__ can embed the absolute build path in
the resulting binary, resulting in different binaries depending on which
path the build was performed in.

Under what scenarios will absolute paths be embed.

As Vagrant said:

"A recent commit 12753d22563f7d2d01f2c6644c7b66b06eb5c90f introduced a use of __FILE__ to display the file in some debugging output.?

The lazy BUG macro, added in that commit, uses __FILE__.

Jess

#3 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

In reply to Xiang W
Quoted 9 lines · Xiang W.

? 2021-11-18???? 17:05 -0800?Vagrant Cascadian???

Quoted 6 lines

Unfortunately, the use of __FILE__ can embed the absolute build path in
the resulting binary, resulting in different binaries depending on which
path the build was performed in.

Under what scenarios will absolute paths be embed.

In each of the directories below, I built using the same toolchain from the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep /home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c
$ cd /home/vagrant/src/opensbi/opensbi2
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep /home/vagrant
/home/vagrant/src/opensbi/opensbi2/lib/sbi/riscv_asm.c

With the patch applied:

$ cd /home/vagrant/src/opensbi/opensbi3
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin  | grep /home/vagrant
$ strings build/platform/generic/firmware/fw_dynamic.bin  | grep riscv_asm.c
./lib/sbi/riscv_asm.c

A relative path within the source should be sufficient for the debug message to properly identify the relevent source files...

An alternate approach would be to not use __FILE__ at all, but that would probably take more code than passing -ffile-prefix-map at compile time.

Thanks!

live well,
vagrant

#4 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???

Quoted 20 lines

of the directories below, I built using the same toolchain from the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c
$ cd /home/vagrant/src/opensbi/opensbi2
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi2/lib/sbi/riscv_asm.c
Quoted 3 lines

With the patch applied:

$ cd /home/vagrant/src/opensbi/opensbi3

This is because the source file uses the absolute path when compiling, you can use the relative path by modifying the Makefile

Makefile +0 -0
index 8623c1c..d017534 100644
@@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p `dirname

$(1)`; \
-MM $(2) >> $(1) || rm -f $(1)
compile_cc = $(CMD_PREFIX)mkdir -p dirname $(1); \
echo " CC $(subst $(build_dir)/,,$(1))"; \
- $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c $(2) -
o $(1)
+ $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c
$(subst $(src_dir)/,,$(2)) -o $(1)
compile_as_dep = $(CMD_PREFIX)mkdir -p dirname $(1); \
echo " AS-DEP $(subst $(build_dir)/,,$(1))"; \
printf %s dirname $(1)/ > $(1) && \

Regards,
Xiang W

#5 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

Quoted 11 lines · Xiang W.

? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???

Quoted 10 lines

of the directories below, I built using the same toolchain from the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c

...

Quoted 17 lines

This is because the source file uses the absolute path when compiling, you can use the relative path by modifying the Makefile

Makefile +1 -1
index 8623c1c..d017534 100644
@@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p `dirname
 $(1)`; \
 	       -MM $(2) >> $(1) || rm -f $(1)
 compile_cc = $(CMD_PREFIX)mkdir -p `dirname $(1)`; \
 	     echo " CC        $(subst $(build_dir)/,,$(1))"; \
-	     $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c $(2) -
 o $(1)
+	     $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c
 $(subst $(src_dir)/,,$(2)) -o $(1)

compile_as_dep = $(CMD_PREFIX)mkdir -p dirname $(1); \

Makefile

Quoted 2 lines
Makefile +0 -0
 	     echo " AS-DEP    $(subst $(build_dir)/,,$(1))"; \
 	     printf %s `dirname $(1)`/ > $(1) && \

That solves the issue and is probably better than passing more compiler arguments, thanks!

live well,
vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/opensbi/attachments/20211121/3ea4d4ac/attachment.sig>

#6 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

Quoted 28 lines · Vagrant Cascadian
Quoted 11 lines · Xiang W.

? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???

Quoted 10 lines

of the directories below, I built using the same toolchain from the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c

...

Quoted 14 lines

This is because the source file uses the absolute path when compiling, you can use the relative path by modifying the Makefile

Makefile +0 -1
index 8623c1c..d017534 100644
@@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p `dirname
 $(1)`; \
             -MM $(2) >> $(1) || rm -f $(1)
 compile_cc = $(CMD_PREFIX)mkdir -p `dirname $(1)`; \
           echo " CC        $(subst $(build_dir)/,,$(1))"; \
-          $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c $(2) -
 o $(1)
Quoted 8 lines
Quoted 5 lines
Makefile +1 -0
+          $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c
 $(subst $(src_dir)/,,$(2)) -o $(1)
 compile_as_dep = $(CMD_PREFIX)mkdir -p `dirname $(1)`; \
           echo " AS-DEP    $(subst $(build_dir)/,,$(1))"; \
           printf %s `dirname $(1)`/ > $(1) && \

That solves the issue and is probably better than passing more compiler arguments, thanks!

This will only work if we are compiling from the OpenSBI source directory itself. What if users compile OpenSBI with a different current directory using "make -C <opensbi_source_path>". I guess passing compiler argument will allow "make -C <opensbi_source_path>".

Regards,
Anup

Quoted 8 lines

live well,
vagrant
--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

#7 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

? 2021-11-22???? 10:13 +0530?Anup Patel???

Quoted 43 lines
Quoted 32 lines · Vagrant Cascadian
Quoted 11 lines · Xiang W.

? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???

Quoted 10 lines

of the directories below, I built using the same toolchain from the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c

...

Quoted 18 lines

This is because the source file uses the absolute path when compiling,
you can use the relative path by modifying the Makefile

Makefile +0 -0
index 8623c1c..d017534 100644
@@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p

`dirname
$(1)`; \
???????????? -MM $(2) >> $(1) || rm -f $(1) ?compile_cc = $(CMD_PREFIX)mkdir -p dirname $(1); \
?????????? echo " CC??????? $(subst $(build_dir)/,,$(1))"; \
-????????? $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c $(2) -
o $(1)
+????????? $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c

Quoted 8 lines
Quoted 4 lines

$(subst $(src_dir)/,,$(2)) -o $(1)
?compile_as_dep = $(CMD_PREFIX)mkdir -p dirname $(1); \
?????????? echo " AS-DEP??? $(subst $(build_dir)/,,$(1))"; \
?????????? printf %s dirname $(1)/ > $(1) && \

That solves the issue and is probably better than passing more compiler
arguments, thanks!

Quoted 6 lines

This will only work if we are compiling from the OpenSBI source directory
itself. What if users compile OpenSBI with a different current directory
using "make -C <opensbi_source_path>". I guess passing compiler argument will allow "make -C <opensbi_source_path>".

make -C <directory> will switch the path to <directory> first, so the result is the same.

Regards,
Xiang W

Quoted 12 lines

Regards,
Anup

Quoted 8 lines

live well,
? vagrant
--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

#8 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

Quoted 51 lines · Xiang W.

? 2021-11-22???? 10:13 +0530?Anup Patel???

Quoted 43 lines
Quoted 32 lines · Vagrant Cascadian
Quoted 11 lines · Xiang W.

? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???

Quoted 10 lines

of the directories below, I built using the same toolchain from the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin | grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c

...

Quoted 18 lines

This is because the source file uses the absolute path when compiling,
you can use the relative path by modifying the Makefile

Makefile +0 -0
index 8623c1c..d017534 100644
@@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p

`dirname
$(1)`; \
???????????? -MM $(2) >> $(1) || rm -f $(1) ?compile_cc = $(CMD_PREFIX)mkdir -p dirname $(1); \
?????????? echo " CC??????? $(subst $(build_dir)/,,$(1))"; \
-????????? $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c $(2) -
o $(1)
+????????? $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c

Quoted 8 lines
Quoted 4 lines

$(subst $(src_dir)/,,$(2)) -o $(1)
?compile_as_dep = $(CMD_PREFIX)mkdir -p dirname $(1); \
?????????? echo " AS-DEP??? $(subst $(build_dir)/,,$(1))"; \
?????????? printf %s dirname $(1)/ > $(1) && \

That solves the issue and is probably better than passing more compiler
arguments, thanks!

Quoted 6 lines

This will only work if we are compiling from the OpenSBI source directory
itself. What if users compile OpenSBI with a different current directory
using "make -C <opensbi_source_path>". I guess passing compiler argument will allow "make -C <opensbi_source_path>".

Quoted 2 lines

make -C <directory> will switch the path to <directory> first, so the result is the same.

Any outstanding concerns with proposing the patch by Xiang W to pass relative directories instead of the original patch I submitted?

Should it be submitted by Xiang W or should I merge the commit message from the original patch and submit it myself?

Thanks!

live well,
vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/opensbi/attachments/20211126/7eb1c7fd/attachment.sig>

#9 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

?On 27/11/21, 8:24 AM, "opensbi on behalf of Vagrant Cascadian" <opensbi-bounces at lists.infradead.org on behalf of [email protected]> wrote:

    On 2021-11-22, Xiang W. wrote:
    > ? 2021-11-22???? 10:13 +0530?Anup Patel???
    >> On Mon, Nov 22, 2021 at 10:02 AM Vagrant Cascadian
    >> <[email protected]> wrote:
    >> > 
    >> > On 2021-11-22, Xiang W. wrote:
    >> > > ? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???
    >> > > > of the directories below, I built using the same toolchain from
    >> > > > the same git commit...
    >> > > > 
    >> > > > $ cd /home/vagrant/src/opensbi/opensbi1
    >> > > > $ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
    >> > > > PLATFORM=generic
    >> > > > ...
    >> > > > $ strings build/platform/generic/firmware/fw_dynamic.bin | grep
    >> > > > /home/vagrant
    >> > > > /home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c
    >> > ...
    >> > > This is because the source file uses the absolute path when
    >> > > compiling,
    >> > > you can use the relative path by modifying the Makefile
    >> > > 
    >> > > diff --git a/Makefile b/Makefile
    >> > > index 8623c1c..d017534 100644
    >> > > --- a/Makefile
    >> > > +++ b/Makefile
    >> > > @@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p
    >> > > `dirname
    >> > > $(1)`; \
    >> > >              -MM $(2) >> $(1) || rm -f $(1)
    >> > >  compile_cc = $(CMD_PREFIX)mkdir -p `dirname $(1)`; \
    >> > >            echo " CC        $(subst $(build_dir)/,,$(1))"; \
    >> > > -          $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c
    >> > > $(2) -
    >> > > o $(1)
    >> > > +          $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c
    >> > > $(subst $(src_dir)/,,$(2)) -o $(1)
    >> > >  compile_as_dep = $(CMD_PREFIX)mkdir -p `dirname $(1)`; \
    >> > >            echo " AS-DEP    $(subst $(build_dir)/,,$(1))"; \
    >> > >            printf %s `dirname $(1)`/ > $(1) && \
    >> > 
    >> > That solves the issue and is probably better than passing more
    >> > compiler
    >> > arguments, thanks!
    >> 
    >> This will only work if we are compiling from the OpenSBI source
    >> directory
    >> itself. What if users compile OpenSBI with a different current
    >> directory
    >> using "make -C <opensbi_source_path>". I guess passing compiler
    >> argument will allow "make -C <opensbi_source_path>".
    >
    > make -C <directory> will switch the path to <directory> first, so the
    > result is the same.

    Any outstanding concerns with proposing the patch by Xiang W to pass
    relative directories instead of the original patch I submitted?

    Should it be submitted by Xiang W or should I merge the commit message
    from the original patch and submit it myself?

[Anup] Change the patch subject to "Makefile: strip embedded file paths" and send v2 patch.
[Anup] Please use "git send-email" to send patch to opensbi mailing list. You will have to join the mailing list before posting because it allows posting by mailing list members only.

Regards,
Anup

Thanks!

live well,
vagrant

#10 lore raw

PATCH 1/1 reproducible builds: strip embedded file paths

In reply to Vagrant Cascadian

? 2021-11-26???? 17:48 -0800?Vagrant Cascadian???

Quoted 58 lines · Xiang W.
Quoted 53 lines

? 2021-11-22???? 10:13 +0530?Anup Patel???

Quoted 45 lines
Quoted 34 lines · Vagrant Cascadian
Quoted 13 lines · Xiang W.

? 2021-11-21???? 11:20 -0800?Vagrant Cascadian???

Quoted 12 lines

of the directories below, I built using the same toolchain from
the same git commit...

$ cd /home/vagrant/src/opensbi/opensbi1
$ CROSS_COMPILE=riscv64-linux-gnu- make V=1 FW_PAYLOAD=n
PLATFORM=generic
...
$ strings build/platform/generic/firmware/fw_dynamic.bin |
grep
/home/vagrant
/home/vagrant/src/opensbi/opensbi1/lib/sbi/riscv_asm.c

...

Quoted 18 lines

This is because the source file uses the absolute path when compiling,
you can use the relative path by modifying the Makefile

Makefile +0 -0
index 8623c1c..d017534 100644
@@ -361,7 +361,7 @@ compile_cc_dep = $(CMD_PREFIX)mkdir -p

`dirname
$(1)`; \
???????????? -MM $(2) >> $(1) || rm -f $(1) ?compile_cc = $(CMD_PREFIX)mkdir -p dirname $(1); \
?????????? echo " CC??????? $(subst $(build_dir)/,,$(1))"; \
-????????? $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c $(2) -
o $(1)
+????????? $(CC) $(CFLAGS) $(call dynamic_flags,$(1),$(2)) -c

Quoted 8 lines
Quoted 4 lines

$(subst $(src_dir)/,,$(2)) -o $(1)
?compile_as_dep = $(CMD_PREFIX)mkdir -p dirname $(1); \
?????????? echo " AS-DEP??? $(subst $(build_dir)/,,$(1))"; \
?????????? printf %s dirname $(1)/ > $(1) && \

That solves the issue and is probably better than passing more compiler
arguments, thanks!

Quoted 6 lines

This will only work if we are compiling from the OpenSBI source directory
itself. What if users compile OpenSBI with a different current directory
using "make -C <opensbi_source_path>". I guess passing compiler argument will allow "make -C <opensbi_source_path>".

Quoted 3 lines

make -C <directory> will switch the path to <directory> first, so the
result is the same.

Quoted 6 lines

Any outstanding concerns with proposing the patch by Xiang W to pass relative directories instead of the original patch I submitted?

Should it be submitted by Xiang W or should I merge the commit message
from the original patch and submit it myself?

I just made a suggestion, you can submit it.

Regards,
Xiang W

Quoted 6 lines

Thanks!

live well,
? vagrant