boost: Improve reproducibility

PATCH Superseded [email protected] 5 participants 14 messages May 31, 2018 – Jun 15, 2018

lore

Share & tools

lore

[email protected]

author

2018-05-31 06:20 UTC #0 lore raw

PATCH 1/1 boost: Improve reproducibility

From: Mingli Yu <[email protected]>

boost_context includes some assembly source
files and will export the assembly object file
make_x86_64_sysv_elf_gas.o which contains host
path as below:

$ readelf --wide --symbols /my-build/boost/1.67.0-r0/boost_1_67_0/x86_64-poky-linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38ed/libboost_context.so.1.67
42: 0000000000000000 0 FILE LOCAL DEFAULT ABS /my-build/boost/1.67.0-r0/boost_1_67_0/x86_64-poky-linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38ed/asm/make_x86_64_sysv_elf_gas.o

Enable stirp for boost_context to hidden the
symbols

[YOCTO #12708]

Signed-off-by Mingli Yu <[email protected]>

---
 meta/recipes-support/boost/boost.inc               |  1 +
 .../boost/boost-context-hidden-the-symbols.patch   | 37 ++++++++++++++++++++++
 meta/recipes-support/boost/boost_1.67.0.bb         |  1 +
 3 files changed, 39 insertions(+)
 create mode 100644 meta/recipes-support/boost/boost/boost-context-hidden-the-symbols.patch

meta/recipes-support/boost/boost.inc +1 -0

index 6984bee7777..5a2086b3332 100644
@@ -197,4 +197,5 @@ do_install() {
 
 }
 
+INSANE_SKIP_${PN} = "already-stripped"
 BBCLASSEXTEND = "native nativesdk"

meta/recipes-support/boost/boost/boost-context-hidden-the-symbols.patch new file +37 -0

new file mode 100644
index 00000000000..0b6ed285621
@@ -0,0 +1,37 @@
+From 8e0c2d18a929f05bfadcbf351b244488917af86a Mon Sep 17 00:00:00 2001
+From: Mingli Yu <[email protected]>
+Date: Wed, 30 May 2018 15:05:57 +0800
+Subject: [PATCH] boost_context: strip the symbols
+
+The boost context library exports the 
+make_x86_64_sysv_elf_gas.o which contains
+host path as below:
+$ readelf --wide --symbols /my-build/boost/1.67.0-r0/boost_1_67_0/x86_64-poky-linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38ed/libboost_context.so.1.67.0
+
+42: 0000000000000000 0 FILE LOCAL DEFAULT ABS /my-build/boost/1.67.0-r0/boost_1_67_0/x86_64-poky-linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38ed/asm/make_x86_64_sysv_elf_gas.o
+
+Enable strip to hidden the symbols for 
+boost context library 
+
+Upstream-Status: Pending
+
+Signed-off-by: Mingli Yu <[email protected]>
+---
+ libs/context/build/Jamfile.v2 | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/libs/context/build/Jamfile.v2 b/libs/context/build/Jamfile.v2
+index 82ee76e..a6bd12f 100644
+--- a/libs/context/build/Jamfile.v2
++++ b/libs/context/build/Jamfile.v2
+@@ -822,6 +822,7 @@ lib boost_context
+    : impl_sources
+      execution_context.cpp
+      stack_traits_sources
++   : <strip>on
+    ;
+ 
+ boost-install boost_context ;
+-- 
+2.7.4
+

meta/recipes-support/boost/boost_1.67.0.bb +1 -0

index eb50899d562..8ba32d1574f 100644
@@ -7,4 +7,5 @@ SRC_URI += "\
     file://boost-math-disable-pch-for-gcc.patch \
     file://0001-Apply-boost-1.62.0-no-forced-flags.patch.patch \
     file://0003-Don-t-set-up-arch-instruction-set-flags-we-do-that-o.patch \
+    file://boost-context-hidden-the-symbols.patch \
 "

Signature · 1 line

2.11.0

Richard Purdie

2018-05-31 13:13 UTC #1 lore raw

Quoted 15 lines · On Wed, 2018-05-30 at 23:20 -0700, [email protected] wrote:

From: Mingli Yu <[email protected]>
boost_context includes some assembly source
files and will export the assembly object file
make_x86_64_sysv_elf_gas.o which contains host
path as below:
$ readelf --wide --symbols /my-build/boost/1.67.0-
r0/boost_1_67_0/x86_64-poky-
linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38e
d/libboost_context.so.1.67
42: 0000000000000000 0 FILE LOCAL DEFAULT ABS /my-build/boost/1.67.0-
r0/boost_1_67_0/x86_64-poky-
linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38e
d/asm/make_x86_64_sysv_elf_gas.o

Quoted 6 lines

Enable stirp for boost_context to hidden the
symbols
[YOCTO #12708]
Signed-off-by Mingli Yu <[email protected]>

This looks very unlikely to ever make it upstream. Rather than carrying a patch to boost forever would we want to just strip the file in question with a custom command?

Alternatively, can we stop this path being injected into the binary in the first place?

Cheers,

Richard

Yu, Mingli

author

2018-06-01 01:44 UTC #2 lore raw

Quoted 19 lines · Richard Purdie

Quoted 18 lines · On Wed, 2018-05-30 at 23:20 -0700, [email protected] wrote:
From: Mingli Yu <[email protected]>
boost_context includes some assembly source
files and will export the assembly object file
make_x86_64_sysv_elf_gas.o which contains host
path as below:
$ readelf --wide --symbols /my-build/boost/1.67.0-
r0/boost_1_67_0/x86_64-poky-
linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38e
d/libboost_context.so.1.67
42: 0000000000000000 0 FILE LOCAL DEFAULT ABS /my-build/boost/1.67.0-
r0/boost_1_67_0/x86_64-poky-
linux/boost/bin.v2/libs/context/build/aca09349fdb84d131321425f6c3a38e
d/asm/make_x86_64_sysv_elf_gas.o
Enable stirp for boost_context to hidden the
symbols

Quoted 7 lines

Quoted 3 lines
[YOCTO #12708]
Signed-off-by Mingli Yu <[email protected]>
This looks very unlikely to ever make it upstream. Rather than carrying a patch to boost forever would we want to just strip the file in question with a custom command?

Thanks very much Richard for your respond!
I did try to strip only file which in question, but not find out the solution and then turn to strip the context library.

Alex,
Do you have any suggestion?

Thanks,

Quoted 8 lines

Alternatively, can we stop this path being injected into the binary in the first place?
Cheers,
Richard

Alexander Kanavin

2018-06-01 05:08 UTC #3 lore raw

2018-06-01 4:39 GMT+03:00 Yu, Mingli <[email protected]>:

Quoted 11 lines

Quoted 3 lines
This looks very unlikely to ever make it upstream. Rather than carrying a patch to boost forever would we want to just strip the file in question with a custom command?
Thanks very much Richard for your respond!
I did try to strip only file which in question, but not find out the solution and then turn to strip the context library.
Alex,
Do you have any suggestion?

Yes - as RP said, please investigate how the path gets into this specific file (but not other files) in the first place, and whether we can disable or make it not happen, rather than strip the path after the fact.

Alex

Yu, Mingli

author

2018-06-01 05:43 UTC #4 lore raw

Quoted 11 lines · Alexander Kanavin

2018-06-01 4:39 GMT+03:00 Yu, Mingli <[email protected]>:
Quoted 9 lines
Quoted 3 lines
This looks very unlikely to ever make it upstream. Rather than carrying a patch to boost forever would we want to just strip the file in question with a custom command?
Thanks very much Richard for your respond!
I did try to strip only file which in question, but not find out the solution and then turn to strip the context library.

Quoted 7 lines

Quoted 2 lines
Alex,
Do you have any suggestion?
Yes - as RP said, please investigate how the path gets into this specific file (but not other files) in the first place, and whether we can disable or make it not happen, rather than strip the path after the fact.

Hi Alex,

Thanks for your respond!
I did investigate the path a lot before send out the patch, but didn't figure out why it introduce the path for make_x86_64_sysv_elf_gas.o whose source file is make_x86_64_sysv_elf_gas.S.

Anyway, I will try to dig more.

If you have any ideas, welcome to help provide some hint.

Thanks,

Quoted 3 lines

Alex

Alexander Kanavin

2018-06-01 08:36 UTC #5 lore raw

2018-06-01 8:38 GMT+03:00 Yu, Mingli <[email protected]>:

Quoted 7 lines

I did investigate the path a lot before send out the patch, but didn't figure out why it introduce the path for make_x86_64_sysv_elf_gas.o whose source file is make_x86_64_sysv_elf_gas.S.
Anyway, I will try to dig more.
If you have any ideas, welcome to help provide some hint.

I'm not an expert in symbol tables and what goes into them, but I strongly suspect the full path is introduced by linker because that .S file has two assembly labels (trampoline, finish). The other two assembly files have no such labels, and so their file paths are not added to the .so. I'm also not sure why there needs to be a full path, as opposed to just the file name, as is the case for .c files (maybe to avoid clashes with labels of the same name in different files?).

We should probably just strip that one specific .o file before it's linked into the library. Or if there is a way to prevent the labels from being written into the symbol table by the assembler, we can use that too. Sadly boost has its own build system, so doing such custom tweaks may be tricky.

Alex

Yu, Mingli

author

2018-06-05 02:48 UTC #6 lore raw

Quoted 8 lines · Alexander Kanavin

2018-06-01 8:38 GMT+03:00 Yu, Mingli <[email protected]>:
Quoted 6 lines
I did investigate the path a lot before send out the patch, but didn't figure out why it introduce the path for make_x86_64_sysv_elf_gas.o whose source file is make_x86_64_sysv_elf_gas.S.
Anyway, I will try to dig more.

Quoted 7 lines

Quoted 1 line
If you have any ideas, welcome to help provide some hint.
I'm not an expert in symbol tables and what goes into them, but I strongly suspect the full path is introduced by linker because that .S file has two assembly labels (trampoline, finish). The other two assembly files have no such labels, and so their file paths are not added to the .so. I'm also not sure why there needs to be a full path,

Thanks Alex very much for your comments!
I think the full path is introduced by linker and I'm also confused why other two assembly files' o file not introduced in the final .so.

Quoted 2 lines

as opposed to just the file name, as is the case for .c files (maybe to avoid clashes with labels of the same name in different files?).

I'm also concerned about that maybe it's necessary to have the full path.

Thanks,

Quoted 9 lines

We should probably just strip that one specific .o file before it's linked into the library. Or if there is a way to prevent the labels from being written into the symbol table by the assembler, we can use that too. Sadly boost has its own build system, so doing such custom tweaks may be tricky.
Alex

Khem Raj

2018-06-01 17:07 UTC #7 lore raw

In reply to Yu, Mingli

Quoted 24 lines · Mingli

Quoted 11 lines · Alexander Kanavin
2018-06-01 4:39 GMT+03:00 Yu, Mingli <[email protected]>:
Quoted 9 lines
Quoted 3 lines
This looks very unlikely to ever make it upstream. Rather than carrying a patch to boost forever would we want to just strip the file in question with a custom command?
Thanks very much Richard for your respond!
I did try to strip only file which in question, but not find out the solution and then turn to strip the context library.
Quoted 7 lines
Quoted 2 lines
Alex,
Do you have any suggestion?
Yes - as RP said, please investigate how the path gets into this specific file (but not other files) in the first place, and whether we can disable or make it not happen, rather than strip the path after the fact.
Hi Alex,

Quoted 9 lines

Thanks for your respond!
I did investigate the path a lot before send out the patch, but didn't figure out why it introduce the path for make_x86_64_sysv_elf_gas.o whose source file is make_x86_64_sysv_elf_gas.S.
Anyway, I will try to dig more.
If you have any ideas, welcome to help provide some hint.

these symbols are added by assembler, if you have .file asm directive either explicitly added to .s file or emitted by compiler, compiler would emit it if the path specified to source file is absolute during compiler invocation, it will take it verbatim and put it in generated .s file

you might then want to check if you can see whats happening when this file is being either compiled or assembled and make the changes there such that it can be specified via a topdir relative path. that should fix the issue.

Quoted 6 lines

Thanks,
Quoted 3 lines
Alex

Alexander Kanavin

2018-06-01 17:56 UTC #8 lore raw

2018-06-01 20:07 GMT+03:00 Khem Raj <[email protected]>:

Quoted 5 lines

these symbols are added by assembler, if you have .file asm directive either explicitly added to .s file or emitted by compiler, compiler would emit it if the path specified to source file is absolute during compiler invocation, it will take it verbatim and put it in generated .s file

The .s file is not generated and it does not contain any .file directives. The .o file also does not contain the path. It appears in the final .so library at linking stage.

Alex

Khem Raj

2018-06-01 18:33 UTC #9 lore raw

Quoted 10 lines · Alexander Kanavin

2018-06-01 20:07 GMT+03:00 Khem Raj <[email protected]>:
Quoted 5 lines
these symbols are added by assembler, if you have .file asm directive either explicitly added to .s file or emitted by compiler, compiler would emit it if the path specified to source file is absolute during compiler invocation, it will take it verbatim and put it in generated .s file
The .s file is not generated and it does not contain any .file directives. The .o file also does not contain the path. It appears in the final .so library at linking stage.

Can you share the linker cmdline ?

Quoted 3 lines

Alex

Alexander Kanavin

2018-06-01 19:34 UTC #10 lore raw

Quoted 5 lines · Khem Raj

Quoted 3 lines
The .s file is not generated and it does not contain any .file directives. The .o file also does not contain the path. It appears in the final .so library at linking stage.
Can you share the linker cmdline ?

https://pastebin.com/WK9GVDfj

Note that of the three assembly-generated object files, only the path to the first shows up in the .so - because of assembly labels I believe.

Alex

Khem Raj

2018-06-15 06:44 UTC #11 lore raw

Quoted 11 lines · Alexander Kanavin

Quoted 5 lines · Khem Raj
Quoted 3 lines
The .s file is not generated and it does not contain any .file directives. The .o file also does not contain the path. It appears in the final .so library at linking stage.
Can you share the linker cmdline ?
https://pastebin.com/WK9GVDfj
Note that of the three assembly-generated object files, only the path to the first shows up in the .so - because of assembly labels I believe.

Thanks for sharing this. If there is a no file directive in an object file then linker will use whatever is specified on linker cmdline so thats what we see, We should define FILE symbol explicitly for .S files here, for C/C++ files compiler does that automatically.

Other files dont make a claim in final symbol table since this one file alone has cross referencing to symbols from other objects.

Something like this will help

libs/context/src/asm/make_i386_sysv_elf_gas.S +0 -1

Index: boost_1_67_0/libs/context/src/asm/make_i386_sysv_elf_gas.S
===================================================================
@@ -23,7 +23,7 @@
  * 
---------------------------------------------------------------------------------- 
 *
  * 
                 *

****************************************************************************************/ -
+.file "make_i386_sysv_elf_gas.S"
.text
.globl make_fcontext
.align 2

libs/context/src/asm/make_x86_64_sysv_elf_gas.S +0 -1

Index: boost_1_67_0/libs/context/src/asm/make_x86_64_sysv_elf_gas.S
===================================================================
@@ -23,7 +23,7 @@
  * 
---------------------------------------------------------------------------------- 
 *
  * 
                 *

****************************************************************************************/ -
+.file "make_x86_64_sysv_elf_gas.S"
.text
.globl make_fcontext
.type make_fcontext,@function

Quoted 1 line

Alex

Alexander Kanavin

2018-06-15 06:54 UTC #12 lore raw

Thanks! Mingli, can you try this please?

Alex

2018-06-15 9:44 GMT+03:00 Khem Raj <[email protected]>:

Quoted 71 lines · Alexander Kanavin

Quoted 8 lines · Khem Raj
Quoted 6 lines
Quoted 4 lines
The .s file is not generated and it does not contain any .file directives. The .o file also does not contain the path. It appears in the final .so library at linking stage.
Quoted 8 lines
Quoted 1 line
Can you share the linker cmdline ?
https://pastebin.com/WK9GVDfj
Note that of the three assembly-generated object files, only the path to the first shows up in the .so - because of assembly labels I believe.
Thanks for sharing this. If there is a no file directive in an object file then linker will use whatever is specified on linker cmdline so thats what we see, We should define FILE symbol explicitly for .S files here, for C/C++ files compiler does that automatically.
Other files dont make a claim in final symbol table since this one file alone has cross referencing to symbols from other objects.
Something like this will help
Index: boost_1_67_0/libs/context/src/asm/make_i386_sysv_elf_gas.S
===================================================================
libs/context/src/asm/make_i386_sysv_elf_gas.S +1 -2
@@ -23,7 +23,7 @@
  *
----------------------------------------------------------------------------------
 *
  *                  *
 
 ****************************************************************************************/
-
+.file "make_i386_sysv_elf_gas.S"
 .text
 .globl make_fcontext
 .align 2
Index: boost_1_67_0/libs/context/src/asm/make_x86_64_sysv_elf_gas.S
===================================================================
libs/context/src/asm/make_x86_64_sysv_elf_gas.S +1 -2
@@ -23,7 +23,7 @@
  *
----------------------------------------------------------------------------------
 *
  *                  *
 
 ****************************************************************************************/
-
+.file "make_x86_64_sysv_elf_gas.S"
 .text
 .globl make_fcontext
 .type make_fcontext,@function
Quoted 3 lines
Alex
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Yu, Mingli

author

2018-06-15 07:35 UTC #13 lore raw

Quoted 1 line · Alexander Kanavin

Thanks! Mingli, can you try this please?

Sure,

Quoted 33 lines

Alex
2018-06-15 9:44 GMT+03:00 Khem Raj <[email protected]>:
Quoted 29 lines · Alexander Kanavin
Quoted 8 lines · Khem Raj
Quoted 6 lines
Quoted 4 lines
The .s file is not generated and it does not contain any .file directives. The .o file also does not contain the path. It appears in the final .so library at linking stage.
Quoted 8 lines
Quoted 1 line
Can you share the linker cmdline ?
https://pastebin.com/WK9GVDfj
Note that of the three assembly-generated object files, only the path to the first shows up in the .so - because of assembly labels I believe.
Thanks for sharing this. If there is a no file directive in an object file then linker will use whatever is specified on linker cmdline so thats what we see, We should define FILE symbol explicitly for .S files here, for C/C++ files compiler does that automatically.
Other files dont make a claim in final symbol table since this one file alone has cross referencing to symbols from other objects.
Something like this will help

Thanks Khem very much!

Have set .file section in .S file as you suggested, it works.

Thanks,

Quoted 43 lines

Quoted 42 lines
Index: boost_1_67_0/libs/context/src/asm/make_i386_sysv_elf_gas.S
===================================================================
libs/context/src/asm/make_i386_sysv_elf_gas.S +1 -2
@@ -23,7 +23,7 @@
   *
----------------------------------------------------------------------------------
 *
   *                  *
 
 ****************************************************************************************/
-
+.file "make_i386_sysv_elf_gas.S"
  .text
  .globl make_fcontext
  .align 2
Index: boost_1_67_0/libs/context/src/asm/make_x86_64_sysv_elf_gas.S
===================================================================
libs/context/src/asm/make_x86_64_sysv_elf_gas.S +1 -2
@@ -23,7 +23,7 @@
   *
----------------------------------------------------------------------------------
 *
   *                  *
 
 ****************************************************************************************/
-
+.file "make_x86_64_sysv_elf_gas.S"
  .text
  .globl make_fcontext
  .type make_fcontext,@function
Quoted 3 lines
Alex
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core