meta-toolchain doesn't compile working binaries
Hi,
I'm having trouble building a working toolchain (meta-toolchain) on yocto-1.4 release.
Buildsystem is x86_64, on current Debian Sid.
The problem seems to be with ld-linux[...].so, SDKMACHINE doesn't seem to matter (x86_64 and i686 produce broken binaries).
The effect is, that (almost) every binary in the toolchain segfaults when being run.
Example m4:
(in /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux)
Host ld:
$ usr/bin/m4 --help Segmentation fault
Explicitly using the toolchain ld:
$ lib/ld-linux-x86-64.so.2 usr/bin/m4 --help Usage: usr/bin/m4 [OPTION]... [FILE]... [...]
Comparing the build with a toolchain created on Jul 30th 2013 (which works fine) I see one major difference:
Old (working) toolchain:
$ ldd m4
linux-vdso.so.1 (0x00007fffdae00000)
libc.so.6 => /usr/local/oecore-x86_64.old_201307/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f6a71ed0000)
/usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f6a72280000)New (broken) toolchain:
$ ldd m4
linux-vdso.so.1 (0x00007fff36e00000)
libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f00e7860000)
/lib64/ld-linux-x86-64.so.2 (0x00007f00e7c10000)The internal toolchain (from tmp/build/) is working fine, though the ldd output looks similar to the broken built one:
[...]build/tmp-angstrom_v2013_06-eglibc/sysroots/x86_64-linux/usr/bin$ ldd m4
linux-vdso.so.1 (0x00007fff9fdc8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f481cda8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f481d188000)ATM, on Debian Sid /lib64/ld-linux-x86-64.so.2 is just a link /lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.18.so
I'd appreciate if someone could give me a hint on this matter, or a point me in a direction, where the problem might be. As mentioned I already tried changing the SDKMACHINE and a completely clean build folder with no effect.
Thanks
Björn
Signature · 2 lines
Björn Krombholz pironex GmbH -- http://www.pironex.de
Quoted 1 line · Björn Krombholz
Comparing the build with a toolchain created on Jul 30th 2013 (which works fine) I see one major difference:
what is the delta between the working and non working OE-Core repo
Quoted 51 lines · Björn Krombholz
Hi,
I'm having trouble building a working toolchain (meta-toolchain) on yocto-1.4 release.
Buildsystem is
x86_64, on current Debian Sid.The problem seems to be with ld-linux[...].so, SDKMACHINE doesn't seem to matter (
x86_64and i686 produce broken binaries).The effect is, that (almost) every binary in the toolchain segfaults when being run.
Example m4:
(in/usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux)Host ld:
$ usr/bin/m4 --help Segmentation faultExplicitly using the toolchain ld:
$ lib/ld-linux-x86-64.so.2 usr/bin/m4 --help Usage: usr/bin/m4 [OPTION]... [FILE]... [...]Comparing the build with a toolchain created on Jul 30th 2013 (which works fine) I see one major difference:
Old (working) toolchain:
$ ldd m4 linux-vdso.so.1 (0x00007fffdae00000) libc.so.6 => /usr/local/oecore-x86_64.old_201307/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f6a71ed0000) /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f6a72280000)New (broken) toolchain:
$ ldd m4 linux-vdso.so.1 (0x00007fff36e00000) libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f00e7860000) /lib64/ld-linux-x86-64.so.2 (0x00007f00e7c10000)The internal toolchain (from tmp/build/) is working fine, though the ldd output looks similar to the broken built one:
[...]build/tmp-angstrom_v2013_06-eglibc/sysroots/x86_64-linux/usr/bin$ ldd m4
linux-vdso.so.1 (0x00007fff9fdc8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f481cda8000) /lib64/ld-linux-x86-64.so.2 (0x00007f481d188000)
Quoted 6 lines
ATM, on Debian Sid
/lib64/ld-linux-x86-64.so.2is just a link/lib64/ld-linux-x86-64.so.2->/lib/x86_64-linux-gnu/ld-2.18.soI'd appreciate if someone could give me a hint on this matter, or a point me in a direction, where the problem might be. As mentioned I already tried changing the SDKMACHINE and a completely clean build folder with no effect.
Do you know that OE toolchain is not relocatable? The above symptoms are very similar to when you try to execute toolchain binaries from a different location that the toolchain was built for or installed to. There is a script called relocate_sdk.py that adjusts binaries and libraries to work in the required location - it's called during install of the SDK. See pages 24-27 of my last year ELC presentation about more details on how OE SDK works:
http://elinux.org/images/c/c8/ExternalToolchainsInYocto.pdf
Signature · 1 line
Denys
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
Quoted 5 lines · Khem Raj
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
It's the load path of the executables, they use the libc-nativesdk, so they need a hard path to the correct ld.so to get started.
--Mark
Quoted 4 lines · Mark Hatle
Quoted 2 lines · Khem Raj
On Tue, May 6, 2014 at 1:55 PM, Denys Dmytriyenko <[email protected]>
Quoted 8 lines · wrote:
Quoted 6 lines
Quoted 2 lines
Do you know that OE toolchain is not relocatable?
is that true ?
It's the load path of the executables, they use the libc-nativesdk, so
they need a hard path to the correct ld.so to get started.
isnt this fixed by the installer when it is run
Quoted 8 lines
--Mark
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 12 lines · Khem Raj
Quoted 4 lines · On May 6, 2014 2:25 PM, "Mark Hatle" <[email protected]…
Quoted 2 lines · Khem Raj
On Tue, May 6, 2014 at 1:55 PM, Denys Dmytriyenko <[email protected]
Quoted 4 lines · <mailto:[email protected]>> wrote:
Quoted 4 lines
Quoted 2 lines
Do you know that OE toolchain is not relocatable?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
is that true ?
It's the load path of the executables, they use the libc-nativesdk, so they
need a hard path to the correct ld.so to get started.
isnt this fixed by the installer when it is run
Yes, it sounds like either he moved the files after the the installer ran, or didn't use the installer at all and just tried to execute them out of the build system.
(If the installer -was- used, then it's definitely a bug.)
--Mark
Quoted 10 lines
Quoted 7 lines
--Mark
--
_______________________________________________
Openembedded-core mailing list
[email protected]<mailto:[email protected]>
Quoted 9 lines · Mark Hatle
Quoted 5 lines · Khem Raj
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
It's the load path of the executables, they use the libc-nativesdk, so they need a hard path to the correct ld.so to get started.
More specifically, it's PT_INTERP section of the ELF header in every binary that hardcodes the full path to our own dynamic linker/loader (i.e. mentioned libc-nativesdk). I wish it would support the use of $ORIGIN similar to RPATH and RUNPATH for libraries, but I bet there are all kinds of corner cases with that... :)
Signature · 1 line
Denys
Quoted 16 lines · Mark Hatle
Quoted 15 lines · Khem Raj
Quoted 4 lines · On May 6, 2014 2:25 PM, "Mark Hatle" <[email protected]…
Quoted 2 lines · Khem Raj
On Tue, May 6, 2014 at 1:55 PM, Denys Dmytriyenko <[email protected]
Quoted 7 lines · <mailto:[email protected]>> wrote:
Quoted 6 lines
Quoted 2 lines
Do you know that OE toolchain is not relocatable?
is that true ?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
It's the load path of the executables, they use the libc-nativesdk, so they
need a hard path to the correct ld.so to get started.
isnt this fixed by the installer when it is run
Yes, it sounds like either he moved the files after the the installer ran, or didn't use the installer at all and just tried to execute them out of the build system.
Exactly what I said earlier about using relocate_sdk.py in the installer, which Khem conveniently ignored and removed when replying... :) Just kidding.
Quoted 1 line
(If the installer -was- used, then it's definitely a bug.)
There's no bug as far as I know, as SDK/toolchain is a major part of our product and we test it actively with every release.
Signature · 1 line
Denys
Quoted 4 lines · Khem Raj
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
Is not auto-relocatable - is it better?
Signature · 1 line
Denys
Quoted 19 lines · Mark Hatle
Quoted 8 lines · Denys Dmytriyenko
Quoted 7 lines · Mark Hatle
Quoted 5 lines · Khem Raj
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
Quoted 9 lines
Quoted 2 lines
It's the load path of the executables, they use the libc-nativesdk, so they need a hard path to the correct ld.so to get started.
More specifically, it's
PT_INTERPsection of the ELF header in every binary that hardcodes the full path to our own dynamic linker/loader (i.e. mentioned libc-nativesdk). I wish it would support the use of $ORIGIN similar to RPATH and RUNPATH for libraries, but I bet there are all kinds of corner cases with that... :)
Quoted 2 lines
$ORIGIN, RPATH and RUNPATH are all implemented by the ld.so, there is no way to dynamically assign the interpreter (ld.so).
Yes, the above wish was highly abstract and hypothetical and would probably require kernel to support $ORIGIN used in the ELF header... But I'm not suggesting it, just speculating! :)
Signature · 1 line
Denys
Quoted 8 lines · Denys Dmytriyenko
Quoted 7 lines · Mark Hatle
Quoted 5 lines · Khem Raj
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
Quoted 9 lines
Quoted 2 lines
It's the load path of the executables, they use the libc-nativesdk, so they need a hard path to the correct ld.so to get started.
More specifically, it's
PT_INTERPsection of the ELF header in every binary that hardcodes the full path to our own dynamic linker/loader (i.e. mentioned libc-nativesdk). I wish it would support the use of $ORIGIN similar to RPATH and RUNPATH for libraries, but I bet there are all kinds of corner cases with that... :)
$ORIGIN, RPATH and RUNPATH are all implemented by the ld.so, there is no way to dynamically assign the interpreter (ld.so).
--Mark
Quoted 7 lines · Mark Hatle
Quoted 6 lines · Khem Raj
On May 6, 2014 2:25 PM, "Mark Hatle" <[email protected] <mailto:[email protected]>> wrote:
so they
need a hard path to the correct ld.so to get started.
Quoted 7 lines
Quoted 1 line
isnt this fixed by the installer when it is run
Yes, it sounds like either he moved the files after the the installer ran, or didn't use the installer at all and just tried to execute them out of the build system.
(If the installer -was- used, then it's definitely a bug.)
The installer _was_ used, I didn't move any files and used the default location /usr/local/oecore-[...]
Thanks for the hints.
Additional note, in case it was missed: I am using v2013.06 (yocto 1.4 base, Angstrom variant)
I guess the problem is, that the relocation_sdk.py excludes /lib/ and 32/64 bit variants.
I just
- re-run the installer with -S
- edited the extracted relocation script commented the lines out:
#if fname.startswith("/lib/") or fname.startswith("/lib64/") or fname.startswith("/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib64/"):
# break- re-run the relocation
Now I see the expected result, matching the "old" toolchain from summer 2013:
# ldd /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/m4
linux-vdso.so.1 (0x00007fffe6800000)
libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/../../lib/libc.so.6 (0x00007f86e8110000)
/usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f86e84c0000)Executing the binary works as well.
Comparing the binaries from the current toolchain with those from the old one skipping the relocation step (extracted toolchain with -R) shows:
Old and working:
# ldd m4
linux-vdso.so.1 (0x00007fffb1800000)
libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f50bd210000)
/usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f50bd5c0000)New and broken:
# ldd m4
linux-vdso.so.1 (0x00007fff6c000000)
libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007fefc1de8000)
/lib64/ld-linux-x86-64.so.2 (0x00007fefc2198000)/lib64/ obviously doesn't get replaced.
I have no clue how to debug this any further. Not even sure if the problem is more likely related to eglibc packages or somewhere hidden in the general OE build scripts. :/
I will build a v2013.12 version now and check if there is any difference.
Signature · 2 lines
Björn Krombholz pironex GmbH -- http://www.pironex.de
Quoted 35 lines · Björn Krombholz
Quoted 7 lines · Mark Hatle
Quoted 6 lines · Khem Raj
On May 6, 2014 2:25 PM, "Mark Hatle" <[email protected] <mailto:[email protected]>> wrote:
so they
need a hard path to the correct ld.so to get started.Quoted 7 lines
Quoted 1 line
isnt this fixed by the installer when it is run
Yes, it sounds like either he moved the files after the the installer ran, or didn't use the installer at all and just tried to execute them out of the build system.
(If the installer -was- used, then it's definitely a bug.)
The installer _was_ used, I didn't move any files and used the default location /usr/local/oecore-[...]
Thanks for the hints.
Additional note, in case it was missed: I am using v2013.06 (yocto 1.4 base, Angstrom variant)
I guess the problem is, that the
relocation_sdk.pyexcludes/lib/and 32/64 bit variants.I just
- re-run the installer with -S
- edited the extracted relocation script commented the lines out:#if fname.startswith("/lib/") or fname.startswith("/lib64/") or fname.startswith("/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib64/"): # break
Quoted 7 lines
- re-run the relocation
Now I see the expected result, matching the "old" toolchain from summer 2013:
# ldd /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/m4 linux-vdso.so.1 (0x00007fffe6800000) libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/../../lib/libc.so.6 (0x00007f86e8110000) /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f86e84c0000)
The bove may work, but it's incorrect. It appears to be trying to run:
/lib64/ld-linux-x86-64.so.2, -instead- of
/usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2
The check is there specifically because some distribution creators add software that is expected to use the host system's ld.so (ld-linux-x86-64.so.2 in this case) loader. So we detect that behavior by looking for system paths embedded into a particular binaries executable.
In this case, where m4 is provided as a nativesdk item, there should be no references at all to the system /lib64/ld-linux-x86-64.so.2.
I'm not familiar with the YP 1.4 + Angstrom variant to expand on this and determine where the error actually is. But in general nativesdk elements referencing any type of system ld.so is incorrect.
--Mark
Quoted 23 lines
Executing the binary works as well.
Comparing the binaries from the current toolchain with those from the old one skipping the relocation step (extracted toolchain with -R) shows:
Old and working:
# ldd m4 linux-vdso.so.1 (0x00007fffb1800000) libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f50bd210000) /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f50bd5c0000)New and broken:
# ldd m4 linux-vdso.so.1 (0x00007fff6c000000) libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007fefc1de8000) /lib64/ld-linux-x86-64.so.2 (0x00007fefc2198000)
/lib64/obviously doesn't get replaced.I have no clue how to debug this any further. Not even sure if the problem is more likely related to eglibc packages or somewhere hidden in the general OE build scripts. :/
I will build a v2013.12 version now and check if there is any difference.
Quoted 5 lines · Björn Krombholz
Additional note, in case it was missed: I am using v2013.06 (yocto 1.4 base, Angstrom variant)
I guess the problem is, that the
relocation_sdk.pyexcludes/lib/and 32/64 bit variants.
[...]
Quoted 11 lines
New and broken:
# ldd m4 linux-vdso.so.1 (0x00007fff6c000000) libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007fefc1de8000) /lib64/ld-linux-x86-64.so.2 (0x00007fefc2198000)
/lib64/obviously doesn't get replaced.I have no clue how to debug this any further. Not even sure if the problem is more likely related to eglibc packages or somewhere hidden in the general OE build scripts. :/
I will build a v2013.12 version now and check if there is any difference.
Quick followup.
yocto-1.5 / Angstrom-v2013.12 _does_ work as expected.
Signature · 2 lines
Björn Krombholz pironex GmbH -- http://www.pironex.de
Quoted 20 lines · Denys Dmytriyenko
Quoted 19 lines · Mark Hatle
Quoted 17 lines · Denys Dmytriyenko
Quoted 9 lines · Mark Hatle
Quoted 5 lines · Khem Raj
Quoted 1 line · Denys Dmytriyenko
Do you know that OE toolchain is not relocatable?
is that true ?
It's the load path of the executables, they use the libc-nativesdk, so they need a hard path to the correct ld.so to get started.
More specifically, it's
PT_INTERPsection of the ELF header in every binary that hardcodes the full path to our own dynamic linker/loader (i.e. mentioned libc-nativesdk). I wish it would support the use of $ORIGIN similar to RPATH and RUNPATH for libraries, but I bet there are all kinds of corner cases with that... :)
Quoted 6 lines
Quoted 2 lines
$ORIGIN, RPATH and RUNPATH are all implemented by the ld.so, there is no way to dynamically assign the interpreter (ld.so).
Yes, the above wish was highly abstract and hypothetical and would probably require kernel to support $ORIGIN used in the ELF header... But I'm not suggesting it, just speculating! :)
Right, that would need support in the kernel and would mean that the SDK was restricted to specific kernel versions.
Having said that, I have wondered if we could see if we could get it in and then someday remove this restriction.
Cheers,
Richard
Quoted 22 lines · Björn Krombholz
Quoted 6 lines · Björn Krombholz
Additional note, in case it was missed: I am using v2013.06 (yocto 1.4 base, Angstrom variant)
I guess the problem is, that the
relocation_sdk.pyexcludes/lib/and 32/64 bit variants.[...]
Quoted 14 lines
New and broken:
# ldd m4 linux-vdso.so.1 (0x00007fff6c000000) libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007fefc1de8000) /lib64/ld-linux-x86-64.so.2 (0x00007fefc2198000)
/lib64/obviously doesn't get replaced.I have no clue how to debug this any further. Not even sure if the problem is more likely related to eglibc packages or somewhere hidden in the general OE build scripts. :/
Quoted 6 lines
Quoted 1 line
I will build a v2013.12 version now and check if there is any difference.
Quick followup.
yocto-1.5 / Angstrom-v2013.12 _does_ work as expected.
...meaning, binaries do run without segfaulting, though as Mark suggested, there shouldn't be any reference to the hosts /lib64/...
The ldd output for an installed nativesdk-m4 from yocto-1.5 looks like:
# ldd m4
linux-vdso.so.1 (0x00007fff92700000)
libc.so.6 => /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/usr/bin/./../../lib/libc.so.6 (0x00007f8aa86e8000)
/usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f8aa8a98000)# ls -l /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld* -rwxr-xr-x 1 root root 149304 May 7 16:00 /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-2.18-2013.10.so lrwxrwxrwx 1 root root 18 May 7 16:01 /usr/local/oecore-x86_64/sysroots/x86_64-angstromsdk-linux/lib/ld-linux-x86-64.so.2 -> ld-2.18-2013.10.so
:-/
Signature · 2 lines
Björn Krombholz pironex GmbH -- http://www.pironex.de