package/binutils+elf2flt: fix for noMMU targets

Hello All!

TL;DR: gcc gets confused by our symlinks trick that we use to fix the rpath handling. We replace symlinks with script wrappers instead.

See the first commit for the whole story.

Regards,
Yann E. MORIN.

The following changes since commit 2c57dc0469f7e32182932e5937f61f82b01a743a

cjson: bump to version 1.7.7 (2018-05-25 16:22:11 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 59b3d35d7c805ebee29dc7185ea63e55aca768ca

package/elf2flt: replace hard-links with script wrappers to fix rpath (2018-05-25 22:24:32 +0200)

----------------------------------------------------------------
Yann E. MORIN (2):
package/binutils: switch from symlinks to script wrappers package/elf2flt: replace hard-links with script wrappers to fix rpath

 package/binutils/binutils.mk |  9 ++++++---
 package/elf2flt/elf2flt.mk   | 17 +++++++++++++++++
 2 files changed, 23 insertions(+), 3 deletions(-)

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Commit f9cffb6af464 (binutils: replace hard-links with soft-links to fix rpath) has a side effect that when we build for a noMMU target, elf2flt will in turn replace some of the programs installed by binutils, with its own wrappers.

For example, it will rename host/TUPLE/bin/ld to ld.real, and add its own wrapper in place of the origginal. It does the same for host/bin/TUPLE-ld and host/bin/TUPLE-ld.real.

However, we had already made ld a symlink to ../../bin/TUPLE-ld, so host/TUPLE/bin/ld.real will still point to host/bin/TUPLE-ld when we want it to point to ld.real instead...

This ultimately confuses gcc later on.

Of course, the culprit is also elf2flt, which also installs similar hardlinks that would ultimately exhibit the same rpath issue as the one fixed by f9cffb6af464. Note: we haven't had an issue so far with that, because those tools installed by elf2flt only link with libz, which is most often present on the host system. So, all seem well, but is nonetheless broken; this will be fixed in a subsequent commit.

But back on topic. If we were to fix elf2flt with similar symlinks, gcc still gets confused. The underlying reason for this confusion is not entirely clear, though... It looks like something is trying to dereference symlinks and gets confused by the resut somehow...

So, in an attempt to restore some sanity in all this mess, we try to restore the previous behaviour, we switch from using symlinks to using shell script wrappers. We ensure that the behaviour really mimicks that of a hardlink, by telling bash to set arg[0] to the name of the script, not to that of the wrapped executable:

    #!/usr/bin/env bash
    exec -a "${0}" "${0%/*}/../../bin/TUPLE-${0##*/}" "${@}"

"exec -a FOO" is a bashism that telss bash to use FOO as argv[0] instead of the name of the executable.

With this trick, we're back to working conditions for building for noMMU targets.

Note: the wrappers are effectively all totally identical, indeed. We can optimise that by using hardlinks in a followup commit, if that even makes sense...

---
 package/binutils/binutils.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

index b24e4334c2..25650b31c7 100644
@@ -132,13 +132,16 @@ endif
 
 # Hardlinks between binaries in different directories cause a problem
 # with rpath fixup, so we de-hardlink those binaries, and replace them
-# with symbolic links.
+# with wrapper scripts.
 BINUTILS_TOOLS = ar as ld ld.bfd nm objcopy objdump ranlib readelf strip
 define HOST_BINUTILS_FIXUP_HARDLINKS
 	$(foreach tool,$(BINUTILS_TOOLS),\
-		rm -f $(HOST_DIR)/$(GNU_TARGET_NAME)/bin/$(tool) ; \
-		ln -s ../../bin/$(GNU_TARGET_NAME)-$(tool) $(HOST_DIR)/$(GNU_TARGET_NAME)/bin/$(tool)
+		rm -f $(HOST_DIR)/$(GNU_TARGET_NAME)/bin/$(tool) && \
+		printf '#!/usr/bin/env bash\nexec -a "$${0}" "$${0%%/*}/%s-$${0##*/}" "$${@}"\n' \
+			../../bin/$(GNU_TARGET_NAME) \
+			>$(HOST_DIR)/$(GNU_TARGET_NAME)/bin/$(tool)
 	)
+	chmod +x $(HOST_DIR)/$(GNU_TARGET_NAME)/bin/*
 endef
 HOST_BINUTILS_POST_INSTALL_HOOKS += HOST_BINUTILS_FIXUP_HARDLINKS
 

2.14.1

Do for elf2flt what we did for binutils: replace the hardlinks (which break rpath handling) with script wrappers.

See previous commit (package/binutils: switch from symlinks to script wrappers) and commit f9cffb6af4 (binutils: replace hard-links with soft-links to fix rpath) for the complete story.

---
 package/elf2flt/elf2flt.mk | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

index 995dfedebd..6ce5cd34a2 100644
@@ -29,4 +29,21 @@ endif
 
 HOST_ELF2FLT_CONF_ENV = LIBS="$(HOST_ELF2FLT_LIBS)"
 
+# Hardlinks between binaries in different directories cause a problem
+# # with rpath fixup, so we de-hardlink those binaries, and replace them
+# with wrapper scripts. ld and ld.real are already wrappers, but they
+# point to the binutils, so we have to replace them with wrappers to
+# the elf2flt ones.
+ELF2FLT_TOOLS = elf2flt flthdr ld ld.real
+define HOST_ELF2FLT_FIXUP_HARDLINKS
+	$(foreach tool,$(ELF2FLT_TOOLS),\
+		rm -f $(HOST_DIR)/$(GNU_TARGET_NAME)/bin/$(tool) && \
+		printf '#!/usr/bin/env bash\nexec -a "$${0}" "$${0%%/*}/%s-$${0##*/}" "$${@}"\n' \
+			../../bin/$(GNU_TARGET_NAME) \
+			>$(HOST_DIR)/$(GNU_TARGET_NAME)/bin/$(tool)
+	)
+	chmod +x $(HOST_DIR)/$(GNU_TARGET_NAME)/bin/*
+endef
+HOST_ELF2FLT_POST_INSTALL_HOOKS += HOST_ELF2FLT_FIXUP_HARDLINKS
+
 $(eval $(host-autotools-package))

2.14.1

So maybe, instead of laying hacks upon hacks, we should go back to the drawing
board on the rpath handling? I would feel a lot more comfortable if we could just fix fix-rpath instead of playing the whack-a-mole game of workarounds in packages.

I don't see a full solution yet, but basically I think that we should
- detect when an executable has multiple hardlinks;
- detect if it has already been "fixed" in this fix-rpath run;
- if so, clone the file before fixing it proper.

The weak point is: if we clone, there may be some hardlinks that actually could
still be reused as hardlinks. So you'd have to detect them first.

Alternatively we could consider that this is anyway only about the host tree,
which is normally the smallest of the three trees (when you exclude external toolchain). So maybe we should just always clone for the host tree.

I realize we're very close to the release, but I still think that always
cloning is probably the safest way to do it.

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Arnout, All,

On 2018-05-27 16:34 +0200, Arnout Vandecappelle spake thusly:

Well, we're not really playing whack-a-mole, because the isue is only about packages that install hardlinks in two locations, because of the bin/TUPLE-foo and TUPLE/bin/foo "duplication".

The only ones doing such a thing AFAIK are binutils and elf2flt.

gcc installs hardlinks, but in the same directory. The tools it installs in a TUPLE-named directory are not hardlinks.

What did you mean by "clone" in this context? Remove the hardlinks and replace them by a copy of the file?

Sorry, I don't follow you here. If we replaced hardlinks with actual copies, how come we'd still have hardlinks?

I'm afraid that detecting hardlinks will not be cheap, even when we exclude staging/ (which is a sub-dir of host/) from the search...

Do we envision other impacted packages? I don't think so. Really, binutils and elf2flt really seem to be the only two packages impacted by the problem.

That we missed the noMMU and elf2flt cases previously is just an oversight on our part...

Then. if I am wrong, we're still talking about only two packages for now, and they only break because of the ARC archtecture, for which we use bleeding-edge tools that require we have libfl.so. Two. Single. Packages. We've always refused to add infra when only very few packages were impacted.

Or were you suggesting that we do not add infra, but we replace the current two hacks (Thomas' symlinks or my script wrappers) by an actual copy, still just for binutils and elf2flt? In which case we would not loose time detecting hard-links in a generic way, obviously.

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Arnout, all,

On 2018-05-27 17:35 +0200, Yann E. MORIN spake thusly:

[--SNIP--]

Something like the following, maybe?
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/binutils-elf2flt-fix-2

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Yes, +/- the comments that still refers to wrapper scripts.

Bye, Peter Korsgaard

Hi,

Yes, I guess that was what Arnout meant. "break" the hard link and duplicate the file content.

Just detecting hard linked files should be easy (E.G. st_nlink > 1 when stat'ing). Figuring out where the other hard links are is indeed hard, but we afaik don't need to know that.

I can follow Arnout, but I think the script wrappers (or simply copies of the binaries) is good enough for 2018.05.

Bye, Peter Korsgaard