Skip to contentSkip to conversation

Make the SDK relocatable

PATCH Superseded Wolfgang Grandegger 3 participants 30 messages Jul 4, 2017 – Jul 5, 2017 Kbuild

v6 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Wolfgang Grandegger

author
#0 lore raw

v6 cover Make the SDK relocatable

Hello,

this is v6 of my patch series to make the buildroot SDK (HOST_DIR) relocatable. It sanitizes the RPATH of all ELF files in the "target", "staging" and "host" tree using "patchelf --make-rpath-relative". We now use patchelf v0.9 to still support old Debian and RHEL systems.

v5 did RPATH sanitization per package after package installation into the host, staging or target tree using GLOBAL_INSTRUMENTATION_HOOKS. This approach got more and more complex and inefficient. Therefore it was abandoned in favor of global sanititation at the end of the host, staging and target build (see changes since v5).

Furthermore this patch creates the script "relocate-sdk.sh" in the top directory of the "host" tree allowing to relocate the SDK after it has been moved to a new location. It replaces the old path with the new one in all text files identified by "file --mime-type". The location is stored in "usr/share/buildroot/sdk-location".

Unfortunately, "qmake" uses hard-coded pathes compiled into the QT5 libraries. To overcome this problem, "qt5pase" now creates "qt.conf".

Hope I have not too much pending issues.

Wolfgang.

Things not yet addressed:

- "make toolchain" creates a toolchain tree which still has references to the build system (in ELF and text files).

Changes since v5:
- switch back to v4
- patchelf patches are now based on v0.9
- patchelf now calculates minimal relative path to the ELF file (and not to the rootdir as before).
- patchelf: neededLibFound is now passed to libFoundInRPath
- Makefile: the "host-finalize" target has been added for SDK relocation
- fix-rpath now uses variables to define the list of dirs to be pruned
- fix-rpath: the staging tree is sanitized like the target tree
- various minor fixes (typos)

Changes since v4:

- RPATH sanitation is now done per package and installation step in "pkg-generic.mk".
- After the installation step, the list of the installed files is stored in "<package-build-dir>/.br_[host|staging|target]_filelist".
- The GLOBAL_INSTRUMENTATION_HOOKS "step_sanitize_rpath" then calls "fix-rpath" to do the sanitation.
- DEPENDENCIES_HOST_PREREQ += host-patchelf is set early in the Makefile as we need it for RPATH sanitation. As soon it's available sanitation can start (currently missing some packages).
- The patchelf "file busy" issue is now worked-around differently.

Changes since v3:

- The patchelf patch implementing " --make-rpath-relative" now supports the option "--relative-to-file" instructing to use "$ORIGIN" in RPATHs. Otherwise an absolute path relative to the root directory will be used.
- The staging tree is now sanitized as well using the options "--relative-to-file" and "--no-standard-libs".
- For the "target" tree, relative RPATHs do not use "$ORIGIN" any longer. An absolute path relative to the root directory is used instead.

Changes since v2:

- provide "qt.conf" to make "qmake" relocatable
- sed now uses the separator "\" to substitute the directory path. It's one of the few characters not allowed in file names. To avoid interpreting it as escape character, the "read -r" is used.
- The paranoia substituion check is done before doing the real substituion.

Changes since v1:

- The name SDK has been chosen for the relocatabed "HOST_DIR" (instead of toolchanin).
- The patchelf version bump and patching are now done by 2 patches
- No more helper functions are used in the Makefile to call "fix-rpath" but added directly.
- The staging tree is not touched any more... until we have a good reason to do so.
- The sanitation is now performed by an optimized "fix-rpath" script.
- The relocate-sdk script is now copied for support/misc to the top directory of the host tree.

Samuel Martin (1):
support/scripts: add fix-rpath script to sanitize the rpath

Wolfgang Grandegger (9):
support/scripts: check-host-rpath now handles $ORIGIN/../lib package/patchelf: add patch for rpath sanitization under a root
directory
core: sanitize RPATH in staging tree at the end of target finalization
core: sanitize RPATH in target tree at the end of target finalization
core: sanitize RPATH in host tree at the very end of the build support/scripts: add relocate-sdk.sh script for SDK relocation
core: install relocation script and location at the end of the build
external-toolchain: check if a buildroot SDK has already been
relocated
package/qt5base: provide "qt.conf" to make "qmake" relocatable

 Makefile                                           |  17 +-
 ...move-apparently-incorrect-usage-of-static.patch |  49 +++
 ...unction-for-splitting-a-colon-separated-s.patch |  58 +++
 ...to-make-the-rpath-relative-under-a-specif.patch | 419 +++++++++++++++++++++
 package/qt5/qt5base/qt.conf.in                     |   6 +
 package/qt5/qt5base/qt5base.mk                     |   8 +
 support/misc/relocate-sdk.sh                       |  47 +++
 support/scripts/check-host-rpath                   |   2 +-
 support/scripts/fix-rpath                          | 122 ++++++
 toolchain/helpers.mk                               |  18 +
 .../toolchain-external/pkg-toolchain-external.mk   |   1 +
 11 files changed, 745 insertions(+), 2 deletions(-)
 create mode 100644 package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
 create mode 100644 package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
 create mode 100644 package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch
 create mode 100644 package/qt5/qt5base/qt.conf.in
 create mode 100755 support/misc/relocate-sdk.sh
 create mode 100755 support/scripts/fix-rpath
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v6 1/10 support/scripts: check-host-rpath now handles $ORIGIN/../lib

The new RPath sanitization uses now the relative path "$ORIGIN/../lib" for binaries in "/usr/bin".

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/check-host-rpath +1 -1
index 020c123..e79560b 100755
@@ -58,7 +58,7 @@ check_elf_has_rpath() {
         for dir in ${rpath//:/ }; do
             # Remove duplicate and trailing '/' for proper match
             dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" -o "${dir}" = "\$ORIGIN/../../usr/lib" ] && return 0
+            [ "${dir}" = "${hostdir}/usr/lib" -o "${dir}" = "\$ORIGIN/../lib" ] && return 0
         done
     done < <( readelf -d "${file}"                                              \
               |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
Signature · 1 line
2.7.4
#2 lore raw

PATCH v6 1/10 support/scripts: check-host-rpath now handles $ORIGIN/../lib

Hello,

Quoted 13 lines · Wolfgang Grandegger

The new RPath sanitization uses now the relative path "$ORIGIN/../lib" for binaries in "/usr/bin".

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/check-host-rpath +0 -0
index 020c123..e79560b 100755

support/scripts/check-host-rpath check_elf_has_rpath() { · line 61

Quoted 5 lines
support/scripts/check-host-rpath +1 -1
@@ -58,7 +58,7 @@ check_elf_has_rpath() {
         for dir in ${rpath//:/ }; do
             # Remove duplicate and trailing '/' for proper match
             dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" -o "${dir}" = "\$ORIGIN/../../usr/lib" ] && return 0
+            [ "${dir}" = "${hostdir}/usr/lib" -o "${dir}" = "\$ORIGIN/../lib" ] && return 0

I guess this needs to be revisited with the removal of $(HOST_DIR)/usr:
now everything is under $(HOST_DIR) directly.

Could you rebase your series on the latest master, in which I just merged earlier today Arnout patches removing $(HOST_DIR)/usr ?

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#3 lore raw

PATCH v6 2/10 package/patchelf: add patch for rpath sanitization under a root directory

In reply to Wolfgang Grandegger

The patch allows to use patchelf to sanitize the rpath of the buildroot libraries and binaries using the option "--make-rpath-relative <rootdir>". Recent versions of patchelf will not built on old Debian and RHEL systems due to C++11 constructs. Therefore we stick with v0.9 for the time being.

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 ...move-apparently-incorrect-usage-of-static.patch |  49 +++
 ...unction-for-splitting-a-colon-separated-s.patch |  58 +++
 ...to-make-the-rpath-relative-under-a-specif.patch | 419 +++++++++++++++++++++
 3 files changed, 526 insertions(+)
 create mode 100644 package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
 create mode 100644 package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
 create mode 100644 package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch
package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +49 -0
new file mode 100644
index 0000000..46fd123
@@ -0,0 +1,49 @@
+From a365bcb7d7025da51b33165ef7ebc7180199a05e Mon Sep 17 00:00:00 2001
+From: Eelco Dolstra <[email protected]>
+Date: Mon, 19 Sep 2016 17:31:37 +0200
+Subject: [PATCH 14/30] Remove apparently incorrect usage of "static"
+
+---
+ src/patchelf.cc | 8 +++-----
+ 1 file changed, 3 insertions(+), 5 deletions(-)
+
+Index: patchelf-0.9.old/src/patchelf.cc
+===================================================================
+--- patchelf-0.9.old.orig/src/patchelf.cc	2017-07-03 09:06:12.292069400 +0200
++++ patchelf-0.9.old/src/patchelf.cc	2017-07-03 09:20:57.000000000 +0200
+@@ -941,7 +941,6 @@
+     assert(strTabAddr == rdi(shdrDynStr.sh_addr));
+ 
+     /* Walk through the dynamic section, look for the DT_SONAME entry. */
+-    static vector<string> neededLibs;
+     dyn = (Elf_Dyn *) (contents + rdi(shdrDynamic.sh_offset));
+     Elf_Dyn * dynSoname = 0;
+     char * soname = 0;
+@@ -949,8 +948,7 @@
+         if (rdi(dyn->d_tag) == DT_SONAME) {
+             dynSoname = dyn;
+             soname = strTab + rdi(dyn->d_un.d_val);
+-        } else if (rdi(dyn->d_tag) == DT_INIT)
+-            neededLibs.push_back(string(strTab + rdi(dyn->d_un.d_val)));
++        }
+     }
+ 
+     if (op == printSoname) {
+@@ -1058,7 +1056,7 @@
+        unless you use its `--enable-new-dtag' option, in which case it
+        generates a DT_RPATH and DT_RUNPATH pointing at the same
+        string. */
+-    static vector<string> neededLibs;
++    vector<string> neededLibs;
+     dyn = (Elf_Dyn *) (contents + rdi(shdrDynamic.sh_offset));
+     Elf_Dyn * dynRPath = 0, * dynRunPath = 0;
+     char * rpath = 0;
+@@ -1091,7 +1089,7 @@
+     /* For each directory in the RPATH, check if it contains any
+        needed library. */
+     if (op == rpShrink) {
+-        static vector<bool> neededLibFound(neededLibs.size(), false);
++        vector<bool> neededLibFound(neededLibs.size(), false);
+ 
+         newRPath = "";
+
package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch new file +58 -0

58 lines not on this page · raw message

package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch new file +419 -0

419 lines not on this page · raw message

Signature · 1 line
2.7.4
#4 lore raw

PATCH v6 2/10 package/patchelf: add patch for rpath sanitization under a root directory

Hello,

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch line 4

Quoted 10 lines · Wolfgang Grandegger
package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +4 -0
new file mode 100644
index 0000000..46fd123
@@ -0,0 +1,49 @@
+From a365bcb7d7025da51b33165ef7ebc7180199a05e Mon Sep 17 00:00:00 2001
+From: Eelco Dolstra <[email protected]>
+Date: Mon, 19 Sep 2016 17:31:37 +0200
+Subject: [PATCH 14/30] Remove apparently incorrect usage of "static"

The 14/30 should be removed, just make it [PATCH]

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch line 5

Quoted 1 line
package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +1 -0
+

Please add here:

[Upstream commit: URL of the upstream commit.]
Signed-off-by ...
Quoted 6 lines
package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch new file +0 -0
new file mode 100644
index 0000000..63f3bce

package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch line 8

Quoted 8 lines
package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch new file +8 -0
@@ -0,0 +1,58 @@
+From 2e3fdc2030c75c19df6fc2924083cfad53856562 Mon Sep 17 00:00:00 2001
+From: Tuomas Tynkkynen <[email protected]>
+Date: Fri, 3 Jun 2016 23:03:51 +0300
+Subject: [PATCH 08/30] Extract a function for splitting a colon-separated
+ string
+
+We're going to need this logic in another place, so make a function of
+this.

Same comments here.

Quoted 33 lines
package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch new file +27 -0
new file mode 100644
index 0000000..4f700d9
@@ -0,0 +1,419 @@
+From af8d4a24a0ef613bdb47f0b1c3d962d59c53a4be Mon Sep 17 00:00:00 2001
+From: Wolfgang Grandegger <[email protected]>
+Date: Mon, 20 Feb 2017 16:29:24 +0100
+Subject: [PATCH] Add option to make the rpath relative under a specified root
+ directory
+
+Running "patchelf" with the option "--make-rpath-relative ROOTDIR" will
+modify or delete the RPATHDIRs according the following rules
+similar to Martin's patches [1] making the Buildroot toolchaing/SDK
+relocatable.
+
+RPATHDIR starts with "$ORIGIN":
+    The original build-system already took care of setting a relative
+    RPATH, resolve it and test if it's valid (does exist)
+
+RPATHDIR starts with ROOTDIR:
+    The original build-system added some absolute RPATH (absolute on
+    the build machine). Test if it's valid (does exist).
+
+ROOTDIR/RPATHDIR exists:
+    The original build-system already took care of setting an absolute
+    RPATH (absolute in the final rootfs), resolve it and test if it's
+    valid (does exist).
+
+RPATHDIR points somewhere else:
+    (can be anywhere: build trees, staging tree, host location,
+    non-existing location, etc.). Just discard such a path.

package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch line 37

Quoted 10 lines
package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch new file +10 -0
+
+The option "--no-standard-libs" will discard RPATHDIRs ROOTDIR/lib and
+ROOTDIR/usr/lib. Like "--shrink-rpath", RPATHDIRs are also discarded
+if the directories do not contain a library referenced by the
+DT_NEEDED fields.
+If the option "--relative-to-file" is given, the rpath will start
+with "$ORIGIN" making it relative to the ELF file, otherwise an
+absolute path relative to ROOTDIR will be used.
+
+[1] http://lists.busybox.net/pipermail/buildroot/2016-April/159422.html

If this has been accepted upstream, add the URL to the upstream commit here as well.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#5 lore raw

PATCH v6 2/10 package/patchelf: add patch for rpath sanitization under a root directory

Hello,

Am 05.07.2017 um 13:16 schrieb Thomas Petazzoni:

Quoted 17 lines

Hello,

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch line 4

Quoted 10 lines · Wolfgang Grandegger
package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +4 -0
new file mode 100644
index 0000000..46fd123
@@ -0,0 +1,49 @@
+From a365bcb7d7025da51b33165ef7ebc7180199a05e Mon Sep 17 00:00:00 2001
+From: Eelco Dolstra <[email protected]>
+Date: Mon, 19 Sep 2016 17:31:37 +0200
+Subject: [PATCH 14/30] Remove apparently incorrect usage of "static"

The 14/30 should be removed, just make it [PATCH]

Quoted 6 lines

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch line 5

Quoted 1 line
package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +1 -0
+

Please add here:

[Upstream commit: URL of the upstream commit.]
Signed-off-by ...

They (patchelf developers) don't add "Signed-off-by". Should I add mine anyway?

Quoted 57 lines
Quoted 7 lines
package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch new file +0 -0
new file mode 100644
index 0000000..63f3bce

package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch line 8

Quoted 8 lines
package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch new file +8 -0
@@ -0,0 +1,58 @@
+From 2e3fdc2030c75c19df6fc2924083cfad53856562 Mon Sep 17 00:00:00 2001
+From: Tuomas Tynkkynen <[email protected]>
+Date: Fri, 3 Jun 2016 23:03:51 +0300
+Subject: [PATCH 08/30] Extract a function for splitting a colon-separated
+ string
+
+We're going to need this logic in another place, so make a function of
+this.

Same comments here.

Quoted 38 lines
package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch new file +32 -0
new file mode 100644
index 0000000..4f700d9
@@ -0,0 +1,419 @@
+From af8d4a24a0ef613bdb47f0b1c3d962d59c53a4be Mon Sep 17 00:00:00 2001
+From: Wolfgang Grandegger <[email protected]>
+Date: Mon, 20 Feb 2017 16:29:24 +0100
+Subject: [PATCH] Add option to make the rpath relative under a specified root
+ directory
+
+Running "patchelf" with the option "--make-rpath-relative ROOTDIR" will
+modify or delete the RPATHDIRs according the following rules
+similar to Martin's patches [1] making the Buildroot toolchaing/SDK
+relocatable.
+
+RPATHDIR starts with "$ORIGIN":
+    The original build-system already took care of setting a relative
+    RPATH, resolve it and test if it's valid (does exist)
+
+RPATHDIR starts with ROOTDIR:
+    The original build-system added some absolute RPATH (absolute on
+    the build machine). Test if it's valid (does exist).
+
+ROOTDIR/RPATHDIR exists:
+    The original build-system already took care of setting an absolute
+    RPATH (absolute in the final rootfs), resolve it and test if it's
+    valid (does exist).
+
+RPATHDIR points somewhere else:
+    (can be anywhere: build trees, staging tree, host location,
+    non-existing location, etc.). Just discard such a path.
+
+The option "--no-standard-libs" will discard RPATHDIRs ROOTDIR/lib and
+ROOTDIR/usr/lib. Like "--shrink-rpath", RPATHDIRs are also discarded
+if the directories do not contain a library referenced by the
+DT_NEEDED fields.
Quoted 8 lines

package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch line 37

Quoted 5 lines
package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch new file +5 -0
+If the option "--relative-to-file" is given, the rpath will start
+with "$ORIGIN" making it relative to the ELF file, otherwise an
+absolute path relative to ROOTDIR will be used.
+
+[1] http://lists.busybox.net/pipermail/buildroot/2016-April/159422.html

If this has been accepted upstream, add the URL to the upstream commit here as well.

So far there was now response to my GitHub-Merge-Request. And it's now based on v0.9 for the known reasons.

Wolfgang.

#6 lore raw

PATCH v6 2/10 package/patchelf: add patch for rpath sanitization under a root directory

Hello,

Quoted 5 lines · Wolfgang Grandegger
Quoted 2 lines
[Upstream commit: URL of the upstream commit.]
Signed-off-by ...

They (patchelf developers) don't add "Signed-off-by". Should I add mine anyway?

Yes, add yours. It's a way of describing how the patch has been added to Buildroot.

Quoted 4 lines
Quoted 2 lines

If this has been accepted upstream, add the URL to the upstream commit here as well.

So far there was now response to my GitHub-Merge-Request.

OK.

Quoted 1 line

And it's now based on v0.9 for the known reasons.

Sure, but that doesn't prevent from indicating in this patch that it is available upstream (if it's the case).

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#7 lore raw

PATCH v6 2/10 package/patchelf: add patch for rpath sanitization under a root directory

In reply to Wolfgang Grandegger
Quoted 22 lines · Wolfgang Grandegger

Hello,

Am 05.07.2017 um 13:16 schrieb Thomas Petazzoni:

Quoted 19 lines

Hello,

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch line 4

Quoted 12 lines · Wolfgang Grandegger

diff --git a/package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch b/package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file mode 100644
index 0000000..46fd123

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +4 -0
@@ -0,0 +1,49 @@
+From a365bcb7d7025da51b33165ef7ebc7180199a05e Mon Sep 17 00:00:00 2001
+From: Eelco Dolstra <[email protected]>
+Date: Mon, 19 Sep 2016 17:31:37 +0200
+Subject: [PATCH 14/30] Remove apparently incorrect usage of "static"

The 14/30 should be removed, just make it [PATCH]

Quoted 8 lines
Quoted 6 lines

package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch line 5

Quoted 1 line
package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch new file +1 -0
+

Please add here:

[Upstream commit: URL of the upstream commit.]
Signed-off-by ...

They (patchelf developers) don't add "Signed-off-by". Should I add mine anyway?

You should add yours in the Buildroot patch, yes, as well as a reference to the
upstream commit. See e.g. package/trinity/0002-fix-build-when_USE_BPF-is-not-defined.patch

[snip]

Quoted 5 lines
Quoted 2 lines

If this has been accepted upstream, add the URL to the upstream commit here as well.

So far there was now response to my GitHub-Merge-Request. And it's now based on v0.9 for the known reasons.

Even if the patch differs from the upstream patch, you should still point to
the upstream pull request so people can check it out when they want to bump patchelf.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
author
#8 lore raw

PATCH v6 3/10 support/scripts: add fix-rpath script to sanitize the rpath

In reply to Wolfgang Grandegger

From: Samuel Martin <[email protected]>

This commit introduces the script "fix-rpath" able to scan a tree, detect ELF files, check their RPATH and fix it in a proper way. The RPATH fixup is done by the patchelf utility using the option "--make-rpath-relative <root-directory>".

Signed-off-by Samuel Martin <[email protected]>
Signed-off-by Wolfgang Grandegger <[email protected]>
---
 support/scripts/fix-rpath | 122 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 122 insertions(+)
 create mode 100755 support/scripts/fix-rpath
support/scripts/fix-rpath new file +122 -0
new file mode 100755
index 0000000..6a81193
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+# Copyright (C) 2017 Wolfgang Grandegger <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  cat <<EOF >&2
+Usage:	${0} TREE_KIND
+
+Description:
+
+    This script scans a tree and sanitize ELF files' RPATH found in there.
+
+    Sanitization behaves the same whatever the kind of the processed tree,
+    but the resulting RPATH differs. The rpath sanitization is done using
+    "patchelf --make-rpath-relazive".
+
+Arguments:
+
+    TREE_KIND	Kind of tree to be processed.
+		Allowed values: host, target, staging
+
+Environment:
+
+    PATCHELF	patchelf program to use
+		(default: HOST_DIR/usr/bin/patchelf)
+EOF
+}
+
+: ${PATCHELF:=${HOST_DIR}/usr/bin/patchelf}
+
+# ELF files should not be in these sub-directories
+HOST_EXCLUDEPATHS="/usr/share/terminfo"
+STAGING_EXCLUDEPATHS="/usr/include /usr/share/terminfo"
+
+main() {
+    local rootdir
+    local tree="${1}"
+    local find_args=( )
+    local sanitize_extra_args=( )
+
+    case "${tree}" in
+	host)
+	    rootdir="${HOST_DIR}"
+
+	    # do not process the sysroot (only contains target binaries)
+	    find_args+=( "-path" "${STAGING_DIR}" "-prune" "-o" )
+
+	    # do not process the external toolchain installation directory to
+	    # avoid breaking it.
+	    test "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" != "" && \
+		find_args+=( "-path" "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" "-prune" "-o" )
+
+	    for excludepath in ${HOST_EXCLUDEPATHS}; do
+		find_args+=( "-path" "${HOST_DIR}""${excludepath}" "-prune" "-o" );
+	    done
+
+	    # do not process the patchelf binary but a copy to work-around "file in use"
+	    find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
+	    cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
+
+	    sanitize_extra_args+=( "--relative-to-file" )
+	    ;;
+
+	staging)
+	    rootdir="${STAGING_DIR}"
+
+	    # ELF files should not be in these sub-directories
+	    for excludepath in ${STAGING_EXCLUDEPATHS}; do
+		find_args+=( "-path" "${STAGING_DIR}""${excludepath}" "-prune" "-o" )
+	    done
+
+	    sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
+	    ;;
+
+	target)
+	    rootdir="${TARGET_DIR}"
+	    sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
+	    ;;
+
+	*)
+	    usage
+	    exit 1
+	    ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file ; do
+	# check if it's an ELF file
+	if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
+	    # make files writable if necessary
+	    changed=$(chmod -c u+w "${file}")
+	    # call patchelf to sanitize the rpath
+	    ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}"
+	    # restore the original permission
+	    test "${changed}" != "" && chmod u-w "${file}"
+	fi
+    done < <(find "${rootdir}" ${find_args[@]})
+
+    # Restore patched patchelf utility
+    test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}"
+
+    # ignore errors
+    return 0
+}
+
+main ${@}
Signature · 1 line
2.7.4
author
#9 lore raw

PATCH v6 4/10 core: sanitize RPATH in staging tree at the end of target finalization

In reply to Wolfgang Grandegger
Signed-off-by Wolfgang Grandegger <[email protected]>
---
 Makefile | 3 +++
 1 file changed, 3 insertions(+)
Makefile +3 -0
index b37171f..1d07673 100644
@@ -715,6 +715,9 @@ endif
 		$(call MESSAGE,"Executing post-build script $(s)"); \
 		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
+	$(TOPDIR)/support/scripts/fix-rpath staging
+
 .PHONY: target-post-image
 target-post-image: $(TARGETS_ROOTFS) target-finalize
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
Signature · 1 line
2.7.4
author
#10 lore raw

PATCH v6 5/10 core: sanitize RPATH in target tree at the end of target finalization

In reply to Wolfgang Grandegger
Signed-off-by Wolfgang Grandegger <[email protected]>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)
Makefile +2 -0
index 1d07673..d8db3f4 100644
@@ -717,6 +717,8 @@ endif
 
 	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
 	$(TOPDIR)/support/scripts/fix-rpath staging
+	@$(call MESSAGE,"Sanitizing RPATH in target tree")
+	$(TOPDIR)/support/scripts/fix-rpath target
 
 .PHONY: target-post-image
 target-post-image: $(TARGETS_ROOTFS) target-finalize
Signature · 1 line
2.7.4
author
#11 lore raw

PATCH v6 6/10 core: sanitize RPATH in host tree at the very end of the build

In reply to Wolfgang Grandegger

We introduce the "finalize-host" target for that purpose. We need the patchelf host utility to check and fix the rpath.

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 Makefile | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
Makefile +9 -1
index d8db3f4..f5646ad 100644
@@ -550,7 +550,15 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
 .PHONY: world
-world: target-post-image
+world: target-post-image host-finalize
+
+.PHONY: host-finalize
+host-finalize:
+	@$(call MESSAGE,"Rendering the SDK relocatable")
+	$(TOPDIR)/support/scripts/fix-rpath host
+
+# We need patchelf for RPATH sanitization
+PACKAGES += host-patchelf
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
Signature · 1 line
2.7.4
author
#12 lore raw

PATCH v6 7/10 support/scripts: add relocate-sdk.sh script for SDK relocation

In reply to Wolfgang Grandegger

It will install the script "relocate-sdk.sh" in the HOST_DIR allowing to adjust the path to the SDK directory in all text files after it has been moved to a new location.

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 support/misc/relocate-sdk.sh | 47 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 support/misc/relocate-sdk.sh
support/misc/relocate-sdk.sh new file +47 -0
new file mode 100755
index 0000000..4e3801c
@@ -0,0 +1,47 @@
+#!/bin/bash
+#
+if [ "$#" -ne 0 ]; then
+    echo "Run this script to relocate the buildroot SDK at that location"
+    exit 1
+fi
+
+LOCFILE="usr/share/buildroot/sdk-location"
+FILEPATH="$(readlink -f "$0")"
+NEWPATH="$(dirname "${FILEPATH}")"
+
+cd "${NEWPATH}"
+if [ ! -r "${LOCFILE}" ]; then
+    echo "Previous location of the buildroot SDK not found!"
+    exit 1
+fi
+OLDPATH="$(cat "${LOCFILE}")"
+
+if [ "${NEWPATH}" = "${OLDPATH}" ]; then
+    echo "This buildroot SDK has already been relocated!"
+    exit 0
+fi
+
+# Check if the path substitution does work properly, e.g.
+# a tree "/a/b/c" copied into "/a/b/c/" would not be allowed.
+newpath="$(sed -e "s\\${OLDPATH}\\${NEWPATH}\\g" "${LOCFILE}")"
+if [ "${NEWPATH}" != "${newpath}" ]; then
+    echo "Something went wrong with substituting the path!"
+    echo "Please choose another location for your SDK!"
+    exit 1
+fi
+
+echo "Relocating the buildroot SDK from ${OLDPATH} to ${NEWPATH} ..."
+
+# Make sure file uses the right language
+export LC_ALL=C
+# Replace the old path with the new one in all text files
+while read -r FILE ; do
+    if file -b --mime-type "${FILE}" | grep -q '^text/' && [ "${FILE}" != "${LOCFILE}" ]
+    then
+	sed -i "s\\${OLDPATH}\\${NEWPATH}\\g" "${FILE}"
+    fi;
+done < <(grep -lr "${OLDPATH}" .)
+
+# At the very end, we update the location file to not break the
+# SDK if this script gets interruted.
+sed -i "s\\${OLDPATH}\\${NEWPATH}\\g" ${LOCFILE}
Signature · 1 line
2.7.4
author
#13 lore raw

PATCH v6 8/10 core: install relocation script and location at the end of the build

In reply to Wolfgang Grandegger

The script "relocate-sdk.sh" is installed into the top directory of the SDK (HOST_DIR) and the SDK location path is stored in the file "HOST_DIR/usr/share/buildroot/sdk-location"-

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)
Makefile +2 -0
index f5646ad..7e92428 100644
@@ -556,6 +556,8 @@ world: target-post-image host-finalize
 host-finalize:
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
+	install $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)
+	echo $(HOST_DIR) > $(HOST_DIR)/usr/share/buildroot/sdk-location
 
 # We need patchelf for RPATH sanitization
 PACKAGES += host-patchelf
Signature · 1 line
2.7.4
author
#14 lore raw

PATCH v6 9/10 external-toolchain: check if a buildroot SDK has already been relocated

In reply to Wolfgang Grandegger

The location of the buildroot SDK is stored in the file "sdk-location" in "usr/share/buildroot". If it's content does not match the current SDK location, ask the user to run the script "relocate-sdk.sh" in the top directory once. The external toolchain may be a pre-installed one in a directory that is not writeable by us. Therefore, we can't run the script directly.

Signed-off-by Wolfgang Grandegger <[email protected]>
---
 toolchain/helpers.mk                                   | 18 ++++++++++++++++++
 toolchain/toolchain-external/pkg-toolchain-external.mk |  1 +
 2 files changed, 19 insertions(+)
toolchain/helpers.mk +18 -0
index 90834f4..806968b 100644
@@ -479,3 +479,21 @@ define simplify_symlink
 	ln -sf "$${DOTS}$${REL_DEST}" "$${FULL_SRC}" ; \
 )
 endef
+
+#
+# Check if it's a buildroot toolchain and if it's already relocatable by
+# reading and testing the toolchain location file
+#
+# $1: toolchain installation directory
+#
+define check_buildroot_sdk_relocated
+( \
+	if [ -r $(1)/share/buildroot/sdk-location ]; then \
+		sdkroot=`dirname "$(1)"`; \
+		if [ "`cat $(1)/share/buildroot/sdk-location`" != "$${sdkroot}" ]; then \
+			echo "Please relocate the buildroot SDK by executing \"$${sdkroot}/relocate-sdk.sh\" once!" ; \
+			exit 1 ; \
+		fi \
+	fi \
+)
+endef
toolchain/toolchain-external/pkg-toolchain-external.mk +1 -0
index 8460e37..3dd7185 100644
@@ -545,6 +545,7 @@ endif
 # matches the configuration provided in Buildroot: ABI, C++ support,
 # kernel headers version, type of C library and all C library features.
 define $(2)_CONFIGURE_CMDS
+	$$(Q)$$(call check_buildroot_sdk_relocated,$$(TOOLCHAIN_EXTERNAL_INSTALL_DIR))
 	$$(Q)$$(call check_cross_compiler_exists,$$(TOOLCHAIN_EXTERNAL_CC))
 	$$(Q)$$(call check_unusable_toolchain,$$(TOOLCHAIN_EXTERNAL_CC))
 	$$(Q)SYSROOT_DIR="$$(call toolchain_find_sysroot,$$(TOOLCHAIN_EXTERNAL_CC))" ; \
Signature · 1 line
2.7.4
author
#15 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

In reply to Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

CC Julien Corjon <[email protected]>
CC Peter Seiderer <[email protected]>
Signed-off-by Wolfgang Grandegger <[email protected]>
---
 package/qt5/qt5base/qt.conf.in | 6 ++++++
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 package/qt5/qt5base/qt.conf.in
package/qt5/qt5base/qt.conf.in new file +6 -0
new file mode 100644
index 0000000..48e4b92
@@ -0,0 +1,6 @@
+[Paths]
+Prefix=@@HOST_DIR@@/usr
+Sysroot=@@STAGING_DIR@@
+Headers=/usr/include/qt5
+Plugins=/usr/lib/qt/plugins
+Examples=/usr/lib/qt/examples
package/qt5/qt5base/qt5base.mk +8 -0
index 5fe8bb8..883850b 100644
@@ -258,9 +258,17 @@ define QT5BASE_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
+# The file "qt.conf" can be used to override the hard-coded paths that are
+# compiled into the Qt library. We need it to make "qmake" relocatable.
+define QT5BASE_INSTALL_QT_CONF
+	sed -e "s\\@@HOST_DIR@@\\$(HOST_DIR)\\" -e "s\\@@STAGING_DIR@@\\$(STAGING_DIR)\\" \
+		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/usr/bin/qt.conf
+endef
+
 define QT5BASE_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
 	$(QT5_LA_PRL_FILES_FIXUP)
+	$(QT5BASE_INSTALL_QT_CONF)
 endef
 
 define QT5BASE_INSTALL_TARGET_LIBS
Signature · 1 line
2.7.4
#16 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hi Wolfgang,

Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Can you have a look?

Regards,
Arnout

Quoted 45 lines
CC Julien Corjon <[email protected]>
CC Peter Seiderer <[email protected]>
Signed-off-by Wolfgang Grandegger <[email protected]>
---
 package/qt5/qt5base/qt.conf.in | 6 ++++++
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 package/qt5/qt5base/qt.conf.in
package/qt5/qt5base/qt.conf.in new file +6 -0
new file mode 100644
index 0000000..48e4b92
@@ -0,0 +1,6 @@
+[Paths]
+Prefix=@@HOST_DIR@@/usr
+Sysroot=@@STAGING_DIR@@
+Headers=/usr/include/qt5
+Plugins=/usr/lib/qt/plugins
+Examples=/usr/lib/qt/examples
package/qt5/qt5base/qt5base.mk +8 -0
index 5fe8bb8..883850b 100644
@@ -258,9 +258,17 @@ define QT5BASE_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
+# The file "qt.conf" can be used to override the hard-coded paths that are
+# compiled into the Qt library. We need it to make "qmake" relocatable.
+define QT5BASE_INSTALL_QT_CONF
+	sed -e "s\\@@HOST_DIR@@\\$(HOST_DIR)\\" -e "s\\@@STAGING_DIR@@\\$(STAGING_DIR)\\" \
+		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/usr/bin/qt.conf
+endef
+
 define QT5BASE_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
 	$(QT5_LA_PRL_FILES_FIXUP)
+	$(QT5BASE_INSTALL_QT_CONF)
 endef
 
 define QT5BASE_INSTALL_TARGET_LIBS
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
author
#17 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hello Arnout,

Am 05.07.2017 um 10:59 schrieb Arnout Vandecappelle:

Quoted 6 lines

Hi Wolfgang,

Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

Quoted 8 lines

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Can you have a look?

I know the problem! The patched used an old version (from an old series) of "qt.conf.in"..., sorry!

Wolfgang.

Quoted 50 lines

Regards,
Arnout

Quoted 45 lines
CC Julien Corjon <[email protected]>
CC Peter Seiderer <[email protected]>
Signed-off-by Wolfgang Grandegger <[email protected]>
---
  package/qt5/qt5base/qt.conf.in | 6 ++++++
  package/qt5/qt5base/qt5base.mk | 8 ++++++++
  2 files changed, 14 insertions(+)

create mode 100644 package/qt5/qt5base/qt.conf.in

package/qt5/qt5base/qt.conf.in new file +6 -0
new file mode 100644
index 0000000..48e4b92
@@ -0,0 +1,6 @@
+[Paths]
+Prefix=@@HOST_DIR@@/usr
+Sysroot=@@STAGING_DIR@@
+Headers=/usr/include/qt5
+Plugins=/usr/lib/qt/plugins
+Examples=/usr/lib/qt/examples
package/qt5/qt5base/qt5base.mk +8 -0
index 5fe8bb8..883850b 100644
@@ -258,9 +258,17 @@ define QT5BASE_BUILD_CMDS
  	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
  endef
 
+# The file "qt.conf" can be used to override the hard-coded paths that are
+# compiled into the Qt library. We need it to make "qmake" relocatable.
+define QT5BASE_INSTALL_QT_CONF
+	sed -e "s\\@@HOST_DIR@@\\$(HOST_DIR)\\" -e "s\\@@STAGING_DIR@@\\$(STAGING_DIR)\\" \
+		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/usr/bin/qt.conf
+endef
+
  define QT5BASE_INSTALL_STAGING_CMDS
  	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
  	$(QT5_LA_PRL_FILES_FIXUP)
+	$(QT5BASE_INSTALL_QT_CONF)
  endef
 
  define QT5BASE_INSTALL_TARGET_LIBS
author
#18 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Am 05.07.2017 um 11:30 schrieb Wolfgang Grandegger:

Quoted 18 lines

Hello Arnout,

Am 05.07.2017 um 10:59 schrieb Arnout Vandecappelle:

Quoted 15 lines

Hi Wolfgang,

Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Quoted 4 lines
Quoted 1 line

Can you have a look?

I know the problem! The patched used an old version (from an old series) of "qt.conf.in"..., sorry!

The patch is reverted now. So just for the records, the patch below fixes the problem. I'm going to provide a proper patch with my next series. This also shows, that if "qt.conf" is in place, it's active and will always overwrite the pathes compiled into qmake... which will break the idea of "make sdk" somehow.

author
#19 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hello,

Am 05.07.2017 um 12:28 schrieb Wolfgang Grandegger:

Quoted 21 lines

Am 05.07.2017 um 11:30 schrieb Wolfgang Grandegger:

Quoted 20 lines

Hello Arnout,

Am 05.07.2017 um 10:59 schrieb Arnout Vandecappelle:

Quoted 16 lines

Hi Wolfgang,

Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Can you have a look?

Quoted 8 lines
Quoted 2 lines

I know the problem! The patched used an old version (from an old series) of "qt.conf.in"..., sorry!

The patch is reverted now. So just for the records, the patch below fixes the problem. I'm going to provide a proper patch with my next series. This also shows, that if "qt.conf" is in place, it's active and will always overwrite the pathes compiled into qmake... which will break the idea of "make sdk" somehow.

What about making the relocation stuff configurable as shown in the patch below. The advantage is that it could also control the generation of the "qt.conf", for example.

Wolfgang.

#20 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hello,

Quoted 3 lines · Wolfgang Grandegger

What about making the relocation stuff configurable as shown in the patch below. The advantage is that it could also control the generation of the "qt.conf", for example.

We don't want to make it configurable, through a Config.in option. However, we would like to have a separate make target that prepares the SDK, creates the tarball, etc.

Basically, a normal "make" would not run all the rpath sanitization logic, absolute path replacement, etc. There should be a separate "make sdk" target that does this.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#21 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Am 05.07.2017 um 16:13 schrieb Thomas Petazzoni:

Quoted 8 lines

Hello,

Quoted 3 lines · Wolfgang Grandegger

What about making the relocation stuff configurable as shown in the patch below. The advantage is that it could also control the generation of the "qt.conf", for example.

Quoted 7 lines

We don't want to make it configurable, through a Config.in option. However, we would like to have a separate make target that prepares the SDK, creates the tarball, etc.

Basically, a normal "make" would not run all the rpath sanitization logic, absolute path replacement, etc. There should be a separate "make sdk" target that does this.

Yes I understand, but how do you want to handle static things like the creation of "qt.conf" required for the relocatable SDK?

Wolfgang.

#22 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hello,

Quoted 10 lines · Wolfgang Grandegger
Quoted 7 lines

We don't want to make it configurable, through a Config.in option. However, we would like to have a separate make target that prepares the SDK, creates the tarball, etc.

Basically, a normal "make" would not run all the rpath sanitization logic, absolute path replacement, etc. There should be a separate "make sdk" target that does this.

Yes I understand, but how do you want to handle static things like the creation of "qt.conf" required for the relocatable SDK?

In order to make sure that the SDK situation is as close to a normal Buildroot usage, I think the qt.conf file should always be created and used by qmake. So basically, the Qt package in Buildroot should create and install this file.

Am I missing something here?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#23 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hello,

Am 05.07.2017 um 16:48 schrieb Thomas Petazzoni:

Quoted 15 lines

Hello,

Quoted 10 lines · Wolfgang Grandegger
Quoted 7 lines

We don't want to make it configurable, through a Config.in option. However, we would like to have a separate make target that prepares the SDK, creates the tarball, etc.

Basically, a normal "make" would not run all the rpath sanitization logic, absolute path replacement, etc. There should be a separate "make sdk" target that does this.

Yes I understand, but how do you want to handle static things like the creation of "qt.conf" required for the relocatable SDK?

Quoted 6 lines

In order to make sure that the SDK situation is as close to a normal Buildroot usage, I think the qt.conf file should always be created and used by qmake. So basically, the Qt package in Buildroot should create and install this file.

Am I missing something here?

It works here (with my usecase)! But the qmake internals are not really transparent (not documented) and may break if new path variables are added etc.(by Qt). But well, then we just need to fix it. Let see how good qt.conf works...

Wolfgang.

#24 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

In reply to Wolfgang Grandegger
Quoted 15 lines · Wolfgang Grandegger

Am 05.07.2017 um 16:13 schrieb Thomas Petazzoni:

Quoted 12 lines

Hello,

Quoted 3 lines · Wolfgang Grandegger

What about making the relocation stuff configurable as shown in the patch below. The advantage is that it could also control the generation of the "qt.conf", for example.

We don't want to make it configurable, through a Config.in option. However, we would like to have a separate make target that prepares the SDK, creates the tarball, etc.

Quoted 6 lines
Quoted 3 lines

Basically, a normal "make" would not run all the rpath sanitization logic, absolute path replacement, etc. There should be a separate "make sdk" target that does this.

Yes I understand, but how do you want to handle static things like the creation of "qt.conf" required for the relocatable SDK?

It should still work if you have the relocatable SDK option selected. So
basically we want the option to always be selected and things to still work.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#25 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

In reply to Wolfgang Grandegger
Quoted 33 lines · Wolfgang Grandegger

Hello,

Am 05.07.2017 um 12:28 schrieb Wolfgang Grandegger:

Quoted 21 lines

Am 05.07.2017 um 11:30 schrieb Wolfgang Grandegger:

Quoted 20 lines

Hello Arnout,

Am 05.07.2017 um 10:59 schrieb Arnout Vandecappelle:

Quoted 16 lines

Hi Wolfgang,

Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Can you have a look?

Quoted 8 lines
Quoted 2 lines

I know the problem! The patched used an old version (from an old series) of "qt.conf.in"..., sorry!

The patch is reverted now. So just for the records, the patch below fixes the problem. I'm going to provide a proper patch with my next series. This also shows, that if "qt.conf" is in place, it's active and will always overwrite the pathes compiled into qmake... which will break the idea of "make sdk" somehow.

Quoted 3 lines

What about making the relocation stuff configurable as shown in the patch below. The advantage is that it could also control the generation of the "qt.conf", for example.

I don't really think this is worth it. Much simpler to always have the
possibility to sdk-ize it by running 'make sdk'.

Quoted 28 lines

Wolfgang.

Quoted 1 line

From 43ef9954f1b5b92e97b5774081a263f2ce068c5f Mon Sep 17 00:00:00 2001

From: Wolfgang Grandegger <[email protected]>
Date: Wed, 5 Jul 2017 15:36:21 +0200
Subject: [RFC PATCH] core: add config option to make SDK (host dir) relocatable

---
 Config.in | 18 ++++++++++++++++++
 Makefile  |  5 +++++
 2 files changed, 23 insertions(+)
Config.in +7 -0
index 72ceadf..57f899c 100644
@@ -721,6 +721,24 @@ config BR2_REPRODUCIBLE
 	  This is labeled as an experimental feature, as not all
 	  packages behave properly to ensure reproducibility.
 
+config BR2_RELOCATABLE
+	bool "Make a relocatable SDK (experimental)"
+	help
+	  This option will build a relocatable SDK (host dir). That
+	  is the directory to store all the binary files that are
+	  built for the host but also the cross compilation toolchain
+	  when building the internal buildroot toolchain. It also

Config.in config BR2_REPRODUCIBLE · line 740

Quoted 10 lines
Config.in +10 -0
+	  includes the "sysroot" tree required for cross compilation.
+	  The relocatable host directory can then be moved to a new
+	  location as shown below:
+
+	    $ cd <buildroot-build-directory>
+	    $ cp -r host <the-new-directory>/sdk
+	    $ <the-new-directory>/sdk/relocate-sdk.sh
+
+	  After executing the script "relocate-sdk.sh" once, it can
+	  be used for external builds.

This text is very useful but should be in the manual instead.

Quoted 17 lines
Config.in +1 -0
+
 endmenu
 
 endmenu
Makefile +1 -0
index 57191d1..962f80a 100644
@@ -460,6 +460,7 @@ endif
 # these locations, so export them so it is easier to use
 export BR2_CONFIG
 export BR2_REPRODUCIBLE
+export BR2_RELOCATABLE
 export TARGET_DIR
 export STAGING_DIR
 export HOST_DIR

Makefile world: target-post-image host-finalize · line 557

Quoted 3 lines
Makefile +0 -0
@@ -554,10 +555,12 @@ world: target-post-image host-finalize
 
 .PHONY: host-finalize
 host-finalize:

Make this 'sdk' instead of host-finalize, and add it to help: as well.

Quoted 10 lines
Makefile +2 -0
+ifeq ($(BR2_RELOCATABLE),y)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	install $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+endif
 
 # We need patchelf for RPATH sanitization
 PACKAGES += host-patchelf

Makefile endif · line 740

Quoted 8 lines
Makefile +1 -0
@@ -730,10 +733,12 @@ endif
 		$(call MESSAGE,"Executing post-build script $(s)"); \
 		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+ifeq ($(BR2_RELOCATABLE),y)
 	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	@$(call MESSAGE,"Sanitizing RPATH in target tree")
 	$(TOPDIR)/support/scripts/fix-rpath target

This has nothing to do with making it relocatable, it is always necessary to
sanitize the rpaths in target. Well, it's not really necessary, things work as they are now, but it would be an improvement.

Regards,
Arnout

Quoted 5 lines
Makefile +1 -0
+endif
 
 .PHONY: target-post-image
 target-post-image: $(TARGETS_ROOTFS) target-finalize
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#26 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

In reply to Arnout Vandecappelle

Hello,

Quoted 9 lines · Arnout Vandecappelle
Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Due to the numerous build failures caused by this patch, I reverted the commit.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#27 lore raw

PATCH v6 10/10 package/qt5base: provide "qt.conf" to make "qmake" relocatable

Hello Thomas,

Am 05.07.2017 um 11:30 schrieb Thomas Petazzoni:

Quoted 8 lines

Hello,

Quoted 4 lines · Arnout Vandecappelle
Quoted 2 lines · Wolfgang Grandegger

The file "qt.conf" can be used to override the hard-coded paths that are compiled into the Qt library. We need it to make "qmake" relocatable.

Quoted 8 lines
Quoted 5 lines

All qt5 packages are failing because of this, e.g.
http://autobuild.buildroot.net/results/33e/33eb7b7ef7d232bed2432d1d44fd80a4fc8c4e2d

qt5serialport: installs files in /accts/mlweber1/instance-2/output/host/usr/x86_64-buildroot-linux-uclibc/sysroot//accts/mlweber1/instance-2/output

Due to the numerous build failures caused by this patch, I reverted the commit.

I will resend then a proper patch for QT5 with the next release of my "Make SDK relocatable" patch series.

Wolfgang

#28 lore raw

v6 cover Make the SDK relocatable

In reply to Wolfgang Grandegger

Hi Wolfgang,

I've already marked patch 7 and 10 as superseded in patchwork because they were
already applied by Thomas. Can you check if there is a significant difference with what was applied?

Quoted 19 lines · Wolfgang Grandegger

Hello,

this is v6 of my patch series to make the buildroot SDK (HOST_DIR) relocatable. It sanitizes the RPATH of all ELF files in the "target", "staging" and "host" tree using "patchelf --make-rpath-relative". We now use patchelf v0.9 to still support old Debian and RHEL systems.

v5 did RPATH sanitization per package after package installation into the host, staging or target tree using GLOBAL_INSTRUMENTATION_HOOKS. This approach got more and more complex and inefficient. Therefore it was abandoned in favor of global sanititation at the end of the host, staging and target build (see changes since v5).

Furthermore this patch creates the script "relocate-sdk.sh" in the top directory of the "host" tree allowing to relocate the SDK after it has been moved to a new location. It replaces the old path with the new one in all text files identified by "file --mime-type". The location is stored in "usr/share/buildroot/sdk-location".

Quoted 4 lines

Unfortunately, "qmake" uses hard-coded pathes compiled into the QT5 libraries. To overcome this problem, "qt5pase" now creates "qt.conf".

Hope I have not too much pending issues.

We all do :-) But this is really complicated infrastructural work, so it is to
be expected that it takes time. Just 7 revisions would already be pretty good!

Quoted 8 lines

Wolfgang.

Things not yet addressed:

- "make toolchain" creates a toolchain tree which still has references to the build system (in ELF and text files).

Quoted 6 lines

Changes since v5:
- switch back to v4
- patchelf patches are now based on v0.9
- patchelf now calculates minimal relative path to the ELF file (and not to the rootdir as before).
- patchelf: neededLibFound is now passed to libFoundInRPath

If any more issues are found in the rpath fixup, they can be handled in
follow-up patches.

Regards,
Arnout

Quoted 87 lines

- Makefile: the "host-finalize" target has been added for SDK relocation
- fix-rpath now uses variables to define the list of dirs to be pruned
- fix-rpath: the staging tree is sanitized like the target tree
- various minor fixes (typos)

Changes since v4:

- RPATH sanitation is now done per package and installation step in "pkg-generic.mk".
- After the installation step, the list of the installed files is stored in "<package-build-dir>/.br_[host|staging|target]_filelist".
- The GLOBAL_INSTRUMENTATION_HOOKS "step_sanitize_rpath" then calls "fix-rpath" to do the sanitation.
- DEPENDENCIES_HOST_PREREQ += host-patchelf is set early in the Makefile as we need it for RPATH sanitation. As soon it's available sanitation can start (currently missing some packages).
- The patchelf "file busy" issue is now worked-around differently.

Changes since v3:

- The patchelf patch implementing " --make-rpath-relative" now supports the option "--relative-to-file" instructing to use "$ORIGIN" in RPATHs. Otherwise an absolute path relative to the root directory will be used.
- The staging tree is now sanitized as well using the options "--relative-to-file" and "--no-standard-libs".
- For the "target" tree, relative RPATHs do not use "$ORIGIN" any longer. An absolute path relative to the root directory is used instead.

Changes since v2:

- provide "qt.conf" to make "qmake" relocatable
- sed now uses the separator "\" to substitute the directory path. It's one of the few characters not allowed in file names. To avoid interpreting it as escape character, the "read -r" is used.
- The paranoia substituion check is done before doing the real substituion.

Changes since v1:

- The name SDK has been chosen for the relocatabed "HOST_DIR" (instead of toolchanin).
- The patchelf version bump and patching are now done by 2 patches
- No more helper functions are used in the Makefile to call "fix-rpath" but added directly.
- The staging tree is not touched any more... until we have a good reason to do so.
- The sanitation is now performed by an optimized "fix-rpath" script.
- The relocate-sdk script is now copied for support/misc to the top directory of the host tree.

Samuel Martin (1):
support/scripts: add fix-rpath script to sanitize the rpath

Wolfgang Grandegger (9):
support/scripts: check-host-rpath now handles $ORIGIN/../lib package/patchelf: add patch for rpath sanitization under a root
directory
core: sanitize RPATH in staging tree at the end of target finalization
core: sanitize RPATH in target tree at the end of target finalization
core: sanitize RPATH in host tree at the very end of the build support/scripts: add relocate-sdk.sh script for SDK relocation
core: install relocation script and location at the end of the build
external-toolchain: check if a buildroot SDK has already been
relocated
package/qt5base: provide "qt.conf" to make "qmake" relocatable

 Makefile                                           |  17 +-
 ...move-apparently-incorrect-usage-of-static.patch |  49 +++
 ...unction-for-splitting-a-colon-separated-s.patch |  58 +++
 ...to-make-the-rpath-relative-under-a-specif.patch | 419 +++++++++++++++++++++
 package/qt5/qt5base/qt.conf.in                     |   6 +
 package/qt5/qt5base/qt5base.mk                     |   8 +
 support/misc/relocate-sdk.sh                       |  47 +++
 support/scripts/check-host-rpath                   |   2 +-
 support/scripts/fix-rpath                          | 122 ++++++
 toolchain/helpers.mk                               |  18 +
 .../toolchain-external/pkg-toolchain-external.mk   |   1 +
 11 files changed, 745 insertions(+), 2 deletions(-)
 create mode 100644 package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
 create mode 100644 package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
 create mode 100644 package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch
 create mode 100644 package/qt5/qt5base/qt.conf.in
 create mode 100755 support/misc/relocate-sdk.sh
 create mode 100755 support/scripts/fix-rpath
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
author
#29 lore raw

v6 cover Make the SDK relocatable

Hello Arnout,

Am 04.07.2017 um 19:55 schrieb Arnout Vandecappelle:

Quoted 26 lines

Hi Wolfgang,

I've already marked patch 7 and 10 as superseded in patchwork because they were
already applied by Thomas. Can you check if there is a significant difference with what was applied?

Quoted 19 lines · Wolfgang Grandegger

Hello,

this is v6 of my patch series to make the buildroot SDK (HOST_DIR) relocatable. It sanitizes the RPATH of all ELF files in the "target", "staging" and "host" tree using "patchelf --make-rpath-relative". We now use patchelf v0.9 to still support old Debian and RHEL systems.

v5 did RPATH sanitization per package after package installation into the host, staging or target tree using GLOBAL_INSTRUMENTATION_HOOKS. This approach got more and more complex and inefficient. Therefore it was abandoned in favor of global sanititation at the end of the host, staging and target build (see changes since v5).

Furthermore this patch creates the script "relocate-sdk.sh" in the top directory of the "host" tree allowing to relocate the SDK after it has been moved to a new location. It replaces the old path with the new one in all text files identified by "file --mime-type". The location is stored in "usr/share/buildroot/sdk-location".

Quoted 7 lines
Quoted 4 lines

Unfortunately, "qmake" uses hard-coded pathes compiled into the QT5 libraries. To overcome this problem, "qt5pase" now creates "qt.conf".

Hope I have not too much pending issues.

We all do :-) But this is really complicated infrastructural work, so it is to
be expected that it takes time. Just 7 revisions would already be pretty good!

It's going forward... that's really good news!

Quoted 15 lines
Quoted 8 lines

Wolfgang.

Things not yet addressed:

- "make toolchain" creates a toolchain tree which still has references to the build system (in ELF and text files).

Quoted 6 lines

Changes since v5:
- switch back to v4
- patchelf patches are now based on v0.9
- patchelf now calculates minimal relative path to the ELF file (and not to the rootdir as before).
- patchelf: neededLibFound is now passed to libFoundInRPath

Quoted 2 lines

If any more issues are found in the rpath fixup, they can be handled in
follow-up patches.

Great!

BTW, I have another little patch for patchelf speeding up the open path significantly. Currently, the complete file will be read in regardless if it's an ELF file or not. Then it starts checking size, magic number, etc. Now it just reads the necessary data (ELF Header) for the checks and only if it's an ELF file, the rest of the data will be read as well. More soon...

Wolfgang.