Skip to contentSkip to conversation

Relocatable SDK/Build machine leaks: RPATH fixing

PATCH Superseded Samuel Martin 3 participants 24 messages Apr 17, 2016 – Apr 19, 2016 Kbuild

v8 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v8 cover Relocatable SDK/Build machine leaks: RPATH fixing

Hi all,

Here is yet another round of the series aiming improving the relocatability of the SDK built by Buildroot.

This time, I reduce the series to those related to RPATH (hope a shorter series will be easier to review...). Besides, the noticeable changes are:
- the refactoring of the O definition, as it needs to point to the absolute canonical path for the whole build, so that fix-rpath can do its job, without missing anything;
- and the refactoring of the check-host-leaks script.

For further details, check the per-commit changelog in each commit log.

Patches 1 and 2:
Fix the O variable definition.

Patches 3 to 5:
Add the fix-rpath script, then run it on the Buildroot host, target and staging trees.

Patch 6:
Clean-up speex package WRT its RPATH hook.

Patch 7:
Add a post-install hook to the toolchain package to render it relocatable.

Patch 8 to 10:
Update and fix the check-host-rpath to re-use the shell helpers.

Patches 11 to 13:
Add means to track build machine leaks into the host, target and staging trees. These patches will allow to identify what remains to be fixed WRT build machine leaks.

Patches 14:
Update documentation about how to enable/adjust log level on script using the shell modules introduced in patch 8/18.

Regards,
Samuel

Samuel Martin (14):
core: split variables definition related to in/out-of-tree build from
O itself
core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
path
support/scripts: add fix-rpath script + a bunch of helpers
core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS
core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
TARGET_FINALIZE_HOOKS
package/speex: remove no longer needed hook
toolchain: add post-install hooks making the toolchain relocatable support/scripts: update check-host-rpath to use the shell helpers support/scripts/check-host-rpath: silent find command support/scripts/check-host-rpath: also check HOST_DIR/{bin,sbin} support/scripts: add check-host-leaks script + all needed helpers
core: add check-leaks-in-{target,host,staging} targets support/scripts/check-host-leaks: add option to classify leaks docs/manual: document how to debug shell script

 Makefile                            | 128 ++++++++++++---
 docs/manual/debugging-buildroot.txt |  17 ++
 package/pkg-utils.mk                |   5 +
 package/speex/speex.mk              |   5 -
 support/scripts/check-host-leaks    | 181 +++++++++++++++++++++
 support/scripts/check-host-rpath    |  73 ++++-----
 support/scripts/fix-rpath           | 116 ++++++++++++++
 support/scripts/shell/log.sh        |  61 +++++++
 support/scripts/shell/patchelf.sh   | 178 +++++++++++++++++++++
 support/scripts/shell/readelf.sh    | 306 ++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh        |  75 +++++++++
 support/scripts/shell/source.sh     |  77 +++++++++
 support/scripts/shell/utils.sh      |  74 +++++++++
 toolchain/toolchain/toolchain.mk    |   5 +
 14 files changed, 1230 insertions(+), 71 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

--
2.8.0

#1 lore raw

PATCH v8 1/14 core: split variables definition related to in/out-of-tree build from O itself

This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from the presence of the O variable in the command line.

Now, the condition used to set these variables is the value of O itself.

This change is a preparatory work since the O definition will need to be moved around when we will make Buildroot run with absolute canonical paths for both its root directory and the output location. This will be addressed in a follow-up patch.

---
changes v7->v8:
- new patch

---
 Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Makefile +11 -3
index 0e4beb2..3d86c9b 100644
@@ -110,10 +110,11 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
 ifneq ("$(origin O)", "command line")
 O := output
-CONFIG_DIR := $(TOPDIR)
-NEED_WRAPPER =
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
@@ -126,9 +127,16 @@ MAKEOVERRIDES =
 # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
 # To really make O go away, we have to override it.
 override O := $(O)
-CONFIG_DIR := $(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+endif
+
+# Set variables related to in-tree or out-of-tree build.
+ifeq ($(O),output)
+CONFIG_DIR := $(TOPDIR)
+NEED_WRAPPER =
+else
+CONFIG_DIR := $(O)
 NEED_WRAPPER = y
 endif
 
Signature · 1 line
2.8.0
#2 lore raw

PATCH v8 2/14 core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

In reply to Samuel Martin

When $(CURDIR) or $(O) contain symlinks in their path, they can be resolved differently, depending on each package build-system (whether it uses the given paths or get the absolute canonical ones).

This will make easier tracking down host machine paths leaking into the host, target or staging trees, the CURDIR and O variables are set to their absolute canonical paths.

In order to recall the toplevel makefile with absolute canonical paths for $(CURDIR) and $(O), we need to:
1- Move the O variable definition out of any if-block; so they are
always available.
2- Compute the absolute canonical paths for $(CURDIR) and $(O) that will
be passed to the sub-make. This is achieved using the 'realpath' make primitive. However, some care must be taken when manipulating O:
- the out-of-tree makefile wrapper happens a trailing "/.", we need to strip this part away to not break the comparison driving the sub-make call;
- according to [1,2], realpath returns an empty string in case of non-existing entry. So, to avoid passing an empty O= variable to sub-make, it is necessary to define the output directory and create it prior to call realpath on it (because on the first invocation, $(O) usually does not yet exists), hence the trick doing the mkdir right before calling realpath.
3- Update EXTRAMAKEARGS with the absobulte canonical $(O) and use it
when call recalling the toplevel makefile with umask and paths correctly set.
4- Lastly, update the condition for setting the CONFIG_DIR and
NEED_WRAPPER variables.

Notes:
* This change takes care of the makefile wrapper installed in $(O) to avoid unneeded make recursion.
* Now, only $(O) is strip away from MAKEOVERRIDES whatever the build is done in- or out-of-tree (i.o.w. without or with O set in the command line); wheares is the previous implementation, all variables set on the command line were stripped away only in the case of out-of-tree build.

[1] https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
[2] http://man7.org/linux/man-pages/man3/realpath.3.html

Cc Matthew Weber <[email protected]>
Cc "Yann E. MORIN" <[email protected]>
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
changes v7->v8:
- keep @ at the beginning of the command (Yann)
- make ifneq condition easier to read/parsed (Yann)
- fix O definition before re-entering make (Reported by Matthew)
- use EXTRAMAKEARGS when re-entering make
- update the condition for CONFIG_DIR and NEED_WRAPPER

changes v6->v7:
- none

changes v5->v6:
- new patch

---
 Makefile | 86 +++++++++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 58 insertions(+), 28 deletions(-)
Makefile +58 -28
index 3d86c9b..a05b9e1 100644
@@ -24,18 +24,68 @@
 # You shouldn't need to mess with anything beyond this point...
 #--------------------------------------------------------------
 
-# Trick for always running with a fixed umask
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
+ifneq ("$(origin O)", "command line")
+O := $(CURDIR)/output
+else
+# Other packages might also support Linux-style out of tree builds
+# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
+# forwards command line variable definitions those packages get very
+# confused. Fix this by telling make to not do so, only for O=..., but
+# keep all others (such as BR2_EXTERNAL, BR2_DL_DIR, etc).
+MAKEOVERRIDES := $(filter-out O=%,$(MAKEOVERRIDES))
+# Strangely enough O is still passed to submakes with MAKEOVERRIDES
+# (with make 3.81 atleast), the only thing that changes is the output
+# of the origin function (command line -> environment).
+# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
+# To really make O go away, we have to override it.
+override O := $(O)
+endif
+
+# Check if the current Buildroot execution meets all the pre-requisites.
+# If they are not met, Buildroot will actually do its job in a sub-make meeting
+# its pre-requisites, which are:
+#  1- Permissive enough umask:
+#       Wrong or too restrictive umask will prevent Buildroot and packages from
+#       creating files and directories.
+#  2- Absolute canonical CWD (i.e. $(CURDIR)):
+#       Otherwise, some packages will use CWD as-is, others will compute its
+#       absolute canonical path. This makes harder tracking and fixing host
+#       machine path leaks.
+#  3- Absolute canonical output location (i.e. $(O)):
+#       For the same reason as the one for CWD.
+
+# Current state:
+CUR_UMASK := $(shell umask)
+# Remove the trailing '/.' from $(O) as it can be added by the makefile wrapper
+# installed in the $(O) directory.
+O := $(patsubst %/.,%,$(O))
+
+# Buildroot requirements:
 UMASK = 0022
-ifneq ($(shell umask),$(UMASK))
+REAL_CURDIR := $(realpath $(CURDIR))
+# realpath needs the entry to exists, otherwise an empty string is returned.
+REAL_O := $(shell mkdir -p $(O) >/dev/null 2>&1)$(realpath $(O))
+
+# Make sure O= is passed (with its absolute canonical path) everywhere the
+# toplevel makefile is called back.
+EXTRAMAKEARGS := O=$(REAL_O)
+
+# Check Buildroot execution pre-requisites here.
+ifneq ($(CUR_UMASK):$(CURDIR):$(O),$(UMASK):$(REAL_CURDIR):$(REAL_O))
 .PHONY: _all $(MAKECMDGOALS)
 
 $(MAKECMDGOALS): _all
 	@:
 
 _all:
-	@umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
+	@umask $(UMASK) && \
+		$(MAKE) -C $(REAL_CURDIR) --no-print-directory \
+			$(MAKECMDGOALS) $(EXTRAMAKEARGS)
 
-else # umask
+else # umask / $(CURDIR) / $(O)
 
 # This is our default rule, so must come first
 all:
@@ -110,30 +160,10 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
-# Set O variable if not already done on the command line;
-# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
-# build by preventing it from being forwarded to sub-make calls.
-ifneq ("$(origin O)", "command line")
-O := output
-else
-# other packages might also support Linux-style out of tree builds
-# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
-# forwards command line variable definitions those packages get very
-# confused. Fix this by telling make to not do so
-MAKEOVERRIDES =
-# strangely enough O is still passed to submakes with MAKEOVERRIDES
-# (with make 3.81 atleast), the only thing that changes is the output
-# of the origin function (command line -> environment).
-# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
-# To really make O go away, we have to override it.
-override O := $(O)
-# we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
-endif
-
 # Set variables related to in-tree or out-of-tree build.
-ifeq ($(O),output)
-CONFIG_DIR := $(TOPDIR)
+# Here, both $(O) and $(CURDIR) are absolute canonical paths.
+ifeq ($(O),$(CURDIR)/output)
+CONFIG_DIR := $(CURDIR)
 NEED_WRAPPER =
 else
 CONFIG_DIR := $(O)
@@ -1017,4 +1047,4 @@ include docs/manual/manual.mk
 
 .PHONY: $(noconfig_targets)
 
-endif #umask
+endif #umask / $(CURDIR) / $(O)
Signature · 1 line
2.8.0
#3 lore raw

PATCH v8 3/14 support/scripts: add fix-rpath script + a bunch of helpers

In reply to Samuel Martin

This commit introduces a fix-rpath shell-script able to scan a tree, detect ELF files, check their RPATH and fix it in a proper way.

Along to the fix-rpath script, it also adds a bunch of shell helper functions grouped into modules. This will help writing scripts handling RPATH and other things, while allowing to share and reuse these functions between scripts.

These helpers are namespaced within the filename of the module in which they are gathered.

This change adds 6 modules:
- source.sh: provides simple helper to easily source another module, taking care of not sourcing again an already-sourced one;
- log.sh: provides logging helpers;
- utils.sh: provides simple functions to filter ELF files in a list;
- readelf.sh: provides functions retrieving ELF details from a file;
- patchelf.sh: provides function updating ELF files;
- sdk.sh: provides RPATH computation functions.

These 6 modules are used by the fix-rpath script. Follow-up patches will make some scripts leveraging these modules.

---
changes v7->v8:
- slightly rework log.error (return non-0 instead of exiting; exit decision is deferred to the caller)

changes v6->v7 (includes changes started during BR dev. days):
- log module:
- debug env. var. renamed: DEBUG -> SHELL_DEBUG
- trace now prints the message level
- patchelf module:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- set_rpath now handles ELF files with no .dynamic section
- cleanup and minor fix in sanitize_rpath
- readelf module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_sections and has_section functions (needed in sanitize_rpath)
- sdk modules:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- minor functions doc. fixes
- source module:
- move "Could not load module" error message in load_module (Arnout)
- utils module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_has function
- fix-rpath:
- runs with "set -e" (exit on error)
- improve help text
- slight refactoring because of modules changes

changes v5->v6:
- fully rewritten in shell

changes v4->v5:
- add verbose support
- rename shrink_rpath -> clear_rpath
- add sanitize_rpath function

changes v3->v4:
- fix typos and license (Baruch)

changes v2->v3:
- no change

---
 support/scripts/fix-rpath         | 116 +++++++++++++++++++++++++
 support/scripts/shell/log.sh      |  61 +++++++++++++
 support/scripts/shell/patchelf.sh | 178 ++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh  | 173 ++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh      |  68 +++++++++++++++
 support/scripts/shell/source.sh   |  77 +++++++++++++++++
 support/scripts/shell/utils.sh    |  60 +++++++++++++
 7 files changed, 733 insertions(+)
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh
support/scripts/fix-rpath new file +116 -0
new file mode 100755
index 0000000..fb2bfeb
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+set -e
+
+usage() {
+  cat <<EOF >&2
+Usage:  ${0} TREE_KIND TREE_ROOT
+
+Description:
+
+        This script scans a tree and sanitize ELF files' RPATH found in there.
+
+        Sanitization behaves the same whatever the kindd of the processed tree, but
+        the resulting RPATH differs.
+
+        Sanitization action:
+        - remove RPATH pointing outside of the tree
+        - for RPATH pointing in the tree:
+          - if they point to standard location (/lib, /usr/lib): remove them
+          - otherwise: make them relative using \$ORIGIN
+
+        For the target tree:
+        - scan the whole tree for sanitization
+
+        For the staging tree :
+        - scan the whole tree for sanitization
+
+        For the host tree:
+        - skip the staging tree for sanitization
+        - add \$HOST_DIR/{lib,usr/lib} to RPATH (as relative pathes)
+
+Arguments:
+
+        TREE_KIND   Kind of tree to be processed.
+                    Allowed values: host, target, staging
+
+        TREE_ROOT   Path to the root of the tree to be scaned
+
+Environment:
+
+        HOST_READELF    readelf program to use for host ELF files
+                        (default: readelf)
+
+        HOST_READELF    readelf program to use for host ELF files
+                        (default: readelf)
+
+        PATCHELF        patchelf program to use
+                        (default: patchelf)
+EOF
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
+source.load_module patchelf
+
+: ${HOST_READELF:=readelf}
+: ${TARGET_READELF:=readelf}
+: ${PATCHELF:=patchelf}
+
+main() {
+    local tree="${1}"
+    local basedir="$(readlink -f "${2}")"
+
+    local find_args=( "${basedir}" )
+    local sanitize_extra_args=()
+    local readelf
+
+    case "${tree}" in
+        host)
+            # do not process the sysroot (only contains target binaries)
+            find_args+=( "-name" "sysroot" "-prune" "-o" )
+
+            # do not process the external toolchain installation directory to
+            # to avoid breaking it.
+            find_args+=( "-path" "*/opt/ext-toolchain" "-prune" "-o" )
+
+            # make sure RPATH will point to ${hostdir}/lib and ${hostdir}/usr/lib
+            sanitize_extra_args+=( "keep_lib_and_usr_lib" )
+
+            readelf="${HOST_READELF}"
+            ;;
+        staging|target)
+            readelf="${TARGET_READELF}"
+            ;;
+        *)
+            usage
+            exit 1
+            ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file ; do
+        READELF="${readelf}" PATCHELF="${PATCHELF}" \
+            patchelf.sanitize_rpath "${basedir}" "${file}" ${sanitize_extra_args[@]}
+    done < <(find ${find_args[@]} | readelf.filter_elf)
+}
+
+main ${@}
support/scripts/shell/log.sh new file +61 -0

61 lines not on this page · raw message

support/scripts/shell/patchelf.sh new file +178 -0

178 lines not on this page · raw message

support/scripts/shell/readelf.sh new file +173 -0

173 lines not on this page · raw message

support/scripts/shell/sdk.sh new file +68 -0

68 lines not on this page · raw message

support/scripts/shell/source.sh new file +77 -0

77 lines not on this page · raw message

support/scripts/shell/utils.sh new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
2.8.0
#4 lore raw

PATCH v8 4/14 core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

In reply to Samuel Martin

This patch adds host-patchelf as a target-finalize dependency, and introduces the HOST_SANITIZE_RPATH_HOOK hook to fix the ELF files' RPATH from the HOST_DIR location (excluding the sysroot).

After running this hook, the RPATH from any host ELF files is relative to the binary location itself.

Notes:
- we avoid to fix RPATH in the sysroot.
- we do not try to fix RPATH in the external toolchain installation location as they may have been built in a way, this is already correct; furthermore, fixing RPATH in those programs may result in breaking them.
- the whole host directory is processed because a number of host-package install programs that could be useful in places different from $(HOST_DIR)/{bin,sbin,usr/bin,usr/sbin}.
- the shared libraries are also processed in case they have a 'main' function.

As a step toward a fully relocatable SDK, this change allows to get the toolchain relocatable, but not yet the whole SDK.

---
changes v7->v8:
- none

changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile, now just place the variable in the Buildroot environment, to enable debug logs. This is documented in a follow-up patch of this series.

changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- add verbose support

changes v3->v4:
- add host-patchelf to PACKAGES instead of target-finalize (Baruch)
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)

---
 Makefile | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
Makefile +15 -0
index a05b9e1..e3574cf 100644
@@ -638,6 +638,21 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+# - The target hook sanitizes RPATH in target ELF binaries, removing paths
+#   pointing to package's build directories or the sysroot's libdirs.
+PACKAGES += host-patchelf
+
+define HOST_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=readelf \
+		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
Signature · 1 line
2.8.0
#5 lore raw

PATCH v8 4/14 core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

Hello,

Makefile endef · line 645

Quoted 5 lines · Samuel Martin
Makefile +5 -0
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+# - The target hook sanitizes RPATH in target ELF binaries, removing paths
+#   pointing to package's build directories or the sysroot's libdirs.

There is nothing in this patch that adds a target hook.

Makefile endef · line 654

Quoted 9 lines
Makefile +9 -0
+PACKAGES += host-patchelf
+
+define HOST_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=readelf \
+		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK

I find it somewhat odd that this gets done as a "target finalize" hook since it really has nothing to do with finalizing the target. It really just needs to be done "at the end of the build".

We could add that as a new target, but it would complexify the dependency chain, so maybe adding to TARGET_FINALIZE_HOOKS is the easiest solution after all.

One tricky aspect though is that not all host packages are guaranteed to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed, TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target* packages have been built. But if a host package is needed just to build a filesystem image for example, it is not guaranteed that it will be built before TARGET_FINALIZE_HOOKS is called.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#6 lore raw

PATCH v8 4/14 core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

Quoted 11 lines · Thomas Petazzoni

Hello,

Makefile endef · line 645

Quoted 5 lines · Samuel Martin
Makefile +5 -0
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+# - The target hook sanitizes RPATH in target ELF binaries, removing paths
+#   pointing to package's build directories or the sysroot's libdirs.

There is nothing in this patch that adds a target hook.

Oops, my bad! :-s
This is for the next patch in the series.

Quoted 19 lines

Makefile endef · line 654

Quoted 10 lines
Makefile +9 -0
+PACKAGES += host-patchelf
+
+define HOST_SANITIZE_RPATH_HOOK
+     PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+     READELF=readelf \
+             $(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK

I find it somewhat odd that this gets done as a "target finalize" hook since it really has nothing to do with finalizing the target. It really just needs to be done "at the end of the build".

We could add that as a new target, but it would complexify the dependency chain, so maybe adding to TARGET_FINALIZE_HOOKS is the easiest solution after all.

Quoted 6 lines

One tricky aspect though is that not all host packages are guaranteed to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed, TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target* packages have been built. But if a host package is needed just to build a filesystem image for example, it is not guaranteed that it will be built before TARGET_FINALIZE_HOOKS is called.

Good point!
I'll check and see if I can come up with something.

Regards,

Signature · 1 line
Samuel
#7 lore raw

PATCH v8 4/14 core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

In reply to Thomas Petazzoni
Quoted 30 lines · Thomas Petazzoni

Hello,

Makefile endef · line 645

Quoted 5 lines · Samuel Martin
Makefile +5 -0
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+# - The target hook sanitizes RPATH in target ELF binaries, removing paths
+#   pointing to package's build directories or the sysroot's libdirs.

There is nothing in this patch that adds a target hook.

Makefile endef · line 654

Quoted 9 lines
Makefile +9 -0
+PACKAGES += host-patchelf
+
+define HOST_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=readelf \
+		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK

I find it somewhat odd that this gets done as a "target finalize" hook since it really has nothing to do with finalizing the target. It really just needs to be done "at the end of the build".

We could add that as a new target, but it would complexify the dependency chain, so maybe adding to TARGET_FINALIZE_HOOKS is the easiest solution after all.

Quoted 6 lines

One tricky aspect though is that not all host packages are guaranteed to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed, TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target* packages have been built. But if a host package is needed just to build a filesystem image for example, it is not guaranteed that it will be built before TARGET_FINALIZE_HOOKS is called.

Actually, it is run after all explicitly selected packages have been built. So
this is yet another reason to also add Kconfig options for host packages.

Regards,
Arnout

Quoted 5 lines

Best regards,

Thomas

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#8 lore raw

PATCH v8 4/14 core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

Hello,

Quoted 9 lines · Arnout Vandecappelle
Quoted 6 lines

One tricky aspect though is that not all host packages are guaranteed to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed, TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target* packages have been built. But if a host package is needed just to build a filesystem image for example, it is not guaranteed that it will be built before TARGET_FINALIZE_HOOKS is called.

Actually, it is run after all explicitly selected packages have been built. So
this is yet another reason to also add Kconfig options for host packages.

Well, while I do like the fact that you try to find more arguments to add Kconfig options for host packages, I am not sure this particular one is a good reason. Indeed, as the name TARGET_FINALIZE_HOOKS suggests, it finalizes the _target_, so there is no point in guaranteeing that all host packages have been built *before* the hooks in TARGET_FINALIZE_HOOKS are executed.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#9 lore raw

PATCH v8 5/14 core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

In reply to Samuel Martin

This patch introduces the TARGET_SANITIZE_RPATH_HOOK and STAGING_SANITIZE_RPATH_HOOK hooks fixing the ELF files' RPATH of binaries from, respectively, the TARGET_DIR and the STAGING_DIR locations.

After running this hook, the RPATH from any target ELF files from both the target and the staging locations won't contain any occurence of the sysroot or some build locations.

---
changes v7->v8:
- none

changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile

changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- target hook now sanitizes the rpath (Baruch)
- add verbose support
- update comment

changes v3->v4:
- rebase
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)

---
 Makefile | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
Makefile +18 -0
index e3574cf..38d1076 100644
@@ -653,6 +653,24 @@ endef
 
 TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
 
+# Function sanitizing target/staging ELF files' RPATH.
+# i.e. it removes paths pointing to the staging or build location from the ELF
+# files' RPATH.
+define TARGET_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+		$(TOPDIR)/support/scripts/fix-rpath target $(TARGET_DIR)
+endef
+
+define STAGING_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+		$(TOPDIR)/support/scripts/fix-rpath staging $(STAGING_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += TARGET_SANITIZE_RPATH_HOOK \
+       STAGING_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
Signature · 1 line
2.8.0
#10 lore raw

PATCH v8 6/14 package/speex: remove no longer needed hook

In reply to Samuel Martin

Remove the LIBTOOL_FIXUP hook since RPATH are now sanitized in target-finalize hooks.

---
changes v7->v8:
- none

changes v6->v7:
- none

changes v5->v6:
- none

changes v4->v5:
- new patch (suggested by Baruch)

---
 package/speex/speex.mk | 5 -----
 1 file changed, 5 deletions(-)
package/speex/speex.mk +0 -5
index a34bfac..24782ec 100644
@@ -24,11 +24,6 @@ ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y)
 SPEEX_CONF_OPTS += --enable-arm5e-asm
 endif
 
-define SPEEX_LIBTOOL_FIXUP
-	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
-	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
-endef
-
 SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
 
 define SPEEX_BUILD_CMDS
Signature · 1 line
2.8.0
#11 lore raw

PATCH v8 6/14 package/speex: remove no longer needed hook

Hello,

package/speex/speex.mk ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y) · line 27

Quoted 6 lines · Samuel Martin
package/speex/speex.mk +0 -5
-define SPEEX_LIBTOOL_FIXUP
-	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
-	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
-endef
-
 SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP

So presumably this line is also not useful anymore?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#12 lore raw

PATCH v8 7/14 toolchain: add post-install hooks making the toolchain relocatable

In reply to Samuel Martin

This change adds the required dependency and post-target-install hooks to render the toolchain relocatable.

Notes:
- Although the toolchain package is a virtual package, it does support hooks.
- This has to be post-target-install hooks because the toolchain package is handled as a "target" package.

Ultimately, this patch allows to generate relocatable toolchains, just by doing:

  $ make menuconfig
  $ make toolchain

Then, this toolchain can be used/shared/deployed anywhere*.

*: on any Linux system with an ABI-compatible set of system libraries
(usually including: glibc, libgcc, zlib, flex and libsigsegv).

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>

---
changes v7->v8:
- none

changes v6->v7:
- new patch

---
 toolchain/toolchain/toolchain.mk | 5 +++++
 1 file changed, 5 insertions(+)
toolchain/toolchain/toolchain.mk +5 -0
index c22713b..618aa64 100644
@@ -12,6 +12,11 @@ endif
 
 TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 
+# dependency and hooks needed to make the toolchain relocatable
+TOOLCHAIN_DEPENDENCIES += host-patchelf
+TOOLCHAIN_POST_INSTALL_TARGET_HOOKS += HOST_SANITIZE_RPATH_HOOK \
+	STAGING_SANITIZE_RPATH_HOOK
+
 $(eval $(virtual-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
2.8.0
#13 lore raw

PATCH v8 7/14 toolchain: add post-install hooks making the toolchain relocatable

Hello,

Quoted 15 lines · Samuel Martin

This change adds the required dependency and post-target-install hooks to render the toolchain relocatable.

Notes:
- Although the toolchain package is a virtual package, it does support hooks.
- This has to be post-target-install hooks because the toolchain package is handled as a "target" package.

Ultimately, this patch allows to generate relocatable toolchains, just by doing:

  $ make menuconfig
  $ make toolchain
Quoted 8 lines

Then, this toolchain can be used/shared/deployed anywhere*.

*: on any Linux system with an ABI-compatible set of system libraries
(usually including: glibc, libgcc, zlib, flex and libsigsegv).

Cc Thomas Petazzoni <[email protected]>
Cc Peter Korsgaard <[email protected]>

Why is this needed? You already do such steps in the TARGET_FINALIZE_HOOKS ?

Yes, it means after just a "make toolchain", you don't have a relocatable toolchain. But Buildroot never guaranteed that a partial build will yield fully usable results.

So I would prefer to not have this patch, at least in the initial version of the series.

Again, this stuff is complicated. Please keep things as minimal as possible in a first step.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#14 lore raw

PATCH v8 8/14 support/scripts: update check-host-rpath to use the shell helpers

In reply to Samuel Martin
Cc "Yann E. MORIN" <[email protected]>

---
changes v7->v8:
- none

changes v6->v7:
- minor updates after shell modules shell changes

changes v5->v6:
- new patch

---
 support/scripts/check-host-rpath | 73 +++++++++++++++-------------------------
 support/scripts/shell/readelf.sh | 69 +++++++++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 45 deletions(-)
support/scripts/check-host-rpath +28 -45
index 48d69da..45bece8 100755
@@ -1,12 +1,29 @@
 #!/usr/bin/env bash
 
+# Copyright (C) 2015 Yann E. MORIN <[email protected]>
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
 # This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
 # they have an RPATH to $(HOST_DIR)/usr/lib if they need libraries from
 # there.
 
-# Override the user's locale so we are sure we can parse the output of
-# readelf(1) and file(1)
-export LC_ALL=C
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
 
 main() {
     local pkg="${1}"
@@ -14,58 +31,24 @@ main() {
     local file ret
 
     # Remove duplicate and trailing '/' for proper match
-    hostdir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}" )"
+    hostdir="$(sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}")"
 
     ret=0
     while read file; do
-        elf_needs_rpath "${file}" "${hostdir}" || continue
-        check_elf_has_rpath "${file}" "${hostdir}" && continue
+        READELF=readelf readelf.needs_rpath "${file}" "${hostdir}" || continue
+        READELF=readelf readelf.has_rpath "${file}" "${hostdir}" && continue
         if [ ${ret} -eq 0 ]; then
             ret=1
             printf "***\n"
-            printf "*** ERROR: package %s installs executables without proper RPATH:\n" "${pkg}"
+            printf \
+                "*** ERROR: package %s installs executables without proper RPATH:\n" \
+                "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <( find "${hostdir}"/usr/{bin,sbin} -type f -exec file {} + 2>/dev/null \
-              |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d'                \
-                      -e 's//\1/'                                                  \
-            )
+    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+                  readelf.filter_elf_executable)
 
     return ${ret}
 }
 
-elf_needs_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local lib
-
-    while read lib; do
-        [ -e "${hostdir}/usr/lib/${lib}" ] && return 0
-    done < <( readelf -d "${file}"                                         \
-              |sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
-                     -e 's//\1/;'                                          \
-            )
-
-    return 1
-}
-
-check_elf_has_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local rpath dir
-
-    while read rpath; do
-        for dir in ${rpath//:/ }; do
-            # Remove duplicate and trailing '/' for proper match
-            dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" ] && return 0
-        done
-    done < <( readelf -d "${file}"                                              \
-              |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
-                      -e 's//\3/;'                                              \
-            )
-
-    return 1
-}
-
 main "${@}"
support/scripts/shell/readelf.sh +69 -0

86 lines not on this page · raw message

Signature · 1 line
2.8.0
#15 lore raw

PATCH v8 8/14 support/scripts: update check-host-rpath to use the shell helpers

Hello,

So the commit title says that you update check-host-rpath to use the shell helpers...

Quoted 2 lines · Samuel Martin
 support/scripts/check-host-rpath | 73 +++++++++++++++-------------------------
 support/scripts/shell/readelf.sh | 69 +++++++++++++++++++++++++++++++++++++

but in practice, you also add a lot of new functions to the shell helpers.

Either this is intended and it should be mentioned in the commit log, or it is not intended.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#16 lore raw

PATCH v8 9/14 support/scripts/check-host-rpath: silent find command

In reply to Samuel Martin

Silent find command when HOST_DIR/usr/sbin does not exist.

Cc "Yann E. MORIN" <[email protected]>

---
changes v7->v8:
- none

changes v6->v7:
- new patch

---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/check-host-rpath +1 -1
index 45bece8..3df70bd 100755
@@ -45,7 +45,7 @@ main() {
                 "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
                   readelf.filter_elf_executable)
 
     return ${ret}
Signature · 1 line
2.8.0
#17 lore raw

PATCH v8 9/14 support/scripts/check-host-rpath: silent find command

Hello,

support/scripts/check-host-rpath main() { · line 49

Quoted 3 lines · Samuel Martin
support/scripts/check-host-rpath +1 -1
-    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
                   readelf.filter_elf_executable)

This patch is useless. Your patch 08/14 is doing:

- done < <( find "${hostdir}"/usr/{bin,sbin} -type f -exec file {} + 2>/dev/null \
- |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d' \
- -e 's//\1/' \
- )
+ done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+ readelf.filter_elf_executable)

So essentially the 2>/dev/null was already here, and you remove it. And then you add it again in your patch 09/14. Simply squash both patches.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#18 lore raw

PATCH v8 10/14 support/scripts/check-host-rpath: also check HOST_DIR/{bin, sbin}

In reply to Samuel Martin

At least syslinux is installing stuff in HOST_DIR/sbin.

Cc "Yann E. MORIN" <[email protected]>

---
changes v7->v8:
- none

changes v6->v7:
- new patch

---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/check-host-rpath +1 -1
index 3df70bd..022f01b 100755
@@ -45,7 +45,7 @@ main() {
                 "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
+    done < <(find "${hostdir}"/{,usr/}{bin,sbin} -type f -print 2>/dev/null |
                   readelf.filter_elf_executable)
 
     return ${ret}
Signature · 1 line
2.8.0
#19 lore raw

PATCH v8 10/14 support/scripts/check-host-rpath: also check HOST_DIR/{bin, sbin}

Hello,

Quoted 20 lines · Samuel Martin

At least syslinux is installing stuff in HOST_DIR/sbin.

Cc "Yann E. MORIN" <[email protected]>

---
changes v7->v8:
- none

changes v6->v7:
- new patch

---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/check-host-rpath +0 -0
index 3df70bd..022f01b 100755

support/scripts/check-host-rpath main() { · line 49

Quoted 6 lines
support/scripts/check-host-rpath +1 -1
@@ -45,7 +45,7 @@ main() {
                 "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
+    done < <(find "${hostdir}"/{,usr/}{bin,sbin} -type f -print 2>/dev/null |
                   readelf.filter_elf_executable)

Please put this patch as PATCH 01 in your series, so that we can apply it right now as a preparation fix. This way, you reduce the size of your patch series, and things move forward.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#20 lore raw

PATCH v8 11/14 support/scripts: add check-host-leaks script + all needed helpers

In reply to Samuel Martin

---
changes v7->v8:
- move sdk.check_host_leaks in the check-host-leaks script itself (Arnout)
- drop leak classification (Arnout)
- rework readelf.is_elf_{static_library,object} (Arnout)
- add utils.assert_absolute_canonical_path
- drop utils.guess_gnu_target_name
- rework/simplify the check-host-leaks script command line

changes v6->v7:
- {filer,is}_elf* functions moved from utils to readelf module
- update sdk.check_host_leaks

changes v5->v6:
- new patch

---
 support/scripts/check-host-leaks | 126 +++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh |  77 ++++++++++++++++++++++--
 support/scripts/shell/sdk.sh     |   7 +++
 support/scripts/shell/utils.sh   |  14 +++++
 4 files changed, 218 insertions(+), 6 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
support/scripts/check-host-leaks new file +126 -0
new file mode 100755
index 0000000..9f363af
@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  local ret="${1:-0}"
+  cat <<EOF
+
+Usage:  ${0##*/} [options] ROOT_DIR LEAK_PATH...
+
+Description:
+
+        This script scans a tree for host paths leaked into it.
+        Prints out leaks alongside with the kind of leak.
+
+Arguments:
+
+        ROOT_DIR        Path to the root of the tree to be scanned
+
+        LEAK_PATH       Paths to check for occurence. Several paths can be
+                        passed.
+                        A LEAK_PATH must be an absolute canonical path.
+                        In the Buildroot context, the LEAK_PATH are Buildroot's
+                        TOPDIR, BASE_DIR, HOST_DIR and STAGING_DIR.
+
+Options:
+
+        --exclude=PATH
+                Excludes files from ROOT_DIR starting with PATH. Obviously, the
+                excluded paths should point to some sub-location of ROOT_DIR.
+                Can be set more than once.
+                An excluded path must be an absolute canonical path.
+
+
+EOF
+  return ${ret}
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module log
+source.load_module utils
+source.load_module sdk
+
+main() {
+    local root_dir
+    local class_leaks
+    local excluded=()
+    local leak_paths=()
+    while test ${#} -gt 0 ; do
+        case "${1}" in
+            --exclude=*) excluded+=( "${1#*=}" )
+                ;;
+            --exclude)   shift ; excluded+=( "${1}" )
+                ;;
+            -h|--help)
+                usage
+                exit 0
+                ;;
+            *)  if test -z "${root_dir}" ; then root_dir="${1}"
+                else leak_paths+=( "${1}" )
+                fi
+                ;;
+        esac
+        shift
+    done
+
+    excluded=( $(utils.list_reduce ${excluded[@]}) )
+    leak_paths=( $(utils.list_reduce ${leak_paths[@]}) )
+
+    # sanity checks
+    if test -z "${root_dir}" ; then
+        log.error "Not enough argument: no ROOT_DIR given.\n" ||
+            usage ${?} || exit ${?}
+    fi
+    if test ${#leak_paths[@]} -lt 1 ; then
+        log.error "Not enough argument. No LEAK_PATH given.\n" ||
+            usage ${?} || exit ${?}
+    fi
+
+    local path has_non_abspath
+    for path in ${leak_paths[@]} ; do
+        utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+    done
+    if test -n "${has_non_abspath}" ; then
+        log.error "Some LEAK_PATH are not absolute canonical paths.\n" ||
+            usage ${?} || exit ${?}
+    fi
+    has_non_abspath=
+    for path in ${excluded[@]} ; do
+        utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+    done
+    if test -n "${has_non_abspath}" ; then
+        log.error "Some excluded path are not absolute canonical paths.\n" ||
+            usage ${?} || exit ${?}
+    fi
+
+    local re_leaks="$(sed -re 's/^/(/ ; s/$/)/ ; s/ +/|/g' <<<"${leak_paths[*]}")"
+    local re_excl="$(sed -re 's/ +/|/g' <<<"${excluded[*]}")"
+    if test -n "${re_excl}" ; then
+      re_excl="\\:^(${re_excl}):d"
+    fi
+    pushd "${root_dir}" >/dev/null
+    local f
+    grep -raEl "${re_leaks}" . |
+        sed -re "${re_excl} ; s:^\.:${root_dir}:" |
+        while read f ; do
+            printf "%s\n" "${f}"
+        done | sort
+    popd >/dev/null
+}
+main "${@}"
support/scripts/shell/readelf.sh +71 -6

119 lines not on this page · raw message

support/scripts/shell/sdk.sh +7 -0
index b2f699c..6cb3bbe 100644
@@ -19,9 +19,16 @@
 # This module defines the following functions:
 #   sdk.compute_relative_path
 #   sdk.compute_rpath
+#   sdk.check_host_leaks
+#
+# This module is sensitive to the following environment variables:
+#   READELF
 
 source.declare_module sdk
 
+source.load_module utils
+source.load_module readelf
+
 # sdk.compute_relative_path basedir path start
 #
 # Computes and prints the relative path between $start and $path within $basedir.
support/scripts/shell/utils.sh +14 -0
index 9e9aaab..eac3158 100644
@@ -19,6 +19,7 @@
 # This module defines the following functions:
 #   utils.list_has
 #   utils.list_reduce
+#   utils.assert_absolute_canonical_path
 
 source.declare_module utils
 
@@ -58,3 +59,16 @@ utils.list_reduce() {
 
     echo ${lout[@]}
 }
+
+# utils.assert_absolute_canonical_path path
+#
+# Returns 0 if 'path' is the absolute canonical path, returns non-0
+# otherwise.
+#
+# If the test failed, an error message will be issued to stderr.
+#
+# path : path to be tested
+utils.assert_absolute_canonical_path() {
+    test "$(readlink -f "${1}")" = "${1}" ||
+        log.error "%s is not the absolute canonical path.\n" "${1}" >&2
+}
Signature · 1 line
2.8.0
#21 lore raw

PATCH v8 12/14 core: add check-leaks-in-{target, host, staging} targets

In reply to Samuel Martin

This new targets allow identifying what need to be fixed to fulfill the ultimate goal of removing any trace of the host in the target filesystem or the SDK.

---
changes v7->v8:
- update the check-host-leak script call

changes v6->v7:
- none

changes v5->v6:
- new patch

---
 Makefile             | 15 +++++++++++++++
 package/pkg-utils.mk |  5 +++++
 2 files changed, 20 insertions(+)
Makefile +15 -0
index 38d1076..8309eae 100644
@@ -638,6 +638,21 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# Checks for build machine leaks into target filesystem, SDK
+check-leaks-in-target:
+	@$(call MESSAGE,Checking leaks in the target)
+	$(Q)$(call check-for-build-machine-leaks-in,$(TARGET_DIR))
+
+check-leaks-in-host:
+	@$(call MESSAGE,Checking leaks in the host)
+	$(Q)$(call check-for-build-machine-leaks-in,$(HOST_DIR),--exclude=$(STAGING_DIR))
+
+check-leaks-in-staging:
+	@$(call MESSAGE,Checking leaks in the staging)
+	$(Q)$(call check-for-build-machine-leaks-in,$(STAGING_DIR))
+
+check-leaks:	check-leaks-in-target check-leaks-in-host check-leaks-in-staging
+
 # RPATH fixing
 # - The host hook sets RPATH in host ELF binaries, using relative paths to the
 #   library locations.
package/pkg-utils.mk +5 -0
index f88313a..8b80028 100644
@@ -150,3 +150,8 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET}
 	mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \
 	cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2)
 endef
+
+define check-for-build-machine-leaks-in
+	$(TOPDIR)/support/scripts/check-host-leaks $(1) \
+		$(TOPDIR) $(BASE_DIR) $(HOST_DIR) $(STAGING_DIR) $(2)
+endef
Signature · 1 line
2.8.0
#22 lore raw

PATCH v8 13/14 support/scripts/check-host-leaks: add option to classify leaks

In reply to Samuel Martin

---
changes v7->v8:
- new patch. Move this feature in its own change-set as it seems a bit controversial.
- make leaks classification optional

---
 support/scripts/check-host-leaks | 59 ++++++++++++++++++++++++++++++++++++++--
 support/scripts/shell/readelf.sh |  1 -
 2 files changed, 57 insertions(+), 3 deletions(-)
support/scripts/check-host-leaks +57 -2
index 9f363af..bd92493 100755
@@ -45,6 +45,12 @@ Options:
                 Can be set more than once.
                 An excluded path must be an absolute canonical path.
 
+        --classify-leaks
+                Enables leak classification. Categories are defined according to
+                the matched files' type or ELF kinds.
+                For ELF files, the sections containing the leaks will displayed.
+                Enabling this option substantially increases the execution time
+                of the script.
 
 EOF
   return ${ret}
@@ -56,9 +62,51 @@ source.load_module log
 source.load_module utils
 source.load_module sdk
 
+classify_leak() {
+    local f="${1}" regexp="${2}"
+    local leak
+    if test -h "${f}" ; then leak="symlink"
+    elif readelf.is_elf "${f}" ; then
+        if readelf.is_elf_executable "${f}" ; then leak="ELF/exe"
+        elif readelf.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
+        elif readelf.is_elf_static_library "${f}" ; then leak="ELF/*.a"
+        elif readelf.is_elf_object "${f}" ; then
+            case "${f}" in
+                *.ko) leak="ELF/*.ko" ;;
+                *) leak="ELF/*.o" ;;
+            esac
+        else leak="ELF/?"
+        fi
+        local section
+        local sections=()
+        for section in $(readelf.list_sections "${f}") ; do
+            if readelf.string_section "${f}" "${section}" |
+                    grep -qaE "${regexp}" ; then
+                if ! utils.list_has ${section} ${sections[@]} ; then
+                    sections+=( "${section}" )
+                fi
+            fi
+        done
+        leak="${leak} [${sections[*]}]"
+    else
+        case "${f}" in
+            *-config) leak="*-config script" ;;
+            *.la) leak="*.la" ;;
+            *.pc) leak="*.pc" ;;
+            *.py) leak="*.py" ;;
+            *.pyc|*.pyo) leak="*.py[co]" ;;
+        esac
+    fi
+    if test -z "${leak}" ; then
+        leak="? [$(file -z "${f}" | sed -e 's/.*: //')]"
+    fi
+    printf "${leak}"
+}
+
+
 main() {
     local root_dir
-    local class_leaks
+    local classify_leaks
     local excluded=()
     local leak_paths=()
     while test ${#} -gt 0 ; do
@@ -67,6 +115,8 @@ main() {
                 ;;
             --exclude)   shift ; excluded+=( "${1}" )
                 ;;
+            --classify-leaks) classify_leaks=y
+                ;;
             -h|--help)
                 usage
                 exit 0
@@ -119,7 +169,12 @@ main() {
     grep -raEl "${re_leaks}" . |
         sed -re "${re_excl} ; s:^\.:${root_dir}:" |
         while read f ; do
-            printf "%s\n" "${f}"
+            if test -n "${classify_leaks}" ; then
+                local leak="$(classify_leak "${f}" "${regexp}")"
+                printf "%-70s : %-120s\n" "${leak}" "${f}"
+            else
+                printf "%s\n" "${f}"
+            fi
         done | sort
     popd >/dev/null
 }
support/scripts/shell/readelf.sh +0 -1
index 87599bb..acb522f 100644
@@ -54,7 +54,6 @@ source.declare_module readelf
 # environment:
 #   READELF: readelf program path
 readelf._match_elf_regexp() {
-    log._trace_func
     local regexp="${1}" file="${2}"
     LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
 }
Signature · 1 line
2.8.0
#23 lore raw

PATCH v8 14/14 docs/manual: document how to debug shell script

In reply to Samuel Martin

The method describes here is only true for scripts using the shell modules from support/scripts/shell/*.

---
changes v7->v8:
- put instrumentation in a subsection (Arnout)
- minor rewording

changes v6->v7:
- new patch

---
 docs/manual/debugging-buildroot.txt | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
docs/manual/debugging-buildroot.txt +17 -0
index f575fc4..b809c2d 100644
@@ -5,6 +5,8 @@
 
 == Debugging Buildroot
 
+=== Running scripts before and after each build step
+
 It is possible to instrument the steps +Buildroot+ does when building
 packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
 the path of one or more scripts (or other executables), in a
@@ -41,3 +43,18 @@ The script has access to the following variables:
   - +BINARIES_DIR+: the place where all binary files (aka images) are
     stored
   - +BASE_DIR+: the base output directory
+
+=== Debugging shell scripts in Buildroot
+
+For shell scripts using the _shell modules_ (available in
++support/scripts/shell/+), an environment variable is available to trace
+the scripts:
+
+  - +SHELL_DEBUG+ adjustes the debug level, from 0 to 3 (only errors to
+    all debug messages)
+
+To enable these debug outputs:
+
+----
+SHELL_DEBUG=3 make
+----
Signature · 1 line
2.8.0