Skip to contentSkip to conversation

Relocatable SDK/Build machine leaks: RPATH fixing

PATCH Superseded Samuel Martin 5 participants 22 messages Apr 22, 2016 – Jul 7, 2016 Kbuild

v9 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v9 cover Relocatable SDK/Build machine leaks: RPATH fixing

Hi all,

Here is yet another round of the series aiming improving the relocatability of the SDK built by Buildroot.

Still slowly reducing the number of patch in the series, as they got merged... For this round, just fixes from comments, nothing really noticeable, except maybe the move of the HOST_DIR RPATH sanitation.

For further details, check the per-commit changelog in each commit log.

Patches 1 and 2:
Fix the O variable definition.

Patches 3 to 5:
Add the fix-rpath script, then run it on the Buildroot host, target and staging trees.

Patch 6:
Clean-up speex package WRT its RPATH hook.

Patch 7:
Update and fix the check-host-rpath to re-use the shell helpers.

Patches 8 to 10:
Add means to track build machine leaks into the host, target and staging trees. These patches will allow to identify what remains to be fixed WRT build machine leaks.

Patches 11:
Update documentation about how to enable/adjust log level on script using the shell modules introduced in patch 3/11.

Regards,
Samuel

Samuel Martin (11):
core: split variables definition related to in/out-of-tree build from
O itself
core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
path
support/scripts: add fix-rpath script + a bunch of helpers
core: sanitize HOST_DIR at the very end of the build
core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
TARGET_FINALIZE_HOOKS
package/speex: remove no longer needed hook support/scripts: update check-host-rpath to use the shell helpers support/scripts: add check-host-leaks script + all needed helpers
core: add check-leaks-in-{target,host,staging} targets support/scripts/check-host-leaks: add option to classify leaks docs/manual: document how to debug shell script

 Makefile                            | 126 ++++++++++++---
 docs/manual/debugging-buildroot.txt |  17 ++
 package/pkg-utils.mk                |   5 +
 package/speex/speex.mk              |   7 -
 support/scripts/check-host-leaks    | 181 +++++++++++++++++++++
 support/scripts/check-host-rpath    |  73 ++++-----
 support/scripts/fix-rpath           | 116 ++++++++++++++
 support/scripts/shell/log.sh        |  61 +++++++
 support/scripts/shell/patchelf.sh   | 178 +++++++++++++++++++++
 support/scripts/shell/readelf.sh    | 306 ++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh        |  75 +++++++++
 support/scripts/shell/source.sh     |  77 +++++++++
 support/scripts/shell/utils.sh      |  74 +++++++++
 13 files changed, 1223 insertions(+), 73 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

--
2.8.0

#1 lore raw

PATCH v9 1/11 core: split variables definition related to in/out-of-tree build from O itself

This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from the presence of the O variable in the command line.

Now, the condition used to set these variables is the value of O itself.

This change is a preparatory work since the O definition will need to be moved around when we will make Buildroot run with absolute canonical paths for both its root directory and the output location. This will be addressed in a follow-up patch.

---
changes v8->v9:
- none

changes v7->v8:
- new patch

---
 Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Makefile +11 -3
index 0e4beb2..3d86c9b 100644
@@ -110,10 +110,11 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
 ifneq ("$(origin O)", "command line")
 O := output
-CONFIG_DIR := $(TOPDIR)
-NEED_WRAPPER =
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
@@ -126,9 +127,16 @@ MAKEOVERRIDES =
 # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
 # To really make O go away, we have to override it.
 override O := $(O)
-CONFIG_DIR := $(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+endif
+
+# Set variables related to in-tree or out-of-tree build.
+ifeq ($(O),output)
+CONFIG_DIR := $(TOPDIR)
+NEED_WRAPPER =
+else
+CONFIG_DIR := $(O)
 NEED_WRAPPER = y
 endif
 
Signature · 1 line
2.8.0
#2 lore raw

PATCH v9 1/11 core: split variables definition related to in/out-of-tree build from O itself

Quoted 11 lines · Samuel Martin

This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from the presence of the O variable in the command line.

Now, the condition used to set these variables is the value of O itself.

This change is a preparatory work since the O definition will need to be moved around when we will make Buildroot run with absolute canonical paths for both its root directory and the output location. This will be addressed in a follow-up patch.

(finally :-) but don't count on me being able to finish the series tonight)

Regards,
Arnout

Quoted 48 lines

---
changes v8->v9:
- none

changes v7->v8:
- new patch

---
 Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Makefile +11 -3
index 0e4beb2..3d86c9b 100644
@@ -110,10 +110,11 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
 ifneq ("$(origin O)", "command line")
 O := output
-CONFIG_DIR := $(TOPDIR)
-NEED_WRAPPER =
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
@@ -126,9 +127,16 @@ MAKEOVERRIDES =
 # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
 # To really make O go away, we have to override it.
 override O := $(O)
-CONFIG_DIR := $(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+endif
+
+# Set variables related to in-tree or out-of-tree build.
+ifeq ($(O),output)
+CONFIG_DIR := $(TOPDIR)
+NEED_WRAPPER =
+else
+CONFIG_DIR := $(O)
 NEED_WRAPPER = y
 endif
 
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#3 lore raw

PATCH v9 1/11 core: split variables definition related to in/out-of-tree build from O itself

Quoted 11 lines · Arnout Vandecappelle
Quoted 10 lines · Samuel Martin

This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from the presence of the O variable in the command line.

Now, the condition used to set these variables is the value of O itself.

This change is a preparatory work since the O definition will need to be moved around when we will make Buildroot run with absolute canonical paths for both its root directory and the output location. This will be addressed in a follow-up patch.

Quoted 6 lines
Quoted 1 line

(finally :-) but don't count on me being able to finish the series tonight)

Thanks, it's ok :)
Anyway, I think I'll have to resend this series after rebasing on top of master (potential conflicts on Makefile due to some recent patches).

Regards,

Signature · 1 line
Samuel
#4 lore raw

PATCH v9 1/11 core: split variables definition related to in/out-of-tree build from O itself

In reply to Samuel Martin

Samuel, All,

On 2016-04-22 22:50 +0200, Samuel Martin spake thusly:

Quoted 11 lines

This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from the presence of the O variable in the command line.

Now, the condition used to set these variables is the value of O itself.

This change is a preparatory work since the O definition will need to be moved around when we will make Buildroot run with absolute canonical paths for both its root directory and the output location. This will be addressed in a follow-up patch.

This can probably go in now, without waiting for the rest of the series.

Regards,
Yann E. MORIN.

Quoted 53 lines

---
changes v8->v9:
- none

changes v7->v8:
- new patch

---
 Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Makefile +11 -3
index 0e4beb2..3d86c9b 100644
@@ -110,10 +110,11 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
 ifneq ("$(origin O)", "command line")
 O := output
-CONFIG_DIR := $(TOPDIR)
-NEED_WRAPPER =
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
@@ -126,9 +127,16 @@ MAKEOVERRIDES =
 # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
 # To really make O go away, we have to override it.
 override O := $(O)
-CONFIG_DIR := $(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+endif
+
+# Set variables related to in-tree or out-of-tree build.
+ifeq ($(O),output)
+CONFIG_DIR := $(TOPDIR)
+NEED_WRAPPER =
+else
+CONFIG_DIR := $(O)
 NEED_WRAPPER = y
 endif
 
Signature · 6 lines
2.8.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH v9 2/11 core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

In reply to Samuel Martin

When $(CURDIR) or $(O) contain symlinks in their path, they can be resolved differently, depending on each package build-system (whether it uses the given paths or get the absolute canonical ones).

This will make easier tracking down host machine paths leaking into the host, target or staging trees, the CURDIR and O variables are set to their absolute canonical paths.

In order to recall the toplevel makefile with absolute canonical paths for $(CURDIR) and $(O), we need to:
1- Move the O variable definition out of any if-block; so they are
always available.
2- Compute the absolute canonical paths for $(CURDIR) and $(O) that will
be passed to the sub-make. This is achieved using the 'realpath' make primitive. However, some care must be taken when manipulating O:
- the out-of-tree makefile wrapper happens a trailing "/.", we need to strip this part away to not break the comparison driving the sub-make call;
- according to [1,2], realpath returns an empty string in case of non-existing entry. So, to avoid passing an empty O= variable to sub-make, it is necessary to define the output directory and create it prior to call realpath on it (because on the first invocation, $(O) usually does not yet exists), hence the trick doing the mkdir right before calling realpath.
3- Update EXTRAMAKEARGS with the absobulte canonical $(O) and use it
when call recalling the toplevel makefile with umask and paths correctly set.
4- Lastly, update the condition for setting the CONFIG_DIR and
NEED_WRAPPER variables.

Notes:
* This change takes care of the makefile wrapper installed in $(O) to avoid unneeded make recursion.
* Now, only $(O) is strip away from MAKEOVERRIDES whatever the build is done in- or out-of-tree (i.o.w. without or with O set in the command line); wheares is the previous implementation, all variables set on the command line were stripped away only in the case of out-of-tree build.

[1] https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
[2] http://man7.org/linux/man-pages/man3/realpath.3.html

Cc Matthew Weber <[email protected]>
Cc "Yann E. MORIN" <[email protected]>
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
changes v8->v9:
- none

changes v7->v8:
- keep @ at the beginning of the command (Yann)
- make ifneq condition easier to read/parsed (Yann)
- fix O definition before re-entering make (Reported by Matthew)
- use EXTRAMAKEARGS when re-entering make
- update the condition for CONFIG_DIR and NEED_WRAPPER

changes v6->v7:
- none

changes v5->v6:
- new patch

---
 Makefile | 86 +++++++++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 58 insertions(+), 28 deletions(-)
Makefile +58 -28
index 3d86c9b..a05b9e1 100644
@@ -24,18 +24,68 @@
 # You shouldn't need to mess with anything beyond this point...
 #--------------------------------------------------------------
 
-# Trick for always running with a fixed umask
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
+ifneq ("$(origin O)", "command line")
+O := $(CURDIR)/output
+else
+# Other packages might also support Linux-style out of tree builds
+# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
+# forwards command line variable definitions those packages get very
+# confused. Fix this by telling make to not do so, only for O=..., but
+# keep all others (such as BR2_EXTERNAL, BR2_DL_DIR, etc).
+MAKEOVERRIDES := $(filter-out O=%,$(MAKEOVERRIDES))
+# Strangely enough O is still passed to submakes with MAKEOVERRIDES
+# (with make 3.81 atleast), the only thing that changes is the output
+# of the origin function (command line -> environment).
+# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
+# To really make O go away, we have to override it.
+override O := $(O)
+endif
+
+# Check if the current Buildroot execution meets all the pre-requisites.
+# If they are not met, Buildroot will actually do its job in a sub-make meeting
+# its pre-requisites, which are:
+#  1- Permissive enough umask:
+#       Wrong or too restrictive umask will prevent Buildroot and packages from
+#       creating files and directories.
+#  2- Absolute canonical CWD (i.e. $(CURDIR)):
+#       Otherwise, some packages will use CWD as-is, others will compute its
+#       absolute canonical path. This makes harder tracking and fixing host
+#       machine path leaks.
+#  3- Absolute canonical output location (i.e. $(O)):
+#       For the same reason as the one for CWD.
+
+# Current state:
+CUR_UMASK := $(shell umask)
+# Remove the trailing '/.' from $(O) as it can be added by the makefile wrapper
+# installed in the $(O) directory.
+O := $(patsubst %/.,%,$(O))
+
+# Buildroot requirements:
 UMASK = 0022
-ifneq ($(shell umask),$(UMASK))
+REAL_CURDIR := $(realpath $(CURDIR))
+# realpath needs the entry to exists, otherwise an empty string is returned.
+REAL_O := $(shell mkdir -p $(O) >/dev/null 2>&1)$(realpath $(O))
+
+# Make sure O= is passed (with its absolute canonical path) everywhere the
+# toplevel makefile is called back.
+EXTRAMAKEARGS := O=$(REAL_O)
+
+# Check Buildroot execution pre-requisites here.
+ifneq ($(CUR_UMASK):$(CURDIR):$(O),$(UMASK):$(REAL_CURDIR):$(REAL_O))
 .PHONY: _all $(MAKECMDGOALS)
 
 $(MAKECMDGOALS): _all
 	@:
 
 _all:
-	@umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
+	@umask $(UMASK) && \
+		$(MAKE) -C $(REAL_CURDIR) --no-print-directory \
+			$(MAKECMDGOALS) $(EXTRAMAKEARGS)
 
-else # umask
+else # umask / $(CURDIR) / $(O)
 
 # This is our default rule, so must come first
 all:
@@ -110,30 +160,10 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
-# Set O variable if not already done on the command line;
-# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
-# build by preventing it from being forwarded to sub-make calls.
-ifneq ("$(origin O)", "command line")
-O := output
-else
-# other packages might also support Linux-style out of tree builds
-# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
-# forwards command line variable definitions those packages get very
-# confused. Fix this by telling make to not do so
-MAKEOVERRIDES =
-# strangely enough O is still passed to submakes with MAKEOVERRIDES
-# (with make 3.81 atleast), the only thing that changes is the output
-# of the origin function (command line -> environment).
-# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
-# To really make O go away, we have to override it.
-override O := $(O)
-# we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
-endif
-
 # Set variables related to in-tree or out-of-tree build.
-ifeq ($(O),output)
-CONFIG_DIR := $(TOPDIR)
+# Here, both $(O) and $(CURDIR) are absolute canonical paths.
+ifeq ($(O),$(CURDIR)/output)
+CONFIG_DIR := $(CURDIR)
 NEED_WRAPPER =
 else
 CONFIG_DIR := $(O)
@@ -1017,4 +1047,4 @@ include docs/manual/manual.mk
 
 .PHONY: $(noconfig_targets)
 
-endif #umask
+endif #umask / $(CURDIR) / $(O)
Signature · 1 line
2.8.0
#6 lore raw

PATCH v9 2/11 core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

Quoted 5 lines · Samuel Martin

When $(CURDIR) or $(O) contain symlinks in their path, they can be resolved differently, depending on each package build-system (whether it uses the given paths or get the absolute canonical ones).

This will make easier tracking down host machine paths leaking into the

^^^^To

Quoted 11 lines

host, target or staging trees, the CURDIR and O variables are set to their absolute canonical paths.

In order to recall the toplevel makefile with absolute canonical paths for $(CURDIR) and $(O), we need to:
1- Move the O variable definition out of any if-block; so they are
always available.
2- Compute the absolute canonical paths for $(CURDIR) and $(O) that will
be passed to the sub-make. This is achieved using the 'realpath' make primitive. However, some care must be taken when manipulating O:
- the out-of-tree makefile wrapper happens a trailing "/.", we need

                                        ^^^^^^^appends
Quoted 2 lines

to strip this part away to not break the comparison driving the sub-make call;

This is just to avoid another level of recursion, right? It doesn't seem to
work, see below.

Quoted 7 lines

- according to [1,2], realpath returns an empty string in case of non-existing entry. So, to avoid passing an empty O= variable to sub-make, it is necessary to define the output directory and create it prior to call realpath on it (because on the first invocation, $(O) usually does not yet exists), hence the trick doing the mkdir right before calling realpath.
3- Update EXTRAMAKEARGS with the absobulte canonical $(O) and use it

                                   ^^^^^^^^^absolute
Quoted 9 lines

when call recalling the toplevel makefile with umask and paths correctly set.
4- Lastly, update the condition for setting the CONFIG_DIR and
NEED_WRAPPER variables.

Notes:
* This change takes care of the makefile wrapper installed in $(O) to avoid unneeded make recursion.
* Now, only $(O) is strip away from MAKEOVERRIDES whatever the build is

                      ^^^^^stripped                 ^^^^^^^^whereever
Quoted 4 lines

done in- or out-of-tree (i.o.w. without or with O set in the command line); wheares is the previous implementation, all variables set on the command line were stripped away only in the case of out-of-tree build.

I don't see how this change is related to the rest, actually.

Maybe this patch is doing a little too much in one go...

There is a lot of stuff being moved around by this patch, and very little
actual changes. It would help to separate that to start with - now the diff is a little large. You can make such a patch that has no use at all except for making the next patch simpler.

Quoted 11 lines

[1] https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
[2] http://man7.org/linux/man-pages/man3/realpath.3.html

Cc Matthew Weber <[email protected]>
Cc "Yann E. MORIN" <[email protected]>
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---

[snip]

Quoted 34 lines
Makefile +25 -1
index 3d86c9b..a05b9e1 100644
@@ -24,18 +24,68 @@
 # You shouldn't need to mess with anything beyond this point...
 #--------------------------------------------------------------
 
-# Trick for always running with a fixed umask
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
+ifneq ("$(origin O)", "command line")
+O := $(CURDIR)/output
+else
+# Other packages might also support Linux-style out of tree builds
+# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
+# forwards command line variable definitions those packages get very
+# confused. Fix this by telling make to not do so, only for O=..., but
+# keep all others (such as BR2_EXTERNAL, BR2_DL_DIR, etc).
+MAKEOVERRIDES := $(filter-out O=%,$(MAKEOVERRIDES))
+# Strangely enough O is still passed to submakes with MAKEOVERRIDES
+# (with make 3.81 atleast), the only thing that changes is the output
+# of the origin function (command line -> environment).
+# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
+# To really make O go away, we have to override it.
+override O := $(O)
+endif
+
+# Check if the current Buildroot execution meets all the pre-requisites.
+# If they are not met, Buildroot will actually do its job in a sub-make meeting
+# its pre-requisites, which are:
+#  1- Permissive enough umask:
+#       Wrong or too restrictive umask will prevent Buildroot and packages from

Makefile line 61

Quoted 10 lines
Makefile +10 -0
+#       creating files and directories.
+#  2- Absolute canonical CWD (i.e. $(CURDIR)):
+#       Otherwise, some packages will use CWD as-is, others will compute its
+#       absolute canonical path. This makes harder tracking and fixing host
+#       machine path leaks.
+#  3- Absolute canonical output location (i.e. $(O)):
+#       For the same reason as the one for CWD.
+
+# Current state:
+CUR_UMASK := $(shell umask)

We normally use = rather than := unless strictly necessary. Here it is not
strictly necessary, becauce CUR_UMASK is used only once anyway.

Makefile line 64

Quoted 3 lines
Makefile +3 -0
+# Remove the trailing '/.' from $(O) as it can be added by the makefile wrapper
+# installed in the $(O) directory.
+O := $(patsubst %/.,%,$(O))

This actually doesn't work, because of the override above you need to use
override here as well. Well, things will still _work_, you just have a redundant recursion level.

Makefile line 66

Quoted 2 lines
Makefile +2 -0
+
+# Buildroot requirements:

This comment (and the 'Current state' above) isn't very well placed here,
because there is a bit too much other stuff (comments etc.) intermingled. I think it would be better to keep the corresponding options together, so put REAL_O right below the last O replacement above, then REAL_CURDIR, and finally both UMASKs. I'm just not sure where to put the EXTRAMAKEARGS.

Makefile line 68

Quoted 3 lines
Makefile +1 -1
 UMASK = 0022
-ifneq ($(shell umask),$(UMASK))
+REAL_CURDIR := $(realpath $(CURDIR))

Here also no real need to use := even though it's used twice, realpath is not
an expensive function.

Makefile line 70

Quoted 2 lines
Makefile +2 -0
+# realpath needs the entry to exists, otherwise an empty string is returned.
+REAL_O := $(shell mkdir -p $(O) >/dev/null 2>&1)$(realpath $(O))

Here the := is needed because of the mkdir.

I don't think redirecting the stderr is a good idea, because the error you get
from it is actually interesting. Instead, I think it's best to use the same command as for BASE_DIR and just slap a $(realpath ...) around it. Also, we need that check like for BASE_DIR to error out immediately if the mkdir failed. And finally, the equivalent for BASE_DIR can be removed. Actually, the whole of BASE_DIR can be removed - O is now exactly the same as BASE_DIR.

I propose to split off a separate patch that moves the mkdir from BASE_DIR up
here to REAL_O, and just sets BASE_DIR = $(REAL_O). Then a second patch can add the recursion, and set BASE_DIR = $(O). And finally a third patch can remove BASE_DIR completely. It's possible that that third patch gets rejected in the end BTW.

Regards,
Arnout

Quoted 64 lines
Makefile +15 -26
+
+# Make sure O= is passed (with its absolute canonical path) everywhere the
+# toplevel makefile is called back.
+EXTRAMAKEARGS := O=$(REAL_O)
+
+# Check Buildroot execution pre-requisites here.
+ifneq ($(CUR_UMASK):$(CURDIR):$(O),$(UMASK):$(REAL_CURDIR):$(REAL_O))
 .PHONY: _all $(MAKECMDGOALS)
 
 $(MAKECMDGOALS): _all
 	@:
 
 _all:
-	@umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
+	@umask $(UMASK) && \
+		$(MAKE) -C $(REAL_CURDIR) --no-print-directory \
+			$(MAKECMDGOALS) $(EXTRAMAKEARGS)
 
-else # umask
+else # umask / $(CURDIR) / $(O)
 
 # This is our default rule, so must come first
 all:
@@ -110,30 +160,10 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
-# Set O variable if not already done on the command line;
-# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
-# build by preventing it from being forwarded to sub-make calls.
-ifneq ("$(origin O)", "command line")
-O := output
-else
-# other packages might also support Linux-style out of tree builds
-# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
-# forwards command line variable definitions those packages get very
-# confused. Fix this by telling make to not do so
-MAKEOVERRIDES =
-# strangely enough O is still passed to submakes with MAKEOVERRIDES
-# (with make 3.81 atleast), the only thing that changes is the output
-# of the origin function (command line -> environment).
-# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
-# To really make O go away, we have to override it.
-override O := $(O)
-# we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
-endif
-
 # Set variables related to in-tree or out-of-tree build.
-ifeq ($(O),output)
-CONFIG_DIR := $(TOPDIR)
+# Here, both $(O) and $(CURDIR) are absolute canonical paths.
+ifeq ($(O),$(CURDIR)/output)
+CONFIG_DIR := $(CURDIR)
 NEED_WRAPPER =
 else
 CONFIG_DIR := $(O)
@@ -1017,4 +1047,4 @@ include docs/manual/manual.mk
 
 .PHONY: $(noconfig_targets)
 
-endif #umask
+endif #umask / $(CURDIR) / $(O)
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#7 lore raw

PATCH v9 3/11 support/scripts: add fix-rpath script + a bunch of helpers

In reply to Samuel Martin

This commit introduces a fix-rpath shell-script able to scan a tree, detect ELF files, check their RPATH and fix it in a proper way.

Along to the fix-rpath script, it also adds a bunch of shell helper functions grouped into modules. This will help writing scripts handling RPATH and other things, while allowing to share and reuse these functions between scripts.

These helpers are namespaced within the filename of the module in which they are gathered.

This change adds 6 modules:
- source.sh: provides simple helper to easily source another module, taking care of not sourcing again an already-sourced one;
- log.sh: provides logging helpers;
- utils.sh: provides simple functions to filter ELF files in a list;
- readelf.sh: provides functions retrieving ELF details from a file;
- patchelf.sh: provides function updating ELF files;
- sdk.sh: provides RPATH computation functions.

These 6 modules are used by the fix-rpath script. Follow-up patches will make some scripts leveraging these modules.

---
changes v8->v9:
- none

changes v7->v8:
- slightly rework log.error (return non-0 instead of exiting; exit decision is deferred to the caller)

changes v6->v7 (includes changes started during BR dev. days):
- log module:
- debug env. var. renamed: DEBUG -> SHELL_DEBUG
- trace now prints the message level
- patchelf module:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- set_rpath now handles ELF files with no .dynamic section
- cleanup and minor fix in sanitize_rpath
- readelf module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_sections and has_section functions (needed in sanitize_rpath)
- sdk modules:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- minor functions doc. fixes
- source module:
- move "Could not load module" error message in load_module (Arnout)
- utils module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_has function
- fix-rpath:
- runs with "set -e" (exit on error)
- improve help text
- slight refactoring because of modules changes

changes v5->v6:
- fully rewritten in shell

changes v4->v5:
- add verbose support
- rename shrink_rpath -> clear_rpath
- add sanitize_rpath function

changes v3->v4:
- fix typos and license (Baruch)

changes v2->v3:
- no change

---
 support/scripts/fix-rpath         | 116 +++++++++++++++++++++++++
 support/scripts/shell/log.sh      |  61 +++++++++++++
 support/scripts/shell/patchelf.sh | 178 ++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh  | 173 ++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh      |  68 +++++++++++++++
 support/scripts/shell/source.sh   |  77 +++++++++++++++++
 support/scripts/shell/utils.sh    |  60 +++++++++++++
 7 files changed, 733 insertions(+)
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh
support/scripts/fix-rpath new file +116 -0
new file mode 100755
index 0000000..fb2bfeb
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+set -e
+
+usage() {
+  cat <<EOF >&2
+Usage:  ${0} TREE_KIND TREE_ROOT
+
+Description:
+
+        This script scans a tree and sanitize ELF files' RPATH found in there.
+
+        Sanitization behaves the same whatever the kindd of the processed tree, but
+        the resulting RPATH differs.
+
+        Sanitization action:
+        - remove RPATH pointing outside of the tree
+        - for RPATH pointing in the tree:
+          - if they point to standard location (/lib, /usr/lib): remove them
+          - otherwise: make them relative using \$ORIGIN
+
+        For the target tree:
+        - scan the whole tree for sanitization
+
+        For the staging tree :
+        - scan the whole tree for sanitization
+
+        For the host tree:
+        - skip the staging tree for sanitization
+        - add \$HOST_DIR/{lib,usr/lib} to RPATH (as relative pathes)
+
+Arguments:
+
+        TREE_KIND   Kind of tree to be processed.
+                    Allowed values: host, target, staging
+
+        TREE_ROOT   Path to the root of the tree to be scaned
+
+Environment:
+
+        HOST_READELF    readelf program to use for host ELF files
+                        (default: readelf)
+
+        HOST_READELF    readelf program to use for host ELF files
+                        (default: readelf)
+
+        PATCHELF        patchelf program to use
+                        (default: patchelf)
+EOF
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
+source.load_module patchelf
+
+: ${HOST_READELF:=readelf}
+: ${TARGET_READELF:=readelf}
+: ${PATCHELF:=patchelf}
+
+main() {
+    local tree="${1}"
+    local basedir="$(readlink -f "${2}")"
+
+    local find_args=( "${basedir}" )
+    local sanitize_extra_args=()
+    local readelf
+
+    case "${tree}" in
+        host)
+            # do not process the sysroot (only contains target binaries)
+            find_args+=( "-name" "sysroot" "-prune" "-o" )
+
+            # do not process the external toolchain installation directory to
+            # to avoid breaking it.
+            find_args+=( "-path" "*/opt/ext-toolchain" "-prune" "-o" )
+
+            # make sure RPATH will point to ${hostdir}/lib and ${hostdir}/usr/lib
+            sanitize_extra_args+=( "keep_lib_and_usr_lib" )
+
+            readelf="${HOST_READELF}"
+            ;;
+        staging|target)
+            readelf="${TARGET_READELF}"
+            ;;
+        *)
+            usage
+            exit 1
+            ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file ; do
+        READELF="${readelf}" PATCHELF="${PATCHELF}" \
+            patchelf.sanitize_rpath "${basedir}" "${file}" ${sanitize_extra_args[@]}
+    done < <(find ${find_args[@]} | readelf.filter_elf)
+}
+
+main ${@}
support/scripts/shell/log.sh new file +61 -0

61 lines not on this page · raw message

support/scripts/shell/patchelf.sh new file +178 -0

178 lines not on this page · raw message

support/scripts/shell/readelf.sh new file +173 -0

173 lines not on this page · raw message

support/scripts/shell/sdk.sh new file +68 -0

68 lines not on this page · raw message

support/scripts/shell/source.sh new file +77 -0

77 lines not on this page · raw message

support/scripts/shell/utils.sh new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
2.8.0
#8 lore raw

PATCH v9 3/11 support/scripts: add fix-rpath script + a bunch of helpers

Quoted 6 lines · Samuel Martin

This commit introduces a fix-rpath shell-script able to scan a tree, detect ELF files, check their RPATH and fix it in a proper way.

Along to the fix-rpath script, it also adds a bunch of shell helper functions grouped into modules. This will help writing scripts

handling

Quoted 4 lines

RPATH and other things, while allowing to share and reuse these functions between scripts.

These helpers are namespaced within the filename of the module in

which

Quoted 4 lines

they are gathered.

This change adds 6 modules:
- source.sh: provides simple helper to easily source another module,

taking

Quoted 11 lines

care of not sourcing again an already-sourced one;
- log.sh: provides logging helpers;
- utils.sh: provides simple functions to filter ELF files in a list;
- readelf.sh: provides functions retrieving ELF details from a file;
- patchelf.sh: provides function updating ELF files;
- sdk.sh: provides RPATH computation functions.

These 6 modules are used by the fix-rpath script. Follow-up patches will make some scripts leveraging these modules.

Quoted 45 lines

---
changes v8->v9:
- none

changes v7->v8:
- slightly rework log.error (return non-0 instead of exiting; exit decision is deferred to the caller)

changes v6->v7 (includes changes started during BR dev. days):
- log module:
- debug env. var. renamed: DEBUG -> SHELL_DEBUG
- trace now prints the message level
- patchelf module:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- set_rpath now handles ELF files with no .dynamic section
- cleanup and minor fix in sanitize_rpath
- readelf module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_sections and has_section functions (needed in sanitize_rpath)
- sdk modules:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- minor functions doc. fixes
- source module:
- move "Could not load module" error message in load_module (Arnout)
- utils module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_has function
- fix-rpath:
- runs with "set -e" (exit on error)
- improve help text
- slight refactoring because of modules changes

changes v5->v6:
- fully rewritten in shell

changes v4->v5:
- add verbose support
- rename shrink_rpath -> clear_rpath
- add sanitize_rpath function

changes v3->v4:
- fix typos and license (Baruch)

Quoted 6 lines

changes v2->v3:
- no change

---
 support/scripts/fix-rpath         | 116 +++++++++++++++++++++++++
 support/scripts/shell/log.sh      |  61 +++++++++++++
 support/scripts/shell/patchelf.sh | 178 

++++++++++++++++++++++++++++++++++++++

Quoted 1 line
 support/scripts/shell/readelf.sh  | 173 

++++++++++++++++++++++++++++++++++++

Quoted 12 lines
 support/scripts/shell/sdk.sh      |  68 +++++++++++++++
 support/scripts/shell/source.sh   |  77 +++++++++++++++++
 support/scripts/shell/utils.sh    |  60 +++++++++++++
 7 files changed, 733 insertions(+)
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

I tested the fix-rpath script on multiple toolchains, and it successfully made them relocatable. One thing I did notice was that once the TREE_ROOT was set and called with the script once, I could no longer call the script to set a more specific TREE_ROOT, i.e. moving ${O} down a folder. This isn't a huge deal, but did mean I needed to rebuild the toolchain before packaging it the way I wanted to. Another option would be trim automatically, so if the user selected a TREE_ROOT as $(some_path)/foo, but there weren't any elfs until a depth of $(some_path)/foo/bar/cow, automatically readjust the TREE_ROOT to be the more specific path.

This is a minor issue though, and it was pretty handy to be able to make relocatable host tools so I think functionally it is still very useful as is.

Also there were quite a few spelling typos in the scripts themselves / commit messages, so I'd run it through a quick spellcheck as well.

Cheers,
Ben

#9 lore raw

PATCH v9 4/11 core: sanitize HOST_DIR at the very end of the build

In reply to Samuel Martin

This patch adds host-patchelf to the list of package to build, and use it in the SANITIZE_RPATH_HOST helper.

This SANITIZE_RPATH_HOST helper is executed in the world target, which ensure:
- to be built after the target-post-image target (because of the dependency);
- to always be built since it is the only dependency of the all (and default) target.

The SANITIZE_RPATH_HOST helper fixes the ELF files' RPATH from the HOST_DIR location (excluding the sysroot).

After running this helper is run, the RPATH from any host ELF files is relative to the binary location itself.

Notes:
- we avoid to fix RPATH in the sysroot.
- we do not try to fix RPATH in the external toolchain installation location as they may have been built in a way, this is already correct; furthermore, attempting to fix RPATH in those programs may result in breaking them.
- the whole host directory is processed because a number of host-package install programs that could be useful in places different from $(HOST_DIR)/{bin,sbin,usr/bin,usr/sbin}.
- the shared libraries are also processed in case they have a 'main' function.

As a step toward a fully relocatable SDK, this change allows to get the toolchain relocatable, but not yet the whole SDK.

---
changes v8->v9:
- remove irrelevant comment (ThomasP)
- move the helper call to the world target instead of as a target-finalize hooks (ThomasP)

changes v7->v8:
- none

changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile, now just place the variable in the Buildroot environment, to enable debug logs. This is documented in a follow-up patch of this series.

changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- add verbose support

changes v3->v4:
- add host-patchelf to PACKAGES instead of target-finalize (Baruch)
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)

---
 Makefile | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Makefile +13 -0
index a05b9e1..0c9bb98 100644
@@ -526,7 +526,20 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+PACKAGES += host-patchelf
+
+define SANITIZE_RPATH_HOST
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=readelf \
+	$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
 world: target-post-image
+	@$(call MESSAGE,"Rendering the SDK relocatable")
+	$(SANITIZE_RPATH_HOST)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars help \
Signature · 1 line
2.8.0
#10 lore raw

PATCH v9 5/11 core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

In reply to Samuel Martin

This patch introduces the TARGET_SANITIZE_RPATH_HOOK and STAGING_SANITIZE_RPATH_HOOK hooks fixing the ELF files' RPATH of binaries from, respectively, the TARGET_DIR and the STAGING_DIR locations.

It is a fair assumption that all target package has been built before reaching the target-finalize target; hence the execution of these hooks as TARGET_FINALIZE_HOOKS.

After running these hooks, the RPATH from any target ELF files from both the target and the staging locations won't contain any occurrence of the sysroot or some build locations.

---
changes v8->v9:
- none

changes v7->v8:
- none

changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile

changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- target hook now sanitizes the rpath (Baruch)
- add verbose support
- update comment

changes v3->v4:
- rebase
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)

---
 Makefile | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
Makefile +18 -0
index 0c9bb98..e92826f 100644
@@ -651,6 +651,24 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# Function sanitizing target/staging ELF files' RPATH.
+# i.e. it removes paths pointing to the staging or build location from the ELF
+# files' RPATH.
+define TARGET_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+	$(TOPDIR)/support/scripts/fix-rpath target $(TARGET_DIR)
+endef
+
+define STAGING_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+	$(TOPDIR)/support/scripts/fix-rpath staging $(STAGING_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += TARGET_SANITIZE_RPATH_HOOK \
+       STAGING_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
Signature · 1 line
2.8.0
#11 lore raw

PATCH v9 6/11 package/speex: remove no longer needed hook

In reply to Samuel Martin

Remove the LIBTOOL_FIXUP hook since RPATH are now sanitized in target-finalize hooks.

---
changes v8->v9:
- remove hook addition (ThomasP)

changes v7->v8:
- none

changes v6->v7:
- none

changes v5->v6:
- none

changes v4->v5:
- new patch (suggested by Baruch)

---
 package/speex/speex.mk | 7 -------
 1 file changed, 7 deletions(-)
package/speex/speex.mk +0 -7
index a34bfac..86c5cef 100644
@@ -24,13 +24,6 @@ ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y)
 SPEEX_CONF_OPTS += --enable-arm5e-asm
 endif
 
-define SPEEX_LIBTOOL_FIXUP
-	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
-	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
-endef
-
-SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
-
 define SPEEX_BUILD_CMDS
 	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_MAKE_OPTS) -C $(@D)/$($(PKG)_SUBDIR)
 endef
Signature · 1 line
2.8.0
#12 lore raw

PATCH v9 7/11 support/scripts: update check-host-rpath to use the shell helpers

In reply to Samuel Martin

This change moves and refactors the helper functions from the check-host-rpath into the readelf module.

The replacement of the elf_needs_rpath function: readelf.needs_rpath does not fundamentally change the implementation, it only uses the APIs offered by the readelf module.

The replacement of the check_elf_has_rpath function: readelf.has_rpath does a bit more than the original implementation - adding the support for relative RPATH (those strating with '$ORIGIN').

Cc "Yann E. MORIN" <[email protected]>

---
changes v8->v9:
- update commit log (ThomasP)
- minor fix

changes v7->v8:
- none

changes v6->v7:
- minor updates after shell modules shell changes

changes v5->v6:
- new patch

---
 support/scripts/check-host-rpath | 73 +++++++++++++++-------------------------
 support/scripts/shell/readelf.sh | 69 +++++++++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 45 deletions(-)
support/scripts/check-host-rpath +28 -45
index 6ce547c..e6e82d6 100755
@@ -1,12 +1,29 @@
 #!/usr/bin/env bash
 
+# Copyright (C) 2015 Yann E. MORIN <[email protected]>
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
 # This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
 # they have an RPATH to $(HOST_DIR)/usr/lib if they need libraries from
 # there.
 
-# Override the user's locale so we are sure we can parse the output of
-# readelf(1) and file(1)
-export LC_ALL=C
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
 
 main() {
     local pkg="${1}"
@@ -14,58 +31,24 @@ main() {
     local file ret
 
     # Remove duplicate and trailing '/' for proper match
-    hostdir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}" )"
+    hostdir="$(sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}")"
 
     ret=0
     while read file; do
-        elf_needs_rpath "${file}" "${hostdir}" || continue
-        check_elf_has_rpath "${file}" "${hostdir}" && continue
+        READELF=readelf readelf.needs_rpath "${file}" "${hostdir}" || continue
+        READELF=readelf readelf.has_rpath "${file}" "${hostdir}" && continue
         if [ ${ret} -eq 0 ]; then
             ret=1
             printf "***\n"
-            printf "*** ERROR: package %s installs executables without proper RPATH:\n" "${pkg}"
+            printf \
+                "*** ERROR: package %s installs executables without proper RPATH:\n" \
+                "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <( find "${hostdir}"/{,usr/}{bin,sbin} -type f -exec file {} + 2>/dev/null \
-              |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d'                \
-                      -e 's//\1/'                                                  \
-            )
+    done < <(find "${hostdir}"/{,usr/}{bin,sbin} -type f -print 2>/dev/null |
+              readelf.filter_elf_executable)
 
     return ${ret}
 }
 
-elf_needs_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local lib
-
-    while read lib; do
-        [ -e "${hostdir}/usr/lib/${lib}" ] && return 0
-    done < <( readelf -d "${file}"                                         \
-              |sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
-                     -e 's//\1/;'                                          \
-            )
-
-    return 1
-}
-
-check_elf_has_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local rpath dir
-
-    while read rpath; do
-        for dir in ${rpath//:/ }; do
-            # Remove duplicate and trailing '/' for proper match
-            dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" ] && return 0
-        done
-    done < <( readelf -d "${file}"                                              \
-              |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
-                      -e 's//\3/;'                                              \
-            )
-
-    return 1
-}
-
 main "${@}"
support/scripts/shell/readelf.sh +69 -0

86 lines not on this page · raw message

Signature · 1 line
2.8.0
#13 lore raw

PATCH v9 8/11 support/scripts: add check-host-leaks script + all needed helpers

In reply to Samuel Martin

---
changes v8->v9:
- none

changes v7->v8:
- move sdk.check_host_leaks in the check-host-leaks script itself (Arnout)
- drop leak classification (Arnout)
- rework readelf.is_elf_{static_library,object} (Arnout)
- add utils.assert_absolute_canonical_path
- drop utils.guess_gnu_target_name
- rework/simplify the check-host-leaks script command line

changes v6->v7:
- {filer,is}_elf* functions moved from utils to readelf module
- update sdk.check_host_leaks

changes v5->v6:
- new patch

---
 support/scripts/check-host-leaks | 126 +++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh |  77 ++++++++++++++++++++++--
 support/scripts/shell/sdk.sh     |   7 +++
 support/scripts/shell/utils.sh   |  14 +++++
 4 files changed, 218 insertions(+), 6 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
support/scripts/check-host-leaks new file +126 -0
new file mode 100755
index 0000000..9f363af
@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  local ret="${1:-0}"
+  cat <<EOF
+
+Usage:  ${0##*/} [options] ROOT_DIR LEAK_PATH...
+
+Description:
+
+        This script scans a tree for host paths leaked into it.
+        Prints out leaks alongside with the kind of leak.
+
+Arguments:
+
+        ROOT_DIR        Path to the root of the tree to be scanned
+
+        LEAK_PATH       Paths to check for occurence. Several paths can be
+                        passed.
+                        A LEAK_PATH must be an absolute canonical path.
+                        In the Buildroot context, the LEAK_PATH are Buildroot's
+                        TOPDIR, BASE_DIR, HOST_DIR and STAGING_DIR.
+
+Options:
+
+        --exclude=PATH
+                Excludes files from ROOT_DIR starting with PATH. Obviously, the
+                excluded paths should point to some sub-location of ROOT_DIR.
+                Can be set more than once.
+                An excluded path must be an absolute canonical path.
+
+
+EOF
+  return ${ret}
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module log
+source.load_module utils
+source.load_module sdk
+
+main() {
+    local root_dir
+    local class_leaks
+    local excluded=()
+    local leak_paths=()
+    while test ${#} -gt 0 ; do
+        case "${1}" in
+            --exclude=*) excluded+=( "${1#*=}" )
+                ;;
+            --exclude)   shift ; excluded+=( "${1}" )
+                ;;
+            -h|--help)
+                usage
+                exit 0
+                ;;
+            *)  if test -z "${root_dir}" ; then root_dir="${1}"
+                else leak_paths+=( "${1}" )
+                fi
+                ;;
+        esac
+        shift
+    done
+
+    excluded=( $(utils.list_reduce ${excluded[@]}) )
+    leak_paths=( $(utils.list_reduce ${leak_paths[@]}) )
+
+    # sanity checks
+    if test -z "${root_dir}" ; then
+        log.error "Not enough argument: no ROOT_DIR given.\n" ||
+            usage ${?} || exit ${?}
+    fi
+    if test ${#leak_paths[@]} -lt 1 ; then
+        log.error "Not enough argument. No LEAK_PATH given.\n" ||
+            usage ${?} || exit ${?}
+    fi
+
+    local path has_non_abspath
+    for path in ${leak_paths[@]} ; do
+        utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+    done
+    if test -n "${has_non_abspath}" ; then
+        log.error "Some LEAK_PATH are not absolute canonical paths.\n" ||
+            usage ${?} || exit ${?}
+    fi
+    has_non_abspath=
+    for path in ${excluded[@]} ; do
+        utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+    done
+    if test -n "${has_non_abspath}" ; then
+        log.error "Some excluded path are not absolute canonical paths.\n" ||
+            usage ${?} || exit ${?}
+    fi
+
+    local re_leaks="$(sed -re 's/^/(/ ; s/$/)/ ; s/ +/|/g' <<<"${leak_paths[*]}")"
+    local re_excl="$(sed -re 's/ +/|/g' <<<"${excluded[*]}")"
+    if test -n "${re_excl}" ; then
+      re_excl="\\:^(${re_excl}):d"
+    fi
+    pushd "${root_dir}" >/dev/null
+    local f
+    grep -raEl "${re_leaks}" . |
+        sed -re "${re_excl} ; s:^\.:${root_dir}:" |
+        while read f ; do
+            printf "%s\n" "${f}"
+        done | sort
+    popd >/dev/null
+}
+main "${@}"
support/scripts/shell/readelf.sh +71 -6

119 lines not on this page · raw message

support/scripts/shell/sdk.sh +7 -0
index b2f699c..6cb3bbe 100644
@@ -19,9 +19,16 @@
 # This module defines the following functions:
 #   sdk.compute_relative_path
 #   sdk.compute_rpath
+#   sdk.check_host_leaks
+#
+# This module is sensitive to the following environment variables:
+#   READELF
 
 source.declare_module sdk
 
+source.load_module utils
+source.load_module readelf
+
 # sdk.compute_relative_path basedir path start
 #
 # Computes and prints the relative path between $start and $path within $basedir.
support/scripts/shell/utils.sh +14 -0
index 9e9aaab..eac3158 100644
@@ -19,6 +19,7 @@
 # This module defines the following functions:
 #   utils.list_has
 #   utils.list_reduce
+#   utils.assert_absolute_canonical_path
 
 source.declare_module utils
 
@@ -58,3 +59,16 @@ utils.list_reduce() {
 
     echo ${lout[@]}
 }
+
+# utils.assert_absolute_canonical_path path
+#
+# Returns 0 if 'path' is the absolute canonical path, returns non-0
+# otherwise.
+#
+# If the test failed, an error message will be issued to stderr.
+#
+# path : path to be tested
+utils.assert_absolute_canonical_path() {
+    test "$(readlink -f "${1}")" = "${1}" ||
+        log.error "%s is not the absolute canonical path.\n" "${1}" >&2
+}
Signature · 1 line
2.8.0
#14 lore raw

PATCH v9 9/11 core: add check-leaks-in-{target, host, staging} targets

In reply to Samuel Martin

This new targets allow identifying what need to be fixed to fulfill the ultimate goal of removing any trace of the host in the target filesystem or the SDK.

---
changes v8->v9:
- none

changes v7->v8:
- update the check-host-leak script call

changes v6->v7:
- none

changes v5->v6:
- new patch

---
 Makefile             | 15 +++++++++++++++
 package/pkg-utils.mk |  5 +++++
 2 files changed, 20 insertions(+)
Makefile +15 -0
index e92826f..a986c49 100644
@@ -651,6 +651,21 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# Checks for build machine leaks into target filesystem, SDK
+check-leaks-in-target:
+	@$(call MESSAGE,Checking leaks in the target)
+	$(Q)$(call check-for-build-machine-leaks-in,$(TARGET_DIR))
+
+check-leaks-in-host:
+	@$(call MESSAGE,Checking leaks in the host)
+	$(Q)$(call check-for-build-machine-leaks-in,$(HOST_DIR),--exclude=$(STAGING_DIR))
+
+check-leaks-in-staging:
+	@$(call MESSAGE,Checking leaks in the staging)
+	$(Q)$(call check-for-build-machine-leaks-in,$(STAGING_DIR))
+
+check-leaks:	check-leaks-in-target check-leaks-in-host check-leaks-in-staging
+
 # Function sanitizing target/staging ELF files' RPATH.
 # i.e. it removes paths pointing to the staging or build location from the ELF
 # files' RPATH.
package/pkg-utils.mk +5 -0
index f88313a..8b80028 100644
@@ -150,3 +150,8 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET}
 	mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \
 	cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2)
 endef
+
+define check-for-build-machine-leaks-in
+	$(TOPDIR)/support/scripts/check-host-leaks $(1) \
+		$(TOPDIR) $(BASE_DIR) $(HOST_DIR) $(STAGING_DIR) $(2)
+endef
Signature · 1 line
2.8.0
#15 lore raw

PATCH v9 10/11 support/scripts/check-host-leaks: add option to classify leaks

In reply to Samuel Martin

This option can be useful to figure out patterns in the leaks.

---
changes v8->v9:
- update commit log

changes v7->v8:
- new patch. Move this feature in its own change-set as it seems a bit controversial.
- make leaks classification optional

---
 support/scripts/check-host-leaks | 59 ++++++++++++++++++++++++++++++++++++++--
 support/scripts/shell/readelf.sh |  1 -
 2 files changed, 57 insertions(+), 3 deletions(-)
support/scripts/check-host-leaks +57 -2
index 9f363af..bd92493 100755
@@ -45,6 +45,12 @@ Options:
                 Can be set more than once.
                 An excluded path must be an absolute canonical path.
 
+        --classify-leaks
+                Enables leak classification. Categories are defined according to
+                the matched files' type or ELF kinds.
+                For ELF files, the sections containing the leaks will displayed.
+                Enabling this option substantially increases the execution time
+                of the script.
 
 EOF
   return ${ret}
@@ -56,9 +62,51 @@ source.load_module log
 source.load_module utils
 source.load_module sdk
 
+classify_leak() {
+    local f="${1}" regexp="${2}"
+    local leak
+    if test -h "${f}" ; then leak="symlink"
+    elif readelf.is_elf "${f}" ; then
+        if readelf.is_elf_executable "${f}" ; then leak="ELF/exe"
+        elif readelf.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
+        elif readelf.is_elf_static_library "${f}" ; then leak="ELF/*.a"
+        elif readelf.is_elf_object "${f}" ; then
+            case "${f}" in
+                *.ko) leak="ELF/*.ko" ;;
+                *) leak="ELF/*.o" ;;
+            esac
+        else leak="ELF/?"
+        fi
+        local section
+        local sections=()
+        for section in $(readelf.list_sections "${f}") ; do
+            if readelf.string_section "${f}" "${section}" |
+                    grep -qaE "${regexp}" ; then
+                if ! utils.list_has ${section} ${sections[@]} ; then
+                    sections+=( "${section}" )
+                fi
+            fi
+        done
+        leak="${leak} [${sections[*]}]"
+    else
+        case "${f}" in
+            *-config) leak="*-config script" ;;
+            *.la) leak="*.la" ;;
+            *.pc) leak="*.pc" ;;
+            *.py) leak="*.py" ;;
+            *.pyc|*.pyo) leak="*.py[co]" ;;
+        esac
+    fi
+    if test -z "${leak}" ; then
+        leak="? [$(file -z "${f}" | sed -e 's/.*: //')]"
+    fi
+    printf "${leak}"
+}
+
+
 main() {
     local root_dir
-    local class_leaks
+    local classify_leaks
     local excluded=()
     local leak_paths=()
     while test ${#} -gt 0 ; do
@@ -67,6 +115,8 @@ main() {
                 ;;
             --exclude)   shift ; excluded+=( "${1}" )
                 ;;
+            --classify-leaks) classify_leaks=y
+                ;;
             -h|--help)
                 usage
                 exit 0
@@ -119,7 +169,12 @@ main() {
     grep -raEl "${re_leaks}" . |
         sed -re "${re_excl} ; s:^\.:${root_dir}:" |
         while read f ; do
-            printf "%s\n" "${f}"
+            if test -n "${classify_leaks}" ; then
+                local leak="$(classify_leak "${f}" "${regexp}")"
+                printf "%-70s : %-120s\n" "${leak}" "${f}"
+            else
+                printf "%s\n" "${f}"
+            fi
         done | sort
     popd >/dev/null
 }
support/scripts/shell/readelf.sh +0 -1
index b23666e..a7d790e 100644
@@ -54,7 +54,6 @@ source.declare_module readelf
 # environment:
 #   READELF: readelf program path
 readelf._match_elf_regexp() {
-    log._trace_func
     local regexp="${1}" file="${2}"
     LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
 }
Signature · 1 line
2.8.0
#16 lore raw

PATCH v9 10/11 support/scripts/check-host-leaks: add option to classify leaks

Quoted 22 lines · Samuel Martin

This option can be useful to figure out patterns in the leaks.

---
changes v8->v9:
- update commit log

changes v7->v8:
- new patch. Move this feature in its own change-set as it seems a bit controversial.
- make leaks classification optional

---
 support/scripts/check-host-leaks | 59 ++++++++++++++++++++++++++++++++++++++--
 support/scripts/shell/readelf.sh |  1 -
 2 files changed, 57 insertions(+), 3 deletions(-)
support/scripts/check-host-leaks +0 -0
index 9f363af..bd92493 100755

support/scripts/check-host-leaks Options: · line 51

Quoted 7 lines
support/scripts/check-host-leaks +4 -0
@@ -45,6 +45,12 @@ Options:
                 Can be set more than once.
                 An excluded path must be an absolute canonical path.
 
+        --classify-leaks
+                Enables leak classification. Categories are defined according to
+                the matched files' type or ELF kinds.
+                For ELF files, the sections containing the leaks will displayed.

s/will displayed/will be displayed

Quoted 100 lines
support/scripts/check-host-leaks +53 -2
+                Enabling this option substantially increases the execution time
+                of the script.
 
 EOF
   return ${ret}
@@ -56,9 +62,51 @@ source.load_module log
 source.load_module utils
 source.load_module sdk
 
+classify_leak() {
+    local f="${1}" regexp="${2}"
+    local leak
+    if test -h "${f}" ; then leak="symlink"
+    elif readelf.is_elf "${f}" ; then
+        if readelf.is_elf_executable "${f}" ; then leak="ELF/exe"
+        elif readelf.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
+        elif readelf.is_elf_static_library "${f}" ; then leak="ELF/*.a"
+        elif readelf.is_elf_object "${f}" ; then
+            case "${f}" in
+                *.ko) leak="ELF/*.ko" ;;
+                *) leak="ELF/*.o" ;;
+            esac
+        else leak="ELF/?"
+        fi
+        local section
+        local sections=()
+        for section in $(readelf.list_sections "${f}") ; do
+            if readelf.string_section "${f}" "${section}" |
+                    grep -qaE "${regexp}" ; then
+                if ! utils.list_has ${section} ${sections[@]} ; then
+                    sections+=( "${section}" )
+                fi
+            fi
+        done
+        leak="${leak} [${sections[*]}]"
+    else
+        case "${f}" in
+            *-config) leak="*-config script" ;;
+            *.la) leak="*.la" ;;
+            *.pc) leak="*.pc" ;;
+            *.py) leak="*.py" ;;
+            *.pyc|*.pyo) leak="*.py[co]" ;;
+        esac
+    fi
+    if test -z "${leak}" ; then
+        leak="? [$(file -z "${f}" | sed -e 's/.*: //')]"
+    fi
+    printf "${leak}"
+}
+
+
 main() {
     local root_dir
-    local class_leaks
+    local classify_leaks
     local excluded=()
     local leak_paths=()
     while test ${#} -gt 0 ; do
@@ -67,6 +115,8 @@ main() {
                 ;;
             --exclude)   shift ; excluded+=( "${1}" )
                 ;;
+            --classify-leaks) classify_leaks=y
+                ;;
             -h|--help)
                 usage
                 exit 0
@@ -119,7 +169,12 @@ main() {
     grep -raEl "${re_leaks}" . |
         sed -re "${re_excl} ; s:^\.:${root_dir}:" |
         while read f ; do
-            printf "%s\n" "${f}"
+            if test -n "${classify_leaks}" ; then
+                local leak="$(classify_leak "${f}" "${regexp}")"
+                printf "%-70s : %-120s\n" "${leak}" "${f}"
+            else
+                printf "%s\n" "${f}"
+            fi
         done | sort
     popd >/dev/null
 }
support/scripts/shell/readelf.sh +0 -2
index b23666e..a7d790e 100644
@@ -54,7 +54,6 @@ source.declare_module readelf
 # environment:
 #   READELF: readelf program path
 readelf._match_elf_regexp() {
-    log._trace_func
     local regexp="${1}" file="${2}"
     LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
 }
--

2.8.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#17 lore raw

PATCH v9 11/11 docs/manual: document how to debug shell script

In reply to Samuel Martin

The method describes here is only true for scripts using the shell modules from support/scripts/shell/*.

---
changes v8->v9:
- none

changes v7->v8:
- put instrumentation in a subsection (Arnout)
- minor rewording

changes v6->v7:
- new patch

---
 docs/manual/debugging-buildroot.txt | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
docs/manual/debugging-buildroot.txt +17 -0
index f575fc4..b809c2d 100644
@@ -5,6 +5,8 @@
 
 == Debugging Buildroot
 
+=== Running scripts before and after each build step
+
 It is possible to instrument the steps +Buildroot+ does when building
 packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
 the path of one or more scripts (or other executables), in a
@@ -41,3 +43,18 @@ The script has access to the following variables:
   - +BINARIES_DIR+: the place where all binary files (aka images) are
     stored
   - +BASE_DIR+: the base output directory
+
+=== Debugging shell scripts in Buildroot
+
+For shell scripts using the _shell modules_ (available in
++support/scripts/shell/+), an environment variable is available to trace
+the scripts:
+
+  - +SHELL_DEBUG+ adjustes the debug level, from 0 to 3 (only errors to
+    all debug messages)
+
+To enable these debug outputs:
+
+----
+SHELL_DEBUG=3 make
+----
Signature · 1 line
2.8.0
#18 lore raw

PATCH v9 11/11 docs/manual: document how to debug shell script

Quoted 1 line · Samuel Martin

The method describes here is only true for scripts using the shell

s/describes/described ?

Quoted 3 lines

modules from support/scripts/shell/*.

Quoted 54 lines

---
changes v8->v9:
- none

changes v7->v8:
- put instrumentation in a subsection (Arnout)
- minor rewording

changes v6->v7:
- new patch

---
 docs/manual/debugging-buildroot.txt | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
docs/manual/debugging-buildroot.txt +17 -1
index f575fc4..b809c2d 100644
@@ -5,6 +5,8 @@
 
 == Debugging Buildroot
 
+=== Running scripts before and after each build step
+
 It is possible to instrument the steps +Buildroot+ does when building
 packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
 the path of one or more scripts (or other executables), in a
@@ -41,3 +43,18 @@ The script has access to the following variables:
   - +BINARIES_DIR+: the place where all binary files (aka images) are
     stored
   - +BASE_DIR+: the base output directory
+
+=== Debugging shell scripts in Buildroot
+
+For shell scripts using the _shell modules_ (available in
++support/scripts/shell/+), an environment variable is available to trace
+the scripts:
+
+  - +SHELL_DEBUG+ adjustes the debug level, from 0 to 3 (only errors to
+    all debug messages)
+
+To enable these debug outputs:
+
+----
+SHELL_DEBUG=3 make
+----
--

2.8.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#19 lore raw

v9 cover Relocatable SDK/Build machine leaks: RPATH fixing

In reply to Samuel Martin

Ping?

Quoted 5 lines · Samuel Martin

Hi all,

Here is yet another round of the series aiming improving the

relocatability

Quoted 3 lines

of the SDK built by Buildroot.

Still slowly reducing the number of patch in the series, as they got

merged...

Quoted 1 line

For this round, just fixes from comments, nothing really noticeable,

except

Quoted 18 lines

maybe the move of the HOST_DIR RPATH sanitation.

For further details, check the per-commit changelog in each commit log.

Patches 1 and 2:
Fix the O variable definition.

Patches 3 to 5:
Add the fix-rpath script, then run it on the Buildroot host, target and staging trees.

Patch 6:
Clean-up speex package WRT its RPATH hook.

Patch 7:
Update and fix the check-host-rpath to re-use the shell helpers.

Quoted 7 lines

Patches 8 to 10:
Add means to track build machine leaks into the host, target and staging trees. These patches will allow to identify what remains to be fixed WRT build machine leaks.

Patches 11:
Update documentation about how to enable/adjust log level on script

using

Quoted 23 lines

the shell modules introduced in patch 3/11.

Regards,
Samuel

Samuel Martin (11):
core: split variables definition related to in/out-of-tree build from
O itself
core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
path
support/scripts: add fix-rpath script + a bunch of helpers
core: sanitize HOST_DIR at the very end of the build
core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
TARGET_FINALIZE_HOOKS
package/speex: remove no longer needed hook support/scripts: update check-host-rpath to use the shell helpers support/scripts: add check-host-leaks script + all needed helpers
core: add check-leaks-in-{target,host,staging} targets support/scripts/check-host-leaks: add option to classify leaks docs/manual: document how to debug shell script

Quoted 10 lines
 Makefile                            | 126 ++++++++++++---
 docs/manual/debugging-buildroot.txt |  17 ++
 package/pkg-utils.mk                |   5 +
 package/speex/speex.mk              |   7 -
 support/scripts/check-host-leaks    | 181 +++++++++++++++++++++
 support/scripts/check-host-rpath    |  73 ++++-----
 support/scripts/fix-rpath           | 116 ++++++++++++++
 support/scripts/shell/log.sh        |  61 +++++++
 support/scripts/shell/patchelf.sh   | 178 +++++++++++++++++++++
 support/scripts/shell/readelf.sh    | 306

++++++++++++++++++++++++++++++++++++

Quoted 13 lines
 support/scripts/shell/sdk.sh        |  75 +++++++++
 support/scripts/shell/source.sh     |  77 +++++++++
 support/scripts/shell/utils.sh      |  74 +++++++++
 13 files changed, 1223 insertions(+), 73 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh
Quoted 3 lines

--
2.8.0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160601/c47c2a90/attachment.html>

#20 lore raw

v9 cover Relocatable SDK/Build machine leaks: RPATH fixing

up!

and the shortcuts:
http://patchwork.ozlabs.org/patch/613853/
http://patchwork.ozlabs.org/patch/613854/
http://patchwork.ozlabs.org/patch/613855/
http://patchwork.ozlabs.org/patch/613856/
http://patchwork.ozlabs.org/patch/613857/
http://patchwork.ozlabs.org/patch/613858/
http://patchwork.ozlabs.org/patch/613860/
http://patchwork.ozlabs.org/patch/613859/
http://patchwork.ozlabs.org/patch/613862/
http://patchwork.ozlabs.org/patch/613861/
http://patchwork.ozlabs.org/patch/613863/

Happy review! ;-)

Quoted 92 lines · Samuel Martin

Ping?

Quoted 88 lines · Samuel Martin

Hi all,

Here is yet another round of the series aiming improving the relocatability
of the SDK built by Buildroot.

Still slowly reducing the number of patch in the series, as they got merged...
For this round, just fixes from comments, nothing really noticeable, except
maybe the move of the HOST_DIR RPATH sanitation.

For further details, check the per-commit changelog in each commit log.

Patches 1 and 2:
Fix the O variable definition.

Patches 3 to 5:
Add the fix-rpath script, then run it on the Buildroot host, target and staging trees.

Patch 6:
Clean-up speex package WRT its RPATH hook.

Patch 7:
Update and fix the check-host-rpath to re-use the shell helpers.

Patches 8 to 10:
Add means to track build machine leaks into the host, target and staging trees. These patches will allow to identify what remains to be fixed WRT build machine leaks.

Patches 11:
Update documentation about how to enable/adjust log level on script
using
the shell modules introduced in patch 3/11.

Regards,
Samuel

Samuel Martin (11):
core: split variables definition related to in/out-of-tree build from
O itself
core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
path
support/scripts: add fix-rpath script + a bunch of helpers
core: sanitize HOST_DIR at the very end of the build
core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
TARGET_FINALIZE_HOOKS
package/speex: remove no longer needed hook support/scripts: update check-host-rpath to use the shell helpers support/scripts: add check-host-leaks script + all needed helpers
core: add check-leaks-in-{target,host,staging} targets support/scripts/check-host-leaks: add option to classify leaks docs/manual: document how to debug shell script

 Makefile                            | 126 ++++++++++++---
 docs/manual/debugging-buildroot.txt |  17 ++
 package/pkg-utils.mk                |   5 +
 package/speex/speex.mk              |   7 -
 support/scripts/check-host-leaks    | 181 +++++++++++++++++++++
 support/scripts/check-host-rpath    |  73 ++++-----
 support/scripts/fix-rpath           | 116 ++++++++++++++
 support/scripts/shell/log.sh        |  61 +++++++
 support/scripts/shell/patchelf.sh   | 178 +++++++++++++++++++++
 support/scripts/shell/readelf.sh    | 306
++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh        |  75 +++++++++
 support/scripts/shell/source.sh     |  77 +++++++++
 support/scripts/shell/utils.sh      |  74 +++++++++
 13 files changed, 1223 insertions(+), 73 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

--
2.8.0

Signature · 1 line
Samuel
#21 lore raw

v9 cover Relocatable SDK/Build machine leaks: RPATH fixing

up!

Any feedback (review/comment/tests/etc) is more than welcome! ;-)

Quoted 115 lines · Samuel Martin

up!

and the shortcuts:
http://patchwork.ozlabs.org/patch/613853/
http://patchwork.ozlabs.org/patch/613854/
http://patchwork.ozlabs.org/patch/613855/
http://patchwork.ozlabs.org/patch/613856/
http://patchwork.ozlabs.org/patch/613857/
http://patchwork.ozlabs.org/patch/613858/
http://patchwork.ozlabs.org/patch/613860/
http://patchwork.ozlabs.org/patch/613859/
http://patchwork.ozlabs.org/patch/613862/
http://patchwork.ozlabs.org/patch/613861/
http://patchwork.ozlabs.org/patch/613863/

Happy review! ;-)

Quoted 92 lines · Samuel Martin

Ping?

Quoted 88 lines · Samuel Martin

Hi all,

Here is yet another round of the series aiming improving the relocatability
of the SDK built by Buildroot.

Still slowly reducing the number of patch in the series, as they got merged...
For this round, just fixes from comments, nothing really noticeable, except
maybe the move of the HOST_DIR RPATH sanitation.

For further details, check the per-commit changelog in each commit log.

Patches 1 and 2:
Fix the O variable definition.

Patches 3 to 5:
Add the fix-rpath script, then run it on the Buildroot host, target and staging trees.

Patch 6:
Clean-up speex package WRT its RPATH hook.

Patch 7:
Update and fix the check-host-rpath to re-use the shell helpers.

Patches 8 to 10:
Add means to track build machine leaks into the host, target and staging trees. These patches will allow to identify what remains to be fixed WRT build machine leaks.

Patches 11:
Update documentation about how to enable/adjust log level on script
using
the shell modules introduced in patch 3/11.

Regards,
Samuel

Samuel Martin (11):
core: split variables definition related to in/out-of-tree build from
O itself
core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
path
support/scripts: add fix-rpath script + a bunch of helpers
core: sanitize HOST_DIR at the very end of the build
core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
TARGET_FINALIZE_HOOKS
package/speex: remove no longer needed hook support/scripts: update check-host-rpath to use the shell helpers support/scripts: add check-host-leaks script + all needed helpers
core: add check-leaks-in-{target,host,staging} targets support/scripts/check-host-leaks: add option to classify leaks docs/manual: document how to debug shell script

 Makefile                            | 126 ++++++++++++---
 docs/manual/debugging-buildroot.txt |  17 ++
 package/pkg-utils.mk                |   5 +
 package/speex/speex.mk              |   7 -
 support/scripts/check-host-leaks    | 181 +++++++++++++++++++++
 support/scripts/check-host-rpath    |  73 ++++-----
 support/scripts/fix-rpath           | 116 ++++++++++++++
 support/scripts/shell/log.sh        |  61 +++++++
 support/scripts/shell/patchelf.sh   | 178 +++++++++++++++++++++
 support/scripts/shell/readelf.sh    | 306
++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh        |  75 +++++++++
 support/scripts/shell/source.sh     |  77 +++++++++
 support/scripts/shell/utils.sh      |  74 +++++++++
 13 files changed, 1223 insertions(+), 73 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

--
2.8.0

--
Samuel

Signature · 1 line
Samuel