Build host-libcap with libattr support

PATCH New Ben Boeckel 4 participants 11 messages Oct 24, 2015 – Nov 13, 2015

Share & tools

Ben Boeckel

author

2015-10-24 21:39 UTC #0 lore raw

cover Build host-libcap with libattr support

I've also sent upstream to always define these symbols, so future versions can drop the hard-coded host-attr dependency if wanted.

--Ben

Ben Boeckel (1):
libcap: depend on host-attr for the host

 package/attr/attr.mk     | 1 +
 package/libcap/libcap.mk | 8 ++++----
 2 files changed, 5 insertions(+), 4 deletions(-)

Signature · 1 line

2.6.2

Ben Boeckel

author

2015-10-24 21:39 UTC #1 lore raw

PATCH 1/1 libcap: depend on host-attr for the host

When a host-libcap is built, it is built without libattr support. However, libcap does not define some symbols based on this flag. This becomes a problem when a build-host tool is run (e.g., ls) which uses these symbols gets the host-libcap library via LD_LIBRARY_PATH.

In order to define all of the symbols, add libattr support on the host. The RAISE_SETFCAP setting must also be disabled since the required permissions are not available during install.

Signed-off-by Ben Boeckel <[email protected]>

---
 package/attr/attr.mk     | 1 +
 package/libcap/libcap.mk | 9 ++++-----
 2 files changed, 5 insertions(+), 5 deletions(-)

package/attr/attr.mk +1 -0

index a66891b..7dac7cf 100644
@@ -37,3 +37,4 @@ endef
 ATTR_POST_INSTALL_STAGING_HOOKS += ATTR_FIX_LIBTOOL_LA_LIBDIR
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))

package/libcap/libcap.mk +4 -5

index 54844f8..65057e5 100644
@@ -20,8 +20,7 @@ else
 LIBCAP_HAVE_LIBATTR = no
 endif
 
-# we don't have host-attr
-HOST_LIBCAP_DEPENDENCIES =
+HOST_LIBCAP_DEPENDENCIES = host-attr
 
 ifeq ($(BR2_STATIC_LIBS),y)
 LIBCAP_MAKE_TARGET = libcap.a

@@ -67,12 +66,12 @@ define LIBCAP_INSTALL_TARGET_CMDS
 endef
 
 define HOST_LIBCAP_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D) LIBATTR=no
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)
 endef
 
 define HOST_LIBCAP_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) LIBATTR=no DESTDIR=$(HOST_DIR) \
-		prefix=/usr lib=lib install
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(HOST_DIR) \
+		RAISE_SETFCAP=no prefix=/usr lib=lib install
 endef
 
 $(eval $(generic-package))

Signature · 1 line

2.6.2

Thomas Petazzoni

2015-10-26 08:58 UTC #2 lore raw

PATCH 1/1 libcap: depend on host-attr for the host

Dear Ben Boeckel,

Quoted 4 lines · Ben Boeckel

When a host-libcap is built, it is built without libattr support. However, libcap does not define some symbols based on this flag. This becomes a problem when a build-host tool is run (e.g., ls) which uses these symbols gets the host-libcap library via LD_LIBRARY_PATH.

Ok, so let me try to understand this a bit better. You say that if you have host-libcap built and installed in $(HOST_DIR), then when you build another package, if the build process of that package uses your system 'ls' (i.e /bin/ls), then this ls doesn't work properly because it uses $(HOST_DIR)/usr/lib/libacl.so instead of /usr/lib/libacl.so.

Is that correct ?

If that is correct, then I think your proposed change is not the right solution, because it is just a workaround for the specific situation, on your distribution, where ls gets linked against an acl that has attr support. A similar problem can appear with other binaries/libraries, and we don't want to clutter our host package dependencies to solve such problems IMO.

Instead, I believe we should look at removing $(HOST_DIR)/usr/lib from LD_LIBRARY_PATH, i.e do something like:

package/Makefile.in +0 -2

index 8a592d4..10f3a47 100644
@@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
        PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
        PKG_CONFIG_SYSROOT_DIR="/" \
        PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig" \
-       LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
        INTLTOOL_PERL=$(PERL)
 
 HOST_MAKE_ENV = \
        PATH=$(BR_PATH) \
-       LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
        PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
        PKG_CONFIG_SYSROOT_DIR="/" \
        PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"

We do build all host binaries in $(HOST_DIR) with a rpath that points to $(HOST_DIR)/usr/lib, so this LD_LIBRARY_PATH should not be necessary. And this LD_LIBRARY_PATH is indeed wrong as it means that any system binary will first use the Buildroot-built host libraries rather than the system ones, as you experienced. These LD_LIBRARY_PATH were introduced back in February 2010, at a time when we were not building host binaries with a rpath. So I believe they should by now be unnecessary, and as you found out, potentially harmful.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Ben Boeckel

author

2015-10-26 14:27 UTC #3 lore raw

PATCH 1/1 libcap: depend on host-attr for the host

Quoted 6 lines · Thomas Petazzoni

Quoted 4 lines · Ben Boeckel
When a host-libcap is built, it is built without libattr support. However, libcap does not define some symbols based on this flag. This becomes a problem when a build-host tool is run (e.g., ls) which uses these symbols gets the host-libcap library via LD_LIBRARY_PATH.

Quoted 7 lines

Ok, so let me try to understand this a bit better. You say that if you have host-libcap built and installed in $(HOST_DIR), then when you build another package, if the build process of that package uses your system 'ls' (i.e /bin/ls), then this ls doesn't work properly because it uses $(HOST_DIR)/usr/lib/libacl.so instead of /usr/lib/libacl.so.
Is that correct ?

Yes.

Quoted 9 lines

If that is correct, then I think your proposed change is not the right solution, because it is just a workaround for the specific situation, on your distribution, where ls gets linked against an acl that has attr support. A similar problem can appear with other binaries/libraries, and we don't want to clutter our host package dependencies to solve such problems IMO.
Instead, I believe we should look at removing $(HOST_DIR)/usr/lib from LD_LIBRARY_PATH, i.e do something like:

<snip>

This sounds like a much better solution.

Thanks,

--Ben

Thomas Petazzoni

2015-10-27 06:29 UTC #4 lore raw

PATCH 1/1 libcap: depend on host-attr for the host

Ben,

Quoted 7 lines · Ben Boeckel

Quoted 7 lines
If that is correct, then I think your proposed change is not the right solution, because it is just a workaround for the specific situation, on your distribution, where ls gets linked against an acl that has attr support. A similar problem can appear with other binaries/libraries, and we don't want to clutter our host package dependencies to solve such problems IMO.

Quoted 5 lines

Quoted 2 lines
Instead, I believe we should look at removing $(HOST_DIR)/usr/lib from LD_LIBRARY_PATH, i.e do something like:
<snip>
This sounds like a much better solution.

Does this mean you will send a patch for it, after doing some testing ?

Thanks!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Ben Boeckel

author

2015-10-29 02:49 UTC #5 lore raw

PATCH 1/1 libcap: depend on host-attr for the host

Quoted 1 line · Thomas Petazzoni

Does this mean you will send a patch for it, after doing some testing ?

Patch sent. Tested locally by building util-linux.

Thanks,

--Ben

Ben Boeckel

author

2015-10-29 02:48 UTC #6 lore raw

PATCH 1/1 makefile: remove LD_LIBRARY_PATH from the environment

In reply to Ben Boeckel

If system tools are selected, the host's lib/ directory may shadow libraries from the system which are configured differently and do not have all of the symbols required by the system tool.

Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be necessary anyways.

Signed-off-by Ben Boeckel <[email protected]>

---
 package/Makefile.in | 2 --
 1 file changed, 2 deletions(-)

package/Makefile.in +0 -2

index 8a592d4..10f3a47 100644
@@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig" \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
 	INTLTOOL_PERL=$(PERL)
 
 HOST_MAKE_ENV = \
 	PATH=$(BR_PATH) \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"

Signature · 1 line

2.6.2

Ben Boeckel

author

2015-11-13 01:37 UTC #7 lore raw

PATCH 1/1 makefile: remove LD_LIBRARY_PATH from the environment

Quoted 8 lines · Ben Boeckel

If system tools are selected, the host's lib/ directory may shadow libraries from the system which are configured differently and do not have all of the symbols required by the system tool.
Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be necessary anyways.
Signed-off-by Ben Boeckel <[email protected]>

Bump.

--Ben

Arnout Vandecappelle

2015-11-13 06:37 UTC #8 lore raw

PATCH 1/1 makefile: remove LD_LIBRARY_PATH from the environment

In reply to Ben Boeckel

Quoted 8 lines · Ben Boeckel

If system tools are selected, the host's lib/ directory may shadow libraries from the system which are configured differently and do not have all of the symbols required by the system tool.
Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be necessary anyways.
Signed-off-by Ben Boeckel <[email protected]>

Acked-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

As I mentioned in a discussion with Yann (don't remember if it was on IRC or on
the list), I think we should apply this now no master to see what the autobuilders say. We currently break the build on Fedora 23 for any configuration including libattr, which IMHO is worse than the (probably small) number of host-tools that are missing rpath.

If we apply this patch _now_, we still have a few weeks to look at autobuild
results and handle the fallout.

Note that all the host tools that are mentioned in Config.in.host will have to
be tested manually. Anybody up for that? Also the boot loaders will have to be tested manually since they're not handled by the autobuilders.

Regards,
Arnout

Quoted 22 lines

---
 package/Makefile.in | 2 --
 1 file changed, 2 deletions(-)

package/Makefile.in +0 -2

index 8a592d4..10f3a47 100644
@@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig" \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
 	INTLTOOL_PERL=$(PERL)
 
 HOST_MAKE_ENV = \
 	PATH=$(BR_PATH) \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if $(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Yann E. MORIN

2015-11-13 09:00 UTC #9 lore raw

PATCH 1/1 makefile: remove LD_LIBRARY_PATH from the environment

Quoted 51 lines · Arnout Vandecappelle

Quoted 8 lines · Ben Boeckel
If system tools are selected, the host's lib/ directory may shadow libraries from the system which are configured differently and do not have all of the symbols required by the system tool.
Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be necessary anyways.
Signed-off-by Ben Boeckel <[email protected]>
Acked-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
As I mentioned in a discussion with Yann (don't remember if it was on IRC or on
the list), I think we should apply this now no master to see what the autobuilders say. We currently break the build on Fedora 23 for any configuration including libattr, which IMHO is worse than the (probably small)
number of host-tools that are missing rpath.
If we apply this patch _now_, we still have a few weeks to look at autobuild
results and handle the fallout.
Note that all the host tools that are mentioned in Config.in.host will have to
be tested manually. Anybody up for that? Also the boot loaders will have to be
tested manually since they're not handled by the autobuilders.
Regards,
Arnout
package/Makefile.in HOST_CONFIGURE_OPTS = \ · line 300
Quoted 12 lines
---
 package/Makefile.in | 2 --
 1 file changed, 2 deletions(-)
package/Makefile.in +0 -0
index 8a592d4..10f3a47 100644
@@ -298,12 +298,10 @@ HOST_CONFIGURE_OPTS = \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 
	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig:$(HOST_DIR)/usr/share/pkgconfig"
\
package/Makefile.in HOST_CONFIGURE_OPTS = \ · line 301
Quoted 1 line
package/Makefile.in +0 -1
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if
$(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \
Quoted 2 lines
package/Makefile.in +0 -0
 	INTLTOOL_PERL=$(PERL)
 

Quoted 21 lines

package/Makefile.in HOST_CONFIGURE_OPTS = \

Quoted 3 lines

package/Makefile.in +0 -1

 HOST_MAKE_ENV = \
 	PATH=$(BR_PATH) \
-	LD_LIBRARY_PATH="$(HOST_DIR)/usr/lib$(if

$(LD_LIBRARY_PATH),:$(LD_LIBRARY_PATH))" \

Quoted 4 lines

package/Makefile.in +0 -0

 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	PKG_CONFIG_SYSROOT_DIR="/" \
 	PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"

Signature · 10 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Ben, Arnout, All,

NAK for this one is the current state.

I have been working on it in the past few days (current branch not pushed, I'm away... Will push tonight):
http://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/no-ld-library-path

We still have packages that are missing the rpath, but that's not the only issue.

Look at libcurl: we're fiddling with LD_LIBRARY_PATH. Since we're setting it today, it is not empty. But if we no longer set it, we'd have the current directory in it (see the dependency check we do on it too for info).

So we should at least fix the known issues first, which I'm working on... More on this tonight when I land back home...

Regards,
Yann E. MORIN (on the move).

Signature · 4 lines

Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151113/270389b1/attachment.html>

Yann E. MORIN

2015-11-13 22:19 UTC #10 lore raw

PATCH 1/1 makefile: remove LD_LIBRARY_PATH from the environment

Ben, Arnout, Peter, All,

+Peter on this, because it is important he reads Arnout's comments and my reply about it (for once, I agree with Arnout! ;-] ).

On 2015-11-13 10:00 +0100, Yann E. MORIN spake thusly:

Quoted 6 lines · Arnout Vandecappelle

On 29-10-15 03:48, Ben Boeckel wrote:
If system tools are selected, the host's lib/ directory may shadow libraries from the system which are configured differently and do not have all of the symbols required by the system tool.

Quoted 6 lines

Since buildroot uses rpath everywhere, LD_LIBRARY_PATH should not be necessary anyways.
Signed-off-by Ben Boeckel <[email protected]>
Acked-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

I've marked your patch as superseded in Pathwork, since I have now included it in my series:

    http://lists.busybox.net/pipermail/buildroot/2015-November/144362.html

Thanks! :-)

Quoted 5 lines

As I mentioned in a discussion with Yann (don't remember if it was on IRC or on
the list), I think we should apply this now no master to see what the autobuilders say. We currently break the build on Fedora 23 for any configuration including libattr, which IMHO is worse than the (probably small) number of host-tools that are missing rpath.

And we should normally no longer have any host tool missing an RPATH. I have built all of them now, and the one that was broken wrt. rpath, I fixed.

Quoted 2 lines

If we apply this patch _now_, we still have a few weeks to look at autobuild
results and handle the fallout.

And at worst, we can simply revert it. Even if the patch touches a critical aspect of the build system, it is very well contained, so will be very easy to revert at the last minute if need be.

Quoted 3 lines

Note that all the host tools that are mentioned in Config.in.host will have to
be tested manually. Anybody up for that? Also the boot loaders will have to be tested manually since they're not handled by the autobuilders.

Ah, I forgot about the bootloaders... ;-/

However, for the host tools, here's what I did;

    $ git grep -l -E '^\$\(eval \$\(host-'    \
    |sed 'magic-to-extract-pkg-name'        \

Quoted 1 line

host-pkg-list

    $ vi host-pkg-list  # remove gcc, binutils, gdb

    $ make $(cat host-pkg-list)

And after ~73min, all the host packages in the lit have been built, and none were installing executables missing an RPATH when they are using our host libs.

Note: openssl install libraries that are missing an RPATH: libssl.so and all the engines libraries. But that's in practice not an issue, since the executables that uses those libraries do have an RPATH (as they were able to find those libs in the first place!) so they will also find the dependencies of those libs.

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'