Skip to contentSkip to conversation

Fix TARGET_PATH variable with external toolchain

PATCH New Lionel Landwerlin 3 participants 10 messages Oct 24, 2009 – Oct 27, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Some time ago, an external toolchain improvement patch added $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH variable. This patch also removed $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin to TARGET_PATH which breaks the build of the WebKit package for example.

WebKit depends on the icu package, and when configuring the package, there is a need to run icu-config which is installed inside the staging directory.

---
 package/Makefile.in |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
package/Makefile.in +1 -1
index ccd7480..57bd6f5 100644
@@ -119,7 +119,7 @@ else
 TOOLCHAIN_EXTERNAL_PREFIX:=$(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_PREFIX))
 TOOLCHAIN_EXTERNAL_PATH:=$(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_PATH))
 TOOLCHAIN_DIR=$(BASE_DIR)/toolchain
-TARGET_PATH="$(HOST_DIR)/bin:$(HOST_DIR)/usr/bin:$(TOOLCHAIN_DIR)/bin:$(TOOLCHAIN_EXTERNAL_PATH)/bin:$(PATH)"
+TARGET_PATH="$(HOST_DIR)/bin:$(HOST_DIR)/usr/bin:$(TOOLCHAIN_DIR)/bin:$(TOOLCHAIN_EXTERNAL_PATH)/bin:$(STAGING_DIR)/bin:$(STAGING_DIR)/usr/bin:$(PATH)"
 #IMAGE:=$(BINARIES_DIR)/$(BR2_ROOTFS_PREFIX).$(TOOLCHAIN_EXTERNAL_PREFIX)$(ROOTFS_SUFFIX)
 IMAGE:=$(BINARIES_DIR)/$(BR2_ROOTFS_PREFIX).$(ARCH)$(ROOTFS_SUFFIX)
 
Signature · 1 line
1.6.4.3
#1 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Lionel" == Lionel Landwerlin <[email protected]> writes:

Lionel> Some time ago, an external toolchain improvement patch added Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH variable. Lionel> This patch also removed $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin Lionel> to TARGET_PATH which breaks the build of the WebKit package for Lionel> example.

Lionel> WebKit depends on the icu package, and when configuring the package, Lionel> there is a need to run icu-config which is installed inside the Lionel> staging directory.

I don't like this, as it mixes binaries/scripts for the host and the target - I think it's better for the icu package to install the host tool into host/usr/bin as well instead.

Signature · 1 line
Bye, Peter Korsgaard
#2 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Le samedi 24 octobre 2009 ? 20:37 +0200, Peter Korsgaard a ?crit :

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Lionel" == Lionel Landwerlin <[email protected]> writes:

Lionel> Some time ago, an external toolchain improvement patch added Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH variable. Lionel> This patch also removed $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin Lionel> to TARGET_PATH which breaks the build of the WebKit package for Lionel> example.

Quoted 8 lines

Lionel> WebKit depends on the icu package, and when configuring the package, Lionel> there is a need to run icu-config which is installed inside the Lionel> staging directory.

I don't like this, as it mixes binaries/scripts for the host and the target - I think it's better for the icu package to install the host tool into host/usr/bin as well instead.

First let me argue that when buildroot is not using an external toolchain, the $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin paths are a part of the TARGET_PATH variable.

But anyway, you're right, mixing binaries sucks.

Then, in this case, I don't think we can use the host script.

Let's right icu-config on my system :

djdeath at coalu:~$ icu-config --ldflags
-lm -L/usr/lib -licui18n -licuuc -licudata -lm

Same thing inside the staging directory :

djdeath at coalu:~/src/buildroot/buildroot_rebase$ ./output/staging/usr/bin/icu-config --ldflags -lpthread -lm -L/home/djdeath/src/buildroot/buildroot_rebase/output/staging/usr/lib -licui18n -licuuc -licudata -lpthread -lm

The script returns paths relative to the host. This is going to be a problem when linking the webkit binaries because the linker will have to deal with host binaries... That's the same kind of problem we are fixing with the libtool patch.

I'm wondering whether adding the following to package/webkit/webkit.mk should fix the problem :

WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

This way the configure script should use the pre-filled variable and not try to detect the icu-config's path. And we don't need to add the staging bin directories to the TARGET_PATH.

A long term fix could be to use pkg-config instead of self made tool. Patching/contributing to libicu is required.

Signature · 1 line
Lionel Landwerlin <[email protected]>
#3 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Lionel" == Lionel Landwerlin <[email protected]> writes:

Lionel> First let me argue that when buildroot is not using an external Lionel> toolchain, the $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin Lionel> paths are a part of the TARGET_PATH variable.

Hmm, you're right - what a mess. There's no good reason why these should be different in the regard.

Lionel> The script returns paths relative to the host. This is going to Lionel> be a problem when linking the webkit binaries because the Lionel> linker will have to deal with host binaries... That's the same Lionel> kind of problem we are fixing with the libtool patch.

Yes, but copying the script from staging to host shouldn't change that.

Lionel> I'm wondering whether adding the following to package/webkit/webkit.mk Lionel> should fix the problem :

Lionel> WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

If the webkit configure script handles that (didn't check), then that would certainly be the easiest/cleanest solution.

Signature · 1 line
Bye, Peter Korsgaard
#4 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Le samedi 24 octobre 2009 ? 22:22 +0200, Peter Korsgaard a ?crit :

Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Lionel" == Lionel Landwerlin <[email protected]> writes:

Lionel> First let me argue that when buildroot is not using an external Lionel> toolchain, the $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin Lionel> paths are a part of the TARGET_PATH variable.

Hmm, you're right - what a mess. There's no good reason why these should be different in the regard.

Quoted 6 lines

Lionel> The script returns paths relative to the host. This is going to Lionel> be a problem when linking the webkit binaries because the Lionel> linker will have to deal with host binaries... That's the same Lionel> kind of problem we are fixing with the libtool patch.

Yes, but copying the script from staging to host shouldn't change that.

Yeah, that an idea. I don't which one is the cleanest.

Quoted 9 lines

Lionel> I'm wondering whether adding the following to package/webkit/webkit.mk Lionel> should fix the problem :

Lionel> WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

If the webkit configure script handles that (didn't check), then that would certainly be the easiest/cleanest solution.

Ok, I just tested it and it works.

So the next step would probably be to remove $(STAGING_DIR)/bin and $(STAGING_DIR)/usr/bin from the TARGET_PATH with source built toolchains, but I'm pretty sure it's going to break some other packages.

Please find a patch as attachment.

Regards,

Signature · 9 lines
Lionel Landwerlin <[email protected]>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-package-Set-icu-config-path.patch
Type: text/x-patch
Size: 748 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20091025/02848f92/attachment.bin>
#5 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

In reply to Peter Korsgaard

Le Sat, 24 Oct 2009 20:37:57 +0200,
Peter Korsgaard <[email protected]> a ?crit :

Quoted 6 lines

Lionel> Some time ago, an external toolchain improvement patch added Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH Lionel> variable. This patch also removed $(STAGING_DIR)/bin and Lionel> $(STAGING_DIR)/usr/bin to TARGET_PATH which breaks the build Lionel> of the WebKit package for example.

Quoted 7 lines

Lionel> WebKit depends on the icu package, and when configuring the Lionel> package, there is a need to run icu-config which is Lionel> installed inside the staging directory.

I don't like this, as it mixes binaries/scripts for the host and the target - I think it's better for the icu package to install the host tool into host/usr/bin as well instead.

As part of my work on the package infrastructure, I converted icu over to the new format.

At first, I'd agree with you that icu-config, being executed on the host, should theorically be installed in $(HOST_DIR). However, this raises an issue : icu is compiled twice, once for the host, once for the target.

In the current package/icu/icu.mk, the host icu is not installed at all, but my package infrastructure work converts it to a more normal package where the host icu is installed in $(HOST_DIR). Therefore, the host icu installs its own icu-config in $(HOST_DIR)/usr/bin.

Then, we build the target icu, which also installs its own icu-config, in $(STAGING_DIR)/usr/bin. It is not possible to install this icu-config into $(HOST_DIR)/usr/bin, otherwise it would overwrite the host-icu one.

Therefore, I think we have no other choice than leaving the *-config stuff into $(STAGING_DIR)/usr/bin and then finding some trick to get them executed even if $(STAGING_DIR)/usr/bin is not in the PATH.

What do you think about this ?

Sincerly,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
#6 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Le mardi 27 octobre 2009 ? 08:44 +0100, Thomas Petazzoni a ?crit :

Quoted 35 lines

Le Sat, 24 Oct 2009 20:37:57 +0200,
Peter Korsgaard <[email protected]> a ?crit :

Quoted 6 lines

Lionel> Some time ago, an external toolchain improvement patch added Lionel> $(HOST_DIR)/bin and $(HOST_DIR)/usr/bin to the TARGET_PATH Lionel> variable. This patch also removed $(STAGING_DIR)/bin and Lionel> $(STAGING_DIR)/usr/bin to TARGET_PATH which breaks the build Lionel> of the WebKit package for example.

Quoted 7 lines

Lionel> WebKit depends on the icu package, and when configuring the Lionel> package, there is a need to run icu-config which is Lionel> installed inside the staging directory.

I don't like this, as it mixes binaries/scripts for the host and the target - I think it's better for the icu package to install the host tool into host/usr/bin as well instead.

As part of my work on the package infrastructure, I converted icu over to the new format.

At first, I'd agree with you that icu-config, being executed on the host, should theorically be installed in $(HOST_DIR). However, this raises an issue : icu is compiled twice, once for the host, once for the target.

In the current package/icu/icu.mk, the host icu is not installed at all, but my package infrastructure work converts it to a more normal package where the host icu is installed in $(HOST_DIR). Therefore, the host icu installs its own icu-config in $(HOST_DIR)/usr/bin.

Then, we build the target icu, which also installs its own icu-config, in $(STAGING_DIR)/usr/bin. It is not possible to install this icu-config into $(HOST_DIR)/usr/bin, otherwise it would overwrite the host-icu one.

Quoted 6 lines

Therefore, I think we have no other choice than leaving the *-config stuff into $(STAGING_DIR)/usr/bin and then finding some trick to get them executed even if $(STAGING_DIR)/usr/bin is not in the PATH.

What do you think about this ?

Sounds right to me.

Let's add

WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

to the webkit makefile.

#7 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Le Tue, 27 Oct 2009 10:45:07 +0100,
Lionel Landwerlin <[email protected]> a ?crit :

Quoted 7 lines

Sounds right to me.

Let's add

WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

to the webkit makefile.

Sure. But we'll have to remember that this is the solution to apply on all similar packages... for some of which a xxx_config configure options might not be present.

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
#8 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Le mardi 27 octobre 2009 ? 10:53 +0100, Thomas Petazzoni a ?crit :

Quoted 7 lines

Le Tue, 27 Oct 2009 10:45:07 +0100,
Lionel Landwerlin <[email protected]> a ?crit :

Quoted 4 lines

Sounds right to me.

Let's add

Quoted 7 lines
Quoted 3 lines

WEBKIT_CONF_ENV = icu_config=$(STAGING_DIR)/usr/bin/icu-config

to the webkit makefile.

Sure. But we'll have to remember that this is the solution to apply on all similar packages... for some of which a xxx_config configure options might not be present.

Right.
In my config I built the following packages :

atk-1.22.0
atk-1.22.0-host
autoconf-2.63-host
automake-1.10-host
buildroot-config
busybox-1.15.1
busybox-1.15.2
cairo-1.6.4
cairo-1.6.4-host
dbus-1.2.16
directfb-1.4.2
directfb-examples-1.2.0
eina-0.0.2.060
enchant-1.5.0
ethtool-6
expat-2.0.1
expat-2.0.1-host
fakeroot-1.9.5
fakeroot-1.9.5-host
fontconfig-2.6.0
fontconfig-2.6.0-host
freetype-2.3.9
freetype-2.3.9-host
gettext-0.16.1
icu
icu-host
iostat-2.2
jpeg-6b
libcurl-7.19.2
libecore-0.9.9.062
libe_dbus-0.5.0.062
libedje-0.9.92.062
libeet-1.2.2
libeina-0.0.2.062
libembryo-0.9.9.062
liberation-fonts-1.04
libevas-0.9.9.062
libglib2-2.20.5
libglib2-2.20.5-host
libgtk2-2.12.12
libgtk2-2.12.12-host
libpng-1.2.38
libsoup-2.26.2
libtool-1.5.24-host
libungif-4.1.4
libxml2-2.7.3
libxslt-1.1.24
lzo-2.03
m4-1.4.9-host
makedevs-host
mtd_orig
openssh-5.1p1
openssl-0.9.8k
pango-1.20.5
pango-1.20.5-host
pixman-0.10.0
pixman-0.10.0-host
pkg-config-0.23-host
sawman-1.4.2
sqlite-3.6.18
strace-4.5.18
sysstat-9.0.5
tiff-3.8.2
udev-114
webkit-r44552
zlib-1.2.3

By the way, is there a set of configuration files that cover the list of package that are known to work with buildroot ?

It could be interesting to setup some kind of non regression builds :)

Signature · 1 line
Lionel Landwerlin <[email protected]>
#9 lore raw

PATCH 1/1 Fix TARGET_PATH variable with external toolchain

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Lionel" == Lionel Landwerlin <[email protected]> writes:

Hi,

Lionel> By the way, is there a set of configuration files that cover Lionel> the list of package that are known to work with buildroot ?

Not a complete list, no. In concept, we would like all packages to work, but as the number of combinations with 600+ packages / 5+ archs is so high, there's bound to be issues.

Lionel> It could be interesting to setup some kind of non regression builds :)

Certainly. I have a buildbot instance running at where I work to test the configs we're using, and I'm planning on setting up a public buildbot soon.

I recently added make all{yes,no,random}packageconfig targets to make it easier to test random package combinations.

Signature · 1 line
Bye, Peter Korsgaard