Does support for external toolchains working in current OE?

For last few months I am working on fixing Linaro binary cross toolchains and their support in OpenEmbedded. Got to point when they work (both AArch64 and ARMv7a ones) for single packages but problem starts when I want to create image...

11:14 hrw@puchatek:build$ bitbake core-image-minimal -g
Parsing of 1302 .bb files complete (1294 cached, 8 parsed). 1670 targets, 45 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: multiple providers are available for virtual/gettext (proxy-libintl, gettext)
NOTE: consider defining a PREFERRED_PROVIDER entry to match virtual/gettext
NOTE: Preparing runqueue
ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
This usually means one provides something the other doesn't and should.
ERROR: Multiple .bb files are due to be built which each provide virtual/arm-linux-gnueabihf-libc-for-gcc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
This usually means one provides something the other doesn't and should.
ERROR: Multiple .bb files are due to be built which each provide virtual/libiconv (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
This usually means one provides something the other doesn't and should.
NOTE: PN build list saved to 'pn-buildlist'
NOTE: PN dependencies saved to 'pn-depends.dot'
NOTE: Package dependencies saved to 'package-depends.dot'
NOTE: Task dependencies saved to 'task-depends.dot'

Summary: There were 3 ERROR messages shown, returning a non-zero exit code.

All three virtual/ are provided by external-linaro-toolchain recipe...

Is there any external toolchain which allows to create image with current OpenEmbedded layers?

TCMODE="external-sourcery" gives exactly same problem.

This means that virtual/libc is providing something which the external toolchain recipes are not. Can you share the "bitbake -DDDv" output for the above command somewhere (I know there will be a lot of it). Buried somewhere in there will be the answer as it does tell you what is going on if you search for the right things.

W dniu 05.04.2013 11:54, Richard Purdie pisze:

Quoted 5 lines

This means that virtual/libc is providing something which the external toolchain recipes are not. Can you share the "bitbake -DDDv" output for the above command somewhere (I know there will be a lot of it). Buried somewhere in there will be the answer as it does tell you what is going on if you search for the right things.

http://tygrysek.juszkiewicz.com.pl/~hrw/tmp/rp-log.txt 3.4MB

Quoted 1 line

http://tygrysek.juszkiewicz.com.pl/~hrw/tmp/rp-log.txt 3.4MB

Some key bits of the log:

DEBUG: Added runtime dependency eglibc
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-dbg
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-dev
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-utils
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-thread-db
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-localedata-i18n
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-ibm850
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-cp1252
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-iso8859-1
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: Added runtime dependency eglibc-gconv-iso8859-15
for /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/packagegroups/packagegroup-core-standalone-sdk-target.bb
DEBUG: providers for eglibc-dev are: ['eglibc']
DEBUG: adding '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb' to satisfy runtime 'eglibc-dev'
DEBUG: providers for eglibc are: ['eglibc']
DEBUG: adding '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb' to satisfy runtime 'eglibc'
DEBUG: providers for eglibc-utils are: ['eglibc']

So where does this come from? packagegroup-core-standalone-sdk-target.bb leads us to LIBC_DEPENDENCIES which leads to:

$ cat conf/distro/include/tclibc-eglibc.inc

LIBC_DEPENDENCIES = "libsegfault \
		     eglibc \
		     eglibc-dbg \
		     eglibc-dev \
		     eglibc-utils \
		     eglibc-thread-db \
		     ${@get_libc_locales_dependencies(d)}"

LIBC_LOCALE_DEPENDENCIES = "\
	eglibc-localedata-i18n \
	eglibc-gconv-ibm850 \
	eglibc-gconv-cp1252 \
	eglibc-gconv-iso8859-1 \
	eglibc-gconv-iso8859-15"

so I suspect you either need to provide some of these things,

or reset those variables...

ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
This usually means one provides something the other doesn't and should.

So I got with this change:

lib/bb/providers.py +0 -0

index fcee6dc..0b37c44 100644

And when I added "eglibc" into PROVIDES in external-linaro-toolchain.bb

W dniu 05.04.2013 11:20, Marcin Juszkiewicz pisze:

Quoted 2 lines

ERROR: Multiple .bb files are due to be built which each provide virtual/libc (/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb).
This usually means one provides something the other doesn't and should.

Let's enable debug:

DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']
NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb']
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc

And now - let's dig deep into BitBake code. We want two files:

- lib/bb/providers.py
- lib/bb/taskdata.py

1. Bitbake jumps into taskdata/add_provider_internal().
2. Then asks bb.providers.filterProviders()
"is there any provider for 'virtual/libc'"?
3. In providers/_filterProviders this output is printed:
"DEBUG: providers for virtual/libc are: ['eglibc', 'external-linaro-toolchain']"
4. providers/filterProviders() (note lack of "_") checks for
PREFERRED_PROVIDER_virtual/libc and outputs:
NOTE: selecting external-linaro-toolchain to satisfy virtual/libc due to PREFERRED_PROVIDERS
sets foundUnique to True, outputs:
DEBUG: sorted providers for virtual/libc are: ['/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb', '/home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb'] and returns all providers + foundUnique
5. We are back at taskdata/add_provider_internal() and we
have more then one entry in "eligible" array plus
foundUnique set to True.
6. Here we are happy of results and skips two "if" checks.
7. Then we add each entry as good one with this output:
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/meta-linaro/meta-linaro-toolchain/recipes-devtools/external-linaro-toolchain/external-linaro-toolchain.bb to satisfy virtual/libc
DEBUG: adding /home/hrw/HDD/devel/canonical/aarch64/openembedded/repos/openembedded-core/meta/recipes-core/eglibc/eglibc_2.17.bb to satisfy virtual/libc

The question is - why providers/filterProviders() returns more then one entry when it founds that one of them is preferred?

For last few months I am working on fixing Linaro binary cross toolchains and their support in OpenEmbedded. Got to point when they work (both AArch64 and ARMv7a ones) for single packages but problem starts when I want to create image...

Quoted 4 lines · Marcin Juszkiewicz

For last few months I am working on fixing Linaro binary cross toolchains and their support in OpenEmbedded. Got to point when they work (both AArch64 and ARMv7a ones) for single packages but problem starts when I want to create image...

Today I got different king of issue with external toolchain.

systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
but host prefix in OE is not the same as in binary toolchain (different TARGET_VENDOR)
so it looks for configure:14545: checking for arm-foo-linux-gnueabi-objcopy
while binary toolchain has only arm-bar-linux-gnueabi-objcopy

and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
configure:14588: checking for objcopy
configure:14606: found /usr/bin/objcopy
and bam a lot later when it tries objcopy from host on some arm lib

Interesting that systemd is first recipe in our image where I've noticed issues like this

Any idea how to easily resolve this without checking every configure.ac how it's looking for e.g. objcopy?

Quoted 21 lines · Martin Jansa

Quoted 4 lines · Marcin Juszkiewicz

For last few months I am working on fixing Linaro binary cross toolchains and their support in OpenEmbedded. Got to point when they work (both AArch64 and ARMv7a ones) for single packages but problem starts when I want to create image...

Today I got different king of issue with external toolchain.

systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
but host prefix in OE is not the same as in binary toolchain (different TARGET_VENDOR)
so it looks for configure:14545: checking for arm-foo-linux-gnueabi-objcopy
while binary toolchain has only arm-bar-linux-gnueabi-objcopy

and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
configure:14588: checking for objcopy
configure:14606: found /usr/bin/objcopy
and bam a lot later when it tries objcopy from host on some arm lib

Quoted 5 lines

Interesting that systemd is first recipe in our image where I've noticed issues like this

Any idea how to easily resolve this without checking every configure.ac how it's looking for e.g. objcopy?

export OBJCOPY in the environment to the one based on TARGET_PREFIX rather than TARGET_SYS, and it'll use that.
--
Christopher Larson

I've already tried that, systemd's configure does not respect that.

Quoted 37 lines · Martin Jansa

Quoted 22 lines · Marcin Juszkiewicz

Quoted 4 lines

For last few months I am working on fixing Linaro binary cross toolchains and their support in OpenEmbedded. Got to point when they work (both AArch64 and ARMv7a ones) for single packages but problem starts when I want to create image...

Today I got different king of issue with external toolchain.

systemd is using AC_PATH_TOOL(OBJCOPY, objcopy)
but host prefix in OE is not the same as in binary toolchain (different TARGET_VENDOR)
so it looks for configure:14545: checking for arm-foo-linux-gnueabi-objcopy
while binary toolchain has only arm-bar-linux-gnueabi-objcopy

and because arm-foo-linux-gnueabi-objcopy wasn't found it continues:
configure:14588: checking for objcopy
configure:14606: found /usr/bin/objcopy
and bam a lot later when it tries objcopy from host on some arm lib

Quoted 5 lines

Interesting that systemd is first recipe in our image where I've noticed issues like this

Any idea how to easily resolve this without checking every configure.ac how it's looking for e.g. objcopy?

export OBJCOPY in the environment to the one based on TARGET_PREFIX rather than TARGET_SYS, and it'll use that.
--
Christopher Larson