Adding an external toolchain to oe-core

Hi folks,

We have been working to support a Codesourcery Lite MIPS toolchain recently. RP__ (on IRC) asked me to write up the process such that others may benefit from our experiences.

Aside: I think this information would probably be best hosted on a wiki, so once we've made any required changes I will seek to post this on the OpenEmbedded wiki. Let me know where I should put it.

Toolchain addition writeup follows:

We wanted to add support in oe-core for a Codesourcery Lite MIPS toolchain. The toolchain we targetted was a legacy release based around gcc-4.2. Toolchain binaries and release notes can be obtained from the Codesourcery website:

http://www.codesourcery.com/sgpp/lite/mips/portal/subscription3130

# Step 0: install the toolchain on your development host

The recipes we create for the external toolchain expect that the toolchain has been downloaded from the Codesourcery website and installed onto the development host.

We downloaded the tarball version of the toolchain and installed it under /usr.

# Step 1: create recipes for the new toolchain

Taking the existing recipes for the CSL 2008q3 as a template, we cloned the following files:

   cp meta/conf/distro/include/tcmode-external-csl2008q3.inc \
      meta/conf/distro/include/tcmode-external-csl-mips-4.2-177.inc

   cp meta/recipes-core/meta/external-csl-toolchain_2008q3-72.bb \
      meta/recipes-core/meta/external-csl-toolchain_mips-4.2-177.bb

Unlike OpenEmbedded (or Poky), oe-core doesn't directly make use of the "distro" concept; and so toolchain and libc choices are controlled directly from local.conf. To use the new MIPS toolchain, add the following to local.conf:

TCMODE = "external-csl-mips-4.2-177"
TCLIBC = "glibc"

# Step 2: fix up toolchain recipe bitrot

The CSL 2008q3 recipe we cloned seems to have bitrotted a bit relative to the current oe-core tree. We had to make the following changes to allow our new mips-4.2 recipes to work:

* Rework the do_configure_prepend() function in libc-package.bbclass as a task added to the queue before do_configure(). This allowed us to override the function in the toolchain recipe which inherits from libc-package.

* Add a LIC_FILES_CHECKSUM variable to the toolchain recipe.

* Replace hardcoded target prefixes with TARGET_PREFIX.

* Add libgcc and compilerlibs to the toolchain PROVIDES list.

# Step 3: tell OE that the toolchain provides locale information

The CSL MIPS toolchain provides locale information out of the box, so we modified the toolchain recipe to suit by setting GLIBC_INTERNAL_USE_BINARY_LOCALE to "precompiled". You have to also set ENABLE_BINARY_LOCALE_GENERATION to "0" in local.conf to prevent this setting getting overridden, though!

# Step 4: hack around the non-standard toolchain naming convention

OE expects toolchain naming to align with the target prefix, e.g.

arch-vendor-os-

The CSL toolchain is named slightly differently:

mips-linux-gnu-

To work around this we set the variables TARGET_VENDOR and TARGET_OS appropriately in the toolchain include file:

TARGET_VENDOR = ""
TARGET_OS = "linux-gnu"

To allow us to do this without tclibc-glibc.inc overriding the setting of TARGET_OS we had to change the assignment of TARGET_OS in tclibc-glibc.inc (to use "?=" rather than "=").

In order to have the rest of OE recognise our new target prefix, we had to modify two lookup tables: one in siteinfo.bbclass, and one in insane.bbclass.

# Step 5: fix up packages which depend on features CSL doesn't have

The CSL MIPS toolchain is built around sanitised kernel headers from a 2.6.24 kernel. Various packages in the oe-core tree play poorly with these headers, so we had to make a series of fixes for these issues.

* Update to util-linux-2.19.1 for changes to better deal with kernels which don't support fallocate.

* Add extra patches to the OpenSSL configure script to support a linux-gnu-mips target.

* Add a missing header include in glib for targets which have no inline asm "atomic" implementation in the glib tree. MIPS is such a target!

* Use busybox mdev rather than udev. Udev wants libc to support O_CLOEXEC, which isn't the case for the libc headers included in the CSL toolchain.

# Step 6: endianness specification

Typically cross-compiling MIPS toolchains reflect which endianness they support by default in their naming, e.g. mipsel for little endian targets, and mips for big endian targets.

The CSL MIPS toolchain supports both little and big endian targets with the same toolchain with the "mips" prefix. Endianness is specified using the -EL and -EB arguments to gcc.

OE doesn't expect this to be the case, so we had to make some modifications to ensure that the endianness arguments were correctly provided to gcc throughout the build.

Tom Parkin
Senior Software Engineer
Pace Plc

Bringing Technology Home

Victoria Road, Saltaire, West Yorkshire, BD18 3LF, United Kingdom. www.pace.com

Think before you print! Please consider the environment before printing this e-mail.


This message has been scanned for viruses by BlackSpider MailControl. - www.blackspider.com

Hi,

I really appreciate this writeup, thanks!

Before we put this on the wiki do you think you'd be able to help us clean up this process as I think we can make this easier for people?

If you are willing, I'd like to propose we drop the old 2008q3 .bb files and replace them with ones from the toolchain you've worked with so its a working example.

I'd then like to find ways to allow this to work "out the box" rather than needing as much hacking to OE-Core such as the TARGET_OS change.

These issues should get addressed if you were able to provide new toolchain recipes?

Ouch, we should find a way for the local.conf setting not to be needed.

This is probably going to be the hardest issue to fix. I would like to find a way to do it but that variable is tricky to handle for various reasons...

I wonder if we can use some wildcards or some other mechanism to avoid that.

Its certainly useful to document these but since they're version/toolchain specific, I think these are less interesting from an OE-Core perspective.

Could you get away with using more recent kernel headers or does that cause issues within the toolchain? I guess that ultimately depends what kernel version you're using for your target too.

Did you do this by changing TOOLCHAIN_OPTIONS or TARGET_CC_ARCH out of interest?

Cheers,

Richard

Hi Richard,

Thanks for your feedback. I've added specific comments inline below -- but basically, ACK to the idea of submitting the updated CSL mips-4.2 recipes for inclusion. I think some of the specifics discussed below will be clearer when you see the patches.

I have an upcoming demo in the pipeline which is demanding most of my attention at the moment, but I will endeavor to get a patchset submitted for early review this week.

That sounds like a good idea. I would caution that the recipes we've developed to date are merely "more working" than the 2008q3 versions, but I think they still represent a better starting place for new external toolchains.

Again, I agree.

Since looking at the mips 4.2 CSL toolchain I've gone on to try the same integration process with a few toolchains we have from silicon vendors -- this is my actual goal, as our projects typically have to make use of vendor-provided toolchains for support reasons.

My experience from this process suggests that each new toolchain requires a new quota of recipe hacks across the stack. The nature of the hacks depends on whether the specific kernel header / libc combination included in the toolchain gels well with the selection of packages in oe-core at the time.

So I think any effort to support a new pre-built toolchain will require some effort on the part of the developer. But anything that makes external toolchain support easier across the board would be useful.

Yes. I'll look into how we can fix this properly.

It's not clear to me whether the correct approach is:

* to subtly abuse the variables as described previously and then fix
up the side-effects
or:
* to modify oe-core to give it a more flexible rendering of the
arch/target/libc triplet.

FWIW, with the hacks we added to support the CSL toolchain we have been able to add support for a couple of other MIPS toolchains without issue, so perhaps the approach we followed was appropriate.

Quite possibly -- I'm sure more experienced bitbake/oe devs would be able to give pointers once we post patches.

Agreed. It is probably useful to document the types of issue one might expect to face in supporting a new external toolchain, but the specifics of the changes will vary from toolchain to toolchain as discussed.

It's really a case of what features your libc supports since that's what userspace is linking against at runtime. In some cases one can make use of kernel functionality that libc doesn't know about by means of direct syscall()s in code, but that's pretty ugly.

As a rule I'm more comfortable to take the toolchain as-is, sanitised kernel headers and all.

We did this by means of appending -EL to CXX and LD in the toolchain recipe. We then added bit of hackery in siteinfo_get_endianess() to look at CLAGS for the "mips" case to decide whether the target is little or big endian...

Thanks,
Tom

Tom Parkin
www.thesecretdogproject.com
Morality, like art, means drawing a line someplace /Wilde/