Yocto development with C++11 threads and gcc

The program below built on the target with the MACHINE=beaglebone gcc-4.9.1 compiler from Yocto/OpenEmbedded poky master produces this error:

beaglebone[52]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
pure virtual method called
terminate called without an active exception Aborted (core dumped)

When the program is recompiled with the defines for __GCC_HAVE_SYNC_COMPARE_AND_SWAP_X enabled as suggested at https://groups.google.com/d/msg/automatak-dnp3/Jisp_zGhd5I/ck_Cj6nO8joJ it works:

beaglebone[53]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
doit
done

Preliminary analysis confirms that the built-ins for those defines are not being added by the compiler because it thinks the target doesn't support those operations. Nonetheless, it doesn't use the substitutes that are obviously available.

Can anybody recall anything about the way GCC is built under OE that would explain this?

Peter

/* g++ -std=c++1y -pthread test.cc && ./a.out */
#if 0
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
#endif

#include <iostream>
#include <thread>

void
doit ()
{
std::cerr << "doit\n";
}

int main (int argc,

          char * argv [])

{
std::cerr << "starting\n";
std::thread thr{doit};
std::cerr << "joining\n";
thr.join();
std::cerr << "done\n";
return 0;
}

Quoted 17 lines · Peter A. Bigot

The program below built on the target with the MACHINE=beaglebone gcc-4.9.1 compiler from Yocto/OpenEmbedded poky master produces this error:

beaglebone[52]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
pure virtual method called
terminate called without an active exception Aborted (core dumped)

When the program is recompiled with the defines for __GCC_HAVE_SYNC_COMPARE_AND_SWAP_X enabled as suggested at https://groups.google.com/d/msg/automatak-dnp3/Jisp_zGhd5I/ck_Cj6nO8joJ it works:

beaglebone[53]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
doit
done

Quoted 3 lines

Preliminary analysis confirms that the built-ins for those defines are not being added by the compiler because it thinks the target doesn't support those operations. Nonetheless, it doesn't use the substitutes that are obviously available.

Can anybody recall anything about the way GCC is built under OE that would explain this?

Not an OE problem. See: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=62100

I'm testing a patch locally and awaiting GCC maintainer comment before proposing it for OE.

The program below built on the target with the MACHINE=beaglebone gcc-4.9.1 compiler from Yocto/OpenEmbedded poky master produces this error:

beaglebone[52]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
pure virtual method called
terminate called without an active exception Aborted (core dumped)

When the program is recompiled with the defines for __GCC_HAVE_SYNC_COMPARE_AND_SWAP_X enabled as suggested at
https://groups.google.com/d/msg/automatak-dnp3/Jisp_zGhd5I/ck_Cj6nO8joJ it works:

beaglebone[53]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
doit
done

Preliminary analysis confirms that the built-ins for those defines are not being added by the compiler because it thinks the target doesn't support those operations. Nonetheless, it doesn't use the substitutes that are obviously available.

Can anybody recall anything about the way GCC is built under OE that would explain this?

Peter

/* g++ -std=c++1y -pthread test.cc && ./a.out */
#if 0
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
#endif

#include <iostream>
#include <thread>

void
doit ()
{
std::cerr << "doit\n";
}

int main (int argc,

          char * argv [])

{
std::cerr << "starting\n";
std::thread thr{doit};
std::cerr << "joining\n";
thr.join();
std::cerr << "done\n";
return 0;
}

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 21 lines · Peter A. Bigot

The program below built on the target with the MACHINE=beaglebone gcc-4.9.1 compiler from Yocto/OpenEmbedded poky master produces this error:

beaglebone[52]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
pure virtual method called
terminate called without an active exception Aborted (core dumped)

When the program is recompiled with the defines for __GCC_HAVE_SYNC_COMPARE_AND_SWAP_X enabled as suggested at
https://groups.google.com/d/msg/automatak-dnp3/Jisp_zGhd5I/ck_Cj6nO8joJ it works:

beaglebone[53]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
doit
done

Quoted 7 lines

Preliminary analysis confirms that the built-ins for those defines are not being added by the compiler because it thinks the target doesn't support those operations. Nonetheless, it doesn't use the substitutes that are obviously available.

Can anybody recall anything about the way GCC is built under OE that would explain this?

Can you add -mcpu=cortex-a8 to your cmdline and see if it solves the problem ?

Quoted 35 lines

Peter

/* g++ -std=c++1y -pthread test.cc && ./a.out */
#if 0
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
#endif

#include <iostream>
#include <thread>

void
doit ()

{
   std::cerr << "doit\n";
}

int main (int argc,

           char * argv [])
{
   std::cerr << "starting\n";
   std::thread thr{doit};
   std::cerr << "joining\n";
   thr.join();
   std::cerr << "done\n";
   return 0;
}

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 27 lines · Khem Raj

Quoted 26 lines · Peter A. Bigot

The program below built on the target with the MACHINE=beaglebone gcc-4.9.1 compiler from Yocto/OpenEmbedded poky master produces this error:

beaglebone[52]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
pure virtual method called
terminate called without an active exception Aborted (core dumped)

When the program is recompiled with the defines for __GCC_HAVE_SYNC_COMPARE_AND_SWAP_X enabled as suggested at
https://groups.google.com/d/msg/automatak-dnp3/Jisp_zGhd5I/ck_Cj6nO8joJ it works:

beaglebone[53]$ g++ -std=c++1y -pthread test.cc && ./a.out starting
joining
doit
done

Preliminary analysis confirms that the built-ins for those defines are not being added by the compiler because it thinks the target doesn't support those operations. Nonetheless, it doesn't use the substitutes that are obviously available.

Quoted 3 lines

Quoted 2 lines

Can anybody recall anything about the way GCC is built under OE that would explain this?

Can you add -mcpu=cortex-a8 to your cmdline and see if it solves the problem ?

Yes, it does. That's a good clue.

Peter

Quoted 35 lines

Quoted 35 lines

Peter

/* g++ -std=c++1y -pthread test.cc && ./a.out */
#if 0
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
#endif

#include <iostream>
#include <thread>

void
doit ()

{
   std::cerr << "doit\n";
}

int main (int argc,

           char * argv [])
{
   std::cerr << "starting\n";
   std::thread thr{doit};
   std::cerr << "joining\n";
   thr.join();
   std::cerr << "done\n";
   return 0;
}

Signature · 4 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

In any case, Khem can you run with this? It'd be fixed a lot better that way....

Quoted 2 lines · Peter A. Bigot

In any case, Khem can you run with this? It'd be fixed a lot better that way....

We do not configure target gcc with right matching cpu defaults, atomic instruction strex/ldrex are only added after armv6 but defaults for gcc if not specified is armv5t and hence it does not use the right set as expected by libstdc++ which has been cross compiled. so while you are at it and can reproduce it. Try to add

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", " --with-cpu=armv7-a", "", d)}'

to gcc-target.inc and see if resulting gcc is any better

Quoted 14 lines · Khem Raj

Quoted 3 lines · Peter A. Bigot

In any case, Khem can you run with this? It'd be fixed a lot better that way....

We do not configure target gcc with right matching cpu defaults, atomic instruction strex/ldrex are only added after armv6 but defaults for gcc if not specified is armv5t and hence it does not use the right set as expected by libstdc++ which has been cross compiled. so while you are at it and can reproduce it. Try to add

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", " --with-cpu=armv7-a", "", d)}'

Quoted 2 lines

to gcc-target.inc and see if resulting gcc is any better

I had to make it:

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", "--with-cpu=generic-armv7-a", "", d)}'

beaglebone[16]$ g++ -std=c++11 -pthread test.cc && ./a.out Assembler messages:
Error: unknown cpu `generic-armv7-a'
Error: unrecognized option -mcpu=generic-armv7-a

which indicates the flag's being passed to the assembler which doesn't recognize it even though g++ is happy with it. I suppose we could hack binutils to substitute whatever spelling it wants to see.

(Also tried --with-cpu=arm7, but that generates assembler errors related to unsupported RM mode "bx lr").

The approach bothers me, though. Instead of explicitly changing gcc-target to match gcc-runtime, shouldn't it be a general rule that gcc-runtime not apply OE-specific target flags that aren't going to be used by direct invocations of the compiler outside of the OE build environment?
That seems a little more robust, as the default target flags may be
changed upstream or by bbappends within OE, and having to make them match in gcc-runtime as well would be a headache.

And would we need similar overrides for other architectures? There's something similar already in gcc-configure-common.inc for mips64.

Peter

On Wednesday, August 13, 2014, Peter A. Bigot <[email protected] <mailto:[email protected]>> wrote:

On 08/13/2014 05:05 PM, Khem Raj wrote:

On Wed, Aug 13, 2014 at 2:36 PM, Peter A. Bigot <[email protected]> wrote:

In any case, Khem can you run with this? It'd be fixed a lot better that
way....

We do not configure target gcc with right matching cpu defaults, atomic instruction strex/ldrex are only added after armv6 but defaults
for gcc if not specified is armv5t and hence it does not use the right
set as expected by libstdc++ which has been cross compiled. so while
you are at it and can reproduce it. Try to add

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", " --with-cpu=armv7-a", "", d)}'

to gcc-target.inc and see if resulting gcc is any better

I had to make it:

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", "--with-cpu=generic-armv7-a", "", d)}'

Sorry a typo there you need --with-arch

to get gcc to build but at runtime I then get:

beaglebone[16]$ g++ -std=c++11 -pthread test.cc && ./a.out Assembler messages:
Error: unknown cpu `generic-armv7-a'
Error: unrecognized option -mcpu=generic-armv7-a

which indicates the flag's being passed to the assembler which doesn't recognize it even though g++ is happy with it. I suppose we could hack binutils to substitute whatever spelling it wants to see.

(Also tried --with-cpu=arm7, but that generates assembler errors related to unsupported RM mode "bx lr").

The approach bothers me, though. Instead of explicitly changing gcc-target to match gcc-runtime, shouldn't it be a general rule that gcc-runtime not apply OE-specific target flags that aren't going to be used by direct invocations of the compiler outside of the OE build environment? That seems a little more robust, as the default target flags may be changed upstream or by bbappends within OE, and having to make them match in gcc-runtime as well would be a headache.

And would we need similar overrides for other architectures? There's something similar already in gcc-configure-common.inc for mips64.

Peter

Quoted 3 lines

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", "--with-cpu=generic-armv7-a", "", d)}'

Sorry a typo there you need --with-arch

OK, that works. So do we need to do the same thing for every TUNE_FEATURES element that ends up changing the value of -march= in TUNE_CCARGS which ends up getting passed into gcc-runtime?

If so would it be better to add a TUNE_ARCH setting to all the tune-foo.inc files and use that in both TUNE_CCARGS and the --with-arch= flag passed to gcc? Just to avoid having this stuff hidden inside gcc-target.inc where it's pretty obscure.

Quoted 9 lines · Peter A. Bigot

Quoted 3 lines

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", "--with-cpu=generic-armv7-a", "", d)}'

Sorry a typo there you need --with-arch

OK, that works. So do we need to do the same thing for every TUNE_FEATURES element that ends up changing the value of -march= in TUNE_CCARGS which ends up getting passed into gcc-runtime?

Not really but arm v6+ is an exception here due to reasons explained earlier. gcc as a target package is a bit
different than rest of them since it has mind of its own when it comes to configuring it. It encodes certain
defaults into its driver etc. based on the configure parameters. Ideally when we pass the flags via CFLAGS
it does it for most of packages but not for gcc.

What this means is that the libraries built by gcc-runtime use TARGET_CC_ARCH settings that don't necessarily match the target compiler's defaults, and that ABI conflicts can result by linking in those libraries when the non-default settings were absent in non-OE application builds. ABI can only be "guaranteed" if every one of the -mFOO=BAR passed in TARGET_CC_ARCH (*or defaulted by the compiler*) has a corresponding -with-FOO=BAR option passed to (*or inferred by*) gcc's configure.

That's a pretty strong assumption to make.

It may be that this can be worked around for the specific case I raised by explicitly adding --with-arch=armv7-a to gcc-target's EXTRA_OECONF. I do have to wonder whether the same should be done for any of armv7 armv7-m armv7-r armv7e-m armv7ve armv8-a armv8-a+crc which are other -march options that are armv6+, and whether there are other ABI issues that might be hiding now or in the future because TARGET_CC_ARCH makes more assumptions than gcc-target does.

The solution I propose is to rework gcc-runtime's override of CC/CXX/CPP so the libraries are built the same way they would be if they had been built during gcc-target.

From initial attempts this won't be easy to do. I'd be happy to keep
trying if this worries other people, but if I'm being too picky I'll just suck it up and move on.

Quoted 23 lines · Peter A. Bigot

What this means is that the libraries built by gcc-runtime use TARGET_CC_ARCH settings that don't necessarily match the target compiler's defaults, and that ABI conflicts can result by linking in those libraries when the non-default settings were absent in non-OE application builds. ABI can only be "guaranteed" if every one of the -mFOO=BAR passed in TARGET_CC_ARCH (*or defaulted by the compiler*) has a corresponding -with-FOO=BAR option passed to (*or inferred by*) gcc's configure.

That's a pretty strong assumption to make.

It may be that this can be worked around for the specific case I raised by explicitly adding --with-arch=armv7-a to gcc-target's EXTRA_OECONF. I do have to wonder whether the same should be done for any of armv7 armv7-m armv7-r armv7e-m armv7ve armv8-a armv8-a+crc which are other -march options that are armv6+, and whether there are other ABI issues that might be hiding now or in the future because TARGET_CC_ARCH makes more assumptions than gcc-target does.

The solution I propose is to rework gcc-runtime's override of CC/CXX/CPP so the libraries are built the same way they would be if they had been built during gcc-target.

Quoted 3 lines

From initial attempts this won't be easy to do. I'd be happy to keep
trying if this worries other people, but if I'm being too picky I'll just suck it up and move on.

Its a valid concern, I just don't think anyone else has run into the kinds of issues you're seeing :/.

I'm content with the solution in my v2 gcc patch series, but I expect someday the underlying cause will re-appear as new processors have new features with ABI impacts due to gcc target optimization. Hopefully when that happens the discussion here will be of some help in identifying the problem.

Quoted 15 lines · Khem Raj

Quoted 4 lines · Peter A. Bigot

Quoted 4 lines

EXTRA_OECONF += '${@bb.utils.contains("TUNE_FEATURES", "armv7a", "--with-cpu=generic-armv7-a", "", d)}'

Quoted 8 lines

Sorry a typo there you need --with-arch

OK, that works. So do we need to do the same thing for every TUNE_FEATURES
element that ends up changing the value of -march= in TUNE_CCARGS which ends
up getting passed into gcc-runtime?

Quoted 7 lines

Not really but arm v6+ is an exception here due to reasons explained earlier. gcc as a target package is a bit
different than rest of them since it has mind of its own when it comes to configuring it. It encodes certain
defaults into its driver etc. based on the configure parameters. Ideally when we pass the flags via CFLAGS
it does it for most of packages but not for gcc.

tl;dr: the proposed fix to gcc-target works for my example but I'm concerned it's a bandaid that doesn't address a more long-term risk.

I understand gcc-target is different. I frame the issue as: bitbake (via toolchain-scripts.bbclass) ensures OE builds its applications with TARGET_CC_ARCH flags such as -march=FOO, but building gcc itself requires EXTRA_OECONF to pass different flags such as --with-arch=FOO which set the default behavior for the target compiler.

There are actually a bunch of these gcc configuration options:

--with-schedule=cpu
--with-arch=cpu
--with-arch-32=cpu
--with-arch-64=cpu
--with-tune=cpu
--with-tune-32=cpu
--with-tune-64=cpu
--with-abi=abi
--with-fpu=type
--with-float=type

all of which affect the default for the corresponding -m flags, and any of which may affect each other within gcc's configure script.

It sounds like the intent is that the default behavior of the target gcc should match what bitbake is passing via TARGET_CC_ARCH. Of course, if that were guaranteed there wouldn't be any need for OE to apply TARGET_CC_ARCH at all, but I wouldn't suggest changing that.

My concern is that gcc-runtime invokes configure in subdirectories of gcc's source tree, bypassing the top-level configure where some of the translation from (e.g.) --with-arch=FOO to -march=FOO is done. It then builds the libraries with TARGET_CC_ARCH flags appended to CC, CXX, and CPP.

What this means is that the libraries built by gcc-runtime use TARGET_CC_ARCH settings that don't necessarily match the target compiler's defaults, and that ABI conflicts can result by linking in those libraries when the non-default settings were absent in non-OE application builds. ABI can only be "guaranteed" if every one of the -mFOO=BAR passed in TARGET_CC_ARCH (*or defaulted by the compiler*) has a corresponding -with-FOO=BAR option passed to (*or inferred by*) gcc's configure.

That's a pretty strong assumption to make.

It may be that this can be worked around for the specific case I raised by explicitly adding --with-arch=armv7-a to gcc-target's EXTRA_OECONF. I do have to wonder whether the same should be done for any of armv7 armv7-m armv7-r armv7e-m armv7ve armv8-a armv8-a+crc which are other -march options that are armv6+, and whether there are other ABI issues that might be hiding now or in the future because TARGET_CC_ARCH makes more assumptions than gcc-target does.

The solution I propose is to rework gcc-runtime's override of CC/CXX/CPP so the libraries are built the same way they would be if they had been built during gcc-target.

From initial attempts this won't be easy to do. I'd be happy to keep trying if this worries other people, but if I'm being too picky I'll just suck it up and move on.

Peter