core: fix setting of HOSTARCH
PATCH 1/1 core: fix setting of HOSTARCH
Currently, we set HOSTARCH to the output of uname -m. This gives us the architecture as seen by the running kernel. For example, we would end up with 'x86_64' for a 64-bit kernel running on an x86_64 processor.
We use that value to determine whether we can run some binary tools, like our pre-configured external toolchains.
However, one may be running a userland in a different bitness than that of the running kernel. For example, one may run in a 32-bit chroot, even though the kernel is running in 64-bit.
Up until recently, this was not an issue because the pre-configured external toolchains were all requiring an i386 (x86 in Buildroot parlance).
But since w eintroduced the latest Linaro toolchains, we now have toolchains that require a 64-bit userland.
So, when running on a 64-bit kernel, we believe those toolchains are available, even when the user is running a 32-bit userland. This causes build failures for our autobuilders, like so:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/
with the following symptoms:
Quoted 1 line
Quoted 1 line
Quoted 1 line
toolchain-external undefined Configuring
Cannot execute cross-compiler '/home/test/autobuild/instance-3/output/host/opt/ext-toolchain/bin/aarch64-linux-gnu-gcc'
So, instead of relying on the output of uname -r, look for the host gcc and extract the target it was configured to generate code for.
Fixes:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ (aarch64) http://autobuild.buildroot.org/results/888/8889aa7d9fb48370e4760a6edbc6d3ae945f02f2/ (arm) and many more...
Besides fixing those issues, it will also allow us to add the 64-bit variants of toolchains when they exist, like the upcoming Codescape MTI and IMG toolchains for MIPS from Imagination Technologies.
--- Makefile | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
Makefile +25 -9
@@ -52,15 +52,31 @@ ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA $(error You have make '$(RUNNING_MAKE_VERSION)' installed. GNU make >= $(MIN_MAKE_VERSION) is required) endif -export HOSTARCH := $(shell uname -m | \ - sed -e s/i.86/x86/ \ - -e s/sun4u/sparc64/ \ - -e s/arm.*/arm/ \ - -e s/sa110/arm/ \ - -e s/ppc64/powerpc64/ \ - -e s/ppc/powerpc/ \ - -e s/macppc/powerpc/\ - -e s/sh.*/sh/) +# Determine the userland we are running on. +# +# Note that, despite its name, we are not interested in the actual +# architecture name. This is mostly used to determine whether some +# of the binary tools (e.g. pre-built external toolchains) can run +# on the current host. So we need to know if the userland we're +# running on can actually run those toolchains. +# +# For example, a 64-bit prebuilt toolchain will not run on a 64-bit +# kernel if the userland is 32-bit (e.g. in a chroot for example). +# +# So, we extract the first part of the tuple the host gcc was +# configured to generate code for; we assume this is our userland. +# +export HOSTARCH := $(shell gcc -v 2>&1 | \ + sed -e '/^Target: \([^-]*\).*/!d' \ + -e 's//\1/' \ + -e 's/i.86/x86/' \ + -e 's/sun4u/sparc64/' \ + -e 's/arm.*/arm/' \ + -e 's/sa110/arm/' \ + -e 's/ppc64/powerpc64/' \ + -e 's/ppc/powerpc/' \ + -e 's/macppc/powerpc/' \ + -e 's/sh.*/sh/' ) # Parallel execution of this Makefile is disabled because it changes # the packages building order, that can be a problem for two reasons:
Signature · 1 line
1.9.1
PATCH 1/1 core: fix setting of HOSTARCH
Quoted 37 lines · Yann E. MORIN
Currently, we set HOSTARCH to the output of
uname -m. This gives us the architecture as seen by the running kernel. For example, we would end up with 'x86_64' for a 64-bit kernel running on anx86_64processor.We use that value to determine whether we can run some binary tools, like our pre-configured external toolchains.
However, one may be running a userland in a different bitness than that of the running kernel. For example, one may run in a 32-bit chroot, even though the kernel is running in 64-bit.
Up until recently, this was not an issue because the pre-configured external toolchains were all requiring an i386 (x86 in Buildroot parlance).
But since w eintroduced the latest Linaro toolchains, we now have toolchains that require a 64-bit userland.
So, when running on a 64-bit kernel, we believe those toolchains are available, even when the user is running a 32-bit userland. This causes build failures for our autobuilders, like so:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/with the following symptoms:
Quoted 1 line
Quoted 1 line
Quoted 1 line
toolchain-external undefined Configuring
Cannot execute cross-compiler '
/home/test/autobuild/instance-3/output/host/opt/ext-toolchain/bin/aarch64-linux-gnu-gcc'So, instead of relying on the output of
uname -r, look for the host gcc and extract the target it was configured to generate code for.Fixes:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ (aarch64) http://autobuild.buildroot.org/results/888/8889aa7d9fb48370e4760a6edbc6d3ae945f02f2/ (arm) and many more...
Quoted 5 lines
Besides fixing those issues, it will also allow us to add the 64-bit variants of toolchains when they exist, like the upcoming Codescape MTI and IMG toolchains for MIPS from Imagination Technologies.
Signed-off-by "Yann E. MORIN" <[email protected]>
But see below.
Quoted 38 lines
Cc Thomas Petazzoni <[email protected]>Cc Vicente Olivert Riera <[email protected]>--- Makefile | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)Makefile +15
-9@@ -52,15 +52,31 @@ ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA $(error You have make '$(RUNNING_MAKE_VERSION)' installed. GNU make >= $(MIN_MAKE_VERSION) is required) endif -export HOSTARCH := $(shell uname -m | \ - sed -e s/i.86/x86/ \ - -e s/sun4u/sparc64/ \ - -e s/arm.*/arm/ \ - -e s/sa110/arm/ \ - -e s/ppc64/powerpc64/ \ - -e s/ppc/powerpc/ \ - -e s/macppc/powerpc/\ - -e s/sh.*/sh/) +# Determine the userland we are running on. +# +# Note that, despite its name, we are not interested in the actual +# architecture name. This is mostly used to determine whether some +# of the binary tools (e.g. pre-built external toolchains) can run +# on the current host. So we need to know if the userland we're +# running on can actually run those toolchains. +# +# For example, a 64-bit prebuilt toolchain will not run on a 64-bit +# kernel if the userland is 32-bit (e.g. in a chroot for example). +# +# So, we extract the first part of the tuple the host gcc was +# configured to generate code for; we assume this is our userland. +# +export HOSTARCH := $(shell gcc -v 2>&1 | \
Makefile ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA · line 79
Quoted 10 lines
Makefile +10
-0+ sed -e '/^Target: \([^-]*\).*/!d' \ + -e 's//\1/' \ + -e 's/i.86/x86/' \ + -e 's/sun4u/sparc64/' \ + -e 's/arm.*/arm/' \ + -e 's/sa110/arm/' \ + -e 's/ppc64/powerpc64/' \ + -e 's/ppc/powerpc/' \ + -e 's/macppc/powerpc/' \ + -e 's/sh.*/sh/' )
Since what we get here already are gcc tuples, the de-mangling shouldn't be
needed, right?
Regards,
Arnout
Quoted 4 lines
Makefile +0
-0# Parallel execution of this Makefile is disabled because it changes # the packages building order, that can be a problem for two reasons:
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/1 core: fix setting of HOSTARCH
Arnout, All,
On 2015-11-09 23:01 +0100, Arnout Vandecappelle spake thusly:
Quoted 4 lines · Yann E. MORIN
Quoted 3 lines
Currently, we set HOSTARCH to the output of
uname -m. This gives us the architecture as seen by the running kernel. For example, we would end up with 'x86_64' for a 64-bit kernel running on anx86_64processor.
[--SNIP--]
Quoted 2 lines
Quoted 2 lines
So, instead of relying on the output of
uname -r, look for the host gcc and extract the target it was configured to generate code for.
[--SNIP--]
Quoted 6 lines
Quoted 6 lines
Makefile +6
-0+export HOSTARCH := $(shell gcc -v 2>&1 | \ + sed -e '/^Target: \([^-]*\).*/!d' \ + -e 's//\1/' \ + -e 's/i.86/x86/' \ + -e 's/sun4u/sparc64/' \ + -e 's/arm.*/arm/' \
Quoted 8 lines
Makefile ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA · line 79
Quoted 5 lines
Makefile +5
-0+ -e 's/sa110/arm/' \ + -e 's/ppc64/powerpc64/' \ + -e 's/ppc/powerpc/' \ + -e 's/macppc/powerpc/' \ + -e 's/sh.*/sh/' )Since what we get here already are gcc tuples, the de-mangling shouldn't be
needed, right?
I am not sure either way. Note that I only constrained the check on how to retrieve the value, and expressely did not address the way we played with it, so as to follow the path of least surprise.
However, we still have to de-mangle i.86 into x86 because that what we're using everywhere. Also, we need to demangle arm.* into arm, because arm may have some trailing stuff (like armv6) which still means it's an arm. Ditto sh I guess. macppc I would guess (y the name of it) that we really don't care about. sa110: are we really expecting to run on that machine anyway?
We can further reduce the de-mangling in a later patch, however, it that makes sense...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 core: fix setting of HOSTARCH
Quoted 26 lines · Yann E. MORIN
Arnout, All,
On 2015-11-09 23:01 +0100, Arnout Vandecappelle spake thusly:
Quoted 4 lines · Yann E. MORIN
Quoted 3 lines
Currently, we set HOSTARCH to the output of
uname -m. This gives us the architecture as seen by the running kernel. For example, we would end up with 'x86_64' for a 64-bit kernel running on anx86_64processor.[--SNIP--]
Quoted 2 lines
Quoted 2 lines
So, instead of relying on the output of
uname -r, look for the host gcc and extract the target it was configured to generate code for.[--SNIP--]
Quoted 6 lines
Quoted 6 lines
Makefile +6
-0+export HOSTARCH := $(shell gcc -v 2>&1 | \ + sed -e '/^Target: \([^-]*\).*/!d' \ + -e 's//\1/' \ + -e 's/i.86/x86/' \ + -e 's/sun4u/sparc64/' \ + -e 's/arm.*/arm/' \Quoted 8 lines
Makefile ifneq ($(firstword $(sort $(RUNNING_MAKE_VERSION) $(MIN_MAKE_VERSION))),$(MIN_MA · line 79
Quoted 5 lines
Makefile +5
-0+ -e 's/sa110/arm/' \ + -e 's/ppc64/powerpc64/' \ + -e 's/ppc/powerpc/' \ + -e 's/macppc/powerpc/' \ + -e 's/sh.*/sh/' )Since what we get here already are gcc tuples, the de-mangling shouldn't be
needed, right?
Quoted 6 lines
I am not sure either way. Note that I only constrained the check on how to retrieve the value, and expressely did not address the way we played with it, so as to follow the path of least surprise.
However, we still have to de-mangle i.86 into x86 because that what we're using everywhere.
Right, we'd have to change that.
Quoted 3 lines
Also, we need to demangle arm.* into arm,
because arm may have some trailing stuff (like armv6) which still means it's an arm.
Really? For the kernel arch (uname -m) that's true, but also for the gcc tuple?
But armeb is indeed possible.
Quoted 6 lines
Ditto sh I guess. macppc I would guess (y the name of it) that we really don't care about. sa110: are we really expecting to run on that machine anyway?
We can further reduce the de-mangling in a later patch, however, it that makes sense...
Of course. However, I'm not sure if it makes sense to copy the mangling for the
uname to the mangling of the gcc tuple...
Regards,
Arnout
Quoted 4 lines
Regards,
Yann E. MORIN.
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/1 core: fix setting of HOSTARCH
In reply to Yann E. MORINYann, Peter,
Quoted 10 lines · Yann E. MORIN
Currently, we set HOSTARCH to the output of
uname -m. This gives usBut since w eintroduced the latest Linaro toolchains, we now have toolchains that require a 64-bit userland.
So, when running on a 64-bit kernel, we believe those toolchains are available, even when the user is running a 32-bit userland. This causes build failures for our autobuilders, like so:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/
Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits.
However, the real problem is that the new Linaro toolchains have been built against glibc 2.14 (for the host), while my chroot only uses glibc 2.11 (from Debian 6.0).
I'm not sure how to handle this. Some old distros may not have glibc 2.14.
Suggestions?
In any case, I believe the rework of how we get HOSTARCH is not really needed IMO: 1/ it is not the source of the problem for the Linaro toolchain and 2/ it is most likely appropriate to use setarch/linux32 when running a 32 bits chroot under a 64 bits kernel.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 1/1 core: fix setting of HOSTARCH
Thomas, All,
On 2015-11-11 23:53 +0100, Thomas Petazzoni spake thusly:
Quoted 10 lines · Yann E. MORIN
Quoted 9 lines
Currently, we set HOSTARCH to the output of
uname -m. This gives usBut since w eintroduced the latest Linaro toolchains, we now have toolchains that require a 64-bit userland.
So, when running on a 64-bit kernel, we believe those toolchains are available, even when the user is running a 32-bit userland. This causes build failures for our autobuilders, like so:
Quoted 5 lines
Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits.
OK, but still, in a 32-bit chroot, in the linux32 personality, we were trying to use the 64-bit variant of the toolchain. And I observed the same symptom as the autobuild failures.
So, two different problems, with a single symptom.
This patch adresses the 32-bit vs. 64-bit issue I encountered.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 core: fix setting of HOSTARCH
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits.
OK, but still, in a 32-bit chroot, in the linux32 personality, we were trying to use the 64-bit variant of the toolchain. And I observed the same symptom as the autobuild failures.
How can that be? With linux32 uname -m returns 'ix86', right?
Signature · 2 lines
Venlig hilsen, Peter Korsgaard
PATCH 1/1 core: fix setting of HOSTARCH
Peter, All,
On 2015-11-12 08:52 +0100, Peter Korsgaard spake thusly:
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits.
OK, but still, in a 32-bit chroot, in the linux32 personality, we were trying to use the 64-bit variant of the toolchain. And I observed the same symptom as the autobuild failures.
How can that be? With linux32 uname -m returns 'ix86', right?
Sorry, my bad. I meant: *without* the linux32 personality.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 core: fix setting of HOSTARCH
In reply to Thomas PetazzoniQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Hi,
Quoted 7 lines
Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits.
However, the real problem is that the new Linaro toolchains have been built against glibc 2.14 (for the host), while my chroot only uses glibc 2.11 (from Debian 6.0).
Ahh. Debian 6.0 is getting quite old (2011), but yeah - People on Enterprise distributions might end up with the same problems.
Quoted 2 lines
I'm not sure how to handle this. Some old distros may not have glibc 2.14.
I don't think there's much we can do, besides perhaps adding a comment in the help text.
I'm not sure how realistic it is to convince the Linaro people to build their toolchains on ancient distributions.
Quoted 4 lines
In any case, I believe the rework of how we get HOSTARCH is not really needed IMO: 1/ it is not the source of the problem for the Linaro toolchain and 2/ it is most likely appropriate to use setarch/linux32 when running a 32 bits chroot under a 64 bits kernel.
I agree about linux32, yes.
Signature · 2 lines
Venlig hilsen, Peter Korsgaard
PATCH 1/1 core: fix setting of HOSTARCH
Hello,
Quoted 10 lines · Peter Korsgaard
Quoted 7 lines
Those build failures are not caused by a 32 bits vs. 64 bits issue. In fact, I started running my chroot under "linux32", but it was even worse. Because indeed my chroots *are* 64 bits.
However, the real problem is that the new Linaro toolchains have been built against glibc 2.14 (for the host), while my chroot only uses glibc 2.11 (from Debian 6.0).
Ahh. Debian 6.0 is getting quite old (2011), but yeah - People on Enterprise distributions might end up with the same problems.
Indeed, and that's why I'd like to keep my old distro.
Quoted 8 lines
Quoted 2 lines
I'm not sure how to handle this. Some old distros may not have glibc 2.14.
I don't think there's much we can do, besides perhaps adding a comment in the help text.
I'm not sure how realistic it is to convince the Linaro people to build their toolchains on ancient distributions.
So I guess that the only reasonable option here is that I had some logic in autobuild-run to not use this toolchain when the host C library is too old. This way, we keep using the Linaro toolchain on build slaves that are modern enough, and we don't have build failures on the others.
Thoughts?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 1/1 core: fix setting of HOSTARCH
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Hi,
Quoted 4 lines
Quoted 2 lines
Ahh. Debian 6.0 is getting quite old (2011), but yeah - People on Enterprise distributions might end up with the same problems.
Indeed, and that's why I'd like to keep my old distro.
Good!
Quoted 7 lines
So I guess that the only reasonable option here is that I had some logic in autobuild-run to not use this toolchain when the host C library is too old. This way, we keep using the Linaro toolchain on build slaves that are modern enough, and we don't have build failures on the others.
Thoughts?
Yes, that makes sense.
Signature · 2 lines
Venlig hilsen, Peter Korsgaard
PATCH 1/1 core: fix setting of HOSTARCH
In reply to Yann E. MORINQuoted 41 lines · Yann E. MORIN
Currently, we set HOSTARCH to the output of
uname -m. This gives us the architecture as seen by the running kernel. For example, we would end up with 'x86_64' for a 64-bit kernel running on anx86_64processor.We use that value to determine whether we can run some binary tools, like our pre-configured external toolchains.
However, one may be running a userland in a different bitness than that of the running kernel. For example, one may run in a 32-bit chroot, even though the kernel is running in 64-bit.
Up until recently, this was not an issue because the pre-configured external toolchains were all requiring an i386 (x86 in Buildroot parlance).
But since w eintroduced the latest Linaro toolchains, we now have toolchains that require a 64-bit userland.
So, when running on a 64-bit kernel, we believe those toolchains are available, even when the user is running a 32-bit userland. This causes build failures for our autobuilders, like so:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/with the following symptoms:
Quoted 1 line
Quoted 1 line
Quoted 1 line
toolchain-external undefined Configuring
Cannot execute cross-compiler '
/home/test/autobuild/instance-3/output/host/opt/ext-toolchain/bin/aarch64-linux-gnu-gcc'So, instead of relying on the output of
uname -r, look for the host gcc and extract the target it was configured to generate code for.Fixes:
http://autobuild.buildroot.org/results/9cd/9cdf10ec5b31144b2e03ea09cf128702339895b3/ (aarch64) http://autobuild.buildroot.org/results/888/8889aa7d9fb48370e4760a6edbc6d3ae945f02f2/ (arm) and many more...Besides fixing those issues, it will also allow us to add the 64-bit variants of toolchains when they exist, like the upcoming Codescape MTI and IMG toolchains for MIPS from Imagination Technologies.
Quoted 6 lines
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Vicente Olivert Riera <[email protected]>--- Makefile | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
Committed after changing it to use HOSTCC_NOCCACHE instead of 'gcc', thanks!
--
Bye, Peter Korsgaard