package/go: use host compiler when go-bootstrap unsupported
PATCH 1/1 package/go: use host compiler when go-bootstrap unsupported
All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source.
https://golang.org/doc/install/source#bootstrapFromSource
The process for "bootstrapping" the Go compiler in Buildroot is:
1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.
The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).
This means that arm64 host machines will skip building Go and all go-based packages like Docker.
This patch instead uses the host Go compiler to bootstrap host-go when BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how the host GCC is used to bootstrap the Buildroot toolchain.
--- package/go-bootstrap/go-bootstrap.mk | 6 ++++++ package/go/Config.in.host | 2 -- package/go/go.mk | 4 ++++ 3 files changed, 10 insertions(+), 2 deletions(-)
package/go-bootstrap/go-bootstrap.mk +6 -0
@@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE # host-go-bootstrap. HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain +# If we do not support this architecture with go-bootstrap, depend on the host +# Go compiler to bootstrap the host-go compiler instead. +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION) +else +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go +endif # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE # here. See https://github.com/golang/go/issues/11685.
package/go/Config.in.host +0 -2
@@ -2,7 +2,6 @@ config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \ || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \ || BR2_mips64 || BR2_mips64el || BR2_s390x
@@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
package/go/go.mk +4 -0
@@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause GO_LICENSE_FILES = LICENSE GO_CPE_ID_VENDOR = golang +# Depend on the host Go compiler if go-bootstrap is not available. +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) HOST_GO_DEPENDENCIES = host-go-bootstrap +endif + HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache HOST_GO_ROOT = $(HOST_DIR)/lib/go
Signature · 1 line
2.32.0
PATCH 1/1 package/go: use host compiler when go-bootstrap unsupported
Hi Christian,
Quoted 33 lines · Christian Stewart
All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source.
https://golang.org/doc/install/source#bootstrapFromSource
The process for "bootstrapping" the Go compiler in Buildroot is:
1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86,
x86_64, and arm (32 bit).This means that arm64 host machines will skip building Go and all go-based packages like Docker.
This patch instead uses the host Go compiler to bootstrap host-go when
BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTSis not set. This is similar to how the host GCC is used to bootstrap the Buildroot toolchain.Signed-off-by Christian Stewart <[email protected]>--- package/go-bootstrap/go-bootstrap.mk | 6 ++++++ package/go/Config.in.host | 2 -- package/go/go.mk | 4 ++++ 3 files changed, 10 insertions(+), 2 deletions(-)package/go-bootstrap/go-bootstrap.mk +0
-0
package/go-bootstrap/go-bootstrap.mk GO_BOOTSTRAP_LICENSE_FILES = LICENSE · line 25
Quoted 9 lines
package/go-bootstrap/go-bootstrap.mk +5
-0@@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE # host-go-bootstrap. HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain +# If we do not support this architecture with go-bootstrap, depend on the host +# Go compiler to bootstrap the host-go compiler instead. +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION) +else +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go
I think we need a check for host go existence and maybe also version under support/dependencies/.
baruch
Quoted 37 lines
package/go-bootstrap/go-bootstrap.mk +1
-0+endif # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE # here. See https://github.com/golang/go/issues/11685.package/go/Config.in.host +0
-2@@ -2,7 +2,6 @@ config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \ || BR2_i386 || BR2_x86_64 || BR2_powerpc64le \ || BR2_mips64 || BR2_mips64el || BR2_s390x@@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS bool default y - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTSpackage/go/go.mk +4
-0@@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause GO_LICENSE_FILES = LICENSE GO_CPE_ID_VENDOR = golang +# Depend on the host Go compiler if go-bootstrap is not available. +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y) HOST_GO_DEPENDENCIES = host-go-bootstrap +endif + HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache HOST_GO_ROOT = $(HOST_DIR)/lib/go
Signature · 3 lines
~. .~ Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
- baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -PATCH 1/1 package/go: use host compiler when go-bootstrap unsupported
Hi Baruch,
Quoted 6 lines · Baruch Siach
Quoted 1 line
+HOST_GO_BOOTSTRAP_ROOT =
/usr/lib/goI think we need a check for host go existence and maybe also version under support/dependencies/.
baruch
I had a look at those scripts and am not familiar enough to know the correct way to only check for host-go existence on the system only if all of:
- BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS=y
- BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set
- BR2_PACKAGE_HOST_GO=y
If any of these conditions are not met, it's not necessary to have Go installed on the host system.
Best,
Christian
Hello Christian,
Quoted 5 lines · On Mon, 21 Jun 2021 19:43:37 -0700 Christian Stewart <[email protected]> wrote:
All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source.
Quoted 8 lines
The process for "bootstrapping" the Go compiler in Buildroot is:
1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86,
x86_64, and arm (32 bit).
So perhaps my brain is not smart enough here, but how is the Go compiler then build on 64-bit ARM ?
Quoted 6 lines
Signed-off-by Christian Stewart <[email protected]>--- package/go-bootstrap/go-bootstrap.mk | 6 ++++++ package/go/Config.in.host | 2 -- package/go/go.mk | 4 ++++ 3 files changed, 10 insertions(+), 2 deletions(-)
Implementation-wise, I agree with the feedback from Baruch, we need some checks in support/dependencies/.
Normally, the way we do things is that packages express their dependency on a particular system tool by selecting a hidden boolean. For example packages that need java on the host will select BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if BR2_NEEDS_HOST_JAVA=y, we check if java is installed.
The other model that we have is for optional host tools, where we can use the system-provided one or fallback to building our own if not. That's for example what support/dependencies/check-host-lzip.{mk,sh} is doing.
check-host-lzip.mk calls suitable-host-package, which itself calls the check-host-lzip.sh script to verify if lzip is installed. If there is no suitable lzip available on the host, then check-host-lzip.mk defines BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be empty if the system has a lzip program, or it will be set to host-lzip if Buildroot needs to build its own.
Another feature for the lzip situation is that we can force Buildroot to build the host packages even if the tool is found on the machine, using BR2_FORCE_HOST_BUILD=y.
The thing is that your host-go-bootstrap situation does not exactly fit these situations.
I think the easiest would be to do like we do for Java, but only select BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support building host-go-bootstrap.
If we go this route, then it should look like this:
* Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean
* support/dependencies/dependencies.sh should test if a functional Go compiler is available on the machine if BR2_NEEDS_HOST_GO is defined, and if not bail out with a hard error
* BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have something like:
select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
* go.mk should be modified to not use host-go-bootstrap if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think host-go-bootstrap should be modified like you did however, the changes should be contained into package/go/go.mk.
The one thing that bothers me however is the hardcoding of /usr/lib/go in your proposal as the location of the Go compiler. I guess we could have some logic in the top-level Makefile that looks for the path of the host Go compiler perhaps ?
Does this make sense ?
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Hi Thomas, all,
Quoted 6 lines · Thomas Petazzoni
Quoted 2 lines · Christian Stewart
The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86,
x86_64, and arm (32 bit).So perhaps my brain is not smart enough here, but how is the Go compiler then build on 64-bit ARM ?
It has to be bootstrapped using an existing arm64 go compiler binary, or cross compiled.
Presumably a bootstrap process which produces first an armv7 Go compiler, and then uses it to build a arm64 Go compiler, is also possible.
With gccgo, it should be possible to bootstrap using the Gcc version of the Go compiler (which is included with the Buildroot toolchain today, but not enabled.)
Quoted 4 lines
The one thing that bothers me however is the hardcoding of
/usr/lib/goin your proposal as the location of the Go compiler. I guess we could have some logic in the top-levelMakefilethat looks for the path of the host Go compiler perhaps ?
Yes, it could be done like this:
PATH_TO_GOROOT=$(go env GOROOT)
Quoted 1 line
Does this make sense ?
Yes, this should work fine.
Eventually the gccgo based bootstrap approach could also be added as another alternative.
Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Hi Thomas,
I have implemented your suggestions below & sent as a V2.
Quoted 43 lines · Thomas Petazzoni
Normally, the way we do things is that packages express their dependency on a particular system tool by selecting a hidden boolean. For example packages that need java on the host will select
BR2_NEEDS_HOST_JAVA, and insupport/dependencies/dependencies.sh, if BR2_NEEDS_HOST_JAVA=y, we check if java is installed.The other model that we have is for optional host tools, where we can use the system-provided one or fallback to building our own if not. That's for example what support/dependencies/check-host-lzip.{mk,sh} is doing.
check-host-lzip.mk calls suitable-host-package, which itself calls the
check-host-lzip.shscript to verify if lzip is installed. If there is no suitable lzip available on the host, then check-host-lzip.mk definesBR2_LZIP_HOST_DEPENDENCYto host-lzip, so that packages who need the wo lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be empty if the system has a lzip program, or it will be set to host-lzip if Buildroot needs to build its own.Another feature for the lzip situation is that we can force Buildroot to build the host packages even if the tool is found on the machine, using BR2_FORCE_HOST_BUILD=y.
The thing is that your host-go-bootstrap situation does not exactly fit these situations.
I think the easiest would be to do like we do for Java, but only select
BR2_NEEDS_HOST_GOif we're on a host architecture that doesn't support building host-go-bootstrap.If we go this route, then it should look like this:
* Top-level Config.in defines
BR2_NEEDS_HOST_GOas an hidden boolean*
support/dependencies/dependencies.shshould test if a functional Go compiler is available on the machine ifBR2_NEEDS_HOST_GOis defined, and if not bail out with a hard error*
BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTSandBR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTSshould loose their "depends onBR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have something like:
Quoted 6 lines
select
BR2_NEEDS_HOST_GOif !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS* go.mk should be modified to not use host-go-bootstrap if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think host-go-bootstrap should be modified like you did however, the changes should be contained into
package/go/go.mk.
Thanks & best,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot