Skip to contentSkip to conversation

package/go: use host compiler when go-bootstrap unsupported

PATCH In review Christian Stewart 4 participants 6 messages Jun 22, 2021 – Jul 24, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/go: use host compiler when go-bootstrap unsupported

All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source.

https://golang.org/doc/install/source#bootstrapFromSource

The process for "bootstrapping" the Go compiler in Buildroot is:

1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.

The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).

This means that arm64 host machines will skip building Go and all go-based packages like Docker.

This patch instead uses the host Go compiler to bootstrap host-go when BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how the host GCC is used to bootstrap the Buildroot toolchain.

---
 package/go-bootstrap/go-bootstrap.mk | 6 ++++++
 package/go/Config.in.host            | 2 --
 package/go/go.mk                     | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)
package/go-bootstrap/go-bootstrap.mk +6 -0
index 6710e31561..8324a67795 100644
@@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE
 # host-go-bootstrap.
 HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain
 
+# If we do not support this architecture with go-bootstrap, depend on the host
+# Go compiler to bootstrap the host-go compiler instead.
+ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
 HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION)
+else
+HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go
+endif
 
 # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE
 # here.  See https://github.com/golang/go/issues/11685.
package/go/Config.in.host +0 -2
index e82ab6e81a..c07605dd02 100644
@@ -2,7 +2,6 @@
 config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
 	depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
 		|| BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
 		|| BR2_mips64 || BR2_mips64el || BR2_s390x
@@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
package/go/go.mk +4 -0
index 4252691343..5f501d2d8c 100644
@@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause
 GO_LICENSE_FILES = LICENSE
 GO_CPE_ID_VENDOR = golang
 
+# Depend on the host Go compiler if go-bootstrap is not available.
+ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
 HOST_GO_DEPENDENCIES = host-go-bootstrap
+endif
+
 HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path
 HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache
 HOST_GO_ROOT = $(HOST_DIR)/lib/go
Signature · 1 line
2.32.0
#1 lore raw

PATCH 1/1 package/go: use host compiler when go-bootstrap unsupported

Hi Christian,

Quoted 33 lines · Christian Stewart

All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source.

https://golang.org/doc/install/source#bootstrapFromSource

The process for "bootstrapping" the Go compiler in Buildroot is:

1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.

The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).

This means that arm64 host machines will skip building Go and all go-based packages like Docker.

This patch instead uses the host Go compiler to bootstrap host-go when BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how the host GCC is used to bootstrap the Buildroot toolchain.

---
 package/go-bootstrap/go-bootstrap.mk | 6 ++++++
 package/go/Config.in.host            | 2 --
 package/go/go.mk                     | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)
package/go-bootstrap/go-bootstrap.mk +0 -0
index 6710e31561..8324a67795 100644

package/go-bootstrap/go-bootstrap.mk GO_BOOTSTRAP_LICENSE_FILES = LICENSE · line 25

Quoted 9 lines
package/go-bootstrap/go-bootstrap.mk +5 -0
@@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE
 # host-go-bootstrap.
 HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain
 
+# If we do not support this architecture with go-bootstrap, depend on the host
+# Go compiler to bootstrap the host-go compiler instead.
+ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
 HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION)
+else
+HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go

I think we need a check for host go existence and maybe also version under support/dependencies/.

baruch

Quoted 37 lines
package/go-bootstrap/go-bootstrap.mk +1 -0
+endif
 
 # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE
 # here.  See https://github.com/golang/go/issues/11685.
package/go/Config.in.host +0 -2
index e82ab6e81a..c07605dd02 100644
@@ -2,7 +2,6 @@
 config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
 	depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
 		|| BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
 		|| BR2_mips64 || BR2_mips64el || BR2_s390x
@@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
package/go/go.mk +4 -0
index 4252691343..5f501d2d8c 100644
@@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause
 GO_LICENSE_FILES = LICENSE
 GO_CPE_ID_VENDOR = golang
 
+# Depend on the host Go compiler if go-bootstrap is not available.
+ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
 HOST_GO_DEPENDENCIES = host-go-bootstrap
+endif
+
 HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path
 HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache
 HOST_GO_ROOT = $(HOST_DIR)/lib/go
Signature · 3 lines
                                                     ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -
#2 lore raw

PATCH 1/1 package/go: use host compiler when go-bootstrap unsupported

Hi Baruch,

Quoted 6 lines · Baruch Siach
Quoted 1 line

+HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go

I think we need a check for host go existence and maybe also version under support/dependencies/.

baruch

I had a look at those scripts and am not familiar enough to know the correct way to only check for host-go existence on the system only if all of:

- BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS=y
- BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set
- BR2_PACKAGE_HOST_GO=y

If any of these conditions are not met, it's not necessary to have Go installed on the host system.

Best,
Christian

#3 lore raw
In reply to Christian Stewart

Hello Christian,

Quoted 5 lines · On Mon, 21 Jun 2021 19:43:37 -0700 Christian Stewart <[email protected]> wrote:

All Go compiler versions > 1.4.x (old) are written in Go, and require a existing compiled Go version to use to build from source.

https://golang.org/doc/install/source#bootstrapFromSource

Quoted 8 lines

The process for "bootstrapping" the Go compiler in Buildroot is:

1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.

The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).

So perhaps my brain is not smart enough here, but how is the Go compiler then build on 64-bit ARM ?

Quoted 6 lines
---
 package/go-bootstrap/go-bootstrap.mk | 6 ++++++
 package/go/Config.in.host            | 2 --
 package/go/go.mk                     | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)

Implementation-wise, I agree with the feedback from Baruch, we need some checks in support/dependencies/.

Normally, the way we do things is that packages express their dependency on a particular system tool by selecting a hidden boolean. For example packages that need java on the host will select BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if BR2_NEEDS_HOST_JAVA=y, we check if java is installed.

The other model that we have is for optional host tools, where we can use the system-provided one or fallback to building our own if not. That's for example what support/dependencies/check-host-lzip.{mk,sh} is doing.

check-host-lzip.mk calls suitable-host-package, which itself calls the check-host-lzip.sh script to verify if lzip is installed. If there is no suitable lzip available on the host, then check-host-lzip.mk defines BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be empty if the system has a lzip program, or it will be set to host-lzip if Buildroot needs to build its own.

Another feature for the lzip situation is that we can force Buildroot to build the host packages even if the tool is found on the machine, using BR2_FORCE_HOST_BUILD=y.

The thing is that your host-go-bootstrap situation does not exactly fit these situations.

I think the easiest would be to do like we do for Java, but only select BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support building host-go-bootstrap.

If we go this route, then it should look like this:

* Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean

* support/dependencies/dependencies.sh should test if a functional Go compiler is available on the machine if BR2_NEEDS_HOST_GO is defined, and if not bail out with a hard error

* BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have something like:

select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS

* go.mk should be modified to not use host-go-bootstrap if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think host-go-bootstrap should be modified like you did however, the changes should be contained into package/go/go.mk.

The one thing that bothers me however is the hardcoding of /usr/lib/go in your proposal as the location of the Go compiler. I guess we could have some logic in the top-level Makefile that looks for the path of the host Go compiler perhaps ?

Does this make sense ?

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#4 lore raw

Hi Thomas, all,

Quoted 6 lines · Thomas Petazzoni
Quoted 2 lines · Christian Stewart

The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm. The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).

So perhaps my brain is not smart enough here, but how is the Go compiler then build on 64-bit ARM ?

It has to be bootstrapped using an existing arm64 go compiler binary, or cross compiled.

Presumably a bootstrap process which produces first an armv7 Go compiler, and then uses it to build a arm64 Go compiler, is also possible.

With gccgo, it should be possible to bootstrap using the Gcc version of the Go compiler (which is included with the Buildroot toolchain today, but not enabled.)

Quoted 4 lines

The one thing that bothers me however is the hardcoding of /usr/lib/go in your proposal as the location of the Go compiler. I guess we could have some logic in the top-level Makefile that looks for the path of the host Go compiler perhaps ?

Yes, it could be done like this:

PATH_TO_GOROOT=$(go env GOROOT)

Quoted 1 line

Does this make sense ?

Yes, this should work fine.

Eventually the gccgo based bootstrap approach could also be added as another alternative.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

#5 lore raw
In reply to Thomas Petazzoni

Hi Thomas,

I have implemented your suggestions below & sent as a V2.

Quoted 43 lines · Thomas Petazzoni

Normally, the way we do things is that packages express their dependency on a particular system tool by selecting a hidden boolean. For example packages that need java on the host will select BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if BR2_NEEDS_HOST_JAVA=y, we check if java is installed.

The other model that we have is for optional host tools, where we can use the system-provided one or fallback to building our own if not. That's for example what support/dependencies/check-host-lzip.{mk,sh} is doing.

check-host-lzip.mk calls suitable-host-package, which itself calls the check-host-lzip.sh script to verify if lzip is installed. If there is no suitable lzip available on the host, then check-host-lzip.mk defines BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the wo lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be empty if the system has a lzip program, or it will be set to host-lzip if Buildroot needs to build its own.

Another feature for the lzip situation is that we can force Buildroot to build the host packages even if the tool is found on the machine, using BR2_FORCE_HOST_BUILD=y.

The thing is that your host-go-bootstrap situation does not exactly fit these situations.

I think the easiest would be to do like we do for Java, but only select BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support building host-go-bootstrap.

If we go this route, then it should look like this:

* Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean

* support/dependencies/dependencies.sh should test if a functional Go compiler is available on the machine if BR2_NEEDS_HOST_GO is defined, and if not bail out with a hard error

* BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have something like:

Quoted 6 lines

select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS

* go.mk should be modified to not use host-go-bootstrap if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think host-go-bootstrap should be modified like you did however, the changes should be contained into package/go/go.mk.

Thanks & best,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot