Add go language support

This series adds support for the go programming language. I tested the build of the included flannel package on i386, powerpc64, arm (32 bit) and arm64. I did runtime testing of the generated flannel binary on arm64.

Patch 1 adds a host only bootstrap compiler needed to build the latest go versions.

Patch 2 adds support for a host cross compiler, libraries and tools.

Patch 3 adds a go language package flannel that can be used to test the go compiler.

Changes for v3:

o Rebase to latest (261cef0fe9598eadfe8a8efa80def40640f2e9f8).

Changes for v2:

o Dropped go target support. We can consider it again if there is a strong use case.
o General cleanup of package files. Added comments, fixed variable names, use github helper function.

-Geoff

The following changes since commit 261cef0fe9598eadfe8a8efa80def40640f2e9f8:

wilc1000-firmware: new package (2016-04-02 17:18:32 +0200)

are available in the git repository at:

git at github.com:glevand/buildroot--buildroot.git go-v3

for you to fetch changes up to 762e3d2b6b696656441d9818512ac7ac10221e7f:

flannel: new package (2016-04-04 12:12:48 -0700)

----------------------------------------------------------------
Geoff Levand (3):
go-bootstrap: new host package
go: new host package
flannel: new package

 package/Config.in                      |  1 +
 package/Config.in.host                 |  1 +
 package/flannel/Config.in              |  9 ++++
 package/flannel/flannel.hash           |  2 +
 package/flannel/flannel.mk             | 42 +++++++++++++++++++
 package/go-bootstrap/go-bootstrap.hash |  2 +
 package/go-bootstrap/go-bootstrap.mk   | 38 +++++++++++++++++
 package/go/Config.in.host              | 13 ++++++
 package/go/go.hash                     |  2 +
 package/go/go.mk                       | 75 ++++++++++++++++++++++++++++++++++
 10 files changed, 185 insertions(+)
 create mode 100644 package/flannel/Config.in
 create mode 100644 package/flannel/flannel.hash
 create mode 100644 package/flannel/flannel.mk
 create mode 100644 package/go-bootstrap/go-bootstrap.hash
 create mode 100644 package/go-bootstrap/go-bootstrap.mk
 create mode 100644 package/go/Config.in.host
 create mode 100644 package/go/go.hash
 create mode 100644 package/go/go.mk

2.5.0

Add a new package go-bootstrap which builds a bootstrap compiler for the go programming language.

---
 package/go-bootstrap/go-bootstrap.hash |  2 ++
 package/go-bootstrap/go-bootstrap.mk   | 38 ++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 package/go-bootstrap/go-bootstrap.hash
 create mode 100644 package/go-bootstrap/go-bootstrap.mk

new file mode 100644
index 0000000..1d0566a
@@ -0,0 +1,2 @@
+# Locally computed:
+sha256 299a6fd8f8adfdce15bc06bde926e7b252ae8e24dd5b16b7d8791ed79e7b5e9b go1.4.2.src.tar.gz

new file mode 100644
index 0000000..ccb932d
@@ -0,0 +1,38 @@
+################################################################################
+#
+# go-bootstrap
+#
+################################################################################
+
+GO_BOOTSTRAP_VERSION = 1.4.2
+GO_BOOTSTRAP_SITE = https://storage.googleapis.com/golang
+GO_BOOTSTRAP_SOURCE = go$(GO_BOOTSTRAP_VERSION).src.tar.gz
+
+GO_BOOTSTRAP_LICENSE = BSD-3c
+GO_BOOTSTRAP_LICENSE_FILES = LICENSE
+
+GO_BOOTSTRAP_FINAL = $(HOST_DIR)/usr/lib/go-$(GO_BOOTSTRAP_VERSION)
+
+GO_BOOTSTRAP_MAKE_ENV = \
+	GOOS=linux \
+	GOROOT_FINAL="$(GO_BOOTSTRAP_FINAL)" \
+	GOROOT="$(@D)" \
+	GOBIN="$(@D)/bin"
+
+define HOST_GO_BOOTSTRAP_BUILD_CMDS
+	cd $(@D)/src && $(GO_BOOTSTRAP_MAKE_ENV) ./make.bash
+endef
+
+define HOST_GO_BOOTSTRAP_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/bin/go $(GO_BOOTSTRAP_FINAL)/bin/go
+	$(INSTALL) -D -m 0755 $(@D)/bin/gofmt $(GO_BOOTSTRAP_FINAL)/bin/gofmt
+
+	cp -a $(@D)/lib $(GO_BOOTSTRAP_FINAL)/
+	cp -a $(@D)/pkg $(GO_BOOTSTRAP_FINAL)/
+
+	# There is a known issue which requires the go sources to be installed
+	# https://golang.org/issue/2775
+	cp -a $(@D)/src $(GO_BOOTSTRAP_FINAL)/
+endef
+
+$(eval $(host-generic-package))

2.5.0

Hello,

I've applied to master after doing the following changes:

[Thomas:
- Rename GO_BOOTSTRAP_FINAL to HOST_GO_BOOTSTRAP_ROOT. Use the HOST_ prefix because this variable is specific to the host variant. And it's not "final" at all, but rather the "root" of the Go Bootstrap installation.
- Rename GO_BOOTSTRAP_MAKE_ENV to HOST_GO_BOOTSTRAP_MAKE_ENV, also to have the HOST_ prefix because it's a host package.]

Thanks a lot!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Hi Thomas,

I just started playing with these patches a few days ago and made a local change to this patch that made sense to me - I removed the HOST_GO_BOOTSTRAP_INSTALL_CMDS because I did not see that it was useful to have the bootstrap compiler installed in the HOST_DIR. This mattered for me as I build the toolchain and install it system-wide to use for other builds, which refer to that toolchain as an external one.

I then changed the second patch to use HOST_GO_BOOTSTRAP_BUILDDIR instead of GO_BOOTSTRAP_FINAL.

Does this make sense to have in buildroot?

Hello,

We normally try to not use things from the build directory of package A when building package B. In an ideal world, things should be implemented as if we could remove the build directory of a package once it has been installed in the host/staging/target directories. It should therefore install there everything that might be needed by packages that depend on that package.

That being said:

1/ There are already numerous exceptions to this rule. For example,
all the external kernel modules directly refer to $(LINUX_DIR). And I guess there are other cases as well.

2/ The host-go-bootstrap case is also somewhat special.

Let's see what Arnout/Peter/Yann think about this, whether we really need to enforce this rule or not.

However, back to your original problem, I don't really see why the fact that host-go-bootstrap is installed in $(HOST_DIR) prevents you from installing the toolchain system-wide for use by other builds. Yes, it increases the size of the host directory, but other than that I don't see the technical limitation.

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

I agree that it is not ideal to reach into the builddir of another package. Ludovic Guegan implemented an alternative where the host-golang package encapsulated the bootstrap compiler rather than using a separate package. This makes some sense since the bootstrap compiler is purely internal to the build of the go compiler.

But the implementation meant that extraction and build of the bootstrap compiler was just replicaing what buildroot already does and gave less control (you couldn't run "make host-go-bootstrap-build" for example), which made it more tedious to iterate some change I was trying.

A separate go-bootstrap package makes better use of buildroot's infrastructure.

With two go compiler installed in the host dir, it becomes possible for someone to use the wrong one. Since the sources are also installed in the host dir, two sets of sources - one older than the other - will be installed. That's annoying when you want to index it (ctags, etc) for easy navigation. Having both full compilers (one completely unncessary) just feels like laying a land mine.

When it is just the outputdir, I don't mind so much but when installed system-wide, I'm usually quite careful about what gets installed.

I could also just delete it in a post-install script. But not doing something is better than doing it and undoing it.

Perhaps this is just all personal preference, in which case I'll maintain a local patch. I'm happy that go has made it into buildroot.

Hello,

That's obviously an alternate way: have the host-go compiler download the bootstrap compiler as <pkg>_EXTRA_DOWNLOADS, and take care to extract it / install it temporarily. Not sure it is really better, and I agree with you that the current status make a better use of the Buildroot package infrastructure.

Your arguments make sense. Let's see what others think about it.

I'm happy to have finally found the time to review and merge those patches. If you're interested in go, can you have a look at the build failures that I pointed to Geoff? See:

http://lists.busybox.net/pipermail/buildroot/2016-April/159336.html

Thanks a lot!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Hi,

I also don't really see any better way. This solution is OK with me.

Venlig hilsen,
Peter Korsgaard 

Add a new package 'go' which builds the host cross compiler and libraries for the go programming language.

---
 package/Config.in.host    |  1 +
 package/go/Config.in.host | 13 ++++++++
 package/go/go.hash        |  2 ++
 package/go/go.mk          | 75 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 91 insertions(+)
 create mode 100644 package/go/Config.in.host
 create mode 100644 package/go/go.hash
 create mode 100644 package/go/go.mk

index a91612b..e646dfe 100644
@@ -13,6 +13,7 @@ menu "Host utilities"
 	source "package/genext2fs/Config.in.host"
 	source "package/genimage/Config.in.host"
 	source "package/genpart/Config.in.host"
+	source "package/go/Config.in.host"
 	source "package/gptfdisk/Config.in.host"
 	source "package/imx-usb-loader/Config.in.host"
 	source "package/jq/Config.in.host"

new file mode 100644
index 0000000..e69259c
@@ -0,0 +1,13 @@
+config BR2_PACKAGE_GO_ARCH_SUPPORTS
+	bool
+	default y
+	depends on BR2_arm || BR2_aarch64 || BR2_i386 || BR2_x86_64 || BR2_powerpc
+	depends on !BR2_ARM_CPU_ARMV4
+
+config BR2_PACKAGE_HOST_GO
+	bool "host go"
+	depends on BR2_PACKAGE_GO_ARCH_SUPPORTS
+	help
+	  The Go Programming Language.
+
+	  https://golang.org

new file mode 100644
index 0000000..e7c72de
@@ -0,0 +1,2 @@
+# Locally computed:
+sha256  754e06dab1c31ab168fc9db9e32596734015ea9e24bc44cae7f237f417ce4efe  go1.5.3.src.tar.gz

2.5.0

Hello,

Applied to master after doing the following changes:

[Thomas:
- Put the computation of GO_GOARM inside the ifeq ($(BR2_arm),y) condition rather than duplicating this condition.
- Remove the GO_GOARCH=unknown case, since there is no way to fall in this case as only supported architectures can use host-go.
- Remove the GO_GOARM=unknown case, since we are sure that only ARMv5/6/7 will use host-go.
- Rename HOST_GO_FINAL to HOST_GO_ROOT, since it's really the "root" of the Go installation.
- Remove visible Config.in.host option.]

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Flannel is a virtual network that gives a subnet to each host for use with container runtimes.

---
 package/Config.in            |  1 +
 package/flannel/Config.in    |  9 +++++++++
 package/flannel/flannel.hash |  2 ++
 package/flannel/flannel.mk   | 42 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 54 insertions(+)
 create mode 100644 package/flannel/Config.in
 create mode 100644 package/flannel/flannel.hash
 create mode 100644 package/flannel/flannel.mk

index 64822bf..b7d9a17 100644
@@ -1374,6 +1374,7 @@ endif
 	source "package/faifa/Config.in"
 	source "package/fastd/Config.in"
 	source "package/fcgiwrap/Config.in"
+	source "package/flannel/Config.in"
 	source "package/fmc/Config.in"
 	source "package/fping/Config.in"
 	source "package/gesftpserver/Config.in"

new file mode 100644
index 0000000..0da356d
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_FLANNEL
+	bool "flannel"
+	depends on BR2_PACKAGE_GO_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
+	help
+	  Flannel is a virtual network that gives a subnet to each host for use
+	  with container runtimes.
+
+	  https://github.com/coreos/flannel

new file mode 100644
index 0000000..a19329a
@@ -0,0 +1,2 @@
+# Locally computed:
+sha256 c09a76e80647282687b8db36f28dd3aede650f8f71dad0cf9f453f2ae3a0635d flannel-v0.5.5.tar.gz

2.5.0

Hello,

Applied to master after doing the following changes:

[Thomas:
- Don't select BR2_PACKAGE_HOST_GO since this option was removed.
- Take into account the rename of BR2_PACKAGE_GO_ARCH_SUPPORTS to BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS.
- Rewrap Config.in help text, it was slightly too large.
- Don't use the github function (the link was anyway incorrect), but instead a direct link to the tarball, since the tarballs are uploaded by the maintainers.
- Update the hash accordingly.
- Use $() to reference make variables and not ${}.
- Call the go compiler from $(HOST_DIR)/usr/bin, rather than from HOST_GO_ROOT.]

Thanks a lot!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

All,

I haven't had a chance yet to test these on an actual device. However, I've read through the patches and everything looks good to me. Also, a test compile resulted in success for my target env.

@Geoff you may add my Acked-by to the commits:

In early May we will be returning to work on the Docker-under-Buildroot related stuff, and I will be rebasing my Docker series on top of this. If someone else wants to pick that up, please feel free as my work on that is still a month out.

In the meantime, thanks for the work Geoff, and I look forward to using your patches in our lab.

Best,
Christian Stewart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160406/f249e968/attachment.html>

Hi Thomas,

We haven't heard anything from Ludovic since I started this work. Both yourself and Christian have reviewed my patches favorably. I have added in all the changes you recommended in your review.

I think these are now ready to merge. I pushed out a go-v3.1 branch that adds Christian's acked:

[1] https://github.com/glevand/buildroot--buildroot.git #go-v3.1

Please pull.

-Geoff

Hello,

Now that those patches have been merged, we're starting to see some build issues:

- With host-go:
http://autobuild.buildroot.org/?reason=host-go-1.5.3

- With flannel:
http://autobuild.buildroot.org/?reason=flannel-v0.5.5

Could you have a look at those build problems?

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

I'll look into them.

-Geoff

Just to give an update, I've got fixes for the host-go problems, and am working on the flannel problem.

WIP patches are in my master branch here:

git at github.com:glevand/buildroot--buildroot.git

-Geoff

Hello,

Hum, I'm looking at this repository, and I see only 3 branches, and none of them seem to contain things that look like fixes. Did I miss something?

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Geoff, Thomas,

On Sat, Apr 30, 2016 at 8:03 AM Thomas Petazzoni
<[email protected]> wrote: > Hum, I'm looking at this repository, and I see only 3 branches, and none of them seem to contain things that look like fixes. Did I miss something? If I try to directly compile something that references Go, it seems that Go does
not have the cross compiler as a dependency in the Makefile tree.

# runtime/cgo fork/exec
/home/paralin/Documents/skiffos/buildroot/output/host/usr/bin/arm-buildroot-linux-gnueabi-gcc:
no such file or directory
Indeed, the cross compiler toolchain is NOT compiled before Go. I think a
dependency needs to be added for this.
Thanks, Christian Stewart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160501/a2286545/attachment.html>

I have the patch below that adds BR2_TOOLCHAIN_HAS_THREADS. Is it enough to pull in the toolchain?

-Geoff

Hello,

No, adding dependencies at the Config.in level doesn't change anything in terms of build order.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

It seems one of these in go-bootstrap.mk should work, is either one prefered?

+HOST_GO_BOOTSTRAP_DEPENDENCIES = host-go-bootstrap

+HOST_GO_BOOTSTRAP_ADD_TOOLCHAIN_DEPENDENCY = YES

-Geoff

+HOST_GO_BOOTSTRAP_DEPENDENCIES = host-toolchain

I don't see how either of these could work... AFAICS, host-toolchain doesn't
exist, and _ADD_TOOLCHAIN_DEPENDENCY is only checked for target packages.

What should work however:

HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain

Regards,
Arnout

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Here's a browser link:

https://github.com/glevand/buildroot--buildroot/commits/master

You should see these:

b461b3a package/go: Add BR2_TOOLCHAIN_HAS_THREADS 149dbc4 package/go: Add not BR2_TOOLCHAIN_USES_UCLIBC abb61c6 package/go: Fix powerpc64 config typo 044579e package/go: Upgrade 1.5.3 to 1.6.2

I don't think 'package/go: Add not BR2_TOOLCHAIN_USES_UCLIBC' is the righ fix though. I think the problem is something else. I need to look at that one some more.

-Geoff