Skip to contentSkip to conversation

package/go: bump to version 1.12

PATCH New Christian Stewart 5 participants 10 messages Feb 26, 2019 – Mar 12, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/go: bump to version 1.12

Go 1.12 was released on 2/25/2019: https://blog.golang.org/go1.12

Additional notes on how Go modules will evolve in 2019 are here:
https://blog.golang.org/modules2019

In Go 1.12, module support remains the same as in Go 1.11. GOPATH is scheduled for deprecation in 1.13, however. The Buildroot Go package infrastructure should therefore aim to migrate to a new, currently undefined, Go module friendly compilation approach before Go 1.13 is released.

---
 package/go/go.hash | 2 +-
 package/go/go.mk   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
package/go/go.hash +1 -1
index 29891399c3..e78cfe410c 100644
@@ -1,3 +1,3 @@
 # From https://golang.org/dl/
-sha256	bc1ef02bb1668835db1390a2e478dcbccb5dd16911691af9d75184bbe5aa943e  go1.11.5.src.tar.gz
+sha256	09c43d3336743866f2985f566db0520b36f4992aea2b4b2fd9f52f17049e88f2  go1.12.src.tar.gz
 sha256	2d36597f7117c38b006835ae7f537487207d8ec407aa9d9980794b2030cbc067  LICENSE
package/go/go.mk +1 -1
index 7755a3f785..8709198b63 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-GO_VERSION = 1.11.5
+GO_VERSION = 1.12
 GO_SITE = https://storage.googleapis.com/golang
 GO_SOURCE = go$(GO_VERSION).src.tar.gz
 
Signature · 1 line
2.19.2
#1 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hello Christian,

Thanks for your patch.

Quoted 9 lines · On Mon, 25 Feb 2019 19:45:54 -0800 Christian Stewart <[email protected]> wrote:

Go 1.12 was released on 2/25/2019: https://blog.golang.org/go1.12

Additional notes on how Go modules will evolve in 2019 are here:
https://blog.golang.org/modules2019

In Go 1.12, module support remains the same as in Go 1.11. GOPATH is scheduled for deprecation in 1.13, however. The Buildroot Go package infrastructure should therefore aim to migrate to a new, currently undefined, Go module friendly compilation approach before Go 1.13 is released.

I am puzzled by your statement "currently undefined". Is it undefined in the sense "it is not defined yet how Buildroot will handle this" or "it is not defined in Go yet how the module compilation will work in 1.13" ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hi Thomas,

Quoted 4 lines · Thomas Petazzoni

I am puzzled by your statement "currently undefined". Is it undefined in the sense "it is not defined yet how Buildroot will handle this" or "it is not defined in Go yet how the module compilation will work in 1.13" ?

The Buildroot approach to handling it is currently not defined. Go Modules are built around fetching from the network and verifying downloads within the Go tool itself, while Buildroot likely will avoid any network calls outside of the official Buildroot download mechanism.So, something else will have to be designed to fix this.

Best,
Christian

#3 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hi Christian,

Quoted 10 lines · Christian Stewart

Hi Thomas,

Quoted 4 lines · Thomas Petazzoni

I am puzzled by your statement "currently undefined". Is it undefined in the sense "it is not defined yet how Buildroot will handle this" or "it is not defined in Go yet how the module compilation will work in 1.13" ?

Quoted 5 lines

The Buildroot approach to handling it is currently not defined. Go Modules are built around fetching from the network and verifying downloads within the Go tool itself, while Buildroot likely will avoid any network calls outside of the official Buildroot download mechanism.So, something else will have to be designed to fix this.

It's less of a problem with module support for go now, because the dependencies are proxyed and verified before being delivered to the go download mechanism, so the reproducibility is somewhat guaranteed. The problem remains for breaking the buildroot internal download mechanism and for not a so probably clear licensing of downloaded modules (I have to investigate this).

I proposed the approach of tainting a buildroot configuration in such cases but the series was not accepted...

Quoted 3 lines

Best,
Christian

#4 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hi all,

Quoted 7 lines · Angelo Compagnucci
Quoted 6 lines
Quoted 4 lines · Thomas Petazzoni

I am puzzled by your statement "currently undefined". Is it undefined in the sense "it is not defined yet how Buildroot will handle this" or "it is not defined in Go yet how the module compilation will work in 1.13" ?

Quoted 6 lines

It's less of a problem with module support for go now, because the dependencies are proxyed and verified before being delivered to the go download mechanism, so the reproducibility is somewhat guaranteed. The problem remains for breaking the buildroot internal download mechanism and for not a so probably clear licensing of downloaded modules (I have to investigate this).

I'm beginning to work on this now, would it be acceptable if Buildroot adopted the following approach:

- Remove GOPATH
- Configure Go to download module files to buildroot/dl/go-modules
- Set GOPROXY to allow network during download step only.
- Require go.mod and go.sum hashes for all Go packages.
- During extract step, run "go mod vendor" to extract code to build/
- Use vendored building mode "-mod=vendor"

This should allow Go to download and verify modules correctly during the download step, to the Buildroot dl/ tree, as well as disallowing network fetching during the build phase, and extracting versioned module code to the build directory.

Best,
Christian

#5 lore raw

PATCH 1/1 package/go: bump to version 1.12

Christian, All,

On 2019-03-10 01:23 -0800, Christian Stewart spake thusly:

Quoted 15 lines
Quoted 7 lines · Angelo Compagnucci
Quoted 6 lines
Quoted 4 lines · Thomas Petazzoni

I am puzzled by your statement "currently undefined". Is it undefined in the sense "it is not defined yet how Buildroot will handle this" or "it is not defined in Go yet how the module compilation will work in 1.13" ?

Quoted 6 lines

It's less of a problem with module support for go now, because the dependencies are proxyed and verified before being delivered to the go download mechanism, so the reproducibility is somewhat guaranteed. The problem remains for breaking the buildroot internal download mechanism and for not a so probably clear licensing of downloaded modules (I have to investigate this).

Quoted 5 lines

I'm beginning to work on this now, would it be acceptable if Buildroot adopted the following approach:

- Remove GOPATH
- Configure Go to download module files to buildroot/dl/go-modules

How would you get the modules list and their URIs, when we are just downloading? The package sources are not yet extracted.

Quoted 2 lines

- Set GOPROXY to allow network during download step only.
- Require go.mod and go.sum hashes for all Go packages.

Not sure what go.mod and go.sum are, but if you're suggesting that we have hashes in foo.hash for all modules in package foo, that would be nice, yes!

Quoted 1 line

- During extract step, run "go mod vendor" to extract code to build/

... getting the modules from $(BR2_DL_DIR) instead of downloading them that is?

Quoted 6 lines

- Use vendored building mode "-mod=vendor"

This should allow Go to download and verify modules correctly during the download step, to the Buildroot dl/ tree, as well as disallowing network fetching during the build phase, and extracting versioned module code to the build directory.

If that were possible, I would like that, yes. That'd be good!

However, I'd suggest that this extra module vendoring dance only occur if the package has set a specific variable, like we do for git submodules:

FOO_GO_MODULES = YES

(I almost wrote GOO_MODULES ;-) )

That way, if a package does not explicitly request them, it would fail at build time since they would not be available and would not be already downloaded.

Of course, if we expect the majority of go packages to have modules, that variable would be moot...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hi Yann,

Quoted 2 lines · Yann E. MORIN
Quoted 2 lines

I'm beginning to work on this now, would it be acceptable if Buildroot adopted the following approach:

Here's a brief explanation of what Go modules are from the perspective of a Buildroot developer.

Reference: https://github.com/golang/go/wiki/Modules

Go code uses a package-based imports system. Imports are relative to a root directory, previously called GOPATH.

import "github.com/docker/docker/pkg/myutils"

This would resolve to $GOPATH/src/github.com/docker/docker/pkg/myutils.

Note, although Github is in the import, it is not necessary to fetch the code from Github. This is just a naming convention.

Buildroot packages currently use an additional feature in the Go tool which allows packages to avoid using GOPATH by "vendoring" dependencies
- copying the code directly into the Git repository.

vendor/github.com/docker/docker/pkg/myutils

Today all of our Buildroot Go modules use "vendor" to avoid downloading dependencies. But, this requires that the Go projects we add to Buildroot include all of their dependencies in their repositories, which is a mess. It also does not allow us the opportunity to validate dependency changes when upgrading packages in Buildroot.

The Go module system aims to completely replace the GOPATH mechanism by allowing the Go tool to reason about package versioning, code fetching, proxying, and checksumming.

A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers.

module mymodule

require (
github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible )

The Go tool creates a go.sum file next to the go.mod file. The go.sum file is akin to Node's package-shrinkwrap.json.

github.com/aws/aws-sdk-go v1.15.31/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= github.com/aws/aws-sdk-go v1.17.12 h1:jMFwRUaM0LcfdenfvbDLePNoWSoCdOHqF4RCvSB4xNQ= github.com/aws/aws-sdk-go v1.17.12/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible h1:8fBbhRkI5/0ocLFbrhPgnGUm0ogc+Gko1cRodPWDKX4= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=

The file contains import paths, versions, and file hashes for the download as well as the contained go.mod file.

With replace directives, it's possible to link together local copies of modules:

replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi )

When GO111MODULE=on, all of the Go tool commands become "module-aware." GOPATH is no longer required. Running "go build" for example will fetch dependencies from the Internet, checksum them, and extract to a staging directory (defaulting currently to GOPATH/pkg/gomod or so). Imports are automatically resolved to the appropriately versioned staging directory.

There are various ways to control the download / extract behavior:

The GOPROXY environment variable allows further control over the download source. If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems. Setting GOPROXY to "off" disallows downloading modules from any source. Otherwise, GOPROXY is expected to be the URL of a module proxy, in which case the go command will fetch all modules from that proxy. No matter the source of the modules, downloaded modules must match existing entries in go.sum...

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

GOPROXY additionally supports file:// URLs.

---

With the above in mind, here's how I believe we could integrate this with Buildroot:

Quoted 1 line
Quoted 1 line

- Remove GOPATH

GOPATH is deprecated anyway so this will be necessary. I'd strongly recommend that Buildroot make a hard requirement that all Go-based packages use the go.mod and go.sum construction, and configure Go to ignore the existing vendor/ trees. This would allow us to pin versioning and tightly control download checksumming through the package.hash -> go.sum merkle tree.

If a package does not have a go.mod or go.sum file, a patch would be added to Buildroot to include the files with the correct pinned dependencies.

Quoted 1 line
Quoted 1 line

- Configure Go to download module files to buildroot/dl/go-modules

We can set the paths such that the $GOPATH/pkg/mod/cache/download directory resolves to buildroot/dl/go-modules. According to the above documentation, a Buildroot user could then serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.

Initially, the desired behavior would be implemented by setting GOPROXY to "direct" (or perhaps the Buildroot download mirrors?) during the downloading phase.

Quoted 2 lines

How would you get the modules list and their URIs, when we are just downloading? The package sources are not yet extracted.

This is a good question. There are a couple of options. One conservative approach is to download and extract, and use the "extract" step to do the second stage of module downloading after extracting the package sources.

A more radical approach, which would perhaps be cleaner, would be to disable the Buildroot download mechanism completely for Go packages, include the root go.mod and go.sum in Buildroot next to the .mk and Config files, and allow Go to manage fetching all of the dependencies including the root package.

Quoted 1 line
Quoted 1 line

- Set GOPROXY to allow network during download step only.

This would be, during the download phase, execute "go mod download"

During the extract phase, we would create the build/package/ dir under the output directory as usual in Buildroot, then execute "go mod vendor" to extract the code sources into build/package/vendor

During the build phase, we would then disable the Go moduling system and compile from the extracted "vendor" tree. This would help improve build stability and determinism by only engaging the Module code during download / extract.

Quoted 5 lines
Quoted 1 line

- Require go.mod and go.sum hashes for all Go packages.

Not sure what go.mod and go.sum are, but if you're suggesting that we have hashes in foo.hash for all modules in package foo, that would be nice, yes!

They are explained above - these files specify the required dependency versions, and hash the downloaded code artifacts to ensure integrity.

Quoted 3 lines
Quoted 1 line

- During extract step, run "go mod vendor" to extract code to build/

... getting the modules from $(BR2_DL_DIR) instead of downloading them that is?

Exactly.

Quoted 8 lines
Quoted 1 line

- Use vendored building mode "-mod=vendor"

If that were possible, I would like that, yes. That'd be good!

However, I'd suggest that this extra module vendoring dance only occur if the package has set a specific variable, like we do for git submodules:

Quoted 7 lines

FOO_GO_MODULES = YES

(I almost wrote GOO_MODULES ;-) )

That way, if a package does not explicitly request them, it would fail at build time since they would not be available and would not be already downloaded.

The only situation where we would not want to use Go modules is if the package has no external dependencies, or if we want to use the old-fashioned vendor/ tree approach. I can prototype this in a way where the old behavior will be kept without a FOO_GO_MODULES flag, if this is something that we want.

Quoted 2 lines

Of course, if we expect the majority of go packages to have modules, that variable would be moot...

Best regards,
Christian Stewart

#7 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hi Christian,

Thanks for this excellent write-up! It certainly clarifies things.

On 11/03/2019 01:08, Christian Stewart wrote:
[snip]

Quoted 8 lines

A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers.

Quoted 6 lines

module mymodule

require (
github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible )

So, since the version is explicit, it's actually completely equivalent to the
vendoring: it's still impossible to treat the modules as packages in the Buildroot sense, since Buildroot doesn't allow the same package to have different versions. It's essentially just making our life more difficult :-(.

That said, at least it's a consistent way for Go packages to work.

Another thing: the module specification is just a path component, not a full
URL. Is the https:// implicit? Does it always pull from git? Or does it have some kind of autoconversion similar to our github helper?

[snip]

Quoted 9 lines

With the above in mind, here's how I believe we could integrate this with Buildroot:

Quoted 1 line
Quoted 1 line

- Remove GOPATH

GOPATH is deprecated anyway so this will be necessary. I'd strongly recommend that Buildroot make a hard requirement that all Go-based packages use the go.mod and go.sum construction, and configure Go to ignore the existing vendor/ trees.

I've taken a look at the upstream master branch of the go packages we currently
have, and none of them seems to have a go.mod yet... So I think this is a little bit premature.

Quoted 3 lines

This would allow us to pin versioning
and tightly control download checksumming through the package.hash -> go.sum merkle tree.

I may miss something, but if it's in vendor/ in the upstream git repo (either
as a submodule or as a copy), it's also tightly controlled and checksummed, right? So I don't think this is really bringing any value. For us, I mean - I imagine it *does* bring value for people developing Go packages.

Quoted 5 lines

If a package does not have a go.mod or go.sum file, a patch would be added to Buildroot to include the files with the correct pinned dependencies.

Quoted 1 line
Quoted 1 line

- Configure Go to download module files to buildroot/dl/go-modules

If all modules go together in a single location, there is a risk that there
will be two modules with the same name and version that are in reality different modules. Or are the full paths repeated there? I.e., does the zip file end up under a github.com/foo/bar directory?

Quoted 7 lines

We can set the paths such that the $GOPATH/pkg/mod/cache/download directory resolves to buildroot/dl/go-modules. According to the above documentation, a Buildroot user could then serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.

No need for it to be configurable, it could be the BR2_PRIMARY_SITE.

However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the
full solution would be to add something like go download infra that first sets GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally $(BR2_SECONDARY_SITE)/go-modules.

Except that I have another idea, see below.

Quoted 12 lines

Initially, the desired behavior would be implemented by setting GOPROXY to "direct" (or perhaps the Buildroot download mirrors?) during the downloading phase.

Quoted 2 lines

How would you get the modules list and their URIs, when we are just downloading? The package sources are not yet extracted.

This is a good question. There are a couple of options. One conservative approach is to download and extract, and use the "extract" step to do the second stage of module downloading after extracting the package sources.

Quoted 5 lines

A more radical approach, which would perhaps be cleaner, would be to disable the Buildroot download mechanism completely for Go packages, include the root go.mod and go.sum in Buildroot next to the .mk and Config files, and allow Go to manage fetching all of the dependencies including the root package.

That, combined with the go download infra I mentioned above, would indeed be a
feasible approach.

Ideally, the go.mod would include the package itself too, so we don't need to
download in two steps (i.e. first the package itself, then the modules it uses). But that might be a bit more difficult.

But again, see below for an alternative (cliffhanger...)

Quoted 7 lines
Quoted 1 line
Quoted 1 line

- Set GOPROXY to allow network during download step only.

This would be, during the download phase, execute "go mod download"

During the extract phase, we would create the build/package/ dir under the output directory as usual in Buildroot, then execute "go mod vendor" to extract the code sources into build/package/vendor

Alternatively, we could add a post-extract hook that executes "go mod vendor"
with GOPROXY=file://$(BR2_DL_DIR)/go-modules. But maybe that's what you meant.

Quoted 1 line

During the build phase, we would then disable the Go moduling system and

disable == GOPROXY=off, right?

Quoted 3 lines

compile from the extracted "vendor" tree. This would help improve build stability and determinism by only engaging the Module code during download / extract.

So this gives me the idea of the alternative...

If we anyway end up with everything extracted in the vendor tree, we can just
as well do that during the *download* step. Similar like we do for VCS
downloads: we get the *entire* source tree, including go modules, and tar that up.

We'd still have a go SITE_METHOD and corresponding download helper. This one
would first use another download method to get the base tarball, then extract it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball.

This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make
source-check' would all still work. So it would be (I think) the least invasive way to introduce this.

[snip]

Quoted 5 lines
Quoted 5 lines

However, I'd suggest that this extra module vendoring dance only occur if the package has set a specific variable, like we do for git submodules:

FOO_GO_MODULES = YES

With my proposal, this would be FOO_SITE_METHOD = go

Oh BTW we'd probably also need to support something like FOO_SITE_METHOD =
go+git. dl-wrapper will (should) strip off the first +, the go download helper would have to handle the second +. But that can be done in a follow-up patch :-)

Quoted 7 lines
Quoted 6 lines

(I almost wrote GOO_MODULES ;-) )

That way, if a package does not explicitly request them, it would fail at build time since they would not be available and would not be already downloaded.

Quoted 5 lines

The only situation where we would not want to use Go modules is if the package has no external dependencies, or if we want to use the old-fashioned vendor/ tree approach. I can prototype this in a way where the old behavior will be kept without a FOO_GO_MODULES flag, if this is something that we want.

I think it's worth prototyping something that is not complete yet, and send it
to the list with a lot of comments in the commit message about what is still missing. That's probably a better basis for discussion that English text.

Regards,
Arnout

Quoted 11 lines
Quoted 3 lines

Of course, if we expect the majority of go packages to have modules, that variable would be moot...

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#8 lore raw

PATCH 1/1 package/go: bump to version 1.12

Hi Arnout,

Quoted 4 lines · Arnout Vandecappelle

So, since the version is explicit, it's actually completely equivalent to the
vendoring: it's still impossible to treat the modules as packages in the Buildroot sense, since Buildroot doesn't allow the same package to have different versions. It's essentially just making our life more difficult :-(.

I don't think it makes sense to treat them as packages in this sense anyway.

Quoted 1 line

That said, at least it's a consistent way for Go packages to work.

This also seems to all be in line with the Go developers' intent and philosophy around dependency management: they prioritize compatibility and robust operation over deduplicating dependency instances.

Quoted 3 lines

Another thing: the module specification is just a path component, not a full URL. Is the https:// implicit? Does it always pull from git? Or does it have some kind of autoconversion similar to our github helper?

The Go code fetching mechanisms are complex, but heavily used, and I would assume they always use https shallow Git clones of the desired versions. The go.sum checksumming system ensures everything is consistent as well.

Quoted 8 lines
Quoted 4 lines

GOPATH is deprecated anyway so this will be necessary. I'd strongly recommend that Buildroot make a hard requirement that all Go-based packages use the go.mod and go.sum construction, and configure Go to ignore the existing vendor/ trees.

I've taken a look at the upstream master branch of the go packages we currently
have, and none of them seems to have a go.mod yet... So I think this is a little bit premature.

Removing GOPATH is not as extreme as it sounds. This is just a re-organization of the build code to simplify it quite a bit.

Today, the build code tries to synthesize a gopath (pkg-golang.mk):

$(2)_WORKSPACE ?= _gopath

$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) $(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) $(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR)

mkdir -p $$(dir $$($(2)_SRC_PATH))
ln -sf $$(@D) $$($(2)_SRC_PATH)

GOPATH="$$(@D)/$$($(2)_WORKSPACE)"

This creates the following directory structure:

$(@D)/pkg_gopath/src/github.com/myorg/mypackage/CODE_SYMLINKED_HERE

Then the Go tool is configured to use pkg_gopath as the GOPATH.

With the Go modules system, even if a Go project is not configured to use Go modules at all, it no longer is necessary to build a GOPATH.

With the absence of a go.mod and go.sum file, and the presence of a vendor/ dir, the Go tool will now correctly recognize the vendor/ and build outside GOPATH. This defaults GO111MODULE=on when Go is invoked outside GOPATH. In Go 1.13, GOPATH will be removed completely (or at least, deprecated).

In buildroot, we can simplify the code significantly with this new construction. We can simply extract code to $(@D), enable GO111MODULE, and the build will function correctly.

Quoted 7 lines
Quoted 3 lines

This would allow us to pin versioning
and tightly control download checksumming through the package.hash -> go.sum merkle tree.

I may miss something, but if it's in vendor/ in the upstream git repo (either
as a submodule or as a copy), it's also tightly controlled and checksummed, right?

The build would be executed with go build -mod=vendor, which tells the Go tool to ignore go.mod and use vendor/ only. Then, if vendor/ is already present in the upstream Git repo, we can skip the "go mod download" and "go mod vendor" steps.

The result is that the Go tool uses the following directory layout regardless of if Go modules download/extract was used during the download phase:

- $(@D)/vendor: contains all deps extracted
- $(@D)/go.mod: contains at minimum "module github.com/myorg/mymodule"

Quoted 2 lines

So I don't think this is really bringing any value. For us, I mean - I imagine it *does* bring value for people developing Go packages.

If we ignore vendor/ in the upstream Git repository and introduce our own go.mod, it would allow us to tightly control changes to dependencies (including security updates) within the Buildroot project, even if upstream maintainers do not update their vendor trees. This is especially important when we are using older revisions of projects that do not maintain release branches with vendored code.

In short, this would allow us to centralize the responsibility of Go dependency management in the Go module system, store all dependencies in the GOPROXY mechanism / Buildroot download directory, and force upgrades of dependencies for older revisions of projects without introducing large code vendoring patches into the Buildroot tree.

Quoted 3 lines
Quoted 3 lines

If a package does not have a go.mod or go.sum file, a patch would be added to Buildroot to include the files with the correct pinned dependencies.

This seems like a viable solution -

1. Copy the Buildroot package directory go.mod, go.sum in if exists.
2. If no go.mod exists in the project, write one with the minimum "module github.com/myorg/mymodule" line.
3. Execute "go mod download" and "go mod vendor" as discussed.

In point #1, if vendor/ already exists, and we indicate we want to override vendor/, then we can delete the upstream Git vendor/ tree in this step.

Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

- Configure Go to download module files to buildroot/dl/go-modules

If all modules go together in a single location, there is a risk that there
will be two modules with the same name and version that are in reality different modules. Or are the full paths repeated there? I.e., does the zip file end up under a github.com/foo/bar directory?

No. The Go tool uses an intricate path scheme to avoid collisions, and does not assume integrity of the on-disk cache (as far as I know).

Quoted 8 lines
Quoted 6 lines

We can set the paths such that the $GOPATH/pkg/mod/cache/download directory resolves to buildroot/dl/go-modules. According to the above documentation, a Buildroot user could then serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.

No need for it to be configurable, it could be the BR2_PRIMARY_SITE.

This seems like the best approach.

Here is some reference from go help goproxy:

A Go module proxy is any web server that can respond to GET requests for URLs of a specified form. The requests have no query parameters, so even a site serving from a fixed file system (including a file:/// URL) can be a module proxy.

The GET requests sent to a Go module proxy are:

GET $GOPROXY/<module>/@v/list returns a list of all known versions of the given module, one per line.

GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata about that version of the given module.

GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file for that version of the given module.

GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive for that version of the given module.

To avoid problems when serving from case-sensitive file systems, the <module> and <version> elements are case-encoded, replacing every uppercase letter with an exclamation mark followed by the corresponding lower-case letter: github.com/Azure encodes as github.com/!azure.

So perhaps we would want to add some suffix to PRIMARY_SITE.

Something further down the pipe, but worth looking at for those interested, is the upcoming Go code notary project:

https://go.googlesource.com/proposal/+/master/design/25530-notary.md

Quoted 5 lines

However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the
full solution would be to add something like go download infra that first sets GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally $(BR2_SECONDARY_SITE)/go-modules.

https://github.com/golang/go/issues/26334

Looks like indeed, it doesn't have a fallback yet. However, this is a bit awkward: what if our GOPROXY doesn't have just a single dependency?

The download tree looks like:

??? github.com
??? ??? davecgh
??? ??? ??? go-spew
??? ??? ??? @v

??? ???         ??? list
??? ???         ??? list.lock
??? ???         ??? v1.1.1.info
??? ???         ??? v1.1.1.mod
??? golang.org
    ??? x
        ??? crypto
        ??? ??? @v
        ???     ??? list
        ???     ??? list.lock
        ???     ??? v0.0.0-20180904163835-0709b304e793.info
        ???     ??? v0.0.0-20180904163835-0709b304e793.lock
        ???     ??? v0.0.0-20180904163835-0709b304e793.mod
        ???     ??? v0.0.0-20180904163835-0709b304e793.zip
        ???     ??? v0.0.0-20180904163835-0709b304e793.ziphash

I tried downloading just one dependency, then copying the $GOPATH/pkg/mod/cache/download/* files to GOPROXY=file://path/to/copy.

Then, invoking Go build:

go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793 Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.info Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.mod

Okay, this worked as expected. Now, adding a new dependency:

Fetching file://goproxy/github.com/urfave/cli/@v/list
Fetching file://goproxy/github.com/urfave/@v/list
Fetching file://goproxy/github.com/@v/list
build tmod: cannot load github.com/urfave/cli: cannot find module providing package github.com/urfave/cli

Hmm. Looks like it doesn't have a fallback yet.

Athens is another project addressing this area: https://github.com/gomods/athens

Quoted 8 lines

Except that I have another idea, see below.

Quoted 5 lines

A more radical approach, which would perhaps be cleaner, would be to disable the Buildroot download mechanism completely for Go packages, include the root go.mod and go.sum in Buildroot next to the .mk and Config files, and allow Go to manage fetching all of the dependencies including the root package.

Quoted 6 lines

That, combined with the go download infra I mentioned above, would indeed be a
feasible approach.

Ideally, the go.mod would include the package itself too, so we don't need to
download in two steps (i.e. first the package itself, then the modules it uses). But that might be a bit more difficult.

This is completely doable.

This approach actually opens up a few other interesting possibilties. We can add a command to set the environment variables up for the Go tool, and then execute the tool to automatically maintain the go.mod file for a particular package.

With no package arguments, 'go get' applies to the main module, and to the Go package in the current directory, if any. In particular, 'go get -u' and 'go get -u=patch' update all the dependencies of the main module.

If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod described above. Instead, it fails when any changes to go.mod are needed. This setting is most useful to check that go.mod does not need updates, such as in a continuous integration and testing system. The "go get" command remains permitted to update go.mod even with -mod=readonly, and the "go mod" commands do not take the -mod flag (or any other build flags).

Of course, this is speculative and not necessary for an initial implementation.

Quoted 2 lines

Alternatively, we could add a post-extract hook that executes "go mod vendor"
with GOPROXY=file://$(BR2_DL_DIR)/go-modules. But maybe that's what you meant.

This is correct.

Quoted 3 lines
Quoted 1 line

During the build phase, we would then disable the Go moduling system and

disable == GOPROXY=off, right?

GOPROXY=off in every situation other than when we want to download code.

Additionally, during build, we would use "-mod=vendor."

To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -mod=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.

Quoted 11 lines

If we anyway end up with everything extracted in the vendor tree, we can just
as well do that during the *download* step. Similar like we do for VCS
downloads: we get the *entire* source tree, including go modules, and tar that up.

We'd still have a go SITE_METHOD and corresponding download helper. This one
would first use another download method to get the base tarball, then extract it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball.

This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make
source-check' would all still work. So it would be (I think) the least invasive way to introduce this.

This seems like the right approach. This all feels similar to how Buildroot uses the Git tool to clone a repository, check out a revision, and bundle up a tarball today.

Quoted 5 lines

With my proposal, this would be FOO_SITE_METHOD = go

Oh BTW we'd probably also need to support something like FOO_SITE_METHOD =
go+git. dl-wrapper will (should) strip off the first +, the go download helper would have to handle the second +. But that can be done in a follow-up patch :-)

This is way too fancy for me to implement :)

Quoted 3 lines

I think it's worth prototyping something that is not complete yet, and send it
to the list with a lot of comments in the commit message about what is still missing. That's probably a better basis for discussion that English text.

Something to note: you can set GOMOD variable to the path to a go.mod file.

Okay, I'll have a look at:

- use vendor/ with GO111MODULE=on, remove gopath
- use -mod=vendor during build phase
- set GOCACHE properly (what is it set to today??)
- set GOPATH such that the Go tool downloads to dl/go-modules/pkg/mod

And (pick one):

- post-extract hook with optional go.mod in repository, "go mod vendor"
- additional site method which "go mod vendor" and re-compress to tarball

Best regards,
Christian Stewart

#9 lore raw

PATCH 1/1 package/go: bump to version 1.12

Quoted 9 lines · Christian Stewart

Hi Arnout,

Quoted 4 lines · Arnout Vandecappelle

So, since the version is explicit, it's actually completely equivalent to the
vendoring: it's still impossible to treat the modules as packages in the Buildroot sense, since Buildroot doesn't allow the same package to have different versions. It's essentially just making our life more difficult :-(.

I don't think it makes sense to treat them as packages in this sense anyway.

Actually, below, you do seem to suggest something that goes in that direction...

[snip]

Quoted 33 lines

Removing GOPATH is not as extreme as it sounds. This is just a re-organization of the build code to simplify it quite a bit.

Today, the build code tries to synthesize a gopath (pkg-golang.mk):

$(2)_WORKSPACE ?= _gopath

$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) $(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) $(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR)

mkdir -p $$(dir $$($(2)_SRC_PATH))
ln -sf $$(@D) $$($(2)_SRC_PATH)

GOPATH="$$(@D)/$$($(2)_WORKSPACE)"

This creates the following directory structure:

$(@D)/pkg_gopath/src/github.com/myorg/mypackage/CODE_SYMLINKED_HERE

Then the Go tool is configured to use pkg_gopath as the GOPATH.

With the Go modules system, even if a Go project is not configured to use Go modules at all, it no longer is necessary to build a GOPATH.

With the absence of a go.mod and go.sum file, and the presence of a vendor/ dir, the Go tool will now correctly recognize the vendor/ and build outside GOPATH. This defaults GO111MODULE=on when Go is invoked outside GOPATH. In Go 1.13, GOPATH will be removed completely (or at least, deprecated).

Quoted 3 lines

In buildroot, we can simplify the code significantly with this new construction. We can simply extract code to $(@D), enable GO111MODULE, and the build will function correctly.

Hell yeah! As a first step, that sounds like a great move. So we'd have:

1. Bump package/go
2. Set GO111MODULE=on and remove GOPATH from pkg-golang
3. Remove now-unused PKG_SRC_{DOMAIN,VENDOR,...} from packages
4. Remove this stuff from documentation

Step 2 should probably also set GOPROXY=off in all steps to avoid accidental
downloads. Maybe we should even export GOPROXY=off and enable it only in the specific cases where it needs to be enabled? That would be a fifth patch.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

This would allow us to pin versioning
and tightly control download checksumming through the package.hash -> go.sum merkle tree.

Quoted 8 lines
Quoted 3 lines

I may miss something, but if it's in vendor/ in the upstream git repo (either
as a submodule or as a copy), it's also tightly controlled and checksummed, right?

The build would be executed with go build -mod=vendor, which tells the Go tool to ignore go.mod and use vendor/ only. Then, if vendor/ is already present in the upstream Git repo, we can skip the "go mod download" and "go mod vendor" steps.

So that would be the current sitatuation then.

Quoted 6 lines

The result is that the Go tool uses the following directory layout regardless of if Go modules download/extract was used during the download phase:

- $(@D)/vendor: contains all deps extracted
- $(@D)/go.mod: contains at minimum "module github.com/myorg/mymodule"

Is this go.mod then autogenerated by 'go build -mod=vendor' or do we need to
generate it in a post-patch hook?

Quoted 10 lines
Quoted 2 lines

So I don't think this is really bringing any value. For us, I mean - I imagine it *does* bring value for people developing Go packages.

If we ignore vendor/ in the upstream Git repository and introduce our own go.mod, it would allow us to tightly control changes to dependencies (including security updates) within the Buildroot project, even if upstream maintainers do not update their vendor trees. This is especially important when we are using older revisions of projects that do not maintain release branches with vendored code.

Quoted 5 lines

In short, this would allow us to centralize the responsibility of Go dependency management in the Go module system, store all dependencies in the GOPROXY mechanism / Buildroot download directory, and force upgrades of dependencies for older revisions of projects without introducing large code vendoring patches into the Buildroot tree.

It sounds to me like a bit too big of an endeavor for the small Buildroot
project to take on... Upstream Go just doesn't have a decent security story for not-well-maintained packages, and I don't think that there's much we can do about it.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

If a package does not have a go.mod or go.sum file, a patch would be added to Buildroot to include the files with the correct pinned dependencies.

This seems like a viable solution -

Quoted 8 lines

1. Copy the Buildroot package directory go.mod, go.sum in if exists.
2. If no go.mod exists in the project, write one with the minimum "module github.com/myorg/mymodule" line.
3. Execute "go mod download" and "go mod vendor" as discussed.

In point #1, if vendor/ already exists, and we indicate we want to override vendor/, then we can delete the upstream Git vendor/ tree in this step.

Yes, sounds good to me. Though I would first like to see if there actually is
any project that has both a go.mod and an existing vendor/ tree.

What does "go mod vendor" do if a vendor/ tree already exists? Maybe we just
have to keep that behaviour, i.e. not delete the existing vendor tree and let the go tool take care of things. It should in principle do the right thing.

Quoted 42 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

- Configure Go to download module files to buildroot/dl/go-modules

If all modules go together in a single location, there is a risk that there
will be two modules with the same name and version that are in reality different modules. Or are the full paths repeated there? I.e., does the zip file end up under a github.com/foo/bar directory?

No. The Go tool uses an intricate path scheme to avoid collisions, and does not assume integrity of the on-disk cache (as far as I know).

Quoted 8 lines
Quoted 6 lines

We can set the paths such that the $GOPATH/pkg/mod/cache/download directory resolves to buildroot/dl/go-modules. According to the above documentation, a Buildroot user could then serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.

No need for it to be configurable, it could be the BR2_PRIMARY_SITE.

This seems like the best approach.

Here is some reference from go help goproxy:

A Go module proxy is any web server that can respond to GET requests for URLs of a specified form. The requests have no query parameters, so even a site serving from a fixed file system (including a file:/// URL) can be a module proxy.

The GET requests sent to a Go module proxy are:

GET $GOPROXY/<module>/@v/list returns a list of all known versions of the given module, one per line.

GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata about that version of the given module.

GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file for that version of the given module.

GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive for that version of the given module.

Quoted 6 lines

To avoid problems when serving from case-sensitive file systems, the <module> and <version> elements are case-encoded, replacing every uppercase letter with an exclamation mark followed by the corresponding lower-case letter: github.com/Azure encodes as github.com/!azure.

So perhaps we would want to add some suffix to PRIMARY_SITE.

That's what you proposed: go-modules.

But I'm not sure now: does the go mod download generate these auxiliary files
automatically in pkg/mod/cache/download? I.e. can we just serve $(BR2_DL_DIR) as PRIMARY, set GOPROXY to $(BR2_PRIMARY_SITE)/go-modules, and Bob's your uncle?

Quoted 47 lines

Something further down the pipe, but worth looking at for those interested, is the upcoming Go code notary project:

https://go.googlesource.com/proposal/+/master/design/25530-notary.md

Quoted 5 lines

However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the
full solution would be to add something like go download infra that first sets GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally $(BR2_SECONDARY_SITE)/go-modules.

https://github.com/golang/go/issues/26334

Looks like indeed, it doesn't have a fallback yet. However, this is a bit awkward: what if our GOPROXY doesn't have just a single dependency?

The download tree looks like:

??? github.com
??? ??? davecgh
??? ??? ??? go-spew
??? ??? ??? @v

??? ???         ??? list
??? ???         ??? list.lock
??? ???         ??? v1.1.1.info
??? ???         ??? v1.1.1.mod
??? golang.org
    ??? x
        ??? crypto
        ??? ??? @v
        ???     ??? list
        ???     ??? list.lock
        ???     ??? v0.0.0-20180904163835-0709b304e793.info
        ???     ??? v0.0.0-20180904163835-0709b304e793.lock
        ???     ??? v0.0.0-20180904163835-0709b304e793.mod
        ???     ??? v0.0.0-20180904163835-0709b304e793.zip
        ???     ??? v0.0.0-20180904163835-0709b304e793.ziphash

I tried downloading just one dependency, then copying the $GOPATH/pkg/mod/cache/download/* files to GOPROXY=file://path/to/copy.

Then, invoking Go build:

go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793 Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.info Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.mod

Quoted 8 lines

Okay, this worked as expected. Now, adding a new dependency:

Fetching file://goproxy/github.com/urfave/cli/@v/list
Fetching file://goproxy/github.com/urfave/@v/list
Fetching file://goproxy/github.com/@v/list
build tmod: cannot load github.com/urfave/cli: cannot find module providing package github.com/urfave/cli

Hmm. Looks like it doesn't have a fallback yet.

Heh, I didn't think of that: the tool tries to download *all* modules...

What would help is if there was some kind of keep-going option. Then we can do
all the fallback on the Buildroot side, like I mentioned: first try with PRIMARY (and download some dependencies and put it in the cache), then try with direct, then try with SECONDARY. The last one should succeed fully, reusing things from the cache if they were already downloaded before, and fetching them from direct or secondary if not.

Quoted 2 lines

Athens is another project addressing this area: https://github.com/gomods/athens

[snip]

Quoted 8 lines
Quoted 8 lines

If we anyway end up with everything extracted in the vendor tree, we can just
as well do that during the *download* step. Similar like we do for VCS
downloads: we get the *entire* source tree, including go modules, and tar that up.

We'd still have a go SITE_METHOD and corresponding download helper. This one
would first use another download method to get the base tarball, then extract it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball.

Quoted 7 lines
Quoted 3 lines

This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make
source-check' would all still work. So it would be (I think) the least invasive way to introduce this.

This seems like the right approach. This all feels similar to how Buildroot uses the Git tool to clone a repository, check out a revision, and bundle up a tarball today.

And I realize now: it would also handle legal-info, which would not be handled
correctly with everything that was proposed up to now (because the go modules are not part of $(PKG)_ALL_DOWNLOADS, they don't get copied to legal-info output).

Quoted 8 lines
Quoted 6 lines

With my proposal, this would be FOO_SITE_METHOD = go

Oh BTW we'd probably also need to support something like FOO_SITE_METHOD =
go+git. dl-wrapper will (should) strip off the first +, the go download helper would have to handle the second +. But that can be done in a follow-up patch :-)

This is way too fancy for me to implement :)

This would in fact not be needed if we would have the go.mod in the Buildroot
tree and use go mod to get the source itself too.

Quoted 10 lines
Quoted 4 lines

I think it's worth prototyping something that is not complete yet, and send it
to the list with a lot of comments in the commit message about what is still missing. That's probably a better basis for discussion that English text.

Something to note: you can set GOMOD variable to the path to a go.mod file.

Okay, I'll have a look at:

- use vendor/ with GO111MODULE=on, remove gopath

That one is not controversial for sure.

Quoted 1 line

- use -mod=vendor during build phase

Ditto. It should essentially change nothing for our current packages, right?

Quoted 1 line

- set GOCACHE properly (what is it set to today??)

You already sent a patch for that :-)

Quoted 1 line

- set GOPATH such that the Go tool downloads to dl/go-modules/pkg/mod

That one seems useless - with the steps above, there should never be any
download done by go mod.

Quoted 4 lines

And (pick one):

- post-extract hook with optional go.mod in repository, "go mod vendor"
- additional site method which "go mod vendor" and re-compress to tarball

So these are the more controversial ones. I think you should pick the one you
like most, make a quick prototype that works for one specific package, note what is missing in the commit log, and send it to the list for discussion.

Thanks for this work!

Regards,
Arnout