package/go: implement go modules integration
RFC PATCH 1/6 package/go: implement go modules integration
This commit moves from the GOPATH mechanism to the new GO111MODULE approach for Go based packages. Old Go packages (with the exception of docker-cli) will compile without changes (for example, mender, flanneld).
Cavets with the current implementation:
- "make source" may not produce a valid output
- module downloading occurs ignoring the Buildroot download server
- module downloading occurs in a post-patch hook
Go code uses a package-based imports system. Imports are relative to a root directory, previously called GOPATH.
import "github.com/docker/docker/pkg/myutils"
This would resolve to $GOPATH/src/github.com/docker/docker/pkg/myutils.
Buildroot packages previously used an additional feature in the Go tool which allows packages to avoid using GOPATH by "vendoring" dependencies - copying the code directly into the Git repository.
vendor/github.com/docker/docker/pkg/myutils
Old packages that used the vendor/ approach remain compatible via inferring the root import path from the download URL if no go.mod is present.
All current Buildroot Go modules use "vendor" to avoid downloading dependencies. This requires that the Go projects added to Buildroot include all of their dependencies in their repositories. It also does not allow us the opportunity to validate or adjust dependency versions when upgrading packages in Buildroot.
The Go module system aims to completely replace the GOPATH mechanism by allowing the Go tool to reason about package versioning, code fetching, proxying, and checksumming.
A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers.
module mymodule
require (
github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible )
The Go tool creates a go.sum file next to the go.mod file. The go.sum file is akin to Node's package-shrinkwrap.json.
github.com/aws/aws-sdk-go v1.15.31/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= github.com/aws/aws-sdk-go v1.17.12 h1:jMFwRUaM0LcfdenfvbDLePNoWSoCdOHqF4RCvSB4xNQ= github.com/aws/aws-sdk-go v1.17.12/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible h1:8fBbhRkI5/0ocLFbrhPgnGUm0ogc+Gko1cRodPWDKX4= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
The file contains import paths, versions, and file hashes for the download as well as the contained go.mod file.
With replace directives, it's possible to link together local copies of modules:
replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi )
When GO111MODULE=on, all of the Go tool commands become "module-aware." GOPATH is no longer required. Running "go build" for example will fetch dependencies from the Internet, checksum them, and extract to a staging directory (defaulting currently to GOPATH/pkg/gomod or so). Imports are automatically resolved to the appropriately versioned staging directory.
There are various ways to control the download / extract behavior:
The GOPROXY environment variable allows further control over the download source. If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems. Setting GOPROXY to "off" disallows downloading modules from any source. Otherwise, GOPROXY is expected to be the URL of a module proxy, in which case the go command will fetch all modules from that proxy. No matter the source of the modules, downloaded modules must match existing entries in go.sum...
Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.
GOPROXY additionally supports file:// URLs.
This commit sets GOPATH to $(DL_DIR)/go-module, as the Go module system will download and cache code sources in the GOPATH/pkg/mod path.
The go.mod and go.sum files can optionally be placed in $(2)_PKGDIR next to Config.in and other support files. They are copied in with a post-download hook, and "go mod download" is executed to pull the dependencies from the internet.
A hook is added to execute "go mod vendor".
Upstream vendor trees are still optionally supported, but can be overridden by placing go.mod into the Buildroot tree as described above. Package developers can alternatively specify LIBFOO_GOMOD to indicate the root module path. This allows the Go module tool to compile code using a legacy vendor/ tree, without a GOPATH to indicate the import path for the root module.
DOCKER_ENGINE_GOMOD = github.com/docker/docker
A Buildroot user can serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.
During the build phase, the "-mod=vendor" option is used to indicate that the extracted vendor/ tree (from the post-extract step) is to be used.
Go modules are never disabled with this approach. They are compatible with the legacy vendor/ tree approach by the above mechanisms.
Reference: https://github.com/golang/go/wiki/Modules
--- package/go/go.mk | 7 +++++- package/pkg-golang.mk | 50 ++++++++++++++++++++++++++++--------------- 2 files changed, 39 insertions(+), 18 deletions(-)
package/go/go.mk +6 -1
@@ -36,6 +36,9 @@ else ifeq ($(BR2_mips64el),y) GO_GOARCH = mips64le endif +# GOPATH is used only for Go module downloads. +HOST_GO_GOPATH = $(DL_DIR)/go-module + HOST_GO_DEPENDENCIES = host-go-bootstrap HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache HOST_GO_ROOT = $(HOST_DIR)/lib/go
@@ -44,10 +47,12 @@ HOST_GO_TARGET_CACHE = $(HOST_DIR)/usr/share/go-cache # For the convienience of target packages. HOST_GO_TOOLDIR = $(HOST_GO_ROOT)/pkg/tool/linux_$(GO_GOARCH) HOST_GO_TARGET_ENV = \ - GO111MODULE=off \ + GO111MODULE=on \ GOARCH=$(GO_GOARCH) \ GOCACHE="$(HOST_GO_TARGET_CACHE)" \ + GOPROXY=off \ GOROOT="$(HOST_GO_ROOT)" \ + GOPATH="$(HOST_GO_GOPATH)" \ CC="$(TARGET_CC)" \ CXX="$(TARGET_CXX)" \ GOTOOLDIR="$(HOST_GO_TOOLDIR)"
package/pkg-golang.mk +33 -17
77 lines not on this page · raw message
Signature · 1 line
2.19.2
RFC PATCH 2/6 package/runc: upgrade to go modules
This commit introduces go.mod and go.sum files derived from the vendor.conf in the upstream runc repository.
--- package/runc/go.mod | 26 ++++++++++++++++++++++++ package/runc/go.sum | 47 ++++++++++++++++++++++++++++++++++++++++++++ package/runc/runc.mk | 2 -- 3 files changed, 73 insertions(+), 2 deletions(-) create mode 100644 package/runc/go.mod create mode 100644 package/runc/go.sum
package/runc/go.mod new file +26 -0
@@ -0,0 +1,26 @@ +module github.com/opencontainers/runc + +// Derived from runc @ v1.0.0-rc6. +require ( + github.com/checkpoint-restore/go-criu v0.0.0-20181120144056-17b0214f6c48 + github.com/containerd/console v0.0.0-20180220200639-2748ece16665 + github.com/coreos/go-systemd v0.0.0-20161114122254-48702e0da86b + github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf // indirect + github.com/cyphar/filepath-securejoin v0.2.1 + github.com/docker/go-units v0.2.0 + github.com/godbus/dbus v0.0.0-20151105175453-c7fdd8b5cd55 + github.com/golang/protobuf v0.0.0-20170427213220-18c9bb326172 + github.com/mrunalp/fileutils v0.0.0-20160610222446-ed869b029674 + github.com/opencontainers/runtime-spec v0.0.0-20190207185410-29686dbc5559 + github.com/opencontainers/selinux v1.0.0-rc1 + github.com/pkg/errors v0.8.0 + github.com/seccomp/libseccomp-golang v0.0.0-20170424173420-84e90a91acea + github.com/sirupsen/logrus v0.0.0-20170713114250-a3f95b5c4235 + github.com/stretchr/testify v1.3.0 // indirect + github.com/syndtr/gocapability v0.0.0-20170704070218-db04d3cc01c8 + github.com/urfave/cli v0.0.0-20160917213806-d53eb991652b + github.com/vishvananda/netlink v0.0.0-20150820014904-1e2e08e8a2dc + github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc // indirect + golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 // indirect + golang.org/x/sys v0.0.0-20190204203706-41f3e6584952 +)
package/runc/go.sum new file +47 -0
47 lines not on this page · raw message
package/runc/runc.mk +0 -2
@@ -9,8 +9,6 @@ RUNC_SITE = $(call github,opencontainers,runc,$(RUNC_VERSION)) RUNC_LICENSE = Apache-2.0 RUNC_LICENSE_FILES = LICENSE -RUNC_WORKSPACE = Godeps/_workspace - RUNC_LDFLAGS = -X main.gitCommit=$(RUNC_VERSION) RUNC_TAGS = cgo static_build
Signature · 1 line
2.19.2
RFC PATCH 3/6 package/docker-containerd: upgrade to go modules
In reply to Christian StewartThis commit introduces go.mod and go.sum files derived from the vendor.conf in the upstream containerd repository.
--- .../docker-containerd/docker-containerd.mk | 9 +- package/docker-containerd/go.mod | 93 ++++++++ package/docker-containerd/go.sum | 200 ++++++++++++++++++ 3 files changed, 298 insertions(+), 4 deletions(-) create mode 100644 package/docker-containerd/go.mod create mode 100644 package/docker-containerd/go.sum
package/docker-containerd/docker-containerd.mk +5 -4
@@ -9,12 +9,13 @@ DOCKER_CONTAINERD_SITE = $(call github,containerd,containerd,$(DOCKER_CONTAINERD DOCKER_CONTAINERD_LICENSE = Apache-2.0 DOCKER_CONTAINERD_LICENSE_FILES = LICENSE -DOCKER_CONTAINERD_WORKSPACE = vendor - DOCKER_CONTAINERD_LDFLAGS = \ - -X github.com/docker/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION) + -X github.com/containerd/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION) -DOCKER_CONTAINERD_BUILD_TARGETS = cmd/ctr cmd/containerd cmd/containerd-shim +DOCKER_CONTAINERD_BUILD_TARGETS = \ + github.com/containerd/containerd/cmd/ctr \ + github.com/containerd/containerd/cmd/containerd \ + github.com/containerd/containerd/cmd/containerd-shim DOCKER_CONTAINERD_INSTALL_BINS = containerd containerd-shim
package/docker-containerd/go.mod new file +93 -0
93 lines not on this page · raw message
package/docker-containerd/go.sum new file +200 -0
200 lines not on this page · raw message
Signature · 1 line
2.19.2
RFC PATCH 3/6 package/docker-containerd: upgrade to go modules
package/docker-containerd/docker-containerd.mk DOCKER_CONTAINERD_SITE = $(call github,containerd,containerd,$(DOCKER_CONTAINERD · line 13
Quoted 3 lines · Christian Stewart
DOCKER_CONTAINERD_LDFLAGS= \package/docker-containerd/docker-containerd.mk +1
-1- -X github.com/docker/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION) + -X github.com/containerd/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION)
This looks like a completely independent change. Or is it because we now have a
go.mod that calls the module containerd/containerd instead of docker/containerd?
package/docker-containerd/docker-containerd.mk DOCKER_CONTAINERD_SITE = $(call github,containerd,containerd,$(DOCKER_CONTAINERD · line 18
Quoted 5 lines
package/docker-containerd/docker-containerd.mk +4
-1-DOCKER_CONTAINERD_BUILD_TARGETS = cmd/ctr cmd/containerd cmd/containerd-shim +DOCKER_CONTAINERD_BUILD_TARGETS = \ + github.com/containerd/containerd/cmd/ctr \ + github.com/containerd/containerd/cmd/containerd \ + github.com/containerd/containerd/cmd/containerd-shim
Same question, how is this related to the go modules?
Would it be possible (and useful) to add those prefixes automatically?
Regards,
Arnout
RFC PATCH 3/6 package/docker-containerd: upgrade to go modules
Hi Arnout,
Quoted 7 lines · Arnout Vandecappelle
package/docker-containerd/docker-containerd.mk DOCKER_CONTAINERD_SITE = $(call github,containerd,containerd,$(DOCKER_CONTAINERD · line 13
Quoted 3 lines · Christian Stewart
DOCKER_CONTAINERD_LDFLAGS= \package/docker-containerd/docker-containerd.mk +1
-1- -X github.com/docker/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION) + -X github.com/containerd/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION)This looks like a completely independent change. Or is it because we now have a
go.mod that calls the module containerd/containerd instead of docker/containerd?
This is a fix that could be applied seperately, IIF the root import path is github.com/containerd/containerd in the current in-tree version (I'd have to confirm to be sure).
Quoted 6 lines
package/docker-containerd/docker-containerd.mk DOCKER_CONTAINERD_SITE = $(call github,containerd,containerd,$(DOCKER_CONTAINERD · line 18
Quoted 4 lines
package/docker-containerd/docker-containerd.mk +4
-0+DOCKER_CONTAINERD_BUILD_TARGETS = \ + github.com/containerd/containerd/cmd/ctr \ + github.com/containerd/containerd/cmd/containerd \ + github.com/containerd/containerd/cmd/containerd-shimSame question, how is this related to the go modules?
Before, we specified which targets to build using a relative path like ./cmd/dockerd. Now, we can actually specify the fully qualified path to the "main" package, and the Go tool is smart enough to locate the correct path inside $(@D). It's also more readable to specify it this way, and is considered best practice.
Quoted 1 line
Would it be possible (and useful) to add those prefixes automatically?
I don't see why. Consider this: I can give you a list of packages:
- k8s.io/cmd/apiserver
- k8s.io/cmd/kubectl
- k8s.io/cmd/kubeadm
- github.com/containerd/containerd/cmd/ctr
... and all of these can be passed to "go build," which will resolve them and produce the outputs with the correct binary names (the last component of the path). It's not that much extra text to leave the "github.com/containerd/containerd" part in. It's a point of personal preference at this point though.
Best regards,
Christian
RFC PATCH 4/6 docker-cli: upgrade to go modules
In reply to Christian StewartThis commit introduces go.mod and go.sum files derived from the vendor.conf in the upstream Docker cli repository.
Replace directives are added to indicate the correct version of docker/engine to pull in. Additionally, a patch is added to bring in the upstream modifications to the vendored spf13/pflag package.
--- ...ude-modified-package-under-thirdpart.patch | 5259 +++++++++++++++++ package/docker-cli/docker-cli.mk | 3 +- package/docker-cli/go.mod | 127 + package/docker-cli/go.sum | 419 ++ 4 files changed, 5806 insertions(+), 2 deletions(-) create mode 100644 package/docker-cli/0001-spf13-pflag-include-modified-package-under-thirdpart.patch create mode 100644 package/docker-cli/go.mod create mode 100644 package/docker-cli/go.sum
package/docker-cli/0001-spf13-pflag-include-modified-package-under-thirdpart.patch new file +5259 -0
@@ -0,0 +1,5259 @@ +From c46c6786948f8a183ca3e0717e5a8a258c845712 Mon Sep 17 00:00:00 2001 +From: Christian Stewart <[email protected]> +Date: Sat, 16 Mar 2019 17:19:43 -0700 +Subject: [PATCH] spf13/pflag: include modified package under thirdparty + +The Docker upstream codebase has a vendored version of spf13/goflag with in-tree +modifications. The modifications add a IPNetSliceVar to pflag.FlagSet. + +Signed-off-by: Christian Stewart <[email protected]> +--- + thirdparty/pflag/LICENSE | 28 + + thirdparty/pflag/README.md | 296 +++++++ + thirdparty/pflag/bool.go | 94 ++ + thirdparty/pflag/bool_slice.go | 147 ++++ + thirdparty/pflag/bytes.go | 209 +++++ + thirdparty/pflag/count.go | 96 ++ + thirdparty/pflag/duration.go | 86 ++ + thirdparty/pflag/duration_slice.go | 128 +++ + thirdparty/pflag/flag.go | 1224 ++++++++++++++++++++++++++ + thirdparty/pflag/float32.go | 88 ++ + thirdparty/pflag/float64.go | 84 ++ + thirdparty/pflag/go.mod | 3 + + thirdparty/pflag/golangflag.go | 105 +++ + thirdparty/pflag/int.go | 84 ++ + thirdparty/pflag/int16.go | 88 ++ + thirdparty/pflag/int32.go | 88 ++ + thirdparty/pflag/int64.go | 84 ++ + thirdparty/pflag/int8.go | 88 ++ + thirdparty/pflag/int_slice.go | 128 +++ + thirdparty/pflag/ip.go | 94 ++ + thirdparty/pflag/ip_slice.go | 148 ++++ + thirdparty/pflag/ipmask.go | 122 +++ + thirdparty/pflag/ipnet.go | 98 +++ + thirdparty/pflag/ipnet_slice.go | 147 ++++ + thirdparty/pflag/string.go | 80 ++ + thirdparty/pflag/string_array.go | 103 +++ + thirdparty/pflag/string_slice.go | 149 ++++ + thirdparty/pflag/string_to_int.go | 149 ++++ + thirdparty/pflag/string_to_string.go | 160 ++++ + thirdparty/pflag/uint.go | 88 ++ + thirdparty/pflag/uint16.go | 88 ++ + thirdparty/pflag/uint32.go | 88 ++ + thirdparty/pflag/uint64.go | 88 ++ + thirdparty/pflag/uint8.go | 88 ++ + thirdparty/pflag/uint_slice.go | 126 +++ + 35 files changed, 4964 insertions(+) + create mode 100644 thirdparty/pflag/LICENSE + create mode 100644 thirdparty/pflag/README.md + create mode 100644 thirdparty/pflag/bool.go + create mode 100644 thirdparty/pflag/bool_slice.go + create mode 100644 thirdparty/pflag/bytes.go + create mode 100644 thirdparty/pflag/count.go + create mode 100644 thirdparty/pflag/duration.go + create mode 100644 thirdparty/pflag/duration_slice.go + create mode 100644 thirdparty/pflag/flag.go + create mode 100644 thirdparty/pflag/float32.go + create mode 100644 thirdparty/pflag/float64.go + create mode 100644 thirdparty/pflag/go.mod + create mode 100644 thirdparty/pflag/golangflag.go + create mode 100644 thirdparty/pflag/int.go + create mode 100644 thirdparty/pflag/int16.go + create mode 100644 thirdparty/pflag/int32.go + create mode 100644 thirdparty/pflag/int64.go + create mode 100644 thirdparty/pflag/int8.go + create mode 100644 thirdparty/pflag/int_slice.go + create mode 100644 thirdparty/pflag/ip.go + create mode 100644 thirdparty/pflag/ip_slice.go + create mode 100644 thirdparty/pflag/ipmask.go + create mode 100644 thirdparty/pflag/ipnet.go + create mode 100644 thirdparty/pflag/ipnet_slice.go + create mode 100644 thirdparty/pflag/string.go + create mode 100644 thirdparty/pflag/string_array.go + create mode 100644 thirdparty/pflag/string_slice.go + create mode 100644 thirdparty/pflag/string_to_int.go + create mode 100644 thirdparty/pflag/string_to_string.go + create mode 100644 thirdparty/pflag/uint.go + create mode 100644 thirdparty/pflag/uint16.go + create mode 100644 thirdparty/pflag/uint32.go + create mode 100644 thirdparty/pflag/uint64.go + create mode 100644 thirdparty/pflag/uint8.go + create mode 100644 thirdparty/pflag/uint_slice.go + +diff --git a/thirdparty/pflag/LICENSE b/thirdparty/pflag/LICENSE +new file mode 100644 +index 00000000..63ed1cfe +--- /dev/null ++++ b/thirdparty/pflag/LICENSE +@@ -0,0 +1,28 @@ ++Copyright (c) 2012 Alex Ogier. All rights reserved. ++Copyright (c) 2012 The Go Authors. All rights reserved. ++ ++Redistribution and use in source and binary forms, with or without ++modification, are permitted provided that the following conditions are ++met: ++ ++ * Redistributions of source code must retain the above copyright ++notice, this list of conditions and the following disclaimer. ++ * Redistributions in binary form must reproduce the above ++copyright notice, this list of conditions and the following disclaimer ++in the documentation and/or other materials provided with the ++distribution. ++ * Neither the name of Google Inc. nor the names of its ++contributors may be used to endorse or promote products derived from ++this software without specific prior written permission. ++ ++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ++"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT ++LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR ++A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT ++OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, ++SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT ++LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, ++DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY ++THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT ++(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE ++OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +diff --git a/thirdparty/pflag/README.md b/thirdparty/pflag/README.md +new file mode 100644 +index 00000000..b052414d +--- /dev/null ++++ b/thirdparty/pflag/README.md +@@ -0,0 +1,296 @@ ++[](https://travis-ci.org/spf13/pflag) ++[](https://goreportcard.com/report/github.com/spf13/pflag) ++[](https://godoc.org/github.com/spf13/pflag) ++ ++## Description ++ ++pflag is a drop-in replacement for Go's flag package, implementing ++POSIX/GNU-style --flags. ++ ++pflag is compatible with the [GNU extensions to the POSIX recommendations ++for command-line options][1]. For a more precise description, see the ++"Command-line flag syntax" section below. ++ ++[1]: http://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html ++ ++pflag is available under the same style of BSD license as the Go language, ++which can be found in the LICENSE file. ++ ++## Installation ++ ++pflag is available using the standard `go get` command. ++ ++Install by running: ++ ++ go get github.com/spf13/pflag ++ ++Run tests by running: ++ ++ go test github.com/spf13/pflag ++ ++## Usage ++ ++pflag is a drop-in replacement of Go's native flag package. If you import ++pflag under the name "flag" then all code should continue to function ++with no changes. ++ ++``` go ++import flag "github.com/spf13/pflag" ++``` ++ ++There is one exception to this: if you directly instantiate the Flag struct ++there is one more field "Shorthand" that you will need to set. ++Most code never instantiates this struct directly, and instead uses ++functions such as String(), BoolVar(), and Var(), and is therefore ++unaffected. ++ ++Define flags using flag.String(), Bool(), Int(), etc. ++ ++This declares an integer flag, -flagname, stored in the pointer ip, with type *int. ++ ++``` go ++var ip *int = flag.Int("flagname", 1234, "help message for flagname") ++``` ++ ++If you like, you can bind the flag to a variable using the Var() functions. ++ ++``` go ++var flagvar int ++func init() { ++ flag.IntVar(&flagvar, "flagname", 1234, "help message for flagname") ++} ++``` ++ ++Or you can create custom flags that satisfy the Value interface (with ++pointer receivers) and couple them to flag parsing by ++ ++``` go ++flag.Var(&flagVal, "name", "help message for flagname") ++``` ++ ++For such flags, the default value is just the initial value of the variable. ++ ++After all flags are defined, call ++ ++``` go ++flag.Parse() ++``` ++ ++to parse the command line into the defined flags. ++ ++Flags may then be used directly. If you're using the flags themselves, ++they are all pointers; if you bind to variables, they're values. ++ ++``` go ++fmt.Println("ip has value ", *ip) ++fmt.Println("flagvar has value ", flagvar) ++``` ++ ++There are helpers function to get values later if you have the FlagSet but ++it was difficult to keep up with all of the flag pointers in your code. ++If you have a pflag.FlagSet with a flag called 'flagname' of type int you ++can use GetInt() to get the int value. But notice that 'flagname' must exist ++and it must be an int. GetString("flagname") will fail. ++ ++``` go ++i, err := flagset.GetInt("flagname") ++``` ++ ++After parsing, the arguments after the flag are available as the ++slice flag.Args() or individually as flag.Arg(i). ++The arguments are indexed from 0 through flag.NArg()-1. ++ ++The pflag package also defines some new functions that are not in flag, ++that give one-letter shorthands for flags. You can use these by appending ++'P' to the name of any function that defines a flag. ++ ++``` go ++var ip = flag.IntP("flagname", "f", 1234, "help message") ++var flagvar bool ++func init() { ++ flag.BoolVarP(&flagvar, "boolname", "b", true, "help message") ++} ++flag.VarP(&flagVal, "varname", "v", "help message") ++``` ++ ++Shorthand letters can be used with single dashes on the command line. ++Boolean shorthand flags can be combined with other shorthand flags. ++ ++The default set of command-line flags is controlled by ++top-level functions. The FlagSet type allows one to define ++independent sets of flags, such as to implement subcommands ++in a command-line interface. The methods of FlagSet are ++analogous to the top-level functions for the command-line ++flag set. ++ ++## Setting no option default values for flags ++ ++After you create a flag it is possible to set the pflag.NoOptDefVal for ++the given flag. Doing this changes the meaning of the flag slightly. If ++a flag has a NoOptDefVal and the flag is set on the command line without ++an option the flag will be set to the NoOptDefVal. For example given: ++ ++``` go ++var ip = flag.IntP("flagname", "f", 1234, "help message") ++flag.Lookup("flagname").NoOptDefVal = "4321" ++``` ++ ++Would result in something like ++ ++| Parsed Arguments | Resulting Value | ++| ------------- | ------------- | ++| --flagname=1357 | ip=1357 | ++| --flagname | ip=4321 | ++| [nothing] | ip=1234 | ++ ++## Command line flag syntax ++ ++``` ++--flag // boolean flags, or flags with no option default values ++--flag x // only on flags without a default value ++--flag=x ++``` ++ ++Unlike the flag package, a single dash before an option means something ++different than a double dash. Single dashes signify a series of shorthand ++letters for flags. All but the last shorthand letter must be boolean flags ++or a flag with a default value ++ ++``` ++// boolean or flags where the 'no option default value' is set ++-f ++-f=true ++-abc ++but ++-b true is INVALID ++ ++// non-boolean and flags without a 'no option default value' ++-n 1234 ++-n=1234 ++-n1234 ++ ++// mixed ++-abcs "hello" ++-absd="hello" ++-abcs1234 ++``` ++ ++Flag parsing stops after the terminator "--". Unlike the flag package, ++flags can be interspersed with arguments anywhere on the command line ++before this terminator. ++ ++Integer flags accept 1234, 0664, 0x1234 and may be negative. ++Boolean flags (in their long form) accept 1, 0, t, f, true, false, ++TRUE, FALSE, True, False. ++Duration flags accept any input valid for time.ParseDuration. ++ ++## Mutating or "Normalizing" Flag names ++ ++It is possible to set a custom flag name 'normalization function.' It allows flag names to be mutated both when created in the code and when used on the command line to some 'normalized' form. The 'normalized' form is used for comparison. Two examples of using the custom normalization func follow. ++ ++**Example #1**: You want -, _, and . in flags to compare the same. aka --my-flag == --my_flag == --my.flag ++ ++``` go ++func wordSepNormalizeFunc(f *pflag.FlagSet, name string) pflag.NormalizedName { ++ from := []string{"-", "_"} ++ to := "." ++ for _, sep := range from { ++ name = strings.Replace(name, sep, to, -1) ++ } ++ return pflag.NormalizedName(name) ++} ++ ++myFlagSet.SetNormalizeFunc(wordSepNormalizeFunc) ++``` ++ ++**Example #2**: You want to alias two flags. aka --old-flag-name == --new-flag-name ++ ++``` go ++func aliasNormalizeFunc(f *pflag.FlagSet, name string) pflag.NormalizedName { ++ switch name { ++ case "old-flag-name": ++ name = "new-flag-name" ++ break ++ } ++ return pflag.NormalizedName(name) ++} ++ ++myFlagSet.SetNormalizeFunc(aliasNormalizeFunc) ++``` ++ ++## Deprecating a flag or its shorthand ++It is possible to deprecate a flag, or just its shorthand. Deprecating a flag/shorthand hides it from help text and prints a usage message when the deprecated flag/shorthand is used. ++ ++**Example #1**: You want to deprecate a flag named "badflag" as well as inform the users what flag they should use instead. ++```go ++// deprecate a flag by specifying its name and a usage message ++flags.MarkDeprecated("badflag", "please use --good-flag instead") ++``` ++This hides "badflag" from help text, and prints `Flag --badflag has been deprecated, please use --good-flag instead` when "badflag" is used. ++ ++**Example #2**: You want to keep a flag name "noshorthandflag" but deprecate its shortname "n". ++```go ++// deprecate a flag shorthand by specifying its flag name and a usage message ++flags.MarkShorthandDeprecated("noshorthandflag", "please use --noshorthandflag only") ++``` ++This hides the shortname "n" from help text, and prints `Flag shorthand -n has been deprecated, please use --noshorthandflag only` when the shorthand "n" is used. ++ ++Note that usage message is essential here, and it should not be empty. ++ ++## Hidden flags ++It is possible to mark a flag as hidden, meaning it will still function as normal, however will not show up in usage/help text. ++ ++**Example**: You have a flag named "secretFlag" that you need for internal use only and don't want it showing up in help text, or for its usage text to be available. ++```go ++// hide a flag by specifying its name ++flags.MarkHidden("secretFlag") ++``` ++ ++## Disable sorting of flags ++`pflag` allows you to disable sorting of flags for help and usage message. ++ ++**Example**: ++```go ++flags.BoolP("verbose", "v", false, "verbose output") ++flags.String("coolflag", "yeaah", "it's really cool flag") ++flags.Int("usefulflag", 777, "sometimes it's very useful") ++flags.SortFlags = false ++flags.PrintDefaults() ++``` ++**Output**: ++``` ++ -v, --verbose verbose output ++ --coolflag string it's really cool flag (default "yeaah") ++ --usefulflag int sometimes it's very useful (default 777) ++``` ++ ++ ++## Supporting Go flags when using pflag ++In order to support flags defined using Go's `flag` package, they must be added to the `pflag` flagset. This is usually necessary ++to support flags defined by third-party dependencies (e.g. `golang/glog`). ++ ++**Example**: You want to add the Go flags to the `CommandLine` flagset ++```go ++import ( ++ goflag "flag" ++ flag "github.com/spf13/pflag" ++) ++ ++var ip *int = flag.Int("flagname", 1234, "help message for flagname") ++ ++func main() { ++ flag.CommandLine.AddGoFlagSet(goflag.CommandLine) ++ flag.Parse() ++} ++``` ++ ++## More info ++ ++You can see the full reference documentation of the pflag package ++[at godoc.org][3], or through go's standard documentation system by ++running `godoc -http=:6060` and browsing to ++[http://localhost:6060/pkg/github.com/spf13/pflag][2] after ++installation. ++ ++[2]: http://localhost:6060/pkg/github.com/spf13/pflag ++[3]: http://godoc.org/github.com/spf13/pflag +diff --git a/thirdparty/pflag/bool.go b/thirdparty/pflag/bool.go +new file mode 100644 +index 00000000..c4c5c0bf +--- /dev/null ++++ b/thirdparty/pflag/bool.go +@@ -0,0 +1,94 @@ ++package pflag ++ ++import "strconv" ++ ++// optional interface to indicate boolean flags that can be ++// supplied without "=value" text ++type boolFlag interface { ++ Value ++ IsBoolFlag() bool ++} ++ ++// -- bool Value ++type boolValue bool ++ ++func newBoolValue(val bool, p *bool) *boolValue { ++ *p = val ++ return (*boolValue)(p) ++} ++ ++func (b *boolValue) Set(s string) error { ++ v, err := strconv.ParseBool(s) ++ *b = boolValue(v) ++ return err ++} ++ ++func (b *boolValue) Type() string { ++ return "bool" ++} ++ ++func (b *boolValue) String() string { return strconv.FormatBool(bool(*b)) } ++ ++func (b *boolValue) IsBoolFlag() bool { return true } ++ ++func boolConv(sval string) (interface{}, error) { ++ return strconv.ParseBool(sval) ++} ++ ++// GetBool return the bool value of a flag with the given name ++func (f *FlagSet) GetBool(name string) (bool, error) { ++ val, err := f.getFlagType(name, "bool", boolConv) ++ if err != nil { ++ return false, err ++ } ++ return val.(bool), nil ++} ++ ++// BoolVar defines a bool flag with specified name, default value, and usage string. ++// The argument p points to a bool variable in which to store the value of the flag. ++func (f *FlagSet) BoolVar(p *bool, name string, value bool, usage string) { ++ f.BoolVarP(p, name, "", value, usage) ++} ++ ++// BoolVarP is like BoolVar, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BoolVarP(p *bool, name, shorthand string, value bool, usage string) { ++ flag := f.VarPF(newBoolValue(value, p), name, shorthand, usage) ++ flag.NoOptDefVal = "true" ++} ++ ++// BoolVar defines a bool flag with specified name, default value, and usage string. ++// The argument p points to a bool variable in which to store the value of the flag. ++func BoolVar(p *bool, name string, value bool, usage string) { ++ BoolVarP(p, name, "", value, usage) ++} ++ ++// BoolVarP is like BoolVar, but accepts a shorthand letter that can be used after a single dash. ++func BoolVarP(p *bool, name, shorthand string, value bool, usage string) { ++ flag := CommandLine.VarPF(newBoolValue(value, p), name, shorthand, usage) ++ flag.NoOptDefVal = "true" ++} ++ ++// Bool defines a bool flag with specified name, default value, and usage string. ++// The return value is the address of a bool variable that stores the value of the flag. ++func (f *FlagSet) Bool(name string, value bool, usage string) *bool { ++ return f.BoolP(name, "", value, usage) ++} ++ ++// BoolP is like Bool, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BoolP(name, shorthand string, value bool, usage string) *bool { ++ p := new(bool) ++ f.BoolVarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// Bool defines a bool flag with specified name, default value, and usage string. ++// The return value is the address of a bool variable that stores the value of the flag. ++func Bool(name string, value bool, usage string) *bool { ++ return BoolP(name, "", value, usage) ++} ++ ++// BoolP is like Bool, but accepts a shorthand letter that can be used after a single dash. ++func BoolP(name, shorthand string, value bool, usage string) *bool { ++ b := CommandLine.BoolP(name, shorthand, value, usage) ++ return b ++} +diff --git a/thirdparty/pflag/bool_slice.go b/thirdparty/pflag/bool_slice.go +new file mode 100644 +index 00000000..5af02f1a +--- /dev/null ++++ b/thirdparty/pflag/bool_slice.go +@@ -0,0 +1,147 @@ ++package pflag ++ ++import ( ++ "io" ++ "strconv" ++ "strings" ++) ++ ++// -- boolSlice Value ++type boolSliceValue struct { ++ value *[]bool ++ changed bool ++} ++ ++func newBoolSliceValue(val []bool, p *[]bool) *boolSliceValue { ++ bsv := new(boolSliceValue) ++ bsv.value = p ++ *bsv.value = val ++ return bsv ++} ++ ++// Set converts, and assigns, the comma-separated boolean argument string representation as the []bool value of this flag. ++// If Set is called on a flag that already has a []bool assigned, the newly converted values will be appended. ++func (s *boolSliceValue) Set(val string) error { ++ ++ // remove all quote characters ++ rmQuote := strings.NewReplacer(`"`, "", `'`, "", "`", "") ++ ++ // read flag arguments with CSV parser ++ boolStrSlice, err := readAsCSV(rmQuote.Replace(val)) ++ if err != nil && err != io.EOF { ++ return err ++ } ++ ++ // parse boolean values into slice ++ out := make([]bool, 0, len(boolStrSlice)) ++ for _, boolStr := range boolStrSlice { ++ b, err := strconv.ParseBool(strings.TrimSpace(boolStr)) ++ if err != nil { ++ return err ++ } ++ out = append(out, b) ++ } ++ ++ if !s.changed { ++ *s.value = out ++ } else { ++ *s.value = append(*s.value, out...) ++ } ++ ++ s.changed = true ++ ++ return nil ++} ++ ++// Type returns a string that uniquely represents this flag's type. ++func (s *boolSliceValue) Type() string { ++ return "boolSlice" ++} ++ ++// String defines a "native" format for this boolean slice flag value. ++func (s *boolSliceValue) String() string { ++ ++ boolStrSlice := make([]string, len(*s.value)) ++ for i, b := range *s.value { ++ boolStrSlice[i] = strconv.FormatBool(b) ++ } ++ ++ out, _ := writeAsCSV(boolStrSlice) ++ ++ return "[" + out + "]" ++} ++ ++func boolSliceConv(val string) (interface{}, error) { ++ val = strings.Trim(val, "[]") ++ // Empty string would cause a slice with one (empty) entry ++ if len(val) == 0 { ++ return []bool{}, nil ++ } ++ ss := strings.Split(val, ",") ++ out := make([]bool, len(ss)) ++ for i, t := range ss { ++ var err error ++ out[i], err = strconv.ParseBool(t) ++ if err != nil { ++ return nil, err ++ } ++ } ++ return out, nil ++} ++ ++// GetBoolSlice returns the []bool value of a flag with the given name. ++func (f *FlagSet) GetBoolSlice(name string) ([]bool, error) { ++ val, err := f.getFlagType(name, "boolSlice", boolSliceConv) ++ if err != nil { ++ return []bool{}, err ++ } ++ return val.([]bool), nil ++} ++ ++// BoolSliceVar defines a boolSlice flag with specified name, default value, and usage string. ++// The argument p points to a []bool variable in which to store the value of the flag. ++func (f *FlagSet) BoolSliceVar(p *[]bool, name string, value []bool, usage string) { ++ f.VarP(newBoolSliceValue(value, p), name, "", usage) ++} ++ ++// BoolSliceVarP is like BoolSliceVar, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BoolSliceVarP(p *[]bool, name, shorthand string, value []bool, usage string) { ++ f.VarP(newBoolSliceValue(value, p), name, shorthand, usage) ++} ++ ++// BoolSliceVar defines a []bool flag with specified name, default value, and usage string. ++// The argument p points to a []bool variable in which to store the value of the flag. ++func BoolSliceVar(p *[]bool, name string, value []bool, usage string) { ++ CommandLine.VarP(newBoolSliceValue(value, p), name, "", usage) ++} ++ ++// BoolSliceVarP is like BoolSliceVar, but accepts a shorthand letter that can be used after a single dash. ++func BoolSliceVarP(p *[]bool, name, shorthand string, value []bool, usage string) { ++ CommandLine.VarP(newBoolSliceValue(value, p), name, shorthand, usage) ++} ++ ++// BoolSlice defines a []bool flag with specified name, default value, and usage string. ++// The return value is the address of a []bool variable that stores the value of the flag. ++func (f *FlagSet) BoolSlice(name string, value []bool, usage string) *[]bool { ++ p := []bool{} ++ f.BoolSliceVarP(&p, name, "", value, usage) ++ return &p ++} ++ ++// BoolSliceP is like BoolSlice, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BoolSliceP(name, shorthand string, value []bool, usage string) *[]bool { ++ p := []bool{} ++ f.BoolSliceVarP(&p, name, shorthand, value, usage) ++ return &p ++} ++ ++// BoolSlice defines a []bool flag with specified name, default value, and usage string. ++// The return value is the address of a []bool variable that stores the value of the flag. ++func BoolSlice(name string, value []bool, usage string) *[]bool { ++ return CommandLine.BoolSliceP(name, "", value, usage) ++} ++ ++// BoolSliceP is like BoolSlice, but accepts a shorthand letter that can be used after a single dash. ++func BoolSliceP(name, shorthand string, value []bool, usage string) *[]bool { ++ return CommandLine.BoolSliceP(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/bytes.go b/thirdparty/pflag/bytes.go +new file mode 100644 +index 00000000..67d53045 +--- /dev/null ++++ b/thirdparty/pflag/bytes.go +@@ -0,0 +1,209 @@ ++package pflag ++ ++import ( ++ "encoding/base64" ++ "encoding/hex" ++ "fmt" ++ "strings" ++) ++ ++// BytesHex adapts []byte for use as a flag. Value of flag is HEX encoded ++type bytesHexValue []byte ++ ++// String implements pflag.Value.String. ++func (bytesHex bytesHexValue) String() string { ++ return fmt.Sprintf("%X", []byte(bytesHex)) ++} ++ ++// Set implements pflag.Value.Set. ++func (bytesHex *bytesHexValue) Set(value string) error { ++ bin, err := hex.DecodeString(strings.TrimSpace(value)) ++ ++ if err != nil { ++ return err ++ } ++ ++ *bytesHex = bin ++ ++ return nil ++} ++ ++// Type implements pflag.Value.Type. ++func (*bytesHexValue) Type() string { ++ return "bytesHex" ++} ++ ++func newBytesHexValue(val []byte, p *[]byte) *bytesHexValue { ++ *p = val ++ return (*bytesHexValue)(p) ++} ++ ++func bytesHexConv(sval string) (interface{}, error) { ++ ++ bin, err := hex.DecodeString(sval) ++ ++ if err == nil { ++ return bin, nil ++ } ++ ++ return nil, fmt.Errorf("invalid string being converted to Bytes: %s %s", sval, err) ++} ++ ++// GetBytesHex return the []byte value of a flag with the given name ++func (f *FlagSet) GetBytesHex(name string) ([]byte, error) { ++ val, err := f.getFlagType(name, "bytesHex", bytesHexConv) ++ ++ if err != nil { ++ return []byte{}, err ++ } ++ ++ return val.([]byte), nil ++} ++ ++// BytesHexVar defines an []byte flag with specified name, default value, and usage string. ++// The argument p points to an []byte variable in which to store the value of the flag. ++func (f *FlagSet) BytesHexVar(p *[]byte, name string, value []byte, usage string) { ++ f.VarP(newBytesHexValue(value, p), name, "", usage) ++} ++ ++// BytesHexVarP is like BytesHexVar, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BytesHexVarP(p *[]byte, name, shorthand string, value []byte, usage string) { ++ f.VarP(newBytesHexValue(value, p), name, shorthand, usage) ++} ++ ++// BytesHexVar defines an []byte flag with specified name, default value, and usage string. ++// The argument p points to an []byte variable in which to store the value of the flag. ++func BytesHexVar(p *[]byte, name string, value []byte, usage string) { ++ CommandLine.VarP(newBytesHexValue(value, p), name, "", usage) ++} ++ ++// BytesHexVarP is like BytesHexVar, but accepts a shorthand letter that can be used after a single dash. ++func BytesHexVarP(p *[]byte, name, shorthand string, value []byte, usage string) { ++ CommandLine.VarP(newBytesHexValue(value, p), name, shorthand, usage) ++} ++ ++// BytesHex defines an []byte flag with specified name, default value, and usage string. ++// The return value is the address of an []byte variable that stores the value of the flag. ++func (f *FlagSet) BytesHex(name string, value []byte, usage string) *[]byte { ++ p := new([]byte) ++ f.BytesHexVarP(p, name, "", value, usage) ++ return p ++} ++ ++// BytesHexP is like BytesHex, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BytesHexP(name, shorthand string, value []byte, usage string) *[]byte { ++ p := new([]byte) ++ f.BytesHexVarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// BytesHex defines an []byte flag with specified name, default value, and usage string. ++// The return value is the address of an []byte variable that stores the value of the flag. ++func BytesHex(name string, value []byte, usage string) *[]byte { ++ return CommandLine.BytesHexP(name, "", value, usage) ++} ++ ++// BytesHexP is like BytesHex, but accepts a shorthand letter that can be used after a single dash. ++func BytesHexP(name, shorthand string, value []byte, usage string) *[]byte { ++ return CommandLine.BytesHexP(name, shorthand, value, usage) ++} ++ ++// BytesBase64 adapts []byte for use as a flag. Value of flag is Base64 encoded ++type bytesBase64Value []byte ++ ++// String implements pflag.Value.String. ++func (bytesBase64 bytesBase64Value) String() string { ++ return base64.StdEncoding.EncodeToString([]byte(bytesBase64)) ++} ++ ++// Set implements pflag.Value.Set. ++func (bytesBase64 *bytesBase64Value) Set(value string) error { ++ bin, err := base64.StdEncoding.DecodeString(strings.TrimSpace(value)) ++ ++ if err != nil { ++ return err ++ } ++ ++ *bytesBase64 = bin ++ ++ return nil ++} ++ ++// Type implements pflag.Value.Type. ++func (*bytesBase64Value) Type() string { ++ return "bytesBase64" ++} ++ ++func newBytesBase64Value(val []byte, p *[]byte) *bytesBase64Value { ++ *p = val ++ return (*bytesBase64Value)(p) ++} ++ ++func bytesBase64ValueConv(sval string) (interface{}, error) { ++ ++ bin, err := base64.StdEncoding.DecodeString(sval) ++ if err == nil { ++ return bin, nil ++ } ++ ++ return nil, fmt.Errorf("invalid string being converted to Bytes: %s %s", sval, err) ++} ++ ++// GetBytesBase64 return the []byte value of a flag with the given name ++func (f *FlagSet) GetBytesBase64(name string) ([]byte, error) { ++ val, err := f.getFlagType(name, "bytesBase64", bytesBase64ValueConv) ++ ++ if err != nil { ++ return []byte{}, err ++ } ++ ++ return val.([]byte), nil ++} ++ ++// BytesBase64Var defines an []byte flag with specified name, default value, and usage string. ++// The argument p points to an []byte variable in which to store the value of the flag. ++func (f *FlagSet) BytesBase64Var(p *[]byte, name string, value []byte, usage string) { ++ f.VarP(newBytesBase64Value(value, p), name, "", usage) ++} ++ ++// BytesBase64VarP is like BytesBase64Var, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BytesBase64VarP(p *[]byte, name, shorthand string, value []byte, usage string) { ++ f.VarP(newBytesBase64Value(value, p), name, shorthand, usage) ++} ++ ++// BytesBase64Var defines an []byte flag with specified name, default value, and usage string. ++// The argument p points to an []byte variable in which to store the value of the flag. ++func BytesBase64Var(p *[]byte, name string, value []byte, usage string) { ++ CommandLine.VarP(newBytesBase64Value(value, p), name, "", usage) ++} ++ ++// BytesBase64VarP is like BytesBase64Var, but accepts a shorthand letter that can be used after a single dash. ++func BytesBase64VarP(p *[]byte, name, shorthand string, value []byte, usage string) { ++ CommandLine.VarP(newBytesBase64Value(value, p), name, shorthand, usage) ++} ++ ++// BytesBase64 defines an []byte flag with specified name, default value, and usage string. ++// The return value is the address of an []byte variable that stores the value of the flag. ++func (f *FlagSet) BytesBase64(name string, value []byte, usage string) *[]byte { ++ p := new([]byte) ++ f.BytesBase64VarP(p, name, "", value, usage) ++ return p ++} ++ ++// BytesBase64P is like BytesBase64, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) BytesBase64P(name, shorthand string, value []byte, usage string) *[]byte { ++ p := new([]byte) ++ f.BytesBase64VarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// BytesBase64 defines an []byte flag with specified name, default value, and usage string. ++// The return value is the address of an []byte variable that stores the value of the flag. ++func BytesBase64(name string, value []byte, usage string) *[]byte { ++ return CommandLine.BytesBase64P(name, "", value, usage) ++} ++ ++// BytesBase64P is like BytesBase64, but accepts a shorthand letter that can be used after a single dash. ++func BytesBase64P(name, shorthand string, value []byte, usage string) *[]byte { ++ return CommandLine.BytesBase64P(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/count.go b/thirdparty/pflag/count.go +new file mode 100644 +index 00000000..aa126e44 +--- /dev/null ++++ b/thirdparty/pflag/count.go +@@ -0,0 +1,96 @@ ++package pflag ++ ++import "strconv" ++ ++// -- count Value ++type countValue int ++ ++func newCountValue(val int, p *int) *countValue { ++ *p = val ++ return (*countValue)(p) ++} ++ ++func (i *countValue) Set(s string) error { ++ // "+1" means that no specific value was passed, so increment ++ if s == "+1" { ++ *i = countValue(*i + 1) ++ return nil ++ } ++ v, err := strconv.ParseInt(s, 0, 0) ++ *i = countValue(v) ++ return err ++} ++ ++func (i *countValue) Type() string { ++ return "count" ++} ++ ++func (i *countValue) String() string { return strconv.Itoa(int(*i)) } ++ ++func countConv(sval string) (interface{}, error) { ++ i, err := strconv.Atoi(sval) ++ if err != nil { ++ return nil, err ++ } ++ return i, nil ++} ++ ++// GetCount return the int value of a flag with the given name ++func (f *FlagSet) GetCount(name string) (int, error) { ++ val, err := f.getFlagType(name, "count", countConv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(int), nil ++} ++ ++// CountVar defines a count flag with specified name, default value, and usage string. ++// The argument p points to an int variable in which to store the value of the flag. ++// A count flag will add 1 to its value evey time it is found on the command line ++func (f *FlagSet) CountVar(p *int, name string, usage string) { ++ f.CountVarP(p, name, "", usage) ++} ++ ++// CountVarP is like CountVar only take a shorthand for the flag name. ++func (f *FlagSet) CountVarP(p *int, name, shorthand string, usage string) { ++ flag := f.VarPF(newCountValue(0, p), name, shorthand, usage) ++ flag.NoOptDefVal = "+1" ++} ++ ++// CountVar like CountVar only the flag is placed on the CommandLine instead of a given flag set ++func CountVar(p *int, name string, usage string) { ++ CommandLine.CountVar(p, name, usage) ++} ++ ++// CountVarP is like CountVar only take a shorthand for the flag name. ++func CountVarP(p *int, name, shorthand string, usage string) { ++ CommandLine.CountVarP(p, name, shorthand, usage) ++} ++ ++// Count defines a count flag with specified name, default value, and usage string. ++// The return value is the address of an int variable that stores the value of the flag. ++// A count flag will add 1 to its value evey time it is found on the command line ++func (f *FlagSet) Count(name string, usage string) *int { ++ p := new(int) ++ f.CountVarP(p, name, "", usage) ++ return p ++} ++ ++// CountP is like Count only takes a shorthand for the flag name. ++func (f *FlagSet) CountP(name, shorthand string, usage string) *int { ++ p := new(int) ++ f.CountVarP(p, name, shorthand, usage) ++ return p ++} ++ ++// Count defines a count flag with specified name, default value, and usage string. ++// The return value is the address of an int variable that stores the value of the flag. ++// A count flag will add 1 to its value evey time it is found on the command line ++func Count(name string, usage string) *int { ++ return CommandLine.CountP(name, "", usage) ++} ++ ++// CountP is like Count only takes a shorthand for the flag name. ++func CountP(name, shorthand string, usage string) *int { ++ return CommandLine.CountP(name, shorthand, usage) ++} +diff --git a/thirdparty/pflag/duration.go b/thirdparty/pflag/duration.go +new file mode 100644 +index 00000000..e9debef8 +--- /dev/null ++++ b/thirdparty/pflag/duration.go +@@ -0,0 +1,86 @@ ++package pflag ++ ++import ( ++ "time" ++) ++ ++// -- time.Duration Value ++type durationValue time.Duration ++ ++func newDurationValue(val time.Duration, p *time.Duration) *durationValue { ++ *p = val ++ return (*durationValue)(p) ++} ++ ++func (d *durationValue) Set(s string) error { ++ v, err := time.ParseDuration(s) ++ *d = durationValue(v) ++ return err ++} ++ ++func (d *durationValue) Type() string { ++ return "duration" ++} ++ ++func (d *durationValue) String() string { return (*time.Duration)(d).String() } ++ ++func durationConv(sval string) (interface{}, error) { ++ return time.ParseDuration(sval) ++} ++ ++// GetDuration return the duration value of a flag with the given name ++func (f *FlagSet) GetDuration(name string) (time.Duration, error) { ++ val, err := f.getFlagType(name, "duration", durationConv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(time.Duration), nil ++} ++ ++// DurationVar defines a time.Duration flag with specified name, default value, and usage string. ++// The argument p points to a time.Duration variable in which to store the value of the flag. ++func (f *FlagSet) DurationVar(p *time.Duration, name string, value time.Duration, usage string) { ++ f.VarP(newDurationValue(value, p), name, "", usage) ++} ++ ++// DurationVarP is like DurationVar, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) DurationVarP(p *time.Duration, name, shorthand string, value time.Duration, usage string) { ++ f.VarP(newDurationValue(value, p), name, shorthand, usage) ++} ++ ++// DurationVar defines a time.Duration flag with specified name, default value, and usage string. ++// The argument p points to a time.Duration variable in which to store the value of the flag. ++func DurationVar(p *time.Duration, name string, value time.Duration, usage string) { ++ CommandLine.VarP(newDurationValue(value, p), name, "", usage) ++} ++ ++// DurationVarP is like DurationVar, but accepts a shorthand letter that can be used after a single dash. ++func DurationVarP(p *time.Duration, name, shorthand string, value time.Duration, usage string) { ++ CommandLine.VarP(newDurationValue(value, p), name, shorthand, usage) ++} ++ ++// Duration defines a time.Duration flag with specified name, default value, and usage string. ++// The return value is the address of a time.Duration variable that stores the value of the flag. ++func (f *FlagSet) Duration(name string, value time.Duration, usage string) *time.Duration { ++ p := new(time.Duration) ++ f.DurationVarP(p, name, "", value, usage) ++ return p ++} ++ ++// DurationP is like Duration, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) DurationP(name, shorthand string, value time.Duration, usage string) *time.Duration { ++ p := new(time.Duration) ++ f.DurationVarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// Duration defines a time.Duration flag with specified name, default value, and usage string. ++// The return value is the address of a time.Duration variable that stores the value of the flag. ++func Duration(name string, value time.Duration, usage string) *time.Duration { ++ return CommandLine.DurationP(name, "", value, usage) ++} ++ ++// DurationP is like Duration, but accepts a shorthand letter that can be used after a single dash. ++func DurationP(name, shorthand string, value time.Duration, usage string) *time.Duration { ++ return CommandLine.DurationP(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/duration_slice.go b/thirdparty/pflag/duration_slice.go +new file mode 100644 +index 00000000..52c6b6dc +--- /dev/null ++++ b/thirdparty/pflag/duration_slice.go +@@ -0,0 +1,128 @@ ++package pflag ++ ++import ( ++ "fmt" ++ "strings" ++ "time" ++) ++ ++// -- durationSlice Value ++type durationSliceValue struct { ++ value *[]time.Duration ++ changed bool ++} ++ ++func newDurationSliceValue(val []time.Duration, p *[]time.Duration) *durationSliceValue { ++ dsv := new(durationSliceValue) ++ dsv.value = p ++ *dsv.value = val ++ return dsv ++} ++ ++func (s *durationSliceValue) Set(val string) error { ++ ss := strings.Split(val, ",") ++ out := make([]time.Duration, len(ss)) ++ for i, d := range ss { ++ var err error ++ out[i], err = time.ParseDuration(d) ++ if err != nil { ++ return err ++ } ++ ++ } ++ if !s.changed { ++ *s.value = out ++ } else { ++ *s.value = append(*s.value, out...) ++ } ++ s.changed = true ++ return nil ++} ++ ++func (s *durationSliceValue) Type() string { ++ return "durationSlice" ++} ++ ++func (s *durationSliceValue) String() string { ++ out := make([]string, len(*s.value)) ++ for i, d := range *s.value { ++ out[i] = fmt.Sprintf("%s", d) ++ } ++ return "[" + strings.Join(out, ",") + "]" ++} ++ ++func durationSliceConv(val string) (interface{}, error) { ++ val = strings.Trim(val, "[]") ++ // Empty string would cause a slice with one (empty) entry ++ if len(val) == 0 { ++ return []time.Duration{}, nil ++ } ++ ss := strings.Split(val, ",") ++ out := make([]time.Duration, len(ss)) ++ for i, d := range ss { ++ var err error ++ out[i], err = time.ParseDuration(d) ++ if err != nil { ++ return nil, err ++ } ++ ++ } ++ return out, nil ++} ++ ++// GetDurationSlice returns the []time.Duration value of a flag with the given name ++func (f *FlagSet) GetDurationSlice(name string) ([]time.Duration, error) { ++ val, err := f.getFlagType(name, "durationSlice", durationSliceConv) ++ if err != nil { ++ return []time.Duration{}, err ++ } ++ return val.([]time.Duration), nil ++} ++ ++// DurationSliceVar defines a durationSlice flag with specified name, default value, and usage string. ++// The argument p points to a []time.Duration variable in which to store the value of the flag. ++func (f *FlagSet) DurationSliceVar(p *[]time.Duration, name string, value []time.Duration, usage string) { ++ f.VarP(newDurationSliceValue(value, p), name, "", usage) ++} ++ ++// DurationSliceVarP is like DurationSliceVar, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) DurationSliceVarP(p *[]time.Duration, name, shorthand string, value []time.Duration, usage string) { ++ f.VarP(newDurationSliceValue(value, p), name, shorthand, usage) ++} ++ ++// DurationSliceVar defines a duration[] flag with specified name, default value, and usage string. ++// The argument p points to a duration[] variable in which to store the value of the flag. ++func DurationSliceVar(p *[]time.Duration, name string, value []time.Duration, usage string) { ++ CommandLine.VarP(newDurationSliceValue(value, p), name, "", usage) ++} ++ ++// DurationSliceVarP is like DurationSliceVar, but accepts a shorthand letter that can be used after a single dash. ++func DurationSliceVarP(p *[]time.Duration, name, shorthand string, value []time.Duration, usage string) { ++ CommandLine.VarP(newDurationSliceValue(value, p), name, shorthand, usage) ++} ++ ++// DurationSlice defines a []time.Duration flag with specified name, default value, and usage string. ++// The return value is the address of a []time.Duration variable that stores the value of the flag. ++func (f *FlagSet) DurationSlice(name string, value []time.Duration, usage string) *[]time.Duration { ++ p := []time.Duration{} ++ f.DurationSliceVarP(&p, name, "", value, usage) ++ return &p ++} ++ ++// DurationSliceP is like DurationSlice, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) DurationSliceP(name, shorthand string, value []time.Duration, usage string) *[]time.Duration { ++ p := []time.Duration{} ++ f.DurationSliceVarP(&p, name, shorthand, value, usage) ++ return &p ++} ++ ++// DurationSlice defines a []time.Duration flag with specified name, default value, and usage string. ++// The return value is the address of a []time.Duration variable that stores the value of the flag. ++func DurationSlice(name string, value []time.Duration, usage string) *[]time.Duration { ++ return CommandLine.DurationSliceP(name, "", value, usage) ++} ++ ++// DurationSliceP is like DurationSlice, but accepts a shorthand letter that can be used after a single dash. ++func DurationSliceP(name, shorthand string, value []time.Duration, usage string) *[]time.Duration { ++ return CommandLine.DurationSliceP(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/flag.go b/thirdparty/pflag/flag.go +new file mode 100644 +index 00000000..5cc710cc +--- /dev/null ++++ b/thirdparty/pflag/flag.go +@@ -0,0 +1,1224 @@ ++// Copyright 2009 The Go Authors. All rights reserved. ++// Use of this source code is governed by a BSD-style ++// license that can be found in the LICENSE file. ++ ++/* ++Package pflag is a drop-in replacement for Go's flag package, implementing ++POSIX/GNU-style --flags. ++ ++pflag is compatible with the GNU extensions to the POSIX recommendations ++for command-line options. See ++http://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html ++ ++Usage: ++ ++pflag is a drop-in replacement of Go's native flag package. If you import ++pflag under the name "flag" then all code should continue to function ++with no changes. ++ ++ import flag "github.com/spf13/pflag" ++ ++There is one exception to this: if you directly instantiate the Flag struct ++there is one more field "Shorthand" that you will need to set. ++Most code never instantiates this struct directly, and instead uses ++functions such as String(), BoolVar(), and Var(), and is therefore ++unaffected. ++ ++Define flags using flag.String(), Bool(), Int(), etc. ++ ++This declares an integer flag, -flagname, stored in the pointer ip, with type *int. ++ var ip = flag.Int("flagname", 1234, "help message for flagname") ++If you like, you can bind the flag to a variable using the Var() functions. ++ var flagvar int ++ func init() { ++ flag.IntVar(&flagvar, "flagname", 1234, "help message for flagname") ++ } ++Or you can create custom flags that satisfy the Value interface (with ++pointer receivers) and couple them to flag parsing by ++ flag.Var(&flagVal, "name", "help message for flagname") ++For such flags, the default value is just the initial value of the variable. ++ ++After all flags are defined, call ++ flag.Parse() ++to parse the command line into the defined flags. ++ ++Flags may then be used directly. If you're using the flags themselves, ++they are all pointers; if you bind to variables, they're values. ++ fmt.Println("ip has value ", *ip) ++ fmt.Println("flagvar has value ", flagvar) ++ ++After parsing, the arguments after the flag are available as the ++slice flag.Args() or individually as flag.Arg(i). ++The arguments are indexed from 0 through flag.NArg()-1. ++ ++The pflag package also defines some new functions that are not in flag, ++that give one-letter shorthands for flags. You can use these by appending ++'P' to the name of any function that defines a flag. ++ var ip = flag.IntP("flagname", "f", 1234, "help message") ++ var flagvar bool ++ func init() { ++ flag.BoolVarP("boolname", "b", true, "help message") ++ } ++ flag.VarP(&flagVar, "varname", "v", 1234, "help message") ++Shorthand letters can be used with single dashes on the command line. ++Boolean shorthand flags can be combined with other shorthand flags. ++ ++Command line flag syntax: ++ --flag // boolean flags only ++ --flag=x ++ ++Unlike the flag package, a single dash before an option means something ++different than a double dash. Single dashes signify a series of shorthand ++letters for flags. All but the last shorthand letter must be boolean flags. ++ // boolean flags ++ -f ++ -abc ++ // non-boolean flags ++ -n 1234 ++ -Ifile ++ // mixed ++ -abcs "hello" ++ -abcn1234 ++ ++Flag parsing stops after the terminator "--". Unlike the flag package, ++flags can be interspersed with arguments anywhere on the command line ++before this terminator. ++ ++Integer flags accept 1234, 0664, 0x1234 and may be negative. ++Boolean flags (in their long form) accept 1, 0, t, f, true, false, ++TRUE, FALSE, True, False. ++Duration flags accept any input valid for time.ParseDuration. ++ ++The default set of command-line flags is controlled by ++top-level functions. The FlagSet type allows one to define ++independent sets of flags, such as to implement subcommands ++in a command-line interface. The methods of FlagSet are ++analogous to the top-level functions for the command-line ++flag set. ++*/ ++package pflag ++ ++import ( ++ "bytes" ++ "errors" ++ goflag "flag" ++ "fmt" ++ "io" ++ "os" ++ "sort" ++ "strings" ++) ++ ++// ErrHelp is the error returned if the flag -help is invoked but no such flag is defined. ++var ErrHelp = errors.New("pflag: help requested") ++ ++// ErrorHandling defines how to handle flag parsing errors. ++type ErrorHandling int ++ ++const ( ++ // ContinueOnError will return an err from Parse() if an error is found ++ ContinueOnError ErrorHandling = iota ++ // ExitOnError will call os.Exit(2) if an error is found when parsing ++ ExitOnError ++ // PanicOnError will panic() if an error is found when parsing flags ++ PanicOnError ++) ++ ++// ParseErrorsWhitelist defines the parsing errors that can be ignored ++type ParseErrorsWhitelist struct { ++ // UnknownFlags will ignore unknown flags errors and continue parsing rest of the flags ++ UnknownFlags bool ++} ++ ++// NormalizedName is a flag name that has been normalized according to rules ++// for the FlagSet (e.g. making '-' and '_' equivalent). ++type NormalizedName string ++ ++// A FlagSet represents a set of defined flags. ++type FlagSet struct { ++ // Usage is the function called when an error occurs while parsing flags. ++ // The field is a function (not a method) that may be changed to point to ++ // a custom error handler. ++ Usage func() ++ ++ // SortFlags is used to indicate, if user wants to have sorted flags in ++ // help/usage messages. ++ SortFlags bool ++ ++ // ParseErrorsWhitelist is used to configure a whitelist of errors ++ ParseErrorsWhitelist ParseErrorsWhitelist ++ ++ name string ++ parsed bool ++ actual map[NormalizedName]*Flag ++ orderedActual []*Flag ++ sortedActual []*Flag ++ formal map[NormalizedName]*Flag ++ orderedFormal []*Flag ++ sortedFormal []*Flag ++ shorthands map[byte]*Flag ++ args []string // arguments after flags ++ argsLenAtDash int // len(args) when a '--' was located when parsing, or -1 if no -- ++ errorHandling ErrorHandling ++ output io.Writer // nil means stderr; use out() accessor ++ interspersed bool // allow interspersed option/non-option args ++ normalizeNameFunc func(f *FlagSet, name string) NormalizedName ++ ++ addedGoFlagSets []*goflag.FlagSet ++} ++ ++// A Flag represents the state of a flag. ++type Flag struct { ++ Name string // name as it appears on command line ++ Shorthand string // one-letter abbreviated flag ++ Usage string // help message ++ Value Value // value as set ++ DefValue string // default value (as text); for usage message ++ Changed bool // If the user set the value (or if left to default) ++ NoOptDefVal string // default value (as text); if the flag is on the command line without any options ++ Deprecated string // If this flag is deprecated, this string is the new or now thing to use ++ Hidden bool // used by cobra.Command to allow flags to be hidden from help/usage text ++ ShorthandDeprecated string // If the shorthand of this flag is deprecated, this string is the new or now thing to use ++ Annotations map[string][]string // used by cobra.Command bash autocomple code ++} ++ ++// Value is the interface to the dynamic value stored in a flag. ++// (The default value is represented as a string.) ++type Value interface { ++ String() string ++ Set(string) error ++ Type() string ++} ++ ++// sortFlags returns the flags as a slice in lexicographical sorted order. ++func sortFlags(flags map[NormalizedName]*Flag) []*Flag { ++ list := make(sort.StringSlice, len(flags)) ++ i := 0 ++ for k := range flags { ++ list[i] = string(k) ++ i++ ++ } ++ list.Sort() ++ result := make([]*Flag, len(list)) ++ for i, name := range list { ++ result[i] = flags[NormalizedName(name)] ++ } ++ return result ++} ++ ++// SetNormalizeFunc allows you to add a function which can translate flag names. ++// Flags added to the FlagSet will be translated and then when anything tries to ++// look up the flag that will also be translated. So it would be possible to create ++// a flag named "getURL" and have it translated to "geturl". A user could then pass ++// "--getUrl" which may also be translated to "geturl" and everything will work. ++func (f *FlagSet) SetNormalizeFunc(n func(f *FlagSet, name string) NormalizedName) { ++ f.normalizeNameFunc = n ++ f.sortedFormal = f.sortedFormal[:0] ++ for fname, flag := range f.formal { ++ nname := f.normalizeFlagName(flag.Name) ++ if fname == nname { ++ continue ++ } ++ flag.Name = string(nname) ++ delete(f.formal, fname) ++ f.formal[nname] = flag ++ if _, set := f.actual[fname]; set { ++ delete(f.actual, fname) ++ f.actual[nname] = flag ++ } ++ } ++} ++ ++// GetNormalizeFunc returns the previously set NormalizeFunc of a function which ++// does no translation, if not set previously. ++func (f *FlagSet) GetNormalizeFunc() func(f *FlagSet, name string) NormalizedName { ++ if f.normalizeNameFunc != nil { ++ return f.normalizeNameFunc ++ } ++ return func(f *FlagSet, name string) NormalizedName { return NormalizedName(name) } ++} ++ ++func (f *FlagSet) normalizeFlagName(name string) NormalizedName { ++ n := f.GetNormalizeFunc() ++ return n(f, name) ++} ++ ++func (f *FlagSet) out() io.Writer { ++ if f.output == nil { ++ return os.Stderr ++ } ++ return f.output ++} ++ ++// SetOutput sets the destination for usage and error messages. ++// If output is nil, os.Stderr is used. ++func (f *FlagSet) SetOutput(output io.Writer) { ++ f.output = output ++} ++ ++// VisitAll visits the flags in lexicographical order or ++// in primordial order if f.SortFlags is false, calling fn for each. ++// It visits all flags, even those not set. ++func (f *FlagSet) VisitAll(fn func(*Flag)) { ++ if len(f.formal) == 0 { ++ return ++ } ++ ++ var flags []*Flag ++ if f.SortFlags { ++ if len(f.formal) != len(f.sortedFormal) { ++ f.sortedFormal = sortFlags(f.formal) ++ } ++ flags = f.sortedFormal ++ } else { ++ flags = f.orderedFormal ++ } ++ ++ for _, flag := range flags { ++ fn(flag) ++ } ++} ++ ++// HasFlags returns a bool to indicate if the FlagSet has any flags defined. ++func (f *FlagSet) HasFlags() bool { ++ return len(f.formal) > 0 ++} ++ ++// HasAvailableFlags returns a bool to indicate if the FlagSet has any flags ++// that are not hidden. ++func (f *FlagSet) HasAvailableFlags() bool { ++ for _, flag := range f.formal { ++ if !flag.Hidden { ++ return true ++ } ++ } ++ return false ++} ++ ++// VisitAll visits the command-line flags in lexicographical order or ++// in primordial order if f.SortFlags is false, calling fn for each. ++// It visits all flags, even those not set. ++func VisitAll(fn func(*Flag)) { ++ CommandLine.VisitAll(fn) ++} ++ ++// Visit visits the flags in lexicographical order or ++// in primordial order if f.SortFlags is false, calling fn for each. ++// It visits only those flags that have been set. ++func (f *FlagSet) Visit(fn func(*Flag)) { ++ if len(f.actual) == 0 { ++ return ++ } ++ ++ var flags []*Flag ++ if f.SortFlags { ++ if len(f.actual) != len(f.sortedActual) { ++ f.sortedActual = sortFlags(f.actual) ++ } ++ flags = f.sortedActual ++ } else { ++ flags = f.orderedActual ++ } ++ ++ for _, flag := range flags { ++ fn(flag) ++ } ++} ++ ++// Visit visits the command-line flags in lexicographical order or ++// in primordial order if f.SortFlags is false, calling fn for each. ++// It visits only those flags that have been set. ++func Visit(fn func(*Flag)) { ++ CommandLine.Visit(fn) ++} ++ ++// Lookup returns the Flag structure of the named flag, returning nil if none exists. ++func (f *FlagSet) Lookup(name string) *Flag { ++ return f.lookup(f.normalizeFlagName(name)) ++} ++ ++// ShorthandLookup returns the Flag structure of the short handed flag, ++// returning nil if none exists. ++// It panics, if len(name) > 1. ++func (f *FlagSet) ShorthandLookup(name string) *Flag { ++ if name == "" { ++ return nil ++ } ++ if len(name) > 1 { ++ msg := fmt.Sprintf("can not look up shorthand which is more than one ASCII character: %q", name) ++ fmt.Fprintf(f.out(), msg) ++ panic(msg) ++ } ++ c := name[0] ++ return f.shorthands[c] ++} ++ ++// lookup returns the Flag structure of the named flag, returning nil if none exists. ++func (f *FlagSet) lookup(name NormalizedName) *Flag { ++ return f.formal[name] ++} ++ ++// func to return a given type for a given flag name ++func (f *FlagSet) getFlagType(name string, ftype string, convFunc func(sval string) (interface{}, error)) (interface{}, error) { ++ flag := f.Lookup(name) ++ if flag == nil { ++ err := fmt.Errorf("flag accessed but not defined: %s", name) ++ return nil, err ++ } ++ ++ if flag.Value.Type() != ftype { ++ err := fmt.Errorf("trying to get %s value of flag of type %s", ftype, flag.Value.Type()) ++ return nil, err ++ } ++ ++ sval := flag.Value.String() ++ result, err := convFunc(sval) ++ if err != nil { ++ return nil, err ++ } ++ return result, nil ++} ++ ++// ArgsLenAtDash will return the length of f.Args at the moment when a -- was ++// found during arg parsing. This allows your program to know which args were ++// before the -- and which came after. ++func (f *FlagSet) ArgsLenAtDash() int { ++ return f.argsLenAtDash ++} ++ ++// MarkDeprecated indicated that a flag is deprecated in your program. It will ++// continue to function but will not show up in help or usage messages. Using ++// this flag will also print the given usageMessage. ++func (f *FlagSet) MarkDeprecated(name string, usageMessage string) error { ++ flag := f.Lookup(name) ++ if flag == nil { ++ return fmt.Errorf("flag %q does not exist", name) ++ } ++ if usageMessage == "" { ++ return fmt.Errorf("deprecated message for flag %q must be set", name) ++ } ++ flag.Deprecated = usageMessage ++ flag.Hidden = true ++ return nil ++} ++ ++// MarkShorthandDeprecated will mark the shorthand of a flag deprecated in your ++// program. It will continue to function but will not show up in help or usage ++// messages. Using this flag will also print the given usageMessage. ++func (f *FlagSet) MarkShorthandDeprecated(name string, usageMessage string) error { ++ flag := f.Lookup(name) ++ if flag == nil { ++ return fmt.Errorf("flag %q does not exist", name) ++ } ++ if usageMessage == "" { ++ return fmt.Errorf("deprecated message for flag %q must be set", name) ++ } ++ flag.ShorthandDeprecated = usageMessage ++ return nil ++} ++ ++// MarkHidden sets a flag to 'hidden' in your program. It will continue to ++// function but will not show up in help or usage messages. ++func (f *FlagSet) MarkHidden(name string) error { ++ flag := f.Lookup(name) ++ if flag == nil { ++ return fmt.Errorf("flag %q does not exist", name) ++ } ++ flag.Hidden = true ++ return nil ++} ++ ++// Lookup returns the Flag structure of the named command-line flag, ++// returning nil if none exists. ++func Lookup(name string) *Flag { ++ return CommandLine.Lookup(name) ++} ++ ++// ShorthandLookup returns the Flag structure of the short handed flag, ++// returning nil if none exists. ++func ShorthandLookup(name string) *Flag { ++ return CommandLine.ShorthandLookup(name) ++} ++ ++// Set sets the value of the named flag. ++func (f *FlagSet) Set(name, value string) error { ++ normalName := f.normalizeFlagName(name) ++ flag, ok := f.formal[normalName] ++ if !ok { ++ return fmt.Errorf("no such flag -%v", name) ++ } ++ ++ err := flag.Value.Set(value) ++ if err != nil { ++ var flagName string ++ if flag.Shorthand != "" && flag.ShorthandDeprecated == "" { ++ flagName = fmt.Sprintf("-%s, --%s", flag.Shorthand, flag.Name) ++ } else { ++ flagName = fmt.Sprintf("--%s", flag.Name) ++ } ++ return fmt.Errorf("invalid argument %q for %q flag: %v", value, flagName, err) ++ } ++ ++ if !flag.Changed { ++ if f.actual == nil { ++ f.actual = make(map[NormalizedName]*Flag) ++ } ++ f.actual[normalName] = flag ++ f.orderedActual = append(f.orderedActual, flag) ++ ++ flag.Changed = true ++ } ++ ++ if flag.Deprecated != "" { ++ fmt.Fprintf(f.out(), "Flag --%s has been deprecated, %s\n", flag.Name, flag.Deprecated) ++ } ++ return nil ++} ++ ++// SetAnnotation allows one to set arbitrary annotations on a flag in the FlagSet. ++// This is sometimes used by spf13/cobra programs which want to generate additional ++// bash completion information. ++func (f *FlagSet) SetAnnotation(name, key string, values []string) error { ++ normalName := f.normalizeFlagName(name) ++ flag, ok := f.formal[normalName] ++ if !ok { ++ return fmt.Errorf("no such flag -%v", name) ++ } ++ if flag.Annotations == nil { ++ flag.Annotations = map[string][]string{} ++ } ++ flag.Annotations[key] = values ++ return nil ++} ++ ++// Changed returns true if the flag was explicitly set during Parse() and false ++// otherwise ++func (f *FlagSet) Changed(name string) bool { ++ flag := f.Lookup(name) ++ // If a flag doesn't exist, it wasn't changed.... ++ if flag == nil { ++ return false ++ } ++ return flag.Changed ++} ++ ++// Set sets the value of the named command-line flag. ++func Set(name, value string) error { ++ return CommandLine.Set(name, value) ++} ++ ++// PrintDefaults prints, to standard error unless configured ++// otherwise, the default values of all defined flags in the set. ++func (f *FlagSet) PrintDefaults() { ++ usages := f.FlagUsages() ++ fmt.Fprint(f.out(), usages) ++} ++ ++// defaultIsZeroValue returns true if the default value for this flag represents ++// a zero value. ++func (f *Flag) defaultIsZeroValue() bool { ++ switch f.Value.(type) { ++ case boolFlag: ++ return f.DefValue == "false" ++ case *durationValue: ++ // Beginning in Go 1.7, duration zero values are "0s" ++ return f.DefValue == "0" || f.DefValue == "0s" ++ case *intValue, *int8Value, *int32Value, *int64Value, *uintValue, *uint8Value, *uint16Value, *uint32Value, *uint64Value, *countValue, *float32Value, *float64Value: ++ return f.DefValue == "0" ++ case *stringValue: ++ return f.DefValue == "" ++ case *ipValue, *ipMaskValue, *ipNetValue: ++ return f.DefValue == "<nil>" ++ case *intSliceValue, *stringSliceValue, *stringArrayValue: ++ return f.DefValue == "[]" ++ default: ++ switch f.Value.String() { ++ case "false": ++ return true ++ case "<nil>": ++ return true ++ case "": ++ return true ++ case "0": ++ return true ++ } ++ return false ++ } ++} ++ ++// UnquoteUsage extracts a back-quoted name from the usage ++// string for a flag and returns it and the un-quoted usage. ++// Given "a `name` to show" it returns ("name", "a name to show"). ++// If there are no back quotes, the name is an educated guess of the ++// type of the flag's value, or the empty string if the flag is boolean. ++func UnquoteUsage(flag *Flag) (name string, usage string) { ++ // Look for a back-quoted name, but avoid the strings package. ++ usage = flag.Usage ++ for i := 0; i < len(usage); i++ { ++ if usage[i] == '`' { ++ for j := i + 1; j < len(usage); j++ { ++ if usage[j] == '`' { ++ name = usage[i+1 : j] ++ usage = usage[:i] + name + usage[j+1:] ++ return name, usage ++ } ++ } ++ break // Only one back quote; use type name. ++ } ++ } ++ ++ name = flag.Value.Type() ++ switch name { ++ case "bool": ++ name = "" ++ case "float64": ++ name = "float" ++ case "int64": ++ name = "int" ++ case "uint64": ++ name = "uint" ++ case "stringSlice": ++ name = "strings" ++ case "intSlice": ++ name = "ints" ++ case "uintSlice": ++ name = "uints" ++ case "boolSlice": ++ name = "bools" ++ } ++ ++ return ++} ++ ++// Splits the string `s` on whitespace into an initial substring up to ++// `i` runes in length and the remainder. Will go `slop` over `i` if ++// that encompasses the entire string (which allows the caller to ++// avoid short orphan words on the final line). ++func wrapN(i, slop int, s string) (string, string) { ++ if i+slop > len(s) { ++ return s, "" ++ } ++ ++ w := strings.LastIndexAny(s[:i], " \t\n") ++ if w <= 0 { ++ return s, "" ++ } ++ nlPos := strings.LastIndex(s[:i], "\n") ++ if nlPos > 0 && nlPos < w { ++ return s[:nlPos], s[nlPos+1:] ++ } ++ return s[:w], s[w+1:] ++} ++ ++// Wraps the string `s` to a maximum width `w` with leading indent ++// `i`. The first line is not indented (this is assumed to be done by ++// caller). Pass `w` == 0 to do no wrapping ++func wrap(i, w int, s string) string { ++ if w == 0 { ++ return strings.Replace(s, "\n", "\n"+strings.Repeat(" ", i), -1) ++ } ++ ++ // space between indent i and end of line width w into which ++ // we should wrap the text. ++ wrap := w - i ++ ++ var r, l string ++ ++ // Not enough space for sensible wrapping. Wrap as a block on ++ // the next line instead. ++ if wrap < 24 { ++ i = 16 ++ wrap = w - i ++ r += "\n" + strings.Repeat(" ", i) ++ } ++ // If still not enough space then don't even try to wrap. ++ if wrap < 24 { ++ return strings.Replace(s, "\n", r, -1) ++ } ++ ++ // Try to avoid short orphan words on the final line, by ++ // allowing wrapN to go a bit over if that would fit in the ++ // remainder of the line. ++ slop := 5 ++ wrap = wrap - slop ++ ++ // Handle first line, which is indented by the caller (or the ++ // special case above) ++ l, s = wrapN(wrap, slop, s) ++ r = r + strings.Replace(l, "\n", "\n"+strings.Repeat(" ", i), -1) ++ ++ // Now wrap the rest ++ for s != "" { ++ var t string ++ ++ t, s = wrapN(wrap, slop, s) ++ r = r + "\n" + strings.Repeat(" ", i) + strings.Replace(t, "\n", "\n"+strings.Repeat(" ", i), -1) ++ } ++ ++ return r ++ ++} ++ ++// FlagUsagesWrapped returns a string containing the usage information ++// for all flags in the FlagSet. Wrapped to `cols` columns (0 for no ++// wrapping) ++func (f *FlagSet) FlagUsagesWrapped(cols int) string { ++ buf := new(bytes.Buffer) ++ ++ lines := make([]string, 0, len(f.formal)) ++ ++ maxlen := 0 ++ f.VisitAll(func(flag *Flag) { ++ if flag.Hidden { ++ return ++ } ++ ++ line := "" ++ if flag.Shorthand != "" && flag.ShorthandDeprecated == "" { ++ line = fmt.Sprintf(" -%s, --%s", flag.Shorthand, flag.Name) ++ } else { ++ line = fmt.Sprintf(" --%s", flag.Name) ++ } ++ ++ varname, usage := UnquoteUsage(flag) ++ if varname != "" { ++ line += " " + varname ++ } ++ if flag.NoOptDefVal != "" { ++ switch flag.Value.Type() { ++ case "string": ++ line += fmt.Sprintf("[=\"%s\"]", flag.NoOptDefVal) ++ case "bool": ++ if flag.NoOptDefVal != "true" { ++ line += fmt.Sprintf("[=%s]", flag.NoOptDefVal) ++ } ++ case "count": ++ if flag.NoOptDefVal != "+1" { ++ line += fmt.Sprintf("[=%s]", flag.NoOptDefVal) ++ } ++ default: ++ line += fmt.Sprintf("[=%s]", flag.NoOptDefVal) ++ } ++ } ++ ++ // This special character will be replaced with spacing once the ++ // correct alignment is calculated ++ line += "\x00" ++ if len(line) > maxlen { ++ maxlen = len(line) ++ } ++ ++ line += usage ++ if !flag.defaultIsZeroValue() { ++ if flag.Value.Type() == "string" { ++ line += fmt.Sprintf(" (default %q)", flag.DefValue) ++ } else { ++ line += fmt.Sprintf(" (default %s)", flag.DefValue) ++ } ++ } ++ if len(flag.Deprecated) != 0 { ++ line += fmt.Sprintf(" (DEPRECATED: %s)", flag.Deprecated) ++ } ++ ++ lines = append(lines, line) ++ }) ++ ++ for _, line := range lines { ++ sidx := strings.Index(line, "\x00") ++ spacing := strings.Repeat(" ", maxlen-sidx) ++ // maxlen + 2 comes from + 1 for the \x00 and + 1 for the (deliberate) off-by-one in maxlen-sidx ++ fmt.Fprintln(buf, line[:sidx], spacing, wrap(maxlen+2, cols, line[sidx+1:])) ++ } ++ ++ return buf.String() ++} ++ ++// FlagUsages returns a string containing the usage information for all flags in ++// the FlagSet ++func (f *FlagSet) FlagUsages() string { ++ return f.FlagUsagesWrapped(0) ++} ++ ++// PrintDefaults prints to standard error the default values of all defined command-line flags. ++func PrintDefaults() { ++ CommandLine.PrintDefaults() ++} ++ ++// defaultUsage is the default function to print a usage message. ++func defaultUsage(f *FlagSet) { ++ fmt.Fprintf(f.out(), "Usage of %s:\n", f.name) ++ f.PrintDefaults() ++} ++ ++// NOTE: Usage is not just defaultUsage(CommandLine) ++// because it serves (via godoc flag Usage) as the example ++// for how to write your own usage function. ++ ++// Usage prints to standard error a usage message documenting all defined command-line flags. ++// The function is a variable that may be changed to point to a custom function. ++// By default it prints a simple header and calls PrintDefaults; for details about the ++// format of the output and how to control it, see the documentation for PrintDefaults. ++var Usage = func() { ++ fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0]) ++ PrintDefaults() ++} ++ ++// NFlag returns the number of flags that have been set. ++func (f *FlagSet) NFlag() int { return len(f.actual) } ++ ++// NFlag returns the number of command-line flags that have been set. ++func NFlag() int { return len(CommandLine.actual) } ++ ++// Arg returns the i'th argument. Arg(0) is the first remaining argument ++// after flags have been processed. ++func (f *FlagSet) Arg(i int) string { ++ if i < 0 || i >= len(f.args) { ++ return "" ++ } ++ return f.args[i] ++} ++ ++// Arg returns the i'th command-line argument. Arg(0) is the first remaining argument ++// after flags have been processed. ++func Arg(i int) string { ++ return CommandLine.Arg(i) ++} ++ ++// NArg is the number of arguments remaining after flags have been processed. ++func (f *FlagSet) NArg() int { return len(f.args) } ++ ++// NArg is the number of arguments remaining after flags have been processed. ++func NArg() int { return len(CommandLine.args) } ++ ++// Args returns the non-flag arguments. ++func (f *FlagSet) Args() []string { return f.args } ++ ++// Args returns the non-flag command-line arguments. ++func Args() []string { return CommandLine.args } ++ ++// Var defines a flag with the specified name and usage string. The type and ++// value of the flag are represented by the first argument, of type Value, which ++// typically holds a user-defined implementation of Value. For instance, the ++// caller could create a flag that turns a comma-separated string into a slice ++// of strings by giving the slice the methods of Value; in particular, Set would ++// decompose the comma-separated string into the slice. ++func (f *FlagSet) Var(value Value, name string, usage string) { ++ f.VarP(value, name, "", usage) ++} ++ ++// VarPF is like VarP, but returns the flag created ++func (f *FlagSet) VarPF(value Value, name, shorthand, usage string) *Flag { ++ // Remember the default value as a string; it won't change. ++ flag := &Flag{ ++ Name: name, ++ Shorthand: shorthand, ++ Usage: usage, ++ Value: value, ++ DefValue: value.String(), ++ } ++ f.AddFlag(flag) ++ return flag ++} ++ ++// VarP is like Var, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) VarP(value Value, name, shorthand, usage string) { ++ f.VarPF(value, name, shorthand, usage) ++} ++ ++// AddFlag will add the flag to the FlagSet ++func (f *FlagSet) AddFlag(flag *Flag) { ++ normalizedFlagName := f.normalizeFlagName(flag.Name) ++ ++ _, alreadyThere := f.formal[normalizedFlagName] ++ if alreadyThere { ++ msg := fmt.Sprintf("%s flag redefined: %s", f.name, flag.Name) ++ fmt.Fprintln(f.out(), msg) ++ panic(msg) // Happens only if flags are declared with identical names ++ } ++ if f.formal == nil { ++ f.formal = make(map[NormalizedName]*Flag) ++ } ++ ++ flag.Name = string(normalizedFlagName) ++ f.formal[normalizedFlagName] = flag ++ f.orderedFormal = append(f.orderedFormal, flag) ++ ++ if flag.Shorthand == "" { ++ return ++ } ++ if len(flag.Shorthand) > 1 { ++ msg := fmt.Sprintf("%q shorthand is more than one ASCII character", flag.Shorthand) ++ fmt.Fprintf(f.out(), msg) ++ panic(msg) ++ } ++ if f.shorthands == nil { ++ f.shorthands = make(map[byte]*Flag) ++ } ++ c := flag.Shorthand[0] ++ used, alreadyThere := f.shorthands[c] ++ if alreadyThere { ++ msg := fmt.Sprintf("unable to redefine %q shorthand in %q flagset: it's already used for %q flag", c, f.name, used.Name) ++ fmt.Fprintf(f.out(), msg) ++ panic(msg) ++ } ++ f.shorthands[c] = flag ++} ++ ++// AddFlagSet adds one FlagSet to another. If a flag is already present in f ++// the flag from newSet will be ignored. ++func (f *FlagSet) AddFlagSet(newSet *FlagSet) { ++ if newSet == nil { ++ return ++ } ++ newSet.VisitAll(func(flag *Flag) { ++ if f.Lookup(flag.Name) == nil { ++ f.AddFlag(flag) ++ } ++ }) ++} ++ ++// Var defines a flag with the specified name and usage string. The type and ++// value of the flag are represented by the first argument, of type Value, which ++// typically holds a user-defined implementation of Value. For instance, the ++// caller could create a flag that turns a comma-separated string into a slice ++// of strings by giving the slice the methods of Value; in particular, Set would ++// decompose the comma-separated string into the slice. ++func Var(value Value, name string, usage string) { ++ CommandLine.VarP(value, name, "", usage) ++} ++ ++// VarP is like Var, but accepts a shorthand letter that can be used after a single dash. ++func VarP(value Value, name, shorthand, usage string) { ++ CommandLine.VarP(value, name, shorthand, usage) ++} ++ ++// failf prints to standard error a formatted error and usage message and ++// returns the error. ++func (f *FlagSet) failf(format string, a ...interface{}) error { ++ err := fmt.Errorf(format, a...) ++ if f.errorHandling != ContinueOnError { ++ fmt.Fprintln(f.out(), err) ++ f.usage() ++ } ++ return err ++} ++ ++// usage calls the Usage method for the flag set, or the usage function if ++// the flag set is CommandLine. ++func (f *FlagSet) usage() { ++ if f == CommandLine { ++ Usage() ++ } else if f.Usage == nil { ++ defaultUsage(f) ++ } else { ++ f.Usage() ++ } ++} ++ ++//--unknown (args will be empty) ++//--unknown --next-flag ... (args will be --next-flag ...) ++//--unknown arg ... (args will be arg ...) ++func stripUnknownFlagValue(args []string) []string { ++ if len(args) == 0 { ++ //--unknown ++ return args ++ } ++ ++ first := args[0] ++ if first[0] == '-' { ++ //--unknown --next-flag ... ++ return args ++ } ++ ++ //--unknown arg ... (args will be arg ...) ++ return args[1:] ++} ++ ++func (f *FlagSet) parseLongArg(s string, args []string, fn parseFunc) (a []string, err error) { ++ a = args ++ name := s[2:] ++ if len(name) == 0 || name[0] == '-' || name[0] == '=' { ++ err = f.failf("bad flag syntax: %s", s) ++ return ++ } ++ ++ split := strings.SplitN(name, "=", 2) ++ name = split[0] ++ flag, exists := f.formal[f.normalizeFlagName(name)] ++ ++ if !exists { ++ switch { ++ case name == "help": ++ f.usage() ++ return a, ErrHelp ++ case f.ParseErrorsWhitelist.UnknownFlags: ++ // --unknown=unknownval arg ... ++ // we do not want to lose arg in this case ++ if len(split) >= 2 { ++ return a, nil ++ } ++ ++ return stripUnknownFlagValue(a), nil ++ default: ++ err = f.failf("unknown flag: --%s", name) ++ return ++ } ++ } ++ ++ var value string ++ if len(split) == 2 { ++ // '--flag=arg' ++ value = split[1] ++ } else if flag.NoOptDefVal != "" { ++ // '--flag' (arg was optional) ++ value = flag.NoOptDefVal ++ } else if len(a) > 0 { ++ // '--flag arg' ++ value = a[0] ++ a = a[1:] ++ } else { ++ // '--flag' (arg was required) ++ err = f.failf("flag needs an argument: %s", s) ++ return ++ } ++ ++ err = fn(flag, value) ++ if err != nil { ++ f.failf(err.Error()) ++ } ++ return ++} ++ ++func (f *FlagSet) parseSingleShortArg(shorthands string, args []string, fn parseFunc) (outShorts string, outArgs []string, err error) { ++ outArgs = args ++ ++ if strings.HasPrefix(shorthands, "test.") { ++ return ++ } ++ ++ outShorts = shorthands[1:] ++ c := shorthands[0] ++ ++ flag, exists := f.shorthands[c] ++ if !exists { ++ switch { ++ case c == 'h': ++ f.usage() ++ err = ErrHelp ++ return ++ case f.ParseErrorsWhitelist.UnknownFlags: ++ // '-f=arg arg ...' ++ // we do not want to lose arg in this case ++ if len(shorthands) > 2 && shorthands[1] == '=' { ++ outShorts = "" ++ return ++ } ++ ++ outArgs = stripUnknownFlagValue(outArgs) ++ return ++ default: ++ err = f.failf("unknown shorthand flag: %q in -%s", c, shorthands) ++ return ++ } ++ } ++ ++ var value string ++ if len(shorthands) > 2 && shorthands[1] == '=' { ++ // '-f=arg' ++ value = shorthands[2:] ++ outShorts = "" ++ } else if flag.NoOptDefVal != "" { ++ // '-f' (arg was optional) ++ value = flag.NoOptDefVal ++ } else if len(shorthands) > 1 { ++ // '-farg' ++ value = shorthands[1:] ++ outShorts = "" ++ } else if len(args) > 0 { ++ // '-f arg' ++ value = args[0] ++ outArgs = args[1:] ++ } else { ++ // '-f' (arg was required) ++ err = f.failf("flag needs an argument: %q in -%s", c, shorthands) ++ return ++ } ++ ++ if flag.ShorthandDeprecated != "" { ++ fmt.Fprintf(f.out(), "Flag shorthand -%s has been deprecated, %s\n", flag.Shorthand, flag.ShorthandDeprecated) ++ } ++ ++ err = fn(flag, value) ++ if err != nil { ++ f.failf(err.Error()) ++ } ++ return ++} ++ ++func (f *FlagSet) parseShortArg(s string, args []string, fn parseFunc) (a []string, err error) { ++ a = args ++ shorthands := s[1:] ++ ++ // "shorthands" can be a series of shorthand letters of flags (e.g. "-vvv"). ++ for len(shorthands) > 0 { ++ shorthands, a, err = f.parseSingleShortArg(shorthands, args, fn) ++ if err != nil { ++ return ++ } ++ } ++ ++ return ++} ++ ++func (f *FlagSet) parseArgs(args []string, fn parseFunc) (err error) { ++ for len(args) > 0 { ++ s := args[0] ++ args = args[1:] ++ if len(s) == 0 || s[0] != '-' || len(s) == 1 { ++ if !f.interspersed { ++ f.args = append(f.args, s) ++ f.args = append(f.args, args...) ++ return nil ++ } ++ f.args = append(f.args, s) ++ continue ++ } ++ ++ if s[1] == '-' { ++ if len(s) == 2 { // "--" terminates the flags ++ f.argsLenAtDash = len(f.args) ++ f.args = append(f.args, args...) ++ break ++ } ++ args, err = f.parseLongArg(s, args, fn) ++ } else { ++ args, err = f.parseShortArg(s, args, fn) ++ } ++ if err != nil { ++ return ++ } ++ } ++ return ++} ++ ++// Parse parses flag definitions from the argument list, which should not ++// include the command name. Must be called after all flags in the FlagSet ++// are defined and before flags are accessed by the program. ++// The return value will be ErrHelp if -help was set but not defined. ++func (f *FlagSet) Parse(arguments []string) error { ++ if f.addedGoFlagSets != nil { ++ for _, goFlagSet := range f.addedGoFlagSets { ++ goFlagSet.Parse(nil) ++ } ++ } ++ f.parsed = true ++ ++ if len(arguments) < 0 { ++ return nil ++ } ++ ++ f.args = make([]string, 0, len(arguments)) ++ ++ set := func(flag *Flag, value string) error { ++ return f.Set(flag.Name, value) ++ } ++ ++ err := f.parseArgs(arguments, set) ++ if err != nil { ++ switch f.errorHandling { ++ case ContinueOnError: ++ return err ++ case ExitOnError: ++ fmt.Println(err) ++ os.Exit(2) ++ case PanicOnError: ++ panic(err) ++ } ++ } ++ return nil ++} ++ ++type parseFunc func(flag *Flag, value string) error ++ ++// ParseAll parses flag definitions from the argument list, which should not ++// include the command name. The arguments for fn are flag and value. Must be ++// called after all flags in the FlagSet are defined and before flags are ++// accessed by the program. The return value will be ErrHelp if -help was set ++// but not defined. ++func (f *FlagSet) ParseAll(arguments []string, fn func(flag *Flag, value string) error) error { ++ f.parsed = true ++ f.args = make([]string, 0, len(arguments)) ++ ++ err := f.parseArgs(arguments, fn) ++ if err != nil { ++ switch f.errorHandling { ++ case ContinueOnError: ++ return err ++ case ExitOnError: ++ os.Exit(2) ++ case PanicOnError: ++ panic(err) ++ } ++ } ++ return nil ++} ++ ++// Parsed reports whether f.Parse has been called. ++func (f *FlagSet) Parsed() bool { ++ return f.parsed ++} ++ ++// Parse parses the command-line flags from os.Args[1:]. Must be called ++// after all flags are defined and before flags are accessed by the program. ++func Parse() { ++ // Ignore errors; CommandLine is set for ExitOnError. ++ CommandLine.Parse(os.Args[1:]) ++} ++ ++// ParseAll parses the command-line flags from os.Args[1:] and called fn for each. ++// The arguments for fn are flag and value. Must be called after all flags are ++// defined and before flags are accessed by the program. ++func ParseAll(fn func(flag *Flag, value string) error) { ++ // Ignore errors; CommandLine is set for ExitOnError. ++ CommandLine.ParseAll(os.Args[1:], fn) ++} ++ ++// SetInterspersed sets whether to support interspersed option/non-option arguments. ++func SetInterspersed(interspersed bool) { ++ CommandLine.SetInterspersed(interspersed) ++} ++ ++// Parsed returns true if the command-line flags have been parsed. ++func Parsed() bool { ++ return CommandLine.Parsed() ++} ++ ++// CommandLine is the default set of command-line flags, parsed from os.Args. ++var CommandLine = NewFlagSet(os.Args[0], ExitOnError) ++ ++// NewFlagSet returns a new, empty flag set with the specified name, ++// error handling property and SortFlags set to true. ++func NewFlagSet(name string, errorHandling ErrorHandling) *FlagSet { ++ f := &FlagSet{ ++ name: name, ++ errorHandling: errorHandling, ++ argsLenAtDash: -1, ++ interspersed: true, ++ SortFlags: true, ++ } ++ return f ++} ++ ++// SetInterspersed sets whether to support interspersed option/non-option arguments. ++func (f *FlagSet) SetInterspersed(interspersed bool) { ++ f.interspersed = interspersed ++} ++ ++// Init sets the name and error handling property for a flag set. ++// By default, the zero FlagSet uses an empty name and the ++// ContinueOnError error handling policy. ++func (f *FlagSet) Init(name string, errorHandling ErrorHandling) { ++ f.name = name ++ f.errorHandling = errorHandling ++ f.argsLenAtDash = -1 ++} +diff --git a/thirdparty/pflag/float32.go b/thirdparty/pflag/float32.go +new file mode 100644 +index 00000000..a243f81f +--- /dev/null ++++ b/thirdparty/pflag/float32.go +@@ -0,0 +1,88 @@ ++package pflag ++ ++import "strconv" ++ ++// -- float32 Value ++type float32Value float32 ++ ++func newFloat32Value(val float32, p *float32) *float32Value { ++ *p = val ++ return (*float32Value)(p) ++} ++ ++func (f *float32Value) Set(s string) error { ++ v, err := strconv.ParseFloat(s, 32) ++ *f = float32Value(v) ++ return err ++} ++ ++func (f *float32Value) Type() string { ++ return "float32" ++} ++ ++func (f *float32Value) String() string { return strconv.FormatFloat(float64(*f), 'g', -1, 32) } ++ ++func float32Conv(sval string) (interface{}, error) { ++ v, err := strconv.ParseFloat(sval, 32) ++ if err != nil { ++ return 0, err ++ } ++ return float32(v), nil ++} ++ ++// GetFloat32 return the float32 value of a flag with the given name ++func (f *FlagSet) GetFloat32(name string) (float32, error) { ++ val, err := f.getFlagType(name, "float32", float32Conv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(float32), nil ++} ++ ++// Float32Var defines a float32 flag with specified name, default value, and usage string. ++// The argument p points to a float32 variable in which to store the value of the flag. ++func (f *FlagSet) Float32Var(p *float32, name string, value float32, usage string) { ++ f.VarP(newFloat32Value(value, p), name, "", usage) ++} ++ ++// Float32VarP is like Float32Var, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Float32VarP(p *float32, name, shorthand string, value float32, usage string) { ++ f.VarP(newFloat32Value(value, p), name, shorthand, usage) ++} ++ ++// Float32Var defines a float32 flag with specified name, default value, and usage string. ++// The argument p points to a float32 variable in which to store the value of the flag. ++func Float32Var(p *float32, name string, value float32, usage string) { ++ CommandLine.VarP(newFloat32Value(value, p), name, "", usage) ++} ++ ++// Float32VarP is like Float32Var, but accepts a shorthand letter that can be used after a single dash. ++func Float32VarP(p *float32, name, shorthand string, value float32, usage string) { ++ CommandLine.VarP(newFloat32Value(value, p), name, shorthand, usage) ++} ++ ++// Float32 defines a float32 flag with specified name, default value, and usage string. ++// The return value is the address of a float32 variable that stores the value of the flag. ++func (f *FlagSet) Float32(name string, value float32, usage string) *float32 { ++ p := new(float32) ++ f.Float32VarP(p, name, "", value, usage) ++ return p ++} ++ ++// Float32P is like Float32, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Float32P(name, shorthand string, value float32, usage string) *float32 { ++ p := new(float32) ++ f.Float32VarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// Float32 defines a float32 flag with specified name, default value, and usage string. ++// The return value is the address of a float32 variable that stores the value of the flag. ++func Float32(name string, value float32, usage string) *float32 { ++ return CommandLine.Float32P(name, "", value, usage) ++} ++ ++// Float32P is like Float32, but accepts a shorthand letter that can be used after a single dash. ++func Float32P(name, shorthand string, value float32, usage string) *float32 { ++ return CommandLine.Float32P(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/float64.go b/thirdparty/pflag/float64.go +new file mode 100644 +index 00000000..04b5492a +--- /dev/null ++++ b/thirdparty/pflag/float64.go +@@ -0,0 +1,84 @@ ++package pflag ++ ++import "strconv" ++ ++// -- float64 Value ++type float64Value float64 ++ ++func newFloat64Value(val float64, p *float64) *float64Value { ++ *p = val ++ return (*float64Value)(p) ++} ++ ++func (f *float64Value) Set(s string) error { ++ v, err := strconv.ParseFloat(s, 64) ++ *f = float64Value(v) ++ return err ++} ++ ++func (f *float64Value) Type() string { ++ return "float64" ++} ++ ++func (f *float64Value) String() string { return strconv.FormatFloat(float64(*f), 'g', -1, 64) } ++ ++func float64Conv(sval string) (interface{}, error) { ++ return strconv.ParseFloat(sval, 64) ++} ++ ++// GetFloat64 return the float64 value of a flag with the given name ++func (f *FlagSet) GetFloat64(name string) (float64, error) { ++ val, err := f.getFlagType(name, "float64", float64Conv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(float64), nil ++} ++ ++// Float64Var defines a float64 flag with specified name, default value, and usage string. ++// The argument p points to a float64 variable in which to store the value of the flag. ++func (f *FlagSet) Float64Var(p *float64, name string, value float64, usage string) { ++ f.VarP(newFloat64Value(value, p), name, "", usage) ++} ++ ++// Float64VarP is like Float64Var, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Float64VarP(p *float64, name, shorthand string, value float64, usage string) { ++ f.VarP(newFloat64Value(value, p), name, shorthand, usage) ++} ++ ++// Float64Var defines a float64 flag with specified name, default value, and usage string. ++// The argument p points to a float64 variable in which to store the value of the flag. ++func Float64Var(p *float64, name string, value float64, usage string) { ++ CommandLine.VarP(newFloat64Value(value, p), name, "", usage) ++} ++ ++// Float64VarP is like Float64Var, but accepts a shorthand letter that can be used after a single dash. ++func Float64VarP(p *float64, name, shorthand string, value float64, usage string) { ++ CommandLine.VarP(newFloat64Value(value, p), name, shorthand, usage) ++} ++ ++// Float64 defines a float64 flag with specified name, default value, and usage string. ++// The return value is the address of a float64 variable that stores the value of the flag. ++func (f *FlagSet) Float64(name string, value float64, usage string) *float64 { ++ p := new(float64) ++ f.Float64VarP(p, name, "", value, usage) ++ return p ++} ++ ++// Float64P is like Float64, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Float64P(name, shorthand string, value float64, usage string) *float64 { ++ p := new(float64) ++ f.Float64VarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// Float64 defines a float64 flag with specified name, default value, and usage string. ++// The return value is the address of a float64 variable that stores the value of the flag. ++func Float64(name string, value float64, usage string) *float64 { ++ return CommandLine.Float64P(name, "", value, usage) ++} ++ ++// Float64P is like Float64, but accepts a shorthand letter that can be used after a single dash. ++func Float64P(name, shorthand string, value float64, usage string) *float64 { ++ return CommandLine.Float64P(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/go.mod b/thirdparty/pflag/go.mod +new file mode 100644 +index 00000000..551ee17d +--- /dev/null ++++ b/thirdparty/pflag/go.mod +@@ -0,0 +1,3 @@ ++// Go upstream includes a modified pflag in vendor ++// Added: IPNetSliceVar to pflag.FlagSet. ++module github.com/spf13/pflag +diff --git a/thirdparty/pflag/golangflag.go b/thirdparty/pflag/golangflag.go +new file mode 100644 +index 00000000..d3dd72b7 +--- /dev/null ++++ b/thirdparty/pflag/golangflag.go +@@ -0,0 +1,105 @@ ++// Copyright 2009 The Go Authors. All rights reserved. ++// Use of this source code is governed by a BSD-style ++// license that can be found in the LICENSE file. ++ ++package pflag ++ ++import ( ++ goflag "flag" ++ "reflect" ++ "strings" ++) ++ ++// flagValueWrapper implements pflag.Value around a flag.Value. The main ++// difference here is the addition of the Type method that returns a string ++// name of the type. As this is generally unknown, we approximate that with ++// reflection. ++type flagValueWrapper struct { ++ inner goflag.Value ++ flagType string ++} ++ ++// We are just copying the boolFlag interface out of goflag as that is what ++// they use to decide if a flag should get "true" when no arg is given. ++type goBoolFlag interface { ++ goflag.Value ++ IsBoolFlag() bool ++} ++ ++func wrapFlagValue(v goflag.Value) Value { ++ // If the flag.Value happens to also be a pflag.Value, just use it directly. ++ if pv, ok := v.(Value); ok { ++ return pv ++ } ++ ++ pv := &flagValueWrapper{ ++ inner: v, ++ } ++ ++ t := reflect.TypeOf(v) ++ if t.Kind() == reflect.Interface || t.Kind() == reflect.Ptr { ++ t = t.Elem() ++ } ++ ++ pv.flagType = strings.TrimSuffix(t.Name(), "Value") ++ return pv ++} ++ ++func (v *flagValueWrapper) String() string { ++ return v.inner.String() ++} ++ ++func (v *flagValueWrapper) Set(s string) error { ++ return v.inner.Set(s) ++} ++ ++func (v *flagValueWrapper) Type() string { ++ return v.flagType ++} ++ ++// PFlagFromGoFlag will return a *pflag.Flag given a *flag.Flag ++// If the *flag.Flag.Name was a single character (ex: `v`) it will be accessiblei ++// with both `-v` and `--v` in flags. If the golang flag was more than a single ++// character (ex: `verbose`) it will only be accessible via `--verbose` ++func PFlagFromGoFlag(goflag *goflag.Flag) *Flag { ++ // Remember the default value as a string; it won't change. ++ flag := &Flag{ ++ Name: goflag.Name, ++ Usage: goflag.Usage, ++ Value: wrapFlagValue(goflag.Value), ++ // Looks like golang flags don't set DefValue correctly :-( ++ //DefValue: goflag.DefValue, ++ DefValue: goflag.Value.String(), ++ } ++ // Ex: if the golang flag was -v, allow both -v and --v to work ++ if len(flag.Name) == 1 { ++ flag.Shorthand = flag.Name ++ } ++ if fv, ok := goflag.Value.(goBoolFlag); ok && fv.IsBoolFlag() { ++ flag.NoOptDefVal = "true" ++ } ++ return flag ++} ++ ++// AddGoFlag will add the given *flag.Flag to the pflag.FlagSet ++func (f *FlagSet) AddGoFlag(goflag *goflag.Flag) { ++ if f.Lookup(goflag.Name) != nil { ++ return ++ } ++ newflag := PFlagFromGoFlag(goflag) ++ f.AddFlag(newflag) ++} ++ ++// AddGoFlagSet will add the given *flag.FlagSet to the pflag.FlagSet ++func (f *FlagSet) AddGoFlagSet(newSet *goflag.FlagSet) { ++ if newSet == nil { ++ return ++ } ++ newSet.VisitAll(func(goflag *goflag.Flag) { ++ f.AddGoFlag(goflag) ++ }) ++ if f.addedGoFlagSets == nil { ++ f.addedGoFlagSets = make([]*goflag.FlagSet, 0) ++ } ++ f.addedGoFlagSets = append(f.addedGoFlagSets, newSet) ++} +diff --git a/thirdparty/pflag/int.go b/thirdparty/pflag/int.go +new file mode 100644 +index 00000000..1474b89d +--- /dev/null ++++ b/thirdparty/pflag/int.go +@@ -0,0 +1,84 @@ ++package pflag ++ ++import "strconv" ++ ++// -- int Value ++type intValue int ++ ++func newIntValue(val int, p *int) *intValue { ++ *p = val ++ return (*intValue)(p) ++} ++ ++func (i *intValue) Set(s string) error { ++ v, err := strconv.ParseInt(s, 0, 64) ++ *i = intValue(v) ++ return err ++} ++ ++func (i *intValue) Type() string { ++ return "int" ++} ++ ++func (i *intValue) String() string { return strconv.Itoa(int(*i)) } ++ ++func intConv(sval string) (interface{}, error) { ++ return strconv.Atoi(sval) ++} ++ ++// GetInt return the int value of a flag with the given name ++func (f *FlagSet) GetInt(name string) (int, error) { ++ val, err := f.getFlagType(name, "int", intConv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(int), nil ++} ++ ++// IntVar defines an int flag with specified name, default value, and usage string. ++// The argument p points to an int variable in which to store the value of the flag. ++func (f *FlagSet) IntVar(p *int, name string, value int, usage string) { ++ f.VarP(newIntValue(value, p), name, "", usage) ++} ++ ++// IntVarP is like IntVar, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) IntVarP(p *int, name, shorthand string, value int, usage string) { ++ f.VarP(newIntValue(value, p), name, shorthand, usage) ++} ++ ++// IntVar defines an int flag with specified name, default value, and usage string. ++// The argument p points to an int variable in which to store the value of the flag. ++func IntVar(p *int, name string, value int, usage string) { ++ CommandLine.VarP(newIntValue(value, p), name, "", usage) ++} ++ ++// IntVarP is like IntVar, but accepts a shorthand letter that can be used after a single dash. ++func IntVarP(p *int, name, shorthand string, value int, usage string) { ++ CommandLine.VarP(newIntValue(value, p), name, shorthand, usage) ++} ++ ++// Int defines an int flag with specified name, default value, and usage string. ++// The return value is the address of an int variable that stores the value of the flag. ++func (f *FlagSet) Int(name string, value int, usage string) *int { ++ p := new(int) ++ f.IntVarP(p, name, "", value, usage) ++ return p ++} ++ ++// IntP is like Int, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) IntP(name, shorthand string, value int, usage string) *int { ++ p := new(int) ++ f.IntVarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// Int defines an int flag with specified name, default value, and usage string. ++// The return value is the address of an int variable that stores the value of the flag. ++func Int(name string, value int, usage string) *int { ++ return CommandLine.IntP(name, "", value, usage) ++} ++ ++// IntP is like Int, but accepts a shorthand letter that can be used after a single dash. ++func IntP(name, shorthand string, value int, usage string) *int { ++ return CommandLine.IntP(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/int16.go b/thirdparty/pflag/int16.go +new file mode 100644 +index 00000000..f1a01d05 +--- /dev/null ++++ b/thirdparty/pflag/int16.go +@@ -0,0 +1,88 @@ ++package pflag ++ ++import "strconv" ++ ++// -- int16 Value ++type int16Value int16 ++ ++func newInt16Value(val int16, p *int16) *int16Value { ++ *p = val ++ return (*int16Value)(p) ++} ++ ++func (i *int16Value) Set(s string) error { ++ v, err := strconv.ParseInt(s, 0, 16) ++ *i = int16Value(v) ++ return err ++} ++ ++func (i *int16Value) Type() string { ++ return "int16" ++} ++ ++func (i *int16Value) String() string { return strconv.FormatInt(int64(*i), 10) } ++ ++func int16Conv(sval string) (interface{}, error) { ++ v, err := strconv.ParseInt(sval, 0, 16) ++ if err != nil { ++ return 0, err ++ } ++ return int16(v), nil ++} ++ ++// GetInt16 returns the int16 value of a flag with the given name ++func (f *FlagSet) GetInt16(name string) (int16, error) { ++ val, err := f.getFlagType(name, "int16", int16Conv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(int16), nil ++} ++ ++// Int16Var defines an int16 flag with specified name, default value, and usage string. ++// The argument p points to an int16 variable in which to store the value of the flag. ++func (f *FlagSet) Int16Var(p *int16, name string, value int16, usage string) { ++ f.VarP(newInt16Value(value, p), name, "", usage) ++} ++ ++// Int16VarP is like Int16Var, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Int16VarP(p *int16, name, shorthand string, value int16, usage string) { ++ f.VarP(newInt16Value(value, p), name, shorthand, usage) ++} ++ ++// Int16Var defines an int16 flag with specified name, default value, and usage string. ++// The argument p points to an int16 variable in which to store the value of the flag. ++func Int16Var(p *int16, name string, value int16, usage string) { ++ CommandLine.VarP(newInt16Value(value, p), name, "", usage) ++} ++ ++// Int16VarP is like Int16Var, but accepts a shorthand letter that can be used after a single dash. ++func Int16VarP(p *int16, name, shorthand string, value int16, usage string) { ++ CommandLine.VarP(newInt16Value(value, p), name, shorthand, usage) ++} ++ ++// Int16 defines an int16 flag with specified name, default value, and usage string. ++// The return value is the address of an int16 variable that stores the value of the flag. ++func (f *FlagSet) Int16(name string, value int16, usage string) *int16 { ++ p := new(int16) ++ f.Int16VarP(p, name, "", value, usage) ++ return p ++} ++ ++// Int16P is like Int16, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Int16P(name, shorthand string, value int16, usage string) *int16 { ++ p := new(int16) ++ f.Int16VarP(p, name, shorthand, value, usage) ++ return p ++} ++ ++// Int16 defines an int16 flag with specified name, default value, and usage string. ++// The return value is the address of an int16 variable that stores the value of the flag. ++func Int16(name string, value int16, usage string) *int16 { ++ return CommandLine.Int16P(name, "", value, usage) ++} ++ ++// Int16P is like Int16, but accepts a shorthand letter that can be used after a single dash. ++func Int16P(name, shorthand string, value int16, usage string) *int16 { ++ return CommandLine.Int16P(name, shorthand, value, usage) ++} +diff --git a/thirdparty/pflag/int32.go b/thirdparty/pflag/int32.go +new file mode 100644 +index 00000000..9b95944f +--- /dev/null ++++ b/thirdparty/pflag/int32.go +@@ -0,0 +1,88 @@ ++package pflag ++ ++import "strconv" ++ ++// -- int32 Value ++type int32Value int32 ++ ++func newInt32Value(val int32, p *int32) *int32Value { ++ *p = val ++ return (*int32Value)(p) ++} ++ ++func (i *int32Value) Set(s string) error { ++ v, err := strconv.ParseInt(s, 0, 32) ++ *i = int32Value(v) ++ return err ++} ++ ++func (i *int32Value) Type() string { ++ return "int32" ++} ++ ++func (i *int32Value) String() string { return strconv.FormatInt(int64(*i), 10) } ++ ++func int32Conv(sval string) (interface{}, error) { ++ v, err := strconv.ParseInt(sval, 0, 32) ++ if err != nil { ++ return 0, err ++ } ++ return int32(v), nil ++} ++ ++// GetInt32 return the int32 value of a flag with the given name ++func (f *FlagSet) GetInt32(name string) (int32, error) { ++ val, err := f.getFlagType(name, "int32", int32Conv) ++ if err != nil { ++ return 0, err ++ } ++ return val.(int32), nil ++} ++ ++// Int32Var defines an int32 flag with specified name, default value, and usage string. ++// The argument p points to an int32 variable in which to store the value of the flag. ++func (f *FlagSet) Int32Var(p *int32, name string, value int32, usage string) { ++ f.VarP(newInt32Value(value, p), name, "", usage) ++} ++ ++// Int32VarP is like Int32Var, but accepts a shorthand letter that can be used after a single dash. ++func (f *FlagSet) Int32VarP(p *int32, name, shorthand string, value int32, usage string) { ++ f.VarP(newInt32Value(value, p), name, shorthand, usage) ++} ++ ++// Int32Var defines an int32 flag with specified name, default value, and usage string. ++// The argument p points to an int32 variable in which to store the value of the flag. ++func Int32Var(p *int32, name string, value int32, usage string) { ++ CommandLine.VarP(newInt32Value(value, p), name, "", usage) ++} ++ ++// Int32VarP is like Int32Var, but accepts a shorthand letter that can be used after a single dash. ++func Int32VarP(p *int32, name, shorthand string, value int32, usage string) { ++ CommandLine.VarP(newInt32Value(value, p), name, shorthand, usage) ++}
Lines of this file are not shown. View the raw message.
package/docker-cli/go.mod new file +127 -0
127 lines not on this page · raw message
package/docker-cli/go.sum new file +419 -0
419 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.19.2
Part of the diff is not shown here. View the raw message.
RFC PATCH 5/6 docker-proxy: upgrade to go modules
In reply to Christian StewartThis commit introduces go.mod and go.sum files derived from the vendor.conf in the upstream docker-proxy repository.
--- package/docker-proxy/docker-proxy.mk | 4 +- package/docker-proxy/go.mod | 46 ++++++++++++++++++++ package/docker-proxy/go.sum | 65 ++++++++++++++++++++++++++++ 3 files changed, 112 insertions(+), 3 deletions(-) create mode 100644 package/docker-proxy/go.mod create mode 100644 package/docker-proxy/go.sum
package/docker-proxy/docker-proxy.mk +1 -3
@@ -12,9 +12,7 @@ DOCKER_PROXY_LICENSE_FILES = LICENSE DOCKER_PROXY_DEPENDENCIES = host-pkgconf -DOCKER_PROXY_WORKSPACE = gopath - -DOCKER_PROXY_BUILD_TARGETS = cmd/proxy +DOCKER_PROXY_BUILD_TARGETS = github.com/docker/libnetwork/cmd/proxy define DOCKER_PROXY_INSTALL_TARGET_CMDS $(INSTALL) -D -m 0755 $(@D)/bin/proxy $(TARGET_DIR)/usr/bin/docker-proxy
package/docker-proxy/go.mod new file +46 -0
46 lines not on this page · raw message
package/docker-proxy/go.sum new file +65 -0
65 lines not on this page · raw message
Signature · 1 line
2.19.2
RFC PATCH 6/6 package/docker-engine: upgrade to go modules
In reply to Christian StewartThis commit introduces in-tree go.mod and go.sum files derived from the vendor.conf file from the upstream Docker engine repository.
The Docker vendor/ directory is overwritten by the Go module tool.
--- package/docker-engine/docker-engine.mk | 3 +- package/docker-engine/go.mod | 124 ++++++++++++ package/docker-engine/go.sum | 256 +++++++++++++++++++++++++ 3 files changed, 381 insertions(+), 2 deletions(-) create mode 100644 package/docker-engine/go.mod create mode 100644 package/docker-engine/go.sum
package/docker-engine/docker-engine.mk +1 -2
@@ -11,14 +11,13 @@ DOCKER_ENGINE_LICENSE = Apache-2.0 DOCKER_ENGINE_LICENSE_FILES = LICENSE DOCKER_ENGINE_DEPENDENCIES = host-pkgconf -DOCKER_ENGINE_SRC_SUBDIR = github.com/docker/docker DOCKER_ENGINE_LDFLAGS = \ -X main.GitCommit=$(DOCKER_ENGINE_VERSION) \ -X main.Version=$(DOCKER_ENGINE_VERSION) DOCKER_ENGINE_TAGS = cgo exclude_graphdriver_zfs autogen -DOCKER_ENGINE_BUILD_TARGETS = cmd/dockerd +DOCKER_ENGINE_BUILD_TARGETS = github.com/docker/docker/cmd/dockerd ifeq ($(BR2_PACKAGE_LIBSECCOMP),y) DOCKER_ENGINE_TAGS += seccomp
package/docker-engine/go.mod new file +124 -0
124 lines not on this page · raw message
package/docker-engine/go.sum new file +256 -0
256 lines not on this page · raw message
Signature · 1 line
2.19.2
RFC PATCH 1/6 package/go: implement go modules integration
In reply to Christian StewartHello Christian,
I've been trying to get my head around this topic, but I don't really grasp why we're changing this. What is the advantage of implementing this go module integration compared to what we're doing today ? Looking at your patches 2/6 to 6/6 doesn't make the thing very appealing: we need additional files (go.mod and go.sum) for each package using Go, which were not needed before, and we have a 5k lines patch in docker-cli.
Quoted 3 lines · On Sat, 16 Mar 2019 18:21:37 -0700 Christian Stewart <[email protected]> wrote:
This commit moves from the GOPATH mechanism to the new GO111MODULE approach for Go based packages. Old Go packages (with the exception of docker-cli) will compile without changes (for example, mender, flanneld).
I don't understand this last sentence: your series is converting docker-cli (which requires a huge patch), but not mender and flanneld.
Quoted 5 lines
Cavets with the current implementation:
- "make source" may not produce a valid output
- module downloading occurs ignoring the Buildroot download server
- module downloading occurs in a post-patch hook
If it has all those caveats, so why do we want this ?
Quoted 5 lines
Go code uses a package-based imports system. Imports are relative to a root directory, previously called GOPATH.
import "github.com/docker/docker/pkg/myutils"
Quoted 7 lines
This would resolve to $GOPATH/src/github.com/docker/docker/pkg/myutils.
Buildroot packages previously used an additional feature in the Go tool which allows packages to avoid using GOPATH by "vendoring" dependencies - copying the code directly into the Git repository.
vendor/github.com/docker/docker/pkg/myutils
What does this path means ?
Quoted 6 lines
Old packages that used the vendor/ approach remain compatible via inferring the root import path from the download URL if no go.mod is present.
All current Buildroot Go modules use "vendor" to avoid downloading dependencies. This requires that the Go projects added to Buildroot include all of their dependencies in their repositories.
So some upstream Go projects collect all their dependencies in their Git repo, while some other upstream Go projects do not do that ? Just for my understanding, could you show an example of two projects, one in each situation ?
Quoted 2 lines
It also does not allow us the opportunity to
validate or adjust dependency versions when upgrading packages in Buildroot.
I'm sorry, but I don't understand what you mean here :-/
Quoted 7 lines
A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers.
Quoted 6 lines
module mymodule
require (
github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible )
Please add a sentence before dropping some example code. Something
like: "Here is an example go.mod file that shows ...".
Quoted 15 lines
The Go tool creates a go.sum file next to the go.mod file. The go.sum file is akin to Node's package-shrinkwrap.json.
github.com/aws/aws-sdk-go v1.15.31/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= github.com/aws/aws-sdk-go v1.17.12 h1:jMFwRUaM0LcfdenfvbDLePNoWSoCdOHqF4RCvSB4xNQ= github.com/aws/aws-sdk-go v1.17.12/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible h1:8fBbhRkI5/0ocLFbrhPgnGUm0ogc+Gko1cRodPWDKX4= github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
Ditto, please make it clear that this is an example of a go.sum file.
Quoted 9 lines
The file contains import paths, versions, and file hashes for the download as well as the contained go.mod file.
With replace directives, it's possible to link together local copies of modules:
replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi )
How does this story of "replace directives" fits in the overall picture ? You just give this information about "replace directives", with no connection with the rest of the explanation.
Quoted 26 lines
When GO111MODULE=on, all of the Go tool commands become "module-aware." GOPATH is no longer required. Running "go build" for example will fetch dependencies from the Internet, checksum them, and extract to a staging directory (defaulting currently to GOPATH/pkg/gomod or so). Imports are automatically resolved to the appropriately versioned staging directory.
There are various ways to control the download / extract behavior:
The GOPROXY environment variable allows further control over the download source. If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems. Setting GOPROXY to "off" disallows downloading modules from any source. Otherwise, GOPROXY is expected to be the URL of a module proxy, in which case the go command will fetch all modules from that proxy. No matter the source of the modules, downloaded modules must match existing entries in go.sum...
Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.
Quoted 1 line
GOPROXY additionally supports file:// URLs.
Perhaps this should be concluded by "Buildroot will set GOPROXY to ... in order to achieve a behavior that ...".
Quoted 5 lines
This commit sets GOPATH to $(DL_DIR)/go-module, as the Go module system will download and cache code sources in the GOPATH/pkg/mod path.
The go.mod and go.sum files can optionally be placed in $(2)_PKGDIR next to Config.in and other support files.
How does it work if there is no go.mod/go.sum file ?
Quoted 4 lines
They are copied in with a post-download hook, and "go mod download" is executed to pull the dependencies from the internet.
A hook is added to execute "go mod vendor".
What is "go mod vendor" going to do ?
Quoted 8 lines
Upstream vendor trees are still optionally supported, but can be overridden by placing go.mod into the Buildroot tree as described above. Package developers can alternatively specify
LIBFOO_GOMODto indicate the root module path. This allows the Go module tool to compile code using a legacy vendor/ tree, without a GOPATH to indicate the import path for the root module.
DOCKER_ENGINE_GOMOD= github.com/docker/docker
Again, please don't throw example code in the middle of the commit log without an introduction.
Quoted 11 lines
A Buildroot user can serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.
During the build phase, the "-mod=vendor" option is used to indicate that the extracted vendor/ tree (from the post-extract step) is to be used.
Go modules are never disabled with this approach. They are compatible with the legacy vendor/ tree approach by the above mechanisms.
These last two paragraphs are still very fuzzy for me :-/ I guess I need to read more about Go modules.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/6 package/go: implement go modules integration
Hi Thomas,
Some initial clarifications: this is a RFC and very much a Work in Progress. It is not complete, nor is it intended to be merged anywhere near its current form. The patch series is a reply to Arnout's request for an example of what these approaches might look like in Buildroot.
The original discussion is here:
https://patchwork.ozlabs.org/patch/1048076/
Quoted 7 lines · Thomas Petazzoni
Quoted 5 lines
Cavets with the current implementation:
- "make source" may not produce a valid output
- module downloading occurs ignoring the Buildroot download server
- module downloading occurs in a post-patch hookIf it has all those caveats, so why do we want this ?
Because these are issues with the initial WIP/RFC. Please note the last email in the chain at https://patchwork.ozlabs.org/patch/1048076/
Arnout:
So these are the more controversial ones. I think you should pick the one you like most, make a quick prototype that works for one specific package, note what is missing in the commit log, and send it to the list for discussion.
So here you go. This is the prototype with the list of cavets of the prototype. This is not a list of cavets with Go modules in general.
The introduction to the commit message is a copy of my writeup on what Go modules are.
Quoted 3 lines
I've been trying to get my head around this topic, but I don't really grasp why we're changing this. What is the advantage of implementing this go module integration compared to what we're doing today ?
The "old way" using GOPATH will be removed in Go 1.13 or Go 1.14. We need to plan a solution in advance. However, vendor/ will not stop working anytime soon.
The most minimal version of this patch series would be to enable GO111MODULE, place a go.mod file which specifies the Go import path of the root of the repository, and allow the Go tool to use the upstream vendor/ tree. The patch already does this in the absence of $(PKG_DIR)/go.mod and $(@D)/go.mod with the presence of $(@D)/vendor.
Today, to accomplish the same behavior with GOPATH (which is being removed/deprecated in Go 1.13), we symlink a directory path like this:
$(@D)/gopath/src/github.com/docker/docker/ -> $(@D)
We currently require upstream developers to copy the source code of every dependency into their repository under vendor/. For each Go package we get a separate dependency tree provided by the upstream developer at the version we have the package pinned to in Buildroot.
Quoted 4 lines
Looking at your patches 2/6 to 6/6 doesn't make the thing very
appealing: we need additional files (go.mod and go.sum) for each package using Go, which were not needed before, and we have a 5k lines patch in docker-cli.
The vendor/ mechanism continues to work, however, not every Go package copies dependencies verbatim into their repository. Under the "current way" to support these vendor-less packages, I would need to submit a patch to buildroot that contains a copy of every dependency's Go source.
Quoted 2 lines
I don't understand this last sentence: your series is converting docker-cli (which requires a huge patch), but not mender and flanneld.
Mender and Flanneld are examples of packages with vendor/ trees that are still compatible with the new patch. Docker-cli is an example of how we might override the upstream vendor/ tree with our own dependency pinning. (the go.mod and go.sum files produce the same vendor/ tree as upstream has in their source tree today).
This is a Proof-of-Concept of how we might update a specific dependency with a critical CVE or other bug which is vendored into an old upstream vendor/ tree. The Buildroot maintainers, upon discovering the CVE or other bug, could update the specific dependency to a newer revision with the fix. This a "feature" and more speculative. It's not required for a v1 of this patch.
Quoted 7 lines
Quoted 5 lines
Buildroot packages previously used an additional feature in the Go tool which allows packages to avoid using GOPATH by "vendoring" dependencies - copying the code directly into the Git repository.
vendor/github.com/docker/docker/pkg/myutils
What does this path means ?
It's an example of the sentence immediately before the line. This patch series is intended to be a initial introduction and a example of what something like this might look like. The commit message is not intended to be final yet. I will adjust anything I write in the future to explicitly say "here is an example..." instead of assuming that the reader is looking top-to-bottom.
Quoted 4 lines
So some upstream Go projects collect all their dependencies in their Git repo, while some other upstream Go projects do not do that ? Just for my understanding, could you show an example of two projects, one in each situation ?
Yes, this is correct.
Most major projects still use vendor/ today because it's the most foolproof way (without Go modules) to ensure no incorrect dependency gets into the mix, and because most projects are still supporting Go 1.10 which does not have Go module support.
Go modules were just added in Go 1.11 (we're at 1.12.1 now).
Here's a background on vendor:
https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment
Here's a background on Go modules:
https://github.com/golang/go/wiki/Modules
Docker currently still collects vendor tree. Here is a URL to that:
https://github.com/docker/engine/tree/da823cf3a5a3437763ba12b2fa369c826d4a2bf4/vendor
Here are some examples of projects that do not include vendor/:
- https://github.com/dustin/gomemcached
- https://github.com/syncthing/syncthing
- https://github.com/drone/drone
- https://github.com/nsqio/nsq
- https://github.com/cockroachdb/cockroach
CockroachDB is an interesting example of a common case where vendor/ will NOT work with Buildroot. The vendor/ tree is a submodule. As we currently download the GitHub tarball for GitHub based packages, this source tarball will not contain the contents of vendor/.
However, the project does have Gopkg.lock and Gopkg.toml, which are files for an old tool called "Dep" which was previously created by the community to solve dependency versioning. The Go Modules tool will automatically read and recognize this format and convert it to go.mod.
With this patch, you could add CockroachDB and the other projects above. Without, it would not be able to be added, and I don't see any way to get the submodule into the source tree, so it probably will never be added to Buildroot.
Quoted 4 lines
Quoted 2 lines
It also does not allow us the opportunity to validate or adjust dependency versions when upgrading packages in Buildroot.
I'm sorry, but I don't understand what you mean here :-/
Please see my above comments re: updating a dependency version when a CVE or other bug is found. Specifically, consider the case where we have an older LTS Buildroot branch pointing to an unmaintained older revision of the project. With vendor/ coming from the downloaded upstream source tarball, we would not be able to fix the CVE or bug without copying the patch in the Buildroot tree.
Quoted 9 lines
Quoted 9 lines
A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers.
module mymodule
Quoted 7 lines
Quoted 4 lines
require (
github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible )Please add a sentence before dropping some example code. Something
like: "Here is an example go.mod file that shows ...".
The entire paragraph before the code sample explains what it is. I will explicitly label it as an example in the future.
Quoted 8 lines
Quoted 4 lines
replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi )How does this story of "replace directives" fits in the overall picture ? You just give this information about "replace directives", with no connection with the rest of the explanation.
These can be placed into go.mod and allow you to replace a specific dependency with another one, or to alias the dependency to a local path.
Quoted 2 lines
Perhaps this should be concluded by "Buildroot will set GOPROXY to ... in order to achieve a behavior that ...".
Please note the commit message is a copy/paste of my Go modules explanation from the email thread in the original discussion. I have no idea how we would address GOPROXY in a v1 of this patch.
Quoted 7 lines
Quoted 5 lines
This commit sets GOPATH to $(DL_DIR)/go-module, as the Go module system will download and cache code sources in the GOPATH/pkg/mod path.
The go.mod and go.sum files can optionally be placed in $(2)_PKGDIR next to Config.in and other support files.
How does it work if there is no go.mod/go.sum file ?
We place a go.mod file with a single line, for example:
module github.com/docker/docker
This is sufficient to use the upstream vendor/ tree. The Go tool can also read the package files from other tools like Dep (Gopkg.toml).
Quoted 6 lines
Quoted 4 lines
They are copied in with a post-download hook, and "go mod download" is executed to pull the dependencies from the internet.
A hook is added to execute "go mod vendor".
What is "go mod vendor" going to do ?
It will extract the files from the buildroot go-module download tree into $(@D)/vendor/...
Quoted 11 lines
Quoted 8 lines
Upstream vendor trees are still optionally supported, but can be overridden by placing go.mod into the Buildroot tree as described above. Package developers can alternatively specify
LIBFOO_GOMODto indicate the root module path. This allows the Go module tool to compile code using a legacy vendor/ tree, without a GOPATH to indicate the import path for the root module.
DOCKER_ENGINE_GOMOD= github.com/docker/dockerAgain, please don't throw example code in the middle of the commit log without an introduction.
Please see above.
Quoted 11 lines
Quoted 8 lines
A Buildroot user can serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.
During the build phase, the "-mod=vendor" option is used to indicate that the extracted vendor/ tree (from the post-extract step) is to be used.
These last two paragraphs are still very fuzzy for me :-/ I guess I need to read more about Go modules.
1. Code is downloaded by the Go tool, hashed, and stored in dl/go-modules.
2. Code is extracted to $(@D)/vendor with the "go mod vendor" command.
3. go build -mod=vendor ./ will inform the Go tool that we want to use vendor/
Best regards,
Christian Stewart
RFC PATCH 1/6 package/go: implement go modules integration
Hi Christian,
Thanks again for working on this. And thanks Thomas for taking a look as well.
Quoted 6 lines · Christian Stewart
Hi Thomas,
Some initial clarifications: this is a RFC and very much a Work in Progress. It is not complete, nor is it intended to be merged anywhere near its current form. The patch series is a reply to Arnout's request for an example of what these approaches might look like in Buildroot.
You should have put this sentence prominently in the first patch's commit
message. But the RFC should have been a hint, of course. Also, I've marked the series as RFC in patchwork.
I think that one issue with this gomodules stuff is that the mails (and commit
messages) are getting too long, which makes it very hard for people to understand it all. Of course, it doesn't help that all this go stuff is wildly unfamiliar for most people.
But anyway, it would be good if parts of this problem could be bitten off in
smaller pieces. Suggestions will follow.
Quoted 7 lines
The original discussion is here:
https://patchwork.ozlabs.org/patch/1048076/
Quoted 2 lines · Thomas Petazzoni
Quoted 2 lines
Cavets with the current implementation:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
- "make source" may not produce a valid output
- module downloading occurs ignoring the Buildroot download server
- module downloading occurs in a post-patch hookIf it has all those caveats, so why do we want this ?
Because these are issues with the initial WIP/RFC.
Still, it would be good if you had at least some idea of how these could be solved.
Quoted 9 lines
Please note the last
email in the chain at https://patchwork.ozlabs.org/patch/1048076/Arnout:
So these are the more controversial ones. I think you should pick the one you like most, make a quick prototype that works for one specific package, note what is missing in the commit log, and send it to the list for discussion.
You forgot to mention the two options, and which one you picked:
- post-extract hook with optional go.mod in repository, "go mod vendor"
- additional download method which does "go mod vendor" and re-compress to a tarball (similar to git download)
and IIUC you picked the first one.
Quoted 6 lines
So here you go. This is the prototype with the list of cavets of the prototype. This is not a list of cavets with Go modules in general.
The introduction to the commit message is a copy of my writeup on what Go modules are.
Quoted 7 lines
Quoted 3 lines
I've been trying to get my head around this topic, but I don't really grasp why we're changing this. What is the advantage of implementing this go module integration compared to what we're doing today ?
The "old way" using GOPATH will be removed in Go 1.13 or Go 1.14. We need to plan a solution in advance. However, vendor/ will not stop working anytime soon.
So, here's a bite-sized piece: just convert GOPATH into GO111MODULE, but
assuming that the vendor/ tree is present. I believe that the current patch does exactly that, but it *also* does something else...
Quoted 3 lines
The most minimal version of this patch series would be to enable GO111MODULE, place a go.mod file which specifies the Go import path of
... I understood from our previous conversation that the go.mod file is in fact
not needed. If the go.mod file is missing, then the go tool will just use the vendor tree as is. Which is perfect for our current situation.
So, IMO, the _APPLY_EXTRACT_GOMOD part should be removed from this patch. Then,
patch 1 becomes a really simple patch that people can get their head around. And you get a much better baseline to explain what you want to do, since you no longer have to compare against GOPATH; you only need to compare against a non-instantiated vendor tree.
As usual, if I understand correctly, of course :-)
Quoted 19 lines
the root of the repository, and allow the Go tool to use the upstream vendor/ tree. The patch already does this in the absence of $(PKG_DIR)/go.mod and $(@D)/go.mod with the presence of $(@D)/vendor.
Today, to accomplish the same behavior with GOPATH (which is being removed/deprecated in Go 1.13), we symlink a directory path like this:
$(@D)/gopath/src/github.com/docker/docker/ -> $(@D)
We currently require upstream developers to copy the source code of every dependency into their repository under vendor/. For each Go package we get a separate dependency tree provided by the upstream developer at the version we have the package pinned to in Buildroot.
Quoted 4 lines
Looking at your patches 2/6 to 6/6 doesn't make the thing very
appealing: we need additional files (go.mod and go.sum) for each package using Go, which were not needed before, and we have a 5k lines patch in docker-cli.
Quoted 4 lines
The vendor/ mechanism continues to work, however, not every Go package copies dependencies verbatim into their repository. Under the "current way" to support these vendor-less packages, I would need to submit a patch to buildroot that contains a copy of every dependency's Go source.
Actually, you would need to specify additional downloads, and a post-extract
hook to extract them into the vendor directory.
If you can give an example of a package that only has a few of these, I could
create an example of how it might work without any extra infra. Although I'm not entirely sure if such an example would be very useful, except to show that it would be extremely hard to maintain.
Quoted 8 lines
Quoted 2 lines
I don't understand this last sentence: your series is converting docker-cli (which requires a huge patch), but not mender and flanneld.
Mender and Flanneld are examples of packages with vendor/ trees that are still compatible with the new patch. Docker-cli is an example of how we might override the upstream vendor/ tree with our own dependency pinning. (the go.mod and go.sum files produce the same vendor/ tree as upstream has in their source tree today).
If that is the example, then it shows that it's definitely not the way we want
to go... But I don't think it's the example that you want to show, because you're not yet using go modules to download stuff.
Quoted 6 lines
This is a Proof-of-Concept of how we might update a specific dependency with a critical CVE or other bug which is vendored into an old upstream vendor/ tree. The Buildroot maintainers, upon discovering the CVE or other bug, could update the specific dependency to a newer revision with the fix. This a "feature" and more speculative. It's not required for a v1 of this patch.
Is this something that we want to do? Go has apparently chosen for this
vendoring model where it is the responsibility of the package maintainers to update their dependencies, and not of the distro. In that model, if the package maintainers don't do their job, we're not going to fix the world for them. In general, we only take fixes (even for CVEs) that come from upstream. Maybe we'll backport a fix, but nothing more than that.
By using the vendoring approach, Go has basically chosen to make each CVE in a
library a CVE against all packages using that library, which has to be solved separately in each user. Recursively.
All of the above IMHO.
[snip]
Quoted 8 lines
Here are some examples of projects that do not include vendor/:
- https://github.com/dustin/gomemcached
- https://github.com/syncthing/syncthing
- https://github.com/drone/drone
- https://github.com/nsqio/nsq
- https://github.com/cockroachdb/cockroach
Quoted 4 lines
CockroachDB is an interesting example of a common case where vendor/ will NOT work with Buildroot. The vendor/ tree is a submodule. As we currently download the GitHub tarball for GitHub based packages, this source tarball will not contain the contents of vendor/.
That one would actually still work - we *can* use the git download method for
github packages.
nsq is a better example - it doesn't have a vendor tree nor submodules. And it
has an actual modern go.mod. And it's a nice example that trying to do this manually within buildroot is not doable: the go.mod is huge. Like, npm-huge...
However, the current patch set does nothing to solve that, I think. So it would
be nice to have an example of how we might integrate something like nsq.
Quoted 10 lines
However, the project does have Gopkg.lock and Gopkg.toml, which are files for an old tool called "Dep" which was previously created by the community to solve dependency versioning. The Go Modules tool will automatically read and recognize this format and convert it to go.mod.
With this patch, you could add CockroachDB and the other projects above. Without, it would not be able to be added, and I don't see any way to get the submodule into the source tree, so it probably will never be added to Buildroot.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
It also does not allow us the opportunity to validate or adjust dependency versions when upgrading packages in Buildroot.
I'm sorry, but I don't understand what you mean here :-/
Please see my above comments re: updating a dependency version when a CVE or other bug is found.
As I explained above: Buildroot will not do that.
Quoted 5 lines
Specifically, consider the case where we have
an older LTS Buildroot branch pointing to an unmaintained older revision of the project. With vendor/ coming from the downloaded upstream source tarball, we would not be able to fix the CVE or bug without copying the patch in the Buildroot tree.
Well, actually we can... In any of the approaches we're considering, after the
extract step we will have all the dependencies present in the vendor/ tree. So we can always add a patch for a CVE fix in a dependent package in the usual way (we just have to add some directory components to the upstream patch).
The only situation where this would not be the case is if we would let the go
tool do its downloads during the build step. But that is something we *definitely* want to avoid.
Quoted 25 lines
Quoted 9 lines
Quoted 9 lines
A project can contain any number of go.mod files. A go.mod file is akin to Node's package.json file. It specifies all direct and indirect dependencies of all Go packages in the subtree below the file. The Go tool can manage this file automatically if desired, and specifies a required format. Go.mod additionally requires dependency versions to be explicitly specified. There is no semantic versioning or asterisk-based version specifiers.
module mymodule
Quoted 7 lines
Quoted 4 lines
require (
github.com/aws/aws-sdk-go v1.17.12 // indirect github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible )Please add a sentence before dropping some example code. Something
like: "Here is an example go.mod file that shows ...".The entire paragraph before the code sample explains what it is. I will explicitly label it as an example in the future.
Quoted 5 lines
Quoted 4 lines
replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi )
Quoted 6 lines
Quoted 3 lines
How does this story of "replace directives" fits in the overall picture ? You just give this information about "replace directives", with no connection with the rest of the explanation.
These can be placed into go.mod and allow you to replace a specific dependency with another one, or to alias the dependency to a local path.
Yeah, that bit is not very clear to me either... Is it intended to be able to
override the version of recursive dependencies?
Quoted 6 lines
Quoted 2 lines
Perhaps this should be concluded by "Buildroot will set GOPROXY to ... in order to achieve a behavior that ...".
Please note the commit message is a copy/paste of my Go modules explanation from the email thread in the original discussion. I have no idea how we would address GOPROXY in a v1 of this patch.
Easy: in the current sitatution, we *never* want the go tool to download
anything (because we expect everything to be present in the vendor/ tree), so we can just globally export GOPROXY=off. IIUC.
Note that this is something that would potentially break existing external
packages, so should be mentioned clearly in the release notes. It can be fixed easily, of course, by setting GOPROXY=direct in <PKG>_GO_ENV.
Given that this is a slightly more controversial thing, I would do it in a
separate patch from the GOPATH removal.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
This commit sets GOPATH to $(DL_DIR)/go-module, as the Go module system will download and cache code sources in the GOPATH/pkg/mod path.
The go.mod and go.sum files can optionally be placed in $(2)_PKGDIR next to Config.in and other support files.
Quoted 8 lines
Quoted 1 line
How does it work if there is no go.mod/go.sum file ?
We place a go.mod file with a single line, for example:
module github.com/docker/docker
This is sufficient to use the upstream vendor/ tree. The Go tool can also read the package files from other tools like Dep (Gopkg.toml).
So,
- if a go.mod is present, we can use the Go tool;
- if a legacy .toml or whatever is present, we can use the Go tool;
- if there is a vendor/ tree but no metadata, we need to create an almost-empty go.mod.
Correct?
And all of that is only needed to be able to do the 'go mod vendor' command in
a post-extract hook, right?
What does 'go mod vendor' do if there is no go.mod or legacy file?
What I also don't understand is why we would ever add a go.mod ourselves... If
upstream has one, clearly there is no reason to add one. If upstream has a .toml, then we can just use that. If upstream has nothing, it means that everything is already in the vendor/ tree so implicitly versioned, so we also don't need anything from the go mod logic. We *maybe* should create the one-line go.mod that just declares the module, but even that I'm not sure of.
Can you explain why patch 2 is needed, converting runc's vendor.conf into a
go.mod and go.sum?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
They are copied in with a post-download hook, and "go mod download" is executed to pull the dependencies from the internet.
I may be blind, but I don't see this call to 'go mod download'... I suppose
that it is in fact the 'go mod vendor' call that will implicitly do a download, no?
Quoted 30 lines
Quoted 3 lines
Quoted 1 line
A hook is added to execute "go mod vendor".
What is "go mod vendor" going to do ?
It will extract the files from the buildroot go-module download tree into $(@D)/vendor/...
Quoted 11 lines
Quoted 8 lines
Upstream vendor trees are still optionally supported, but can be overridden by placing go.mod into the Buildroot tree as described above. Package developers can alternatively specify
LIBFOO_GOMODto indicate the root module path. This allows the Go module tool to compile code using a legacy vendor/ tree, without a GOPATH to indicate the import path for the root module.
DOCKER_ENGINE_GOMOD= github.com/docker/dockerAgain, please don't throw example code in the middle of the commit log without an introduction.
Please see above.
Quoted 9 lines
Quoted 8 lines
A Buildroot user can serve the dl/go-modules directory directly with a HTTP server and set GOPROXY such that all module downloads come from that server. This could be configurable eventually via the Buildroot KConfig architecture.
During the build phase, the "-mod=vendor" option is used to indicate that the extracted vendor/ tree (from the post-extract step) is to be used.
Quoted 6 lines
Quoted 2 lines
These last two paragraphs are still very fuzzy for me :-/ I guess I need to read more about Go modules.
1. Code is downloaded by the Go tool, hashed, and stored in dl/go-modules.
2. Code is extracted to $(@D)/vendor with the "go mod vendor" command.
3. go build -mod=vendor ./ will inform the Go tool that we want to use vendor/
So, I think this should be chunked into:
1. Replace GOPATH with -mod=vendor - the same patch will need to update packages if needed to keep them building.
2. Update packages to remove no-longer-needed _WORKSPACE etc. settings.
3. export GOPROXY=off
4. Run 'go mod vendor' in a post-extract hook. This should still be RFC because it violates the offline builds principle. Don't forget to mention some ideas of how the limitations could be overcome.
5. Add a package (nsq?) that would be very difficult to support without the go.mod support.
6. Add support for Buildroot-supplied go.mod.
7. Add or modify a package that actually makes sensible use of the Buildroot-supplied go.mod.
Regards,
Arnout
RFC PATCH 1/6 package/go: implement go modules integration
Hi Arnout,
Quoted 8 lines · Arnout Vandecappelle
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Cavets with the current implementation:
- "make source" may not produce a valid output
- module downloading occurs ignoring the Buildroot download server
- module downloading occurs in a post-patch hookIf it has all those caveats, so why do we want this ?
Because these are issues with the initial WIP/RFC.
Still, it would be good if you had at least some idea of how these could be solved.
Create a new package download method that uses the "go mod download" and "go mod vendor" steps seems to be the only way to go.
Quoted 10 lines
So, here's a bite-sized piece: just convert GOPATH into GO111MODULE, but
assuming that the vendor/ tree is present. I believe that the current patch does exactly that, but it *also* does something else...Quoted 3 lines
The most minimal version of this patch series would be to enable GO111MODULE, place a go.mod file which specifies the Go import path of
... I understood from our previous conversation that the go.mod file is in fact
not needed. If the go.mod file is missing, then the go tool will just use the vendor tree as is. Which is perfect for our current situation.
The file IS needed because the Go tool needs to know the root package path at $(@D). For example, in Docker, this is "github.com/docker/docker" even though the code comes from "github.com/docker/engine."
Quoted 6 lines
So, IMO, the
_APPLY_EXTRACT_GOMODpart should be removed from this patch. Then,
patch 1 becomes a really simple patch that people can get their head around. And you get a much better baseline to explain what you want to do, since you no longer have to compare against GOPATH; you only need to compare against a non-instantiated vendor tree.
This part cannot be removed but the go.mod in the Buildroot tree and the module download / vendor steps could be for an initial version. I'll spin a series with this soon.
Quoted 7 lines
Actually, you would need to specify additional downloads, and a post-extract
hook to extract them into the vendor directory.If you can give an example of a package that only has a few of these, I could
create an example of how it might work without any extra infra. Although I'm not entirely sure if such an example would be very useful, except to show that it would be extremely hard to maintain.
This doesn't make any sense to me.
Quoted 3 lines
If that is the example, then it shows that it's definitely not the way we want
to go... But I don't think it's the example that you want to show, because you're not yet using go modules to download stuff.
I am using Go modules to download stuff in this PoC patch.
Quoted 7 lines
Quoted 6 lines
This is a Proof-of-Concept of how we might update a specific dependency with a critical CVE or other bug which is vendored into an old upstream vendor/ tree. The Buildroot maintainers, upon discovering the CVE or other bug, could update the specific dependency to a newer revision with the fix. This a "feature" and more speculative. It's not required for a v1 of this patch.
Quoted 6 lines
Is this something that we want to do? Go has apparently chosen for this
vendoring model where it is the responsibility of the package maintainers to update their dependencies, and not of the distro. In that model, if the package maintainers don't do their job, we're not going to fix the world for them. In general, we only take fixes (even for CVEs) that come from upstream. Maybe we'll backport a fix, but nothing more than that.
I don't see the point. If we discover a security issue, this provides a mechanism to override a dependency at download time. If this is not something that Buildroot wants I'm happy to just remove the 2 lines it takes to implement it in the code.
Quoted 3 lines
By using the vendoring approach, Go has basically chosen to make each CVE in a
library a CVE against all packages using that library, which has to be solved separately in each user. Recursively.
They have not chosen to do this. The vendoring model is/was a stop-gap until something better (Modules) was developed.
Quoted 6 lines
nsq is a better example - it doesn't have a vendor tree nor submodules. And it
has an actual modern go.mod. And it's a nice example that trying to do this manually within buildroot is not doable: the go.mod is huge. Like, npm-huge...However, the current patch set does nothing to solve that, I think. So it would
be nice to have an example of how we might integrate something like nsq.
The current patch set DOES solve this. It would work fine AS IS, since the "go mod vendor" step downloads and extracts dependencies from go.mod.
The only awkward part of the RFC is that this behavior is not placed into a proper download mechanism rather than the extract step.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
It also does not allow us the opportunity to validate or adjust dependency versions when upgrading packages in Buildroot.
Please see my above comments re: updating a dependency version when a CVE or other bug is found.
As I explained above: Buildroot will not do that.
Why?
Quoted 10 lines
Quoted 5 lines
Specifically, consider the case where we have
an older LTS Buildroot branch pointing to an unmaintained older revision of the project. With vendor/ coming from the downloaded upstream source tarball, we would not be able to fix the CVE or bug without copying the patch in the Buildroot tree.Well, actually we can... In any of the approaches we're considering, after the
extract step we will have all the dependencies present in the vendor/ tree. So we can always add a patch for a CVE fix in a dependent package in the usual way (we just have to add some directory components to the upstream patch).
I don't see why this is better than overriding go.mod?
Quoted 3 lines
The only situation where this would not be the case is if we would let the go
tool do its downloads during the build step. But that is something we *definitely* want to avoid.
This doesn't make any sense to me either. Who proposed downloading during the build step? Why would this be the only case where adding a patch would not work? With a new download method, it would go:
1. Download base package (containing go.mod)
2. (optional) Copy in go.mod and go.sum from Buildroot tree
3. Execute "go mod vendor." Note, go.sum ensures the result is consistent.
4. Compress the result and store in dl/ tree.
5. Later, in the extract step, extract this tarball
6. Apply any patches (this can also target the vendor/ tree if needed)
7. Build the package with -mod=vendor arg which uses the vendor/ tree.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi )How does this story of "replace directives" fits in the overall picture ? You just give this information about "replace directives", with no connection with the rest of the explanation.
Quoted 5 lines
Quoted 2 lines
These can be placed into go.mod and allow you to replace a specific dependency with another one, or to alias the dependency to a local path.
Yeah, that bit is not very clear to me either... Is it intended to be able to
override the version of recursive dependencies?
Yes. You can also tell Go to use local copies of dependencies out-of-tree.
Quoted 3 lines
Easy: in the current sitatution, we *never* want the go tool to download
anything (because we expect everything to be present in the vendor/ tree), so we can just globally export GOPROXY=off. IIUC.
This is already done in the above patch, with the exception of the "go mod vendor" step which does a download, which would be moved into a download provider in the actual change.
Quoted 3 lines
Note that this is something that would potentially break existing external
packages, so should be mentioned clearly in the release notes. It can be fixed easily, of course, by setting GOPROXY=direct in <PKG>_GO_ENV.
How would this break external packages? The _GOMOD variable is automatically determined using the same logic that the GOPATH directory structure is determined today. This means that existing external packages using vendor/ would still build fine.
Quoted 2 lines
Given that this is a slightly more controversial thing, I would do it in a
separate patch from the GOPATH removal.
I'm not sure what specifically you're referring to here.
Quoted 8 lines
So,
- if a go.mod is present, we can use the Go tool;
- if a legacy .toml or whatever is present, we can use the Go tool;
- if there is a vendor/ tree but no metadata, we need to create an almost-empty go.mod.Correct?
Yes, the go.mod needs to contain a single line to inform the Go tool of the package path of the root of the code.
Quoted 2 lines
And all of that is only needed to be able to do the 'go mod vendor' command in
a post-extract hook, right?
No, this is needed to get Go to use the vendor/ tree without a GOPATH, because it used to derive the root package path of the code from the location in the GOPATH tree.
Quoted 1 line
What does 'go mod vendor' do if there is no go.mod or legacy file?
It fails.
Quoted 2 lines
What I also don't understand is why we would ever add a go.mod ourselves... If
upstream has one, clearly there is no reason to add one.
Upstream does not have one in a lot of cases (yet).
Quoted 5 lines
If upstream has a .toml, then we can just use that. If upstream has nothing, it means that everything is already in the vendor/ tree so implicitly versioned, so we also don't need anything from the go mod logic. We *maybe* should create the one-line go.mod that just declares the module, but even that I'm not sure of.
The goal of copying go.mod in from the Buildroot tree is to allow you to override / provide go.mod if there is not one or if the existing one is outdated with security issues, BEFORE the "go mod download" "go mod vendor" steps performed in a download handler (the download step, before the patch step).
Quoted 2 lines
Can you explain why patch 2 is needed, converting runc's vendor.conf into a go.mod and go.sum?
Converting vendor.conf does network lookups with a non-deterministic result.
Quoted 2 lines
I may be blind, but I don't see this call to 'go mod download'... I suppose that it is in fact the 'go mod vendor' call that will implicitly do a download, no?
Yes, when GOPROXY=direct.
Quoted 4 lines
So, I think this should be chunked into:
1. Replace GOPATH with -mod=vendor - the same patch will need to update packages if needed to keep them building.
No updates to existing packages should be needed, and it's roughly the same patch as what is in this RFC, except with the "copy in go.mod" lines removed, and the "go mod vendor" lines removed.
Quoted 2 lines
2. Update packages to remove no-longer-needed
_WORKSPACEetc. settings.
3. export GOPROXY=off
Already done in this patch, and would be done in step #1, not after step #2.
Quoted 3 lines
4. Run 'go mod vendor' in a post-extract hook. This should still be RFC because it violates the offline builds principle. Don't forget to mention some ideas of how the limitations could be overcome.
Not going to do this in a "real" solution.
Quoted 2 lines
5. Add a package (nsq?) that would be very difficult to support without the go.mod support.
I don't see why this is required for this, but that is the goal (to allow external and internal packages that use go.mod and to transition off of GOPATH for vendor/ based packages.)
Quoted 1 line
6. Add support for Buildroot-supplied go.mod.
I agree this could be done after the initial minimal commit creating the single-line go.mod file, setting GOPROXY=off, and setting -mod=vendor.
Best regards,
Christian
RFC PATCH 1/6 package/go: implement go modules integration
Quoted 9 lines · Christian Stewart
Hi Arnout,
Quoted 5 lines · Arnout Vandecappelle
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Cavets with the current implementation:
- "make source" may not produce a valid output
- module downloading occurs ignoring the Buildroot download server
- module downloading occurs in a post-patch hook
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
If it has all those caveats, so why do we want this ?
Because these are issues with the initial WIP/RFC.
Still, it would be good if you had at least some idea of how these could be solved.
Create a new package download method that uses the "go mod download" and "go mod vendor" steps seems to be the only way to go.
+1
Quoted 7 lines
Quoted 7 lines
So, here's a bite-sized piece: just convert GOPATH into GO111MODULE, but
assuming that the vendor/ tree is present. I believe that the current patch does exactly that, but it *also* does something else...Quoted 3 lines
The most minimal version of this patch series would be to enable GO111MODULE, place a go.mod file which specifies the Go import path of
Quoted 8 lines
Quoted 3 lines
... I understood from our previous conversation that the go.mod file is in fact
not needed. If the go.mod file is missing, then the go tool will just use the vendor tree as is. Which is perfect for our current situation.The file IS needed because the Go tool needs to know the root package path at $(@D). For example, in Docker, this is "github.com/docker/docker" even though the code comes from "github.com/docker/engine."
Okay, I see... So e.g. in docker-containerd, if we would create a go.mod with
just "module github.com/docker/containerd" (note the 'incorrect' docker instead of containerd), we would not have to change the -X ...GitCommit setting?
Quoted 9 lines
Quoted 5 lines
So, IMO, the
_APPLY_EXTRACT_GOMODpart should be removed from this patch. Then,
patch 1 becomes a really simple patch that people can get their head around. And you get a much better baseline to explain what you want to do, since you no longer have to compare against GOPATH; you only need to compare against a non-instantiated vendor tree.This part cannot be removed but the go.mod in the Buildroot tree and the module download / vendor steps could be for an initial version. I'll spin a series with this soon.
I don't understand why it can't be removed. The APPLY_EXTRACT_GOMOD doesn't do
anything if a vendor tree is already present, and in all our current go packages the vendor tree is already present...
Oh, hang on, you probably still have to create an almost-empty go.mod for the
ones which don't have a vendor.conf.
Quoted 9 lines
Quoted 7 lines
Actually, you would need to specify additional downloads, and a post-extract
hook to extract them into the vendor directory.If you can give an example of a package that only has a few of these, I could
create an example of how it might work without any extra infra. Although I'm not entirely sure if such an example would be very useful, except to show that it would be extremely hard to maintain.This doesn't make any sense to me.
What doesn't make sense to you? Do you mean that you don't understand how it
could be done with additional downloads, or do you mean that you don't understand how it could not be useful, or do you mean that you don't understand how it could be hard to maintain, or do you mean that you fail to parse the entire sentence?
Quoted 5 lines
Quoted 3 lines
If that is the example, then it shows that it's definitely not the way we want
to go... But I don't think it's the example that you want to show, because you're not yet using go modules to download stuff.I am using Go modules to download stuff in this PoC patch.
Can you point out which bit causes the modules to be downloaded?
Quoted 14 lines
Quoted 7 lines
Quoted 6 lines
This is a Proof-of-Concept of how we might update a specific dependency with a critical CVE or other bug which is vendored into an old upstream vendor/ tree. The Buildroot maintainers, upon discovering the CVE or other bug, could update the specific dependency to a newer revision with the fix. This a "feature" and more speculative. It's not required for a v1 of this patch.
Quoted 6 lines
Is this something that we want to do? Go has apparently chosen for this
vendoring model where it is the responsibility of the package maintainers to update their dependencies, and not of the distro. In that model, if the package maintainers don't do their job, we're not going to fix the world for them. In general, we only take fixes (even for CVEs) that come from upstream. Maybe we'll backport a fix, but nothing more than that.
Quoted 2 lines
I don't see the point. If we discover a security issue, this provides a mechanism to override a dependency at download time.
... by keeping a copy of the entire go.mod and go.sum in Buildroot...
But I see your point. It's the docker-cli thing that requires the entire
docker-specific patch for pflag that put me on the wrong foot.
Quoted 10 lines
If this is not
something that Buildroot wants I'm happy to just remove the 2 lines it takes to implement it in the code.Quoted 3 lines
By using the vendoring approach, Go has basically chosen to make each CVE in a
library a CVE against all packages using that library, which has to be solved separately in each user. Recursively.They have not chosen to do this. The vendoring model is/was a stop-gap until something better (Modules) was developed.
"The vendoring model" means that every package bundles its dependencies.
Whether those dependencies are maintained in the same git tree, or as git submodules, or as go modules with an explicit version doesn't make much difference. For the package developers it does make a world of difference, of course. But for the distro maintainer it runs down to the same thing: updates of dependencies are handled by the package developer, not by the distro maintainer.
There are obviously advantages to the vendoring model. It relieves the package
developers from dealing with changing APIs of their dependencies, it allows them to work around bugs in dependencies, and it makes sure that what they test is also what the final user is going to get. Also for library developers it makes things easier, because ABI compatibility is no longer a thing. And even for distro maintainers it makes things easier, because in essence the distro no longer needs to worry about dependencies. However, it has this one big disadvantage that a vulnerability can no longer be fixed by just updating a single package.
Quoted 9 lines
Quoted 6 lines
nsq is a better example - it doesn't have a vendor tree nor submodules. And it
has an actual modern go.mod. And it's a nice example that trying to do this manually within buildroot is not doable: the go.mod is huge. Like, npm-huge...However, the current patch set does nothing to solve that, I think. So it would
be nice to have an example of how we might integrate something like nsq.The current patch set DOES solve this. It would work fine AS IS, since the "go mod vendor" step downloads and extracts dependencies from go.mod.
Yeah, I don't understand why I wrote that... The current patch solves exactly
that...
Quoted 2 lines
The only awkward part of the RFC is that this behavior is not placed into a proper download mechanism rather than the extract step.
Ack. A limitation which you clearly marked in your commit message.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
It also does not allow us the opportunity to validate or adjust dependency versions when upgrading packages in Buildroot.
Please see my above comments re: updating a dependency version when a CVE or other bug is found.
As I explained above: Buildroot will not do that.
Why?
My statement was a little too strong. Let's say that up to now we have not been
in the habit of doing that.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Specifically, consider the case where we have
an older LTS Buildroot branch pointing to an unmaintained older revision of the project. With vendor/ coming from the downloaded upstream source tarball, we would not be able to fix the CVE or bug without copying the patch in the Buildroot tree.
Quoted 6 lines
Quoted 4 lines
Well, actually we can... In any of the approaches we're considering, after the
extract step we will have all the dependencies present in the vendor/ tree. So we can always add a patch for a CVE fix in a dependent package in the usual way (we just have to add some directory components to the upstream patch).I don't see why this is better than overriding go.mod?
Because usually, changing the version brings in a whole lot more changes than
just the CVE fix.
Quoted 5 lines
Quoted 3 lines
The only situation where this would not be the case is if we would let the go
tool do its downloads during the build step. But that is something we *definitely* want to avoid.This doesn't make any sense to me either.
"this would not be the case" refers to your statement that we would not be able
to fix a CVE (by adding a .patch). My statement is: we are able to fix CVEs like we do it for any other package, as long as go modules are instantiated/extracted before the patch step.
Quoted 2 lines
Who proposed downloading
during the build step?
If we don't set GOPROXY=off, the Go tool might go off and download stuff.
Quoted 10 lines
Why would this be the only case where adding a
patch would not work? With a new download method, it would go:1. Download base package (containing go.mod)
2. (optional) Copy in go.mod and go.sum from Buildroot tree
3. Execute "go mod vendor." Note, go.sum ensures the result is consistent.
4. Compress the result and store in dl/ tree.
5. Later, in the extract step, extract this tarball
6. Apply any patches (this can also target the vendor/ tree if needed)
7. Build the package with -mod=vendor arg which uses the vendor/ tree.
Exactly, so it is still possible to use the patch approach instead of changing
the version.
[snip]
Quoted 8 lines
Quoted 3 lines
Easy: in the current sitatution, we *never* want the go tool to download
anything (because we expect everything to be present in the vendor/ tree), so we can just globally export GOPROXY=off. IIUC.This is already done in the above patch, with the exception of the "go mod vendor" step which does a download, which would be moved into a download provider in the actual change.
Quoted 8 lines
Quoted 3 lines
Note that this is something that would potentially break existing external
packages, so should be mentioned clearly in the release notes. It can be fixed easily, of course, by setting GOPROXY=direct in <PKG>_GO_ENV.How would this break external packages? The
_GOMODvariable is automatically determined using the same logic that the GOPATH directory structure is determined today. This means that existing external packages using vendor/ would still build fine.
An existing external package that doesn't use vendor/ but instead uses go.mod
(and thus downloads stuff in the build step, because the go tool will do that automatically) would break. I don't know if this is even possible at the moment, I'm just indicating a possible concern. Maybe I'm exaggerating and setting GOPROXY=off doesn't really carry any risks.
Quoted 16 lines
Quoted 2 lines
Given that this is a slightly more controversial thing, I would do it in a
separate patch from the GOPATH removal.I'm not sure what specifically you're referring to here.>
Quoted 8 lines
So,
- if a go.mod is present, we can use the Go tool;
- if a legacy .toml or whatever is present, we can use the Go tool;
- if there is a vendor/ tree but no metadata, we need to create an almost-empty go.mod.Correct?
Yes, the go.mod needs to contain a single line to inform the Go tool of the package path of the root of the code.
Quoted 6 lines
Quoted 2 lines
And all of that is only needed to be able to do the 'go mod vendor' command in
a post-extract hook, right?No, this is needed to get Go to use the vendor/ tree without a GOPATH, because it used to derive the root package path of the code from the location in the GOPATH tree.
OK, got it!
Quoted 16 lines
Quoted 2 lines
What does 'go mod vendor' do if there is no go.mod or legacy file?
It fails.
Quoted 2 lines
What I also don't understand is why we would ever add a go.mod ourselves... If
upstream has one, clearly there is no reason to add one.Upstream does not have one in a lot of cases (yet).
Quoted 5 lines
If upstream has a .toml, then we can just use that. If upstream has nothing, it means that everything is already in the vendor/ tree so implicitly versioned, so we also don't need anything from the go mod logic. We *maybe* should create the one-line go.mod that just declares the module, but even that I'm not sure of.
Quoted 3 lines
The goal of copying go.mod in from the Buildroot tree is to allow you to override / provide go.mod if there is not one or if the existing one is outdated with security issues,
There are two concerns here, and I think we should consider them separately.
1. Make sure that building with -mod works.
a. Upstream has a go.mod -> nothing needs to be done.
b. Upstream has no go.mod, but has vendor.conf or .toml -> nothing needs to be
done, the Go tool handles these as well.
c. Upstream has no go.mod or vendor.conf or anything -> go.mod with just the
module line needs to be created.
2. Modify dependencies by changing their versions. -> Buildroot-supplied go.mod is required.
So, if my above analysis is correct, a Buildroot-supplied go.mod is only needed
for the 'changing versions' scenario (which I am not convinced is something we need). For the 'upstream has nothing' scenario, it is sufficient to generate the one-line go.mod.
Is that correct?
Quoted 9 lines
BEFORE the "go mod download" "go mod
vendor" steps performed in a download handler (the download step, before the patch step).Quoted 2 lines
Can you explain why patch 2 is needed, converting runc's vendor.conf into a go.mod and go.sum?
Converting vendor.conf does network lookups with a non-deterministic result.
Let me rephrase the question... As far as I understand, after the current patch
series, the Go tool will still not download anything for the existing packages, because the vendor tree is already there. Is that correct? If not, could you point out where the download happens?
Quoted 5 lines
Quoted 2 lines
I may be blind, but I don't see this call to 'go mod download'... I suppose that it is in fact the 'go mod vendor' call that will implicitly do a download, no?
Yes, when GOPROXY=direct.
Quoted 8 lines
Quoted 4 lines
So, I think this should be chunked into:
1. Replace GOPATH with -mod=vendor - the same patch will need to update packages if needed to keep them building.
No updates to existing packages should be needed, and it's roughly the same patch as what is in this RFC, except with the "copy in go.mod" lines removed, and the "go mod vendor" lines removed.
Ah, of course, because the go.mod is autogenerated, it will still work. But
then why are those changes in docker-containerd.mk needed? I still don't get it :-(
Quoted 4 lines
Quoted 2 lines
2. Update packages to remove no-longer-needed
_WORKSPACEetc. settings.
3. export GOPROXY=offAlready done in this patch, and would be done in step #1, not after step #2.
I put this separate because AFAIU setting GOPROXY to off is not strictly
needed. In the current situation, the Go tool is never going to download anything anyway, because the go.mod is empty.
Quoted 5 lines
Quoted 3 lines
4. Run 'go mod vendor' in a post-extract hook. This should still be RFC because it violates the offline builds principle. Don't forget to mention some ideas of how the limitations could be overcome.
Not going to do this in a "real" solution.
The real solution being the go download method, right?
Quoted 6 lines
Quoted 2 lines
5. Add a package (nsq?) that would be very difficult to support without the go.mod support.
I don't see why this is required for this, but that is the goal (to allow external and internal packages that use go.mod and to transition off of GOPATH for vendor/ based packages.)
It's not *required*, it's just to show what is the use case for this work.
Quoted 4 lines
Quoted 1 line
6. Add support for Buildroot-supplied go.mod.
I agree this could be done after the initial minimal commit creating the single-line go.mod file, setting GOPROXY=off, and setting -mod=vendor.
For this one, it would be nice if there could be an alternative that doesn't
require copying the entire go.mod and go.sum, but rather override only one or two. Because I still think that that is going to be the only use case of this feature.
Regards,
Arnout
RFC PATCH 1/6 package/go: implement go modules integration
Hi Arnout,
Quoted 3 lines · Arnout Vandecappelle
Okay, I see... So e.g. in docker-containerd, if we would create a go.mod with
just "module github.com/docker/containerd" (note the 'incorrect' docker instead of containerd), we would not have to change the -X ...GitCommit setting?
This would not work since the code expects a specific import path. Note the difference between an "import path" and the actual path where the code is hosted, and the actual path where the code is located on disk. These are 3 different paths.
It's correct to update the setting to match the code.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
So, IMO, the
_APPLY_EXTRACT_GOMODpart should be removed from this patch. Then,
patch 1 becomes a really simple patch that people can get their head around. And you get a much better baseline to explain what you want to do, since you no longer have to compare against GOPATH; you only need to compare against a non-instantiated vendor tree.
Quoted 7 lines
Quoted 3 lines
This part cannot be removed but the go.mod in the Buildroot tree and the module download / vendor steps could be for an initial version. I'll spin a series with this soon.
I don't understand why it can't be removed. The
APPLY_EXTRACT_GOMODdoesn't do
anything if a vendor tree is already present, and in all our current go packages the vendor tree is already present...
The hook can probably be removed but we need to make the almost-empty go.mod somewhere.
Quoted 2 lines
Oh, hang on, you probably still have to create an almost-empty go.mod for the
ones which don't have a vendor.conf.
I'm almost sorry I mentioned vendor.conf or gopkg.toml in the first place, becuase it seems like you're now distracted by the other formats. It's not really a good idea to use the auto conversion at runtime since the result is not as deterministic as having a predetetermined go.mod, and the conversion requires network lookups.
Pretend the other formats don't exist. They are there for legacy only anyway.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
If you can give an example of a package that only has a few of these, I could
create an example of how it might work without any extra infra. Although I'm not entirely sure if such an example would be very useful, except to show that it would be extremely hard to maintain.
Quoted 7 lines
Quoted 1 line
This doesn't make any sense to me.
What doesn't make sense to you? Do you mean that you don't understand how it
could be done with additional downloads, or do you mean that you don't understand how it could not be useful, or do you mean that you don't understand how it could be hard to maintain, or do you mean that you fail to parse the entire sentence?
I suppose you're saying you want to add more download statements in the package and then download all of the dependencies into the vendor/ tree using the Buildroot mechanism. This is not viable and I don't see why we would even consider it to begin with.
Quoted 1 line
Can you point out which bit causes the modules to be downloaded?
GOPROXY=direct go mod vendor, or you might try executing the build with the series applied, compiling any Go package, and you'll see the download happen during that step.
Quoted 4 lines
Quoted 2 lines
I don't see the point. If we discover a security issue, this provides a mechanism to override a dependency at download time.
... by keeping a copy of the entire go.mod and go.sum in Buildroot...
So what? Is it really that big of a cost?
Quoted 6 lines
"The vendoring model" means that every package bundles its dependencies.
Whether those dependencies are maintained in the same git tree, or as git submodules, or as go modules with an explicit version doesn't make much difference. For the package developers it does make a world of difference, of course. But for the distro maintainer it runs down to the same thing: updates of dependencies are handled by the package developer, not by the distro maintainer.
I suggest you research the original reason why vendor was added, GOPATH, the old "go get" mechanism, and the general history of package versioning in Go. I have not found any of these things to be the original reasoning behind these features in Go, nor have I found it referenced anywhere in their documentation. Vendor was added because the old Go "go get" tool did not offer an opportunity to override broken "master" branches of dependencies. When Go modules come around, it allows us to forget about vendoring, since the module files specify exact version hashes in the go.mod file, so you get an exact result.
I don't see any evidence that distro maintainers do not patch go.mod.
Quoted 10 lines
There are obviously advantages to the vendoring model. It relieves the package developers from dealing with changing APIs of their dependencies, it allows them to work around bugs in dependencies, and it makes sure that what they test is also what the final user is going to get. Also for library developers it makes things easier, because ABI compatibility is no longer a thing. And even for distro maintainers it makes things easier, because in essence the distro no longer needs to worry about dependencies. However, it has this one big disadvantage that a vulnerability can no longer be fixed by just updating a single package.
I agree that it is a good idea to test the final result when you change a dependency. But you can update a single package. You're misunderstanding the behavior of the Go tool. If you update a single line in go.mod, only a single package will be updated, since all of the dependencies, indirect or not, are copied into go.mod to begin with.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Well, actually we can... In any of the approaches we're considering, after the
extract step we will have all the dependencies present in the vendor/ tree. So we can always add a patch for a CVE fix in a dependent package in the usual way (we just have to add some directory components to the upstream patch).I don't see why this is better than overriding go.mod?
Because usually, changing the version brings in a whole lot more changes than
just the CVE fix.
I don't see why this is the case. Maybe in a versioning system like package.json, but please see above on why Go modules are different. Fully qualified version strings make this not the case.
Quoted 4 lines
"this would not be the case" refers to your statement that we would not be able
to fix a CVE (by adding a .patch). My statement is: we are able to fix CVEs like we do it for any other package, as long as go modules are instantiated/extracted before the patch step.
Yeah, this is true, we can fix things this way. But usually, patches in the buildroot tree are meant to be able to be applied with "git am" against the original code repository, correct? This way that would not be possible. Seems awkward.
Quoted 1 line
If we don't set GOPROXY=off, the Go tool might go off and download stuff.
We set GOPROXY=off even today in master. The Go tool will NOT go off and download stuff randomly :)
Quoted 11 lines
Quoted 10 lines
Why would this be the only case where adding a
patch would not work? With a new download method, it would go:1. Download base package (containing go.mod)
2. (optional) Copy in go.mod and go.sum from Buildroot tree
3. Execute "go mod vendor." Note, go.sum ensures the result is consistent.
4. Compress the result and store in dl/ tree.
5. Later, in the extract step, extract this tarball
6. Apply any patches (this can also target the vendor/ tree if needed)
7. Build the package with -mod=vendor arg which uses the vendor/ tree.
Quoted 2 lines
Exactly, so it is still possible to use the patch approach instead of changing
the version.
... which was the point of me listing the order of things happening. However, you can also see why it might be useful to have go.mod and go.sum in the tree, considering that you are not patching BEFORE running "go mod download" or "go mod vendor." You need an opportunity to override those files earlier on, especially if a dependency no longer exists, and you're looking to update the code URL for it.
Here's an example: sometimes, packages use "vanity urls" like mydomain.com/mynamespace/mycode (or similar). The Go tool does send HTTP requests to that domain, which usually will redirect the tool to a Github repo.
It would be impossible with patches + the proposed ordering above to fix the import URL if someone's domain were to go away randomly. The Buildroot proxy helps here, of course.
Quoted 10 lines
Quoted 4 lines
How would this break external packages? The
_GOMODvariable is automatically determined using the same logic that the GOPATH directory structure is determined today. This means that existing external packages using vendor/ would still build fine.An existing external package that doesn't use vendor/ but instead uses go.mod
(and thus downloads stuff in the build step, because the go tool will do that automatically) would break. I don't know if this is even possible at the moment, I'm just indicating a possible concern. Maybe I'm exaggerating and setting GOPROXY=off doesn't really carry any risks.
None use this. This is not a backwards compatibility concern yet. I understand your reasoning but GOPROXY=off in master so nobody should be using this yet.
Quoted 8 lines
Quoted 3 lines
The goal of copying go.mod in from the Buildroot tree is to allow you to override / provide go.mod if there is not one or if the existing one is outdated with security issues,
There are two concerns here, and I think we should consider them separately.
1. Make sure that building with -mod works.
a. Upstream has a go.mod -> nothing needs to be done.
This is the currently implemented logic in the PoC patches.
Quoted 2 lines
b. Upstream has no go.mod, but has vendor.conf or .toml -> nothing needs to be
done, the Go tool handles these as well.
I don't recommend this, since the conversion is not deterministic and does network calls, as I mentioned above.
Copying in the go.mod and go.sum might have spooked you because of the line count and the hashes in the file. But the reason for it is really really intentional. I don't want to introduce points of uncertainty into the build process.
Isn't this a bigger priority for us than a lower line count in the repo??
For this reason I regret mentioning vendor.conf conversion at all.
Quoted 2 lines
c. Upstream has no go.mod or vendor.conf or anything -> go.mod with just the
module line needs to be created.
Yup, currently implemented.
Quoted 2 lines
2. Modify dependencies by changing their versions. -> Buildroot-supplied go.mod is required.
Sort-of, unless we can make a way to patch it before we run the download process.
Quoted 6 lines
So, if my above analysis is correct, a Buildroot-supplied go.mod is only needed
for the 'changing versions' scenario (which I am not convinced is something we need). For the 'upstream has nothing' scenario, it is sufficient to generate the one-line go.mod.Is that correct?
Sure, and I'm happy to NOT allow package maintainers to override it by placing a go.mod and go.sum next to their .mk file. I don't know why limiting this is an advantage. You could just say that developers can place those files if they want in their external packages, and never do it in the Buildroot tree. But why close off the functionality? I'm going to have to keep a custom patch on top of Buildroot to add it back in again for my own projects and external packages otherwise.
Maybe you think you need to manually maintain the go.mod and go.sum files. This isn't the case. We would use the Go tool to convert the vendor.conf or whatever from upstream, check the result for consistency, then update the go.mod and go.sum in the Buildroot tree with the output from this. But, as far as I can see, none of the packages in the mainline Buildroot tree will need this (yet). But you may as well leave the 2 lines in the code to enable external packages to make this choice.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Can you explain why patch 2 is needed, converting runc's vendor.conf into a go.mod and go.sum?
Converting vendor.conf does network lookups with a non-deterministic result.
This is an exact example of what I'm describing in the previous paragraph.
Quoted 4 lines
Let me rephrase the question... As far as I understand, after the current patch
series, the Go tool will still not download anything for the existing packages, because the vendor tree is already there. Is that correct? If not, could you point out where the download happens?
"go mod vendor"
Although, if you look at the code, if the vendor tree is present, this is not called.
When compiling, go build -mod=vendor will tell the tool to use vendor/ and use the go.mod file only for the module path line (roughly equiv to just the 1-line go.mod file).
Quoted 2 lines
Ah, of course, because the go.mod is autogenerated, it will still work. But
then why are those changes in docker-containerd.mk needed? I still don't get it :-(
This series could be done without those, using vendor in the tree. It's an example.
Quoted 3 lines
I put this separate because AFAIU setting GOPROXY to off is not strictly needed. In the current situation, the Go tool is never going to download anything anyway, because the go.mod is empty.
It's strictly needed. The Go tool will analyze the code and determine the import paths of all the code files, and then generate / update go.mod. You might want to run the patch series and observe the behavior yourself.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
4. Run 'go mod vendor' in a post-extract hook. This should still be RFC because it violates the offline builds principle. Don't forget to mention some ideas of how the limitations could be overcome.
Not going to do this in a "real" solution.
The real solution being the go download method, right?
Most likely, but I also would just rule out doing any downloading outside of the download step. This makes sense, right?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
5. Add a package (nsq?) that would be very difficult to support without the go.mod support.
I don't see why this is required for this, but that is the goal (to allow external and internal packages that use go.mod and to transition off of GOPATH for vendor/ based packages.)
It's not *required*, it's just to show what is the use case for this work.
Hey, I'm all for adding new packages, required or not.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
6. Add support for Buildroot-supplied go.mod.
I agree this could be done after the initial minimal commit creating the single-line go.mod file, setting GOPROXY=off, and setting -mod=vendor.
For this one, it would be nice if there could be an alternative that doesn't
require copying the entire go.mod and go.sum, but rather override only one or two. Because I still think that that is going to be the only use case of this feature.
Maybe, but you're making a lot of absolute statements here without actually using the Go modules workflow yourself to know why those statements are or aren't true. Please have a look at my reasoning above and see what you think.
Best regards,
Christian
RFC PATCH 1/6 package/go: implement go modules integration
Quoted 3 lines · Christian Stewart
Hi Arnout,
Arnout Vandecappelle <[email protected]> writes:
[snip]
Quoted 9 lines
Quoted 3 lines
I don't understand why it can't be removed. The
APPLY_EXTRACT_GOMODdoesn't do
anything if a vendor tree is already present, and in all our current go packages the vendor tree is already present...The hook can probably be removed but we need to make the almost-empty go.mod somewhere.
Quoted 2 lines
Oh, hang on, you probably still have to create an almost-empty go.mod for the
ones which don't have a vendor.conf.
My bad, I forgot that that was also done in the post-extract hook.
Quoted 2 lines
I'm almost sorry I mentioned vendor.conf or gopkg.toml in the first place, becuase it seems like you're now distracted by the other formats.
:-)
Quoted 3 lines
It's not really a good idea to use the auto conversion at runtime since the result is not as deterministic as having a predetetermined go.mod, and the conversion requires network lookups.
AFAIU, packages which use vendor.conf or gopkg.toml will have their vendor
trees bundled, no? Otherwise other users wouldn't have a reliable way of downloading the package anyway, right?
Quoted 1 line
Pretend the other formats don't exist. They are there for legacy only anyway.
Okay. Anyway, the go.mod will be autogenerated since it will be missing, and
supposedly the vendor tree is there.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
If you can give an example of a package that only has a few of these, I could
create an example of how it might work without any extra infra. Although I'm not entirely sure if such an example would be very useful, except to show that it would be extremely hard to maintain.Quoted 7 lines
Quoted 1 line
This doesn't make any sense to me.
What doesn't make sense to you? Do you mean that you don't understand how it
could be done with additional downloads, or do you mean that you don't understand how it could not be useful, or do you mean that you don't understand how it could be hard to maintain, or do you mean that you fail to parse the entire sentence?
Quoted 3 lines
I suppose you're saying you want to add more download statements in the package and then download all of the dependencies into the vendor/ tree using the Buildroot mechanism.
Not really download statements; _EXTRA_DOWNLOADS, and a post-extract hook to
extract them. In terms of complexity, it's pretty similar as carrying the go.mod in Buildroot. But of course, this was would not be able to use an existing upstream go.mod.
Quoted 1 line
This is not viable
Why is it not viable?
Quoted 2 lines
and I don't see why we
would even consider it to begin with.
Because we have the option of either using the existing mechanisms, or to
extend the infra to cover go modules. To evaluate whether the extra infra is useful, it is interesting to look at what it would look at what would be needed without it.
Quoted 5 lines
Quoted 1 line
Can you point out which bit causes the modules to be downloaded?
GOPROXY=direct go mod vendor, or you might try executing the build with the series applied, compiling any Go package, and you'll see the download happen during that step.
I'll admit that I hadn't done that yet. So I applied only the first patch and
tried to build docker-cli. It doesn't do any downloading of dependencies because the vendor/ tree is already there (the patch has an explicit condition for that).
BTW, the build step then does automatic conversion of vendor.conf, but the
build fails with:
can't load package: package cmd/docker: cannot find package "." in:
/home/arnout/src/buildroot/output/host/lib/go/src/cmd/docker
I'm probably doing something wrong :-)
[snip]
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Why would this be the only case where adding a
patch would not work? With a new download method, it would go:1. Download base package (containing go.mod)
2. (optional) Copy in go.mod and go.sum from Buildroot tree
3. Execute "go mod vendor." Note, go.sum ensures the result is consistent.
4. Compress the result and store in dl/ tree.
I just noticed something missing: you'll need to add host-go to theDOWNLOAD_DEPENDENCIES of go packages if you want to use the go tool in the download step.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
5. Later, in the extract step, extract this tarball
6. Apply any patches (this can also target the vendor/ tree if needed)
7. Build the package with -mod=vendor arg which uses the vendor/ tree.Exactly, so it is still possible to use the patch approach instead of changing
the version.
Quoted 6 lines
... which was the point of me listing the order of things happening. However, you can also see why it might be useful to have go.mod and go.sum in the tree, considering that you are not patching BEFORE running "go mod download" or "go mod vendor." You need an opportunity to override those files earlier on, especially if a dependency no longer exists, and you're looking to update the code URL for it.
Agreed.
I'm not communicating my message very well apparently. I am not saying that a
Buildroot-supplied go.mod is not useful. I'm only saying that it is not absolutely needed, and that the current infra probably already covers a lot if not all of the use cases we have in reality. While explaining why I think that, I apparently didn't make it clear that I'm only trying to compare the different options we have, and weigh that against the priorities of the project.
[snip]
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
The goal of copying go.mod in from the Buildroot tree is to allow you to override / provide go.mod if there is not one or if the existing one is outdated with security issues,
Quoted 9 lines
Quoted 4 lines
There are two concerns here, and I think we should consider them separately.
1. Make sure that building with -mod works.
a. Upstream has a go.mod -> nothing needs to be done.This is the currently implemented logic in the PoC patches.
Quoted 2 lines
b. Upstream has no go.mod, but has vendor.conf or .toml -> nothing needs to be
done, the Go tool handles these as well.
I think my analysis was wrong. It should have been:
a. Upstream has a go.mod.
-> Dependencies have to be downloaded. Handled by the current patch, with the 3
limitations mentioned.
b. Upstream has a bundled vendor/ tree (as part of the tarball or as git submodules).
-> go.mod has to be created with just the module name. Handled by the current
patch, no limitations.
c. Upstream doesn't have a bundled vendor/ tree, only vendor.conf or gopkg.toml.
-> not reproducible because the vendor.conf and gopkg.toml don't encode the
version to download. (Note: I looked at the vendor.conf of docker-cli and it does have a version, so I'm not sure if this is true).
For the c case, I indeed see no other way than a Buildroot-provided go.mod that
I can think of.
Quoted 1 line
I don't recommend this, since the conversion is not deterministic and
Could you maybe explain why it is not deterministic?
Quoted 1 line
does network calls, as I mentioned above.
The network calls are the same as in case a I think, no?
Quoted 2 lines
Copying in the go.mod and go.sum might have spooked you because of the line count and the hashes in the file.
Yes indeed :-)
Note that I don't think we need go.sum. We do with the current patch, but if
the go download becomes a download method, we end up with a tarball that contains all dependencies, and the buildroot hashing is done on that tarball. So checksumming is done (though it's a little late, so finding out what went wrong in case of hash mismatch might be tricky).
[snip]
Quoted 9 lines
Quoted 4 lines
Let me rephrase the question... As far as I understand, after the current patch
series, the Go tool will still not download anything for the existing packages, because the vendor tree is already there. Is that correct? If not, could you point out where the download happens?"go mod vendor"
Although, if you look at the code, if the vendor tree is present, this is not called.
That was my point: all current packages have a vendor tree, so it never gets
called.
Quoted 10 lines
When compiling, go build -mod=vendor will tell the tool to use vendor/ and use the go.mod file only for the module path line (roughly equiv to just the 1-line go.mod file).
Quoted 2 lines
Ah, of course, because the go.mod is autogenerated, it will still work. But
then why are those changes in docker-containerd.mk needed? I still don't get it :-(This series could be done without those, using vendor in the tree. It's an example.
Quoted 7 lines
Quoted 3 lines
I put this separate because AFAIU setting GOPROXY to off is not strictly needed. In the current situation, the Go tool is never going to download anything anyway, because the go.mod is empty.
It's strictly needed. The Go tool will analyze the code and determine the import paths of all the code files, and then generate / update go.mod. You might want to run the patch series and observe the behavior yourself.
The behaviour I observe (with only patch 1 and removing GOPROXY=off) is that
there is a single-line go.mod (generated by Buildroot) and no go.sum, and no downloads are done. Which makes sense, because the vendor tree is already there.
Note that I'm not saying that GOPROXY should not be set to off. What I'm saying
is that that is pretty much independent of the rest of the go module support.
[snip]
Quoted 8 lines
Quoted 4 lines
For this one, it would be nice if there could be an alternative that doesn't
require copying the entire go.mod and go.sum, but rather override only one or two. Because I still think that that is going to be the only use case of this feature.Maybe, but you're making a lot of absolute statements here without actually using the Go modules workflow yourself to know why those statements are or aren't true.
I don't intend to make absolute statements, but I can hardly write AFAIU on
every line :-)
Regards,
Arnout
Quoted 6 lines
Please have a look at my reasoning above
and see what you think.Best regards,
Christian
RFC PATCH 1/6 package/go: implement go modules integration
Hi Arnout,
Quoted 7 lines · Arnout Vandecappelle
Quoted 3 lines
It's not really a good idea to use the auto conversion at runtime since the result is not as deterministic as having a predetetermined go.mod, and the conversion requires network lookups.
AFAIU, packages which use vendor.conf or gopkg.toml will have their vendor
trees bundled, no? Otherwise other users wouldn't have a reliable way of downloading the package anyway, right?
No. Not necessarily. Especially in the case of Gopkg.toml.
However, in any case where it is bundled, we can just use it - and this is the current behavior of this PoC series.
Quoted 4 lines
Quoted 1 line
Pretend the other formats don't exist. They are there for legacy only anyway.
Okay. Anyway, the go.mod will be autogenerated since it will be missing, and supposedly the vendor tree is there.
No, the tool will not know what the path to the root module is.
Quoted 8 lines
Quoted 3 lines
I suppose you're saying you want to add more download statements in the package and then download all of the dependencies into the vendor/ tree using the Buildroot mechanism.
Not really download statements;
_EXTRA_DOWNLOADS, and a post-extract hook to
extract them. In terms of complexity, it's pretty similar as carrying the go.mod in Buildroot. But of course, this was would not be able to use an existing upstream go.mod.
It's not similar at all to the complexity of carrying go.mod in Buildroot. The Go tool manages an immensely complex process of resolving the import paths to download URLs, fetching the sparse commits from the source repositories, hashing the code for consistency, and linking it all together so that the imports resolve correctly.
This is particularly complex with aliased imports in go.mod, which Kubernetes uses for example.
Carrying the go.mod and go.sum allows the Go tool to manage all of this for us while still keeping the code source hashes in a file similar to my-package.hash.
Quoted 1 line
Why is it not viable?
The maintenance process for carrying go.mod and go.sum is as follows:
1. Clone the project and checkout the correct revision in GOPATH.
2. Run "go mod init"
3. Copy go.mod and go.sum to Buildroot.
That's it.
The maintenence process for what you're describing is, I'd imagine:
1. Clone the project and checkout the correct revision
2. Read through the dependencies and determine the versions of all.
3. Write the download paths for the code into Buildroot BY HAND.
4. Download all of the sources and hash them BY HAND.
Nobody is going to do this.
Perhaps the misunderstanding here is in how the go.mod and go.sum files are managed. It's almost completely autonomous by the Go tool. We could even write a command in Buildroot that generates them for us. If you're imagining that I wrote all of these for docker-cli by hand, no wonder you're alarmed at the prospect of including them in the Buildroot tree.
Quoted 4 lines
Because we have the option of either using the existing mechanisms, or to
extend the infra to cover go modules. To evaluate whether the extra infra is useful, it is interesting to look at what it would look at what would be needed without it.
That's valid, and I believe it's an intractable approach, due to the immense development overhead as described above.
Quoted 8 lines
Quoted 3 lines
GOPROXY=direct go mod vendor, or you might try executing the build with the series applied, compiling any Go package, and you'll see the download happen during that step.
I'll admit that I hadn't done that yet. So I applied only the first patch and
tried to build docker-cli. It doesn't do any downloading of dependencies because the vendor/ tree is already there (the patch has an explicit condition for that).
Quoted 7 lines
BTW, the build step then does automatic conversion of vendor.conf, but the
build fails with:can't load package: package cmd/docker: cannot find package "." in:
/home/arnout/src/buildroot/output/host/lib/go/src/cmd/dockerI'm probably doing something wrong :-)
Docker CLI requires a patch because they use some non-standard logic for wiring together some adjusted vendored dependencies.
Quoted 3 lines
I just noticed something missing: you'll need to add host-go to the
DOWNLOAD_DEPENDENCIESof go packages if you want to use the go tool in the download step.
Noted, thanks.
Quoted 8 lines
Agreed.
I'm not communicating my message very well apparently. I am not saying that a
Buildroot-supplied go.mod is not useful. I'm only saying that it is not absolutely needed, and that the current infra probably already covers a lot if not all of the use cases we have in reality. While explaining why I think that, I apparently didn't make it clear that I'm only trying to compare the different options we have, and weigh that against the priorities of the project.
It is necessary in some cases when the package is using some "hack" scripts in their tree to wire together dependencies at build time in a non-standard way. The go.mod format allows us to codify these adjustments in a way that the Go tool understands natively.
See docker-cli. I included these patches not to alarm you guys with superfulous hash files, but to show examples of where these adjustments allow us to fix quirks and simplify package build processes as well as improve build consistency and reproducability.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
b. Upstream has no go.mod, but has vendor.conf or .toml -> nothing needs to be
done, the Go tool handles these as well.I think my analysis was wrong. It should have been:
a. Upstream has a go.mod.
-> Dependencies have to be downloaded. Handled by the current patch, with the 3
limitations mentioned.
... unless they include vendor/ in the tree with go.mod, which sometimes happens, and is indeed already handled by the PoC patch series (which should be able to handle ANY condition as written with the quirks I listed before).
Quoted 4 lines
b. Upstream has a bundled vendor/ tree (as part of the tarball or as git submodules).
-> go.mod has to be created with just the module name. Handled by the current
patch, no limitations.
Yes.
Quoted 9 lines
c. Upstream doesn't have a bundled vendor/ tree, only vendor.conf or gopkg.toml.
-> not reproducible because the vendor.conf and gopkg.toml don't encode the
version to download. (Note: I looked at the vendor.conf of docker-cli and it does have a version, so I'm not sure if this is true).For the c case, I indeed see no other way than a Buildroot-provided go.mod that
I can think of.Could you maybe explain why it is not deterministic?
They encode versions, but unfortunately they don't encode the transient dependency versions (in all cases).
Quoted 1 line
The network calls are the same as in case a I think, no?
No, go.mod and go.sum encode all the dependency versions, and also store hashes of the code as well, so it's guaranteed to produce an identical result.
Quoted 6 lines
Note that I don't think we need go.sum. We do with the current patch, but if the go download becomes a download method, we end up with a tarball that contains all dependencies, and the buildroot hashing is done on that tarball. So checksumming is done (though it's a little late, so finding out what went wrong in case of hash mismatch might be tricky).
Maybe so, but the Go tool would be quite happy to do the summing for us.
Please note that this file is NOT hand written.
Quoted 2 lines
That was my point: all current packages have a vendor tree, so it never gets called.
Because it is not currently possible to add a package without a vendor tree.
The fact that all packages work one way today doesn't factor into the optimal solution, at least in my mind, aside from backwards compatiblity, which we do have here.
Note Docker required specific touchups with the GOPATH approach before.
The reason why the build failed when you tried it with just the first patch, is because previously the docker-cli package overrided the WORKSPACE variables to set the Go package path. The Go package path is automatically inferred to be "github.com/docker/cli" instead of the correct "github.com/docker/docker" with just the first patch and not the docker-cli touchup patch.
Quoted 9 lines
Quoted 4 lines
[GOPROXY=off] is strictly needed. The Go tool will analyze the code and determine the import paths of all the code files, and then generate / update go.mod. You might want to run the patch series and observe the behavior yourself.
The behaviour I observe (with only patch 1 and removing GOPROXY=off) is that there is a single-line go.mod (generated by Buildroot) and no go.sum, and no downloads are done. Which makes sense, because the vendor tree is already there.
Yes, this is the correct behavior when vendor/ exists, and there is no go.mod and go.sum in the Buildroot tree to override it. If there is a go.mod and go.sum in the Buildroot tree, the PoC patch deletes vendor/ and replaces it via "go mod vendor."
Quoted 3 lines
Note that I'm not saying that GOPROXY should not be set to off. What
I'm saying is that that is pretty much independent of the rest of the go module support.
If you just want Go module support without go.mod and go.sum and any network fetching, then you have to change the following two environment variables:
- GO111MODULE: off -> on
- GOPROXY: [none] (defaults to "direct") -> "off"
When we turn GO111MODULE to on, we imply GOPROXY=direct. This will cause network calls during the "build" call.
Quoted 6 lines
Quoted 3 lines
Maybe, but you're making a lot of absolute statements here without actually using the Go modules workflow yourself to know why those statements are or aren't true.
I don't intend to make absolute statements, but I can hardly write AFAIU on
every line :-)
I've provided a lot of examples of where overriding go.mod is useful:
- When we add a download provider, we need an opportunity to adjust go.mod before the download step, which is far before the PATCH step.
- Older Buildroot versions will use older releases of packages. Developers do not typically update the vendor/ tree of old releases.
- Some packages use weird monkey-patching of GOPATH in their build scripts, which can be replicated with a single go.mod file instead.
I feel like it's not that big of a deal to copy go.mod and go.sum from the package directory in Buildroot if it exists, and then agree to not commit these files to the mainline Buildroot tree. The "copy them in" step would be used by external packages, then. Even then, it's not that big of a deal, I'm happy to remove the "copy these files in" line, so this feels a bit like a garden shed :)
Best,
Christian
RFC PATCH 1/6 package/go: implement go modules integration
In reply to Christian StewartHello Christian,
Quoted 3 lines · On Sat, 16 Mar 2019 18:21:37 -0700 Christian Stewart <[email protected]> wrote:
This commit moves from the GOPATH mechanism to the new GO111MODULE approach for Go based packages. Old Go packages (with the exception of docker-cli) will compile without changes (for example, mender, flanneld).
While browsing through the list of pending patches, I found the following patch series related to Go and package management:
http://patchwork.ozlabs.org/project/buildroot/list/?series=68102
The first patch in the series,
http://patchwork.ozlabs.org/patch/976322/, introduces:
+ Glide is a tool for managing the _vendor_ directory within a Go + package. This feature, first introduced in Go 1.5, allows each + package to have a _vendor_ directory containing dependent + packages for the project. These vendor packages can be + installed by a tool (e.g. glide), similar to go get or they + can be vendored and distributed with the package.
and it does a fair amount of changes in pkg-golang.mk.
How does this fit (or not) with the changes you're proposing ?
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/6 package/go: implement go modules integration
Hi Thomas,
Quoted 15 lines · Thomas Petazzoni
While browsing through the list of pending patches, I found the following patch series related to Go and package management:
http://patchwork.ozlabs.org/project/buildroot/list/?series=68102
The first patch in the series,
http://patchwork.ozlabs.org/patch/976322/, introduces:(quoted diff) +6
-0+ Glide is a tool for managing the _vendor_ directory within a Go + package. This feature, first introduced in Go 1.5, allows each + package to have a _vendor_ directory containing dependent + packages for the project. These vendor packages can be + installed by a tool (e.g. glide), similar to go get or they + can be vendored and distributed with the package.
Quoted 3 lines
and it does a fair amount of changes in pkg-golang.mk.
How does this fit (or not) with the changes you're proposing ?
Glide and other tools were built before Go modules were built. I do not recommend integrating them into Buildroot. They are on the way out.
Please see the Go module wiki:
Specifically:
go mod init automatically translates the required information from dep, glide, govendor, godep and 5 other pre-existing dependency managers into a go.mod file that produces the equivalent build.
The Go developers left it up to the community to develop a source package manager, which is why Glide, Dep (gopkg.toml), vendor.conf, and other formats exist. These formats are intended to be completely replaced by go.mod.
My suggestion is to instead solve the Go module stuff, and then bring in Glide-based packages with the glide.lock file converted into the Go module format via the above automatic process.
Best regards,
Christian
RFC PATCH 1/6 package/go: implement go modules integration
Hello Christian,
Thanks a lot for your feedback!
Quoted 3 lines · On Sun, 07 Apr 2019 19:04:00 -0700 Christian Stewart <[email protected]> wrote:
My suggestion is to instead solve the Go module stuff, and then bring in Glide-based packages with the glide.lock file converted into the Go module format via the above automatic process.
OK, thanks! What is the specific situation of the three Go packages that Fabio wanted to integrate, namely bitcoin, btcd and neutrino ? Will they be readily usable with the Go module integration ?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/6 package/go: implement go modules integration
Hi Thomas,
Quoted 3 lines · Thomas Petazzoni
OK, thanks! What is the specific situation of the three Go packages that Fabio wanted to integrate, namely bitcoin, btcd and neutrino ? Will they be readily usable with the Go module integration ?
Both of these projects already seem to be using go.mod:
- https://github.com/btcsuite/btcd
- https://github.com/lightninglabs/neutrino
... I can submit PoC patches implementing them trivially with the go module PoC patch in place.
Best regards,
Christian