package/go: set GOCACHE for download post-process
PATCH 1/2 package/go: set GOCACHE for download post-process
The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.
The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:
rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied make[1]: *** [package/pkg-generic.mk:189: /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1
We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.
--- package/go/go.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/go/go.mk +1 -1
@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \ GOFLAGS=-mod=vendor \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ + GOCACHE="$(HOST_GO_TARGET_CACHE)" \ GOPROXY=off \ PATH=$(BR_PATH) \ GOBIN= \
@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \ $(HOST_GO_COMMON_ENV) \ GOOS="linux" \ GOARCH=$(GO_GOARCH) \ - GOCACHE="$(HOST_GO_TARGET_CACHE)" \ CC="$(TARGET_CC)" \ CXX="$(TARGET_CXX)" \ CGO_CFLAGS="$(TARGET_CFLAGS)" \
Signature · 6 lines
2.30.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/2 package/go: explicitly set GOMODCACHE
go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:
https://go.dev/ref/mod#module-cache
So explicitly set GOMODCACHE= for reproducibility.
--- package/go/go.mk | 1 + 1 file changed, 1 insertion(+)
package/go/go.mk +1 -0
@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ GOCACHE="$(HOST_GO_TARGET_CACHE)" \ + GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \ GOPROXY=off \ PATH=$(BR_PATH) \ GOBIN= \
Signature · 6 lines
2.30.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN
AppliedRe: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE
Peter, All,
On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:
Quoted 9 lines
go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:
https://go.dev/ref/mod#module-cache
So explicitly set GOMODCACHE= for reproducibility.
Signed-off-by Peter Korsgaard <[email protected]>
Applied to master, thanks, but see below...
package/go/go.mk HOST_GO_COMMON_ENV = \ · line 29
Quoted 13 lines
--- package/go/go.mk | 1 + 1 file changed, 1 insertion(+)package/go/go.mk +1
-0@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ GOCACHE="$(HOST_GO_TARGET_CACHE)" \ + GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \
HOST_GO_GOPATH is $(HOST_DIR)/share/go-path, however, with PPD, HOST_DIR is per-package, so there will not besharing of the cache between two packages, unless there is a build dependency between them (unlikely in the go ecosystem).
If we wanted to really share the go cache(s), we need to hit somewhere in $(BR2_DL_DIR), like we do for cargo, see 8450b7691870 (package/pkg-cargo:
move CARGO_HOME into DL_DIR).
Still, this patch guarantees that the current situation is reproducible, and does not depend on the user's environment.
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 10 lines
package/go/go.mk +0
-0GOPROXY=off \ PATH=$(BR_PATH) \ GOBIN= \Signature · 6 lines
2.30.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE
Quoted 35 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:Quoted 9 lines
go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:
https://go.dev/ref/mod#module-cache
So explicitly set GOMODCACHE= for reproducibility.
Signed-off-by Peter Korsgaard <[email protected]>Applied to master, thanks, but see below...
package/go/go.mk HOST_GO_COMMON_ENV = \ · line 29
Quoted 13 lines
---
package/go/go.mk| 1 +1 file changed, 1 insertion(+)package/go/go.mk +1
-0@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ GOCACHE="$(HOST_GO_TARGET_CACHE)" \ + GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \
HOST_GO_GOPATHis $(HOST_DIR)/share/go-path, however, with PPD,HOST_DIRis per-package, so there will not besharing of the cache between two packages, unless there is a build dependency between them (unlikely in the go ecosystem).
Quoted 6 lines
If we wanted to really share the go cache(s), we need to hit somewhere in $(BR2_DL_DIR), like we do for cargo, see
8450b7691870(package/pkg-cargo:
moveCARGO_HOMEintoDL_DIR).Still, this patch guarantees that the current situation is reproducible, and does not depend on the user's environment.
Yes, I know. As far as I can see this was already the case (on systems where GOMODCACHE isn't set), and I wanted a "safe" fix we could backport
- But indeed, longer term we should probably put the go module cache in DL_DIR like we recently did for the cargo cache.
Quoted 1 line
Applied to master, thanks.
Thanks!
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE
In reply to Peter KorsgaardHi Peter,
This reminds me of when GO111MODULE was leaking from the host environment as well. I wonder if there's any merit to wiping the host environment variables if br2 reproducible is enabled?
Quoted 26 lines · Peter Korsgaard
go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:
https://go.dev/ref/mod#module-cache
So explicitly set GOMODCACHE= for reproducibility.
Signed-off-by Peter Korsgaard <[email protected]>--- package/go/go.mk | 1 + 1 file changed, 1 insertion(+)package/go/go.mk +1
-1@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ GOCACHE="$(HOST_GO_TARGET_CACHE)" \ + GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \ GOPROXY=off \ PATH=$(BR_PATH) \ GOBIN= \ --
Quoted 2 lines
2.30.2
Quoted 2 lines
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Christian" == Christian Stewart <[email protected]> writes:
Hi Peter,
This reminds me of when GO111MODULE was leaking from the host environment as well. I wonder if there's any merit to wiping the host environment variables if br2 reproducible is enabled?
I definately think it is, ideally we want Buildroot to work the same way on all hosts - But why only for reproducible builds?
Notice that we do override GO111MODULE= in HOST_GO_COMMON_ENV.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE
Hi Peter,
Quoted 8 lines · Peter Korsgaard
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
"Christian" == Christian Stewart <[email protected]> writes:
Hi Peter,
This reminds me of when GO111MODULE was leaking from the host environment as well. I wonder if there's any merit to wiping the host environment variables if br2 reproducible is enabled?
Quoted 4 lines
I definately think it is, ideally we want Buildroot to work the same way on all hosts - But why only for reproducible builds?
Notice that we do override GO111MODULE= in
HOST_GO_COMMON_ENV.
Apologies my wording was a bit ambiguous - we should always drop any Go related environment variables (you can get a full list with "go env") from the host environment.
I was additionally saying it might make sense to drop the entire host environment for all of buildroot, but maybe only if REPRODUCIBLE is enabled.
Best,
Christian
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE
In reply to Peter KorsgaardQuoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Peter" == Peter Korsgaard <[email protected]> writes:
go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:
https://go.dev/ref/mod#module-cache
So explicitly set GOMODCACHE= for reproducibility.
Signed-off-by Peter Korsgaard <[email protected]>
Committed to 2022.11.x and 2022.02.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN
AppliedPeter, All,
On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:
Quoted 14 lines
The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.
The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:
rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at/home/peko/.cache/go-build: mkdir/home/peko/.cache/go-build: permission denied make[1]: *** [package/pkg-generic.mk:189:/home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1
Quoted 5 lines
We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.
Signed-off-by Peter Korsgaard <[email protected]>
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 31 lines
--- package/go/go.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/go/go.mk +1
-1@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \ GOFLAGS=-mod=vendor \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ + GOCACHE="$(HOST_GO_TARGET_CACHE)" \ GOPROXY=off \ PATH=$(BR_PATH) \ GOBIN= \@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \ $(HOST_GO_COMMON_ENV) \ GOOS="linux" \ GOARCH=$(GO_GOARCH) \ - GOCACHE="$(HOST_GO_TARGET_CACHE)" \ CC="$(TARGET_CC)" \ CXX="$(TARGET_CXX)" \ CGO_CFLAGS="$(TARGET_CFLAGS)" \Signature · 6 lines
2.30.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Peter,
Thought this was set already!
Quoted 47 lines · Peter Korsgaard
The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.
The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:
rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at/home/peko/.cache/go-build: mkdir/home/peko/.cache/go-build: permission denied
make[1]: *** [package/pkg-generic.mk:189:/home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.
Signed-off-by Peter Korsgaard <[email protected]>--- package/go/go.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/go/go.mk +1
-2@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \ GOFLAGS=-mod=vendor \ GOROOT="$(HOST_GO_ROOT)" \ GOPATH="$(HOST_GO_GOPATH)" \ + GOCACHE="$(HOST_GO_TARGET_CACHE)" \ GOPROXY=off \ PATH=$(BR_PATH) \ GOBIN= \@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \ $(HOST_GO_COMMON_ENV) \ GOOS="linux" \ GOARCH=$(GO_GOARCH) \ - GOCACHE="$(HOST_GO_TARGET_CACHE)" \ CC="$(TARGET_CC)" \ CXX="$(TARGET_CXX)" \ CGO_CFLAGS="$(TARGET_CFLAGS)" \ --
Quoted 2 lines
2.30.2
Quoted 2 lines
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Christian" == Christian Stewart <[email protected]> writes:
Hi Peter,
Thought this was set already!
It is for the build step - but not for the download step, so we can end up in problems with the go mod vendor call in the download post-process.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Peter" == Peter Korsgaard <[email protected]> writes:
The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.
The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:
rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at/home/peko/.cache/go-build: mkdir/home/peko/.cache/go-build: permission denied make[1]: *** [package/pkg-generic.mk:189:/home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded]
Error 1
Quoted 5 lines
We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.
Signed-off-by Peter Korsgaard <[email protected]>
Committed to 2022.11.x and 2022.02.x, thanks.
Signature · 5 lines
Bye, Peter Korsgaard _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot