Skip to contentSkip to conversation

package/go: set GOCACHE for download post-process

PATCH Applied Peter Korsgaard 3 participants 12 messages Jan 16, 2023 – Jan 17, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 package/go: set GOCACHE for download post-process

The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.

The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:

rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied make[1]: *** [package/pkg-generic.mk:189: /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1

We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.

---
 package/go/go.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/go/go.mk +1 -1
index d3ed07f03b..c4f72d403c 100644
@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \
 	GOFLAGS=-mod=vendor \
 	GOROOT="$(HOST_GO_ROOT)" \
 	GOPATH="$(HOST_GO_GOPATH)" \
+	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 	GOPROXY=off \
 	PATH=$(BR_PATH) \
 	GOBIN= \
@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \
 	$(HOST_GO_COMMON_ENV) \
 	GOOS="linux" \
 	GOARCH=$(GO_GOARCH) \
-	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 	CC="$(TARGET_CC)" \
 	CXX="$(TARGET_CXX)" \
 	CGO_CFLAGS="$(TARGET_CFLAGS)" \
Signature · 6 lines
2.30.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH 2/2 package/go: explicitly set GOMODCACHE

go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:

https://go.dev/ref/mod#module-cache

So explicitly set GOMODCACHE= for reproducibility.

---
 package/go/go.mk | 1 +
 1 file changed, 1 insertion(+)
package/go/go.mk +1 -0
index c4f72d403c..c38ae0b99c 100644
@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
 	GOROOT="$(HOST_GO_ROOT)" \
 	GOPATH="$(HOST_GO_GOPATH)" \
 	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
+	GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \
 	GOPROXY=off \
 	PATH=$(BR_PATH) \
 	GOBIN= \
Signature · 6 lines
2.30.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

Peter, All,

On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:

Quoted 9 lines

go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:

https://go.dev/ref/mod#module-cache

So explicitly set GOMODCACHE= for reproducibility.

Applied to master, thanks, but see below...

package/go/go.mk HOST_GO_COMMON_ENV = \ · line 29

Quoted 13 lines
---
 package/go/go.mk | 1 +
 1 file changed, 1 insertion(+)
package/go/go.mk +1 -0
index c4f72d403c..c38ae0b99c 100644
@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
 	GOROOT="$(HOST_GO_ROOT)" \
 	GOPATH="$(HOST_GO_GOPATH)" \
 	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
+	GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \

HOST_GO_GOPATH is $(HOST_DIR)/share/go-path, however, with PPD, HOST_DIR is per-package, so there will not besharing of the cache between two packages, unless there is a build dependency between them (unlikely in the go ecosystem).

If we wanted to really share the go cache(s), we need to hit somewhere in $(BR2_DL_DIR), like we do for cargo, see 8450b7691870 (package/pkg-cargo:
move CARGO_HOME into DL_DIR).

Still, this patch guarantees that the current situation is reproducible, and does not depend on the user's environment.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 10 lines
package/go/go.mk +0 -0
 	GOPROXY=off \
 	PATH=$(BR_PATH) \
 	GOBIN= \
Signature · 6 lines
2.30.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

Quoted 35 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Peter, All,
On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:

Quoted 9 lines

go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:

https://go.dev/ref/mod#module-cache

So explicitly set GOMODCACHE= for reproducibility.

Applied to master, thanks, but see below...

package/go/go.mk HOST_GO_COMMON_ENV = \ · line 29

Quoted 13 lines

---
package/go/go.mk | 1 +

1 file changed, 1 insertion(+)
package/go/go.mk +1 -0
index c4f72d403c..c38ae0b99c 100644
@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
 GOROOT="$(HOST_GO_ROOT)" \
 GOPATH="$(HOST_GO_GOPATH)" \
 GOCACHE="$(HOST_GO_TARGET_CACHE)" \
+	GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \

HOST_GO_GOPATH is $(HOST_DIR)/share/go-path, however, with PPD, HOST_DIR is per-package, so there will not besharing of the cache between two packages, unless there is a build dependency between them (unlikely in the go ecosystem).

Quoted 6 lines

If we wanted to really share the go cache(s), we need to hit somewhere in $(BR2_DL_DIR), like we do for cargo, see 8450b7691870 (package/pkg-cargo:
move CARGO_HOME into DL_DIR).

Still, this patch guarantees that the current situation is reproducible, and does not depend on the user's environment.

Yes, I know. As far as I can see this was already the case (on systems where GOMODCACHE isn't set), and I wanted a "safe" fix we could backport
- But indeed, longer term we should probably put the go module cache in DL_DIR like we recently did for the cargo cache.

Quoted 1 line

Applied to master, thanks.

Thanks!

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by Christian Stewart on v1 2/2
#4 lore raw

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

In reply to Peter Korsgaard

Hi Peter,

This reminds me of when GO111MODULE was leaking from the host environment as well. I wonder if there's any merit to wiping the host environment variables if br2 reproducible is enabled?

Quoted 26 lines · Peter Korsgaard

go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:

https://go.dev/ref/mod#module-cache

So explicitly set GOMODCACHE= for reproducibility.

---
 package/go/go.mk | 1 +
 1 file changed, 1 insertion(+)
package/go/go.mk +1 -1
index c4f72d403c..c38ae0b99c 100644
@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
        GOROOT="$(HOST_GO_ROOT)" \
        GOPATH="$(HOST_GO_GOPATH)" \
        GOCACHE="$(HOST_GO_TARGET_CACHE)" \
+       GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \
        GOPROXY=off \
        PATH=$(BR_PATH) \
        GOBIN= \
--
Quoted 2 lines

2.30.2

Quoted 2 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#5 lore raw

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Christian" == Christian Stewart <[email protected]> writes:

Hi Peter,
This reminds me of when GO111MODULE was leaking from the host environment as well. I wonder if there's any merit to wiping the host environment variables if br2 reproducible is enabled?

I definately think it is, ideally we want Buildroot to work the same way on all hosts - But why only for reproducible builds?

Notice that we do override GO111MODULE= in HOST_GO_COMMON_ENV.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#6 lore raw

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

Hi Peter,

Quoted 8 lines · Peter Korsgaard
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

"Christian" == Christian Stewart <[email protected]> writes:

Hi Peter,
This reminds me of when GO111MODULE was leaking from the host environment as well. I wonder if there's any merit to wiping the host environment variables if br2 reproducible is enabled?

Quoted 4 lines

I definately think it is, ideally we want Buildroot to work the same way on all hosts - But why only for reproducible builds?

Notice that we do override GO111MODULE= in HOST_GO_COMMON_ENV.

Apologies my wording was a bit ambiguous - we should always drop any Go related environment variables (you can get a full list with "go env") from the host environment.

I was additionally saying it might make sense to drop the entire host environment for all of buildroot, but maybe only if REPRODUCIBLE is enabled.

Best,
Christian
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#7 lore raw

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

In reply to Peter Korsgaard
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Peter" == Peter Korsgaard <[email protected]> writes:

go mod vendor caches downloaded modules to the Go module cache, which defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE environment variable:

https://go.dev/ref/mod#module-cache

So explicitly set GOMODCACHE= for reproducibility.

Committed to 2022.11.x and 2022.02.x, thanks.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#8 lore raw
In reply to Peter Korsgaard

Peter, All,

On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:

Quoted 14 lines

The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.

The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:

rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied make[1]: *** [package/pkg-generic.mk:189: /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1

Quoted 5 lines

We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 31 lines
---
 package/go/go.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/go/go.mk +1 -1
index d3ed07f03b..c4f72d403c 100644
@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \
 	GOFLAGS=-mod=vendor \
 	GOROOT="$(HOST_GO_ROOT)" \
 	GOPATH="$(HOST_GO_GOPATH)" \
+	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 	GOPROXY=off \
 	PATH=$(BR_PATH) \
 	GOBIN= \
@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \
 	$(HOST_GO_COMMON_ENV) \
 	GOOS="linux" \
 	GOARCH=$(GO_GOARCH) \
-	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 	CC="$(TARGET_CC)" \
 	CXX="$(TARGET_CXX)" \
 	CGO_CFLAGS="$(TARGET_CFLAGS)" \
Signature · 6 lines
2.30.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by Christian Stewart on v1 1/2
#9 lore raw
In reply to Peter Korsgaard

Hi Peter,

Thought this was set already!

Quoted 47 lines · Peter Korsgaard

The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.

The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:

rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied
make[1]: *** [package/pkg-generic.mk:189:
/home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1

We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.

---
 package/go/go.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/go/go.mk +1 -2
index d3ed07f03b..c4f72d403c 100644
@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \
        GOFLAGS=-mod=vendor \
        GOROOT="$(HOST_GO_ROOT)" \
        GOPATH="$(HOST_GO_GOPATH)" \
+       GOCACHE="$(HOST_GO_TARGET_CACHE)" \
        GOPROXY=off \
        PATH=$(BR_PATH) \
        GOBIN= \
@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \
        $(HOST_GO_COMMON_ENV) \
        GOOS="linux" \
        GOARCH=$(GO_GOARCH) \
-       GOCACHE="$(HOST_GO_TARGET_CACHE)" \
        CC="$(TARGET_CC)" \
        CXX="$(TARGET_CXX)" \
        CGO_CFLAGS="$(TARGET_CFLAGS)" \
--
Quoted 2 lines

2.30.2

Quoted 2 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#10 lore raw
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Christian" == Christian Stewart <[email protected]> writes:

Hi Peter,
Thought this was set already!

It is for the build step - but not for the download step, so we can end up in problems with the go mod vendor call in the download post-process.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#11 lore raw
In reply to Peter Korsgaard
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Peter" == Peter Korsgaard <[email protected]> writes:

The go mod vendor call in support/download/go-post-process accesses the go cache, so pass GOCACHE= in the environment to ensure our cache directory is used.

The go cache defaults to ~/.cache/go-build if not set, so this fixes builds where that location (or GOCACHE if set in the environment) is not writable:

rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied make[1]: *** [package/pkg-generic.mk:189:
/home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded]
Error 1

Quoted 5 lines

We use two different cache directories for target and host builds, but the download/vendoring should be independent of the architecture, so use the target variant even for host-only packages for simplicity.

Committed to 2022.11.x and 2022.02.x, thanks.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot