go.bbclass: change GOTMPDIR to improve reproducibility
PATCH 1/2 go.bbclass: change GOTMPDIR to improve reproducibility
From: Changqing Li <[email protected]>
The Go toolchain writes temporary source files under GOTMPDIR and compiles them there. To support reproducibility, Go passes options such as -ffile-prefix-map=$WORK/b387=/tmp/go-build to the GCC instance it invokes. The variable WORK is a temporary directory created under GOTMPDIR, refer this example:
WORK=/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build-tmp/go-build377321751
cd $WORK/b387
TERM='dumb' x86_64-wrs-linux-gcc -m64 -march=x86-64-v3 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot -I /tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah/vendor/github.com/proglottis/gpgme -fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=$WORK/b387=/tmp/go-build -gno-record-gcc-switches -v -D_FILE_OFFSET_BITS=64 -I $WORK/b387/ -O2 -g -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot= -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native= -pipe -v -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah/vendor=/_/vendor -frandom-seed=TZkSPVSBUvDMjg4wKjWS -o $WORK/b387/_x004.o -c unset_agent_info.cgo2.c
OE also passes its own DEBUG_PREFIX_MAP to GCC(finally by CGO_CFLAGS), including -ffile-prefix-map=${B}=${TARGET_DBGSRC_DIR}, where B is ${WORKDIR}/build. Because GOTMPDIR defaults to ${WORKDIR}/build-tmp, the Go temporary directory looks like ${WORKDIR}/build-tmp/go-buildXYZ. Its prefix therefore begins with ${WORKDIR}/build, so GCC matches the DEBUG_PREFIX_MAP entry for ${B} first.
As a result, a path such as ${WORKDIR}/build-tmp/go-buildXYZ is rewritten to ${TARGET_DBGSRC_DIR}-tmp/go-buildXYZ. This breaks the -ffile-prefix-map option that Go itself adds, because the original WORK path no longer matches the value Go expects. Since Go creates go-buildXYZ directories randomly and internally, this causes the build non-reproducible.
This patch changes GOTMPDIR from ${WORKDIR}/build-tmp to ${WORKDIR}/go-build-tmp so that the path no longer matches ${B}. This prevents unintended replacements by OE's DEBUG_PREFIX_MAP and restores reproducibility.
--- meta/classes-recipe/go.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/go.bbclass +1 -1
@@ -79,7 +79,7 @@ B = "${WORKDIR}/build" export GOPATH = "${B}" export GOENV = "off" export GOPROXY ??= "https://proxy.golang.org,direct" -export GOTMPDIR ?= "${WORKDIR}/build-tmp" +export GOTMPDIR ?= "${WORKDIR}/go-build-tmp" GOTMPDIR[vardepvalue] = "" GO_SRCURI_DESTSUFFIX = "${@os.path.join(os.path.basename(d.getVar('S')), 'src', d.getVar('GO_IMPORT')) + '/'}"
Signature · 1 line
2.34.1
PATCH 2/2 go.bbclass: set buildid to empty string to improve reproducibility
From: Changqing Li <[email protected]>
Seems there are other things making Go BuildID not reproducible, set buildid to empty string to improve reproducibility, open a new issue [1], and refer an old issue [2].
Refer:
[1] https://github.com/golang/go/issues/76743
[2] https://github.com/golang/go/issues/34186
--- meta/classes-recipe/go.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/go.bbclass +1 -1
@@ -50,7 +50,7 @@ GO_RPATH:class-native = "${@'-r ${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE GO_RPATH_LINK:class-native = "${@'-Wl,-rpath-link=${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE}_dynlink' if d.getVar('GO_DYNLINK') else ''}" GO_EXTLDFLAGS ?= "${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} ${GO_RPATH_LINK} ${LDFLAGS}" GO_LINKMODE ?= "" -GO_EXTRA_LDFLAGS ?= "" +GO_EXTRA_LDFLAGS ?= "-buildid=" GO_LINUXLOADER ?= "-I ${@get_linuxloader(d)}" # Use system loader. If uninative is used, the uninative loader will be patched automatically GO_LINUXLOADER:class-native = ""
Signature · 1 line
2.34.1
RE: [OE-core] [PATCH 2/2] go.bbclass: set buildid to empty string to improve reproducibility
Quoted 30 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Changqing Li via lists.openembedded.org
Sent: den 8 december 2025 10:12
To: [email protected]
Subject: [OE-core] [PATCH 2/2] go.bbclass: set buildid to empty string to improve reproducibilityFrom: Changqing Li <[email protected]>
Seems there are other things making Go BuildID not reproducible, set buildid to empty string to improve reproducibility, open a new issue [1], and refer an old issue [2].
Refer:
[1] https://github.com/golang/go/issues/76743
[2] https://github.com/golang/go/issues/34186Signed-off-by Changqing Li <[email protected]>--- meta/classes-recipe/go.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes-recipe/go.bbclass +0
-0@@ -50,7 +50,7 @@ GO_RPATH:class-native = "${@'-r${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE
GO_RPATH_LINK:class-native = "${@'-Wl,-rpath-link=${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE}_dynlink' ifd.getVar('GO_DYNLINK')else ''}"GO_EXTLDFLAGS?= "${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} ${GO_RPATH_LINK} ${LDFLAGS}"
GO_LINKMODE?= ""
meta/classes-recipe/go.bbclass GO_RPATH:class-native = "${@'-r ${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE · line 53
Quoted 2 lines
meta/classes-recipe/go.bbclass +1
-1-GO_EXTRA_LDFLAGS ?= "" +GO_EXTRA_LDFLAGS ?= "-buildid="
Adding it to GO_EXTRA_LDFLAGS will probably not have the intended effect as some recipes already set that variable and thus will not receive this option. You should probably add a new variable for it instead, like the other options that are set in GO_LDFLAGS.
Quoted 5 lines
GO_LINUXLOADER?= "-I ${@get_linuxloader(d)}"
# Use system loader. If uninative is used, the uninative loader will be patched automatically GO_LINUXLOADER:class-native = ""
--
2.34.1
//Peter
Re: [OE-core] [PATCH 2/2] go.bbclass: set buildid to empty string to improve reproducibility
Quoted 24 lines · Peter Kjellerstedt
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 21 lines
-----Original Message-----
From:[email protected] <[email protected]> On Behalf Of Changqing Li via lists.openembedded.org
Sent: den 8 december 2025 10:12
To:[email protected]
Subject: [OE-core] [PATCH 2/2] go.bbclass: set buildid to empty string to improve reproducibilityFrom: Changqing Li<[email protected]>
Seems there are other things making Go BuildID not reproducible, set buildid to empty string to improve reproducibility, open a new issue [1], and refer an old issue [2].
Refer:
[1]https://github.com/golang/go/issues/76743
[2]https://github.com/golang/go/issues/34186Signed-off-by Changqing Li<[email protected]>--- meta/classes-recipe/go.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
meta/classes-recipe/go.bbclass GO_RPATH:class-native = "${@'-r ${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE · line 53
Quoted 12 lines
meta/classes-recipe/go.bbclass +0
-0@@ -50,7 +50,7 @@ GO_RPATH:class-native = "${@'-r${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE
GO_RPATH_LINK:class-native ="${@'-Wl,-rpath-link=${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE}_dynlink'
ifd.getVar('GO_DYNLINK')else ''}"
GO_EXTLDFLAGS?= "${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} ${GO_RPATH_LINK} ${LDFLAGS}"GO_LINKMODE?= ""meta/classes-recipe/go.bbclass +1
-1-GO_EXTRA_LDFLAGS ?= "" +GO_EXTRA_LDFLAGS ?= "-buildid="Adding it to
GO_EXTRA_LDFLAGSwill probably not have the intended effect as some recipes already set that variable and thus will not receive this option. You should probably add a new variable for it instead, like the other options that are set inGO_LDFLAGS.
Good suggestion. Maybe I can add a varibale GO_BUILDID ?= "-buildid="
//Changqing
Quoted 7 lines
Quoted 5 lines
GO_LINUXLOADER?= "-I ${@get_linuxloader(d)}"
# Use system loader. If uninative is used, the uninative loader will be patched automatically GO_LINUXLOADER:class-native = ""
--
2.34.1//Peter
Quoted 4 lines · Changqing Li via lists.openembedded.org
This patch changes GOTMPDIR from ${WORKDIR}/build-tmp to ${WORKDIR}/go-build-tmp so that the path no longer matches ${B}. This prevents unintended replacements by OE's
DEBUG_PREFIX_MAPand restores reproducibility.
Wait. Go does not have reproducibility problems right now: it is a part of reproducibility test, and not in the exclusion list. So how can the issues that this patch set aims to fix be observed?
Alex
Quoted 6 lines · Alexander Kanavin
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.On Mon, 8 Dec 2025 at 10:11, Changqing Li via lists.openembedded.org <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
This patch changes GOTMPDIR from ${WORKDIR}/build-tmp to ${WORKDIR}/go-build-tmp so that the path no longer matches ${B}. This prevents unintended replacements by OE's
DEBUG_PREFIX_MAPand restores reproducibility.Wait. Go does not have reproducibility problems right now: it is a part of reproducibility test, and not in the exclusion list. So how can the issues that this patch set aims to fix be observed?
I checked autobuilder configs, although we have reproducible test, but only test with openembeded-core
and layers under meta-openembeded, and there is no recipe inherit go.bbclass(except etcd under meta-oe, but already skipped)
So actually, we don't have reproducible test for recipes that compile with go.
And I tested with this step:
on a host, with build path ra, and another build path rb-extend, bitbake buildah under both project,
the rpm of ra, and rb-extend is different, and with these 2 patches, the rpm will be the same.
//Changqing
Quoted 2 lines
Alex
Quoted 9 lines · Changqing Li
Wait. Go does not have reproducibility problems right now: it is a part of reproducibility test, and not in the exclusion list. So how can the issues that this patch set aims to fix be observed?
I checked autobuilder configs, although we have reproducible test, but only test with openembeded-core
and layers under meta-openembeded, and there is no recipe inherit
go.bbclass(except etcd under meta-oe, but already skipped)So actually, we don't have reproducible test for recipes that compile with go.
I see. I was confused and thought this fixes the go toolchain somehow.
There is one recipe on oe-core that does inherit go class (go-helloworld), can you check why reproducibility issues don't happen there? It would be good to have a reproducibility test for things written in go in core, so this doesn't again regress quietly.
Alex
Quoted 12 lines · Alexander Kanavin
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 8 lines · Changqing Li
Wait. Go does not have reproducibility problems right now: it is a part of reproducibility test, and not in the exclusion list. So how can the issues that this patch set aims to fix be observed?
I checked autobuilder configs, although we have reproducible test, but only test with openembeded-core
and layers under meta-openembeded, and there is no recipe inherit
go.bbclass(except etcd under meta-oe, but already skipped)
Quoted 7 lines
Quoted 1 line
So actually, we don't have reproducible test for recipes that compile with go.
I see. I was confused and thought this fixes the go toolchain somehow.
There is one recipe on oe-core that does inherit go class (go-helloworld), can you check why reproducibility issues don't happen there? It would be good to have a reproducibility test for things written in go in core, so this doesn't again regress quietly.
This issue reproduced when cgo is enabled, and go internally add --ffile-prefix-map="WorkDir"=/tmp/go-build" to gcc
to tell gcc not include workdir in object file. for go-helloworld, it should be pure go, does not include the "C", so it will not met this issue.
//Changqing
Quoted 1 line
Alex
Quoted 3 lines · Changqing Li
This issue reproduced when cgo is enabled, and go internally add --ffile-prefix-map="WorkDir"=/tmp/go-build" to gcc
to tell gcc not include workdir in object file. for go-helloworld, it should be pure go, does not include the "C", so it will not met this issue.
Alright. The patches are ok then, please summarize these points in v2 commit messages.
Alex
Quoted 8 lines · Changqing Li
There is one recipe on oe-core that does inherit go class (go-helloworld), can you check why reproducibility issues don't happen there? It would be good to have a reproducibility test for things written in go in core, so this doesn't again regress quietly.
This issue reproduced when cgo is enabled, and go internally add --ffile-prefix-map="WorkDir"=/tmp/go-build" to gcc
to tell gcc not include workdir in object file. for go-helloworld, it should be pure go, does not include the "C", so it will not met this issue.
I have filed a bugzilla ticket for the missong cgo reproducibility
testing: https://bugzilla.yoctoproject.org/show_bug.cgi?id=16100
If you have ideas how to address it (with something minimal) in oe-core, that would be welcome.
Alex