go.bbclass: change GOTMPDIR to improve reproducibility
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 41 lines · Changqing Li via lists.openembedded.org
From: Changqing Li<[email protected]>
The Go toolchain writes temporary source files under GOTMPDIR and compiles them there. To support reproducibility, Go passes options such as -ffile-prefix-map=$WORK/b387=/tmp/go-build to the GCC instance it invokes. The variable WORK is a temporary directory created under GOTMPDIR, refer this example:
WORK=/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build-tmp/go-build377321751
cd $WORK/b387
TERM='dumb' x86_64-wrs-linux-gcc -m64 -march=x86-64-v3 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot -I/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah/vendor/github.com/proglottis/gpgme-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=$WORK/b387=/tmp/go-build -gno-record-gcc-switches -v -D_FILE_OFFSET_BITS=64 -I $WORK/b387/ -O2 -g -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot= -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native= -pipe -v -ffile-prefix-map=/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah/vendor=/_/vendor -frandom-seed=TZkSPVSBUvDMjg4wKjWS -o$WORK/b387/_x004.o-cunset_agent_info.cgo2.cOE also passes its own
DEBUG_PREFIX_MAPtoGCC(finally by CGO_CFLAGS), including -ffile-prefix-map=${B}=${TARGET_DBGSRC_DIR}, where B is ${WORKDIR}/build. Because GOTMPDIR defaults to ${WORKDIR}/build-tmp, the Go temporary directory looks like ${WORKDIR}/build-tmp/go-buildXYZ. Its prefix therefore begins with ${WORKDIR}/build, so GCC matches theDEBUG_PREFIX_MAPentry for ${B} first.As a result, a path such as ${WORKDIR}/build-tmp/go-buildXYZ is rewritten to ${TARGET_DBGSRC_DIR}-tmp/go-buildXYZ. This breaks the -ffile-prefix-map option that Go itself adds, because the original WORK path no longer matches the value Go expects. Since Go creates go-buildXYZ directories randomly and internally, this causes the build non-reproducible.
This patch changes GOTMPDIR from ${WORKDIR}/build-tmp to ${WORKDIR}/go-build-tmp so that the path no longer matches ${B}. This prevents unintended replacements by OE's
DEBUG_PREFIX_MAPand restores reproducibility.Signed-off-by Changqing Li<[email protected]>--- meta/classes-recipe/go.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes-recipe/go.bbclass +0
-0
meta/classes-recipe/go.bbclass B = "${WORKDIR}/build" · line 82
Quoted 5 lines
meta/classes-recipe/go.bbclass +1
-1@@ -79,7 +79,7 @@ B = "${WORKDIR}/build" export GOPATH = "${B}" export GOENV = "off" export GOPROXY ??="https://proxy.golang.org,direct" -export GOTMPDIR ?= "${WORKDIR}/build-tmp" +export GOTMPDIR ?= "${WORKDIR}/go-build-tmp"
Hi, Alex
In the following patch, you said "Additionally, the top level build directory cannot be named 'go-something'."
https://git.openembedded.org/openembedded-core/commit/?id=9985b17a30bb9b9f1bc82a44662687db5cead66e
Can you help to explain why it cannot be named 'go-something'? "go-build-tmp" works well in my env. if it cannot be named
as "go-something", how about "gobuild-tmp"?
Thanks
Changqing
Quoted 13 lines
meta/classes-recipe/go.bbclass +0
-0GOTMPDIR[vardepvalue] = "" GO_SRCURI_DESTSUFFIX ="${@os.path.join(os.path.basename(d.getVar('S')), 'src',
d.getVar('GO_IMPORT')) + '/'}"-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#227380):https://lists.openembedded.org/g/openembedded-core/message/227380 Mute This Topic:https://lists.openembedded.org/mt/116673066/3616873
Group Owner:[email protected] Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 7 lines · Changqing Li via lists.openembedded.org
In the following patch, you said "Additionally, the top level build directory cannot be named 'go-something'."
https://git.openembedded.org/openembedded-core/commit/?id=9985b17a30bb9b9f1bc82a44662687db5cead66e
Can you help to explain why it cannot be named 'go-something'? "go-build-tmp" works well in my env. if it cannot be named
as "go-something", how about "gobuild-tmp"?
This was 4 years ago and honestly I don't remember anything about the patch, other than it was a pain to find the particular settings that work. You can probably set up a yocto build from the revision just before the patch and apply parts of it to see why 'go-something' didn't work at that point.
We should be careful with further tweaks.
Alex
Quoted 7 lines · Alexander Kanavin
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 2 lines · Changqing Li via lists.openembedded.org
In the following patch, you said "Additionally, the top level build directory cannot be named 'go-something'." https://git.openembedded.org/openembedded-core/commit/?id=9985b17a30bb9b9f1bc82a44662687db5cead66e Can you help to explain why it cannot be named 'go-something'? "go-build-tmp" works well in my env. if it cannot be named
Quoted 8 lines
Quoted 1 line
as "go-something", how about "gobuild-tmp"?
This was 4 years ago and honestly I don't remember anything about the patch, other than it was a pain to find the particular settings that work. You can probably set up a yocto build from the revision just before the patch and apply parts of it to see why 'go-something' didn't work at that point.
We should be careful with further tweaks.
Understand, thanks. I tried with kirkstone branch, and set GOTMPDIR to go-tmp, then bitbake buildah, build successfully.
But maybe I should set it to "tmp-go-build" to avoid possible regression.
//Changqing
Quoted 2 lines
Alex