go.bbclass: install dependencies under $GOPATH instead of $GOROOT

To package a go application in Yocto, one needs to also package its dependencies to keep the build reproducible and under control. The default install task of the go class puts the source in ${libdir}/go/src.

${libdir}/go/src is where the standard go packages resides, aka $GOROOT. When a go package has dependencies on other go packages, they must be located under $GOPATH, otherwise errors like this one may happen:

Non-standard package <packagename> in standard package <packagename>

This point of this patch is to trigger a discussion on how this issue can be tackled in Yocto. Following code is working but should not be considered for inclusion.

Signed-off-by Damien Riegel <[email protected]>

---
 meta/classes/go.bbclass | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

meta/classes/go.bbclass +15 -4

index 167d02e3fa..c0ed3211cd 100644
@@ -111,9 +111,9 @@ do_compile_ptest_base() {
 do_compile_ptest_base[dirs] =+ "${GOTMPDIR}"
 
 go_do_install() {
-       install -d ${D}${libdir}/go/src/${GO_IMPORT}
+       install -d ${D}/gopath/src/${GO_IMPORT} ${D}${libdir}/go
        tar -C ${S}/src/${GO_IMPORT} -cf - --exclude-vcs --exclude '*.test' --exclude 'testdata' . | \
-               tar -C ${D}${libdir}/go/src/${GO_IMPORT} --no-same-owner -xf -
+               tar -C ${D}/gopath/src/${GO_IMPORT} --no-same-owner -xf -
        tar -C ${B} -cf - pkg | tar -C ${D}${libdir}/go --no-same-owner -xf -
 
        if [ -n "`ls ${B}/${GO_BUILD_BINDIR}/`" ]; then

@@ -172,9 +172,21 @@ do_install_ptest_base() {
        chown -R root:root ${D}${PTEST_PATH}
 }
 
+do_fixupdeps() {
+       gopath="${WORKDIR}/recipe-sysroot/gopath"
+       if [ -d ${gopath} ]; then
+               tar -C ${gopath} -cf - src | tar -C ${B} --no-same-owner -k -xf -
+       else
+               echo "no dependencies to fixup"
+       fi
+}
+
 EXPORT_FUNCTIONS do_unpack do_configure do_compile do_install
 
-FILES_${PN}-dev = "${libdir}/go/src"
+addtask do_fixupdeps after do_configure before do_compile
+
+SYSROOT_DIRS_append = "/gopath"
+FILES_${PN}-dev = "/gopath/src"
 FILES_${PN}-staticdev = "${libdir}/go/pkg"
 
 INSANE_SKIP_${PN} += "ldflags"
--

2.19.2

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 9 lines · Damien Riegel

To package a go application in Yocto, one needs to also package its dependencies to keep the build reproducible and under control. The default install task of the go class puts the source in ${libdir}/go/src.

${libdir}/go/src is where the standard go packages resides, aka $GOROOT. When a go package has dependencies on other go packages, they must be located under $GOPATH, otherwise errors like this one may happen:

Quoted 5 lines

Non-standard package <packagename> in standard package <packagename>

This point of this patch is to trigger a discussion on how this issue can be tackled in Yocto. Following code is working but should not be considered for inclusion.

Can you describe the context of this a bit more? The way go.bbclass is set up, all of your recipe's dependencies (and their dependencies) should already have been installed in the per-recipe sysroot (where GOROOT lives for the current recipe's build), and should get resolved from there. GOPATH gets set to ${B} for the current recipe's build. Nothing in the GOROOT should be importing packages that reside in GOPATH, which is what that particular error message is indicating. If you're getting that error, then there might be something wrong with recipe dependencies.

-Matt

Quoted 53 lines

Signed-off-by Damien Riegel <[email protected]>

---
 meta/classes/go.bbclass | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

meta/classes/go.bbclass +15 -4

index 167d02e3fa..c0ed3211cd 100644
@@ -111,9 +111,9 @@ do_compile_ptest_base() {
 do_compile_ptest_base[dirs] =+ "${GOTMPDIR}"
 
 go_do_install() {
-       install -d ${D}${libdir}/go/src/${GO_IMPORT}
+       install -d ${D}/gopath/src/${GO_IMPORT} ${D}${libdir}/go
        tar -C ${S}/src/${GO_IMPORT} -cf - --exclude-vcs --exclude '*.test' --exclude 'testdata' . | \
-               tar -C ${D}${libdir}/go/src/${GO_IMPORT} --no-same-owner -xf -
+               tar -C ${D}/gopath/src/${GO_IMPORT} --no-same-owner -xf -
        tar -C ${B} -cf - pkg | tar -C ${D}${libdir}/go --no-same-owner -xf -
 
        if [ -n "`ls ${B}/${GO_BUILD_BINDIR}/`" ]; then

@@ -172,9 +172,21 @@ do_install_ptest_base() {
        chown -R root:root ${D}${PTEST_PATH}
 }
 
+do_fixupdeps() {
+       gopath="${WORKDIR}/recipe-sysroot/gopath"
+       if [ -d ${gopath} ]; then
+               tar -C ${gopath} -cf - src | tar -C ${B} --no-same-owner -k -xf -
+       else
+               echo "no dependencies to fixup"
+       fi
+}
+
 EXPORT_FUNCTIONS do_unpack do_configure do_compile do_install
 
-FILES_${PN}-dev = "${libdir}/go/src"
+addtask do_fixupdeps after do_configure before do_compile
+
+SYSROOT_DIRS_append = "/gopath"
+FILES_${PN}-dev = "/gopath/src"
 FILES_${PN}-staticdev = "${libdir}/go/pkg"
 
 INSANE_SKIP_${PN} += "ldflags"
--

2.19.2

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 17 lines · Matt Madison

Quoted 9 lines · Damien Riegel

To package a go application in Yocto, one needs to also package its dependencies to keep the build reproducible and under control. The default install task of the go class puts the source in ${libdir}/go/src.

${libdir}/go/src is where the standard go packages resides, aka $GOROOT. When a go package has dependencies on other go packages, they must be located under $GOPATH, otherwise errors like this one may happen:

Quoted 5 lines

Non-standard package <packagename> in standard package <packagename>

This point of this patch is to trigger a discussion on how this issue can be tackled in Yocto. Following code is working but should not be considered for inclusion.

Quoted 9 lines

Can you describe the context of this a bit more? The way go.bbclass is set up, all of your recipe's dependencies (and their dependencies) should already have been installed in the per-recipe sysroot (where GOROOT lives for the current recipe's build), and should get resolved from there. GOPATH gets set to ${B} for the current recipe's build. Nothing in the GOROOT should be importing packages that reside in GOPATH, which is what that particular error message is indicating. If you're getting that error, then there might be something wrong with recipe dependencies.

Sure. I think the issue is because I have the following chain of dependencies:

A --> B --> C
`-> C

Building C is fine, building B is fine, but building A triggers the error I mentioned, because the Go compiler complains about B using a non-standard package C. It thinks B is a standard package because it's located in GOROOT.

Regarding dependencies, I created recipes named foocorp.com-{A,B,C}.bb and used DEPENDS_append = " foocorp.com-A" for instance.

I understand it's convenient to install dependencies in the GOROOT directory of recipes' sysroot, but we should keep in mind that when one calls go get, dependencies are installed in GOPATH, not GOROOT. I think that if we want to avoid (a bit obscure) build failures and cover all use cases, we should try to mimic that behaviour as close as we can.

Quoted 56 lines

-Matt

Quoted 53 lines

Signed-off-by Damien Riegel <[email protected]>

---
 meta/classes/go.bbclass | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

meta/classes/go.bbclass +15 -4

index 167d02e3fa..c0ed3211cd 100644
@@ -111,9 +111,9 @@ do_compile_ptest_base() {
 do_compile_ptest_base[dirs] =+ "${GOTMPDIR}"
 
 go_do_install() {
-       install -d ${D}${libdir}/go/src/${GO_IMPORT}
+       install -d ${D}/gopath/src/${GO_IMPORT} ${D}${libdir}/go
        tar -C ${S}/src/${GO_IMPORT} -cf - --exclude-vcs --exclude '*.test' --exclude 'testdata' . | \
-               tar -C ${D}${libdir}/go/src/${GO_IMPORT} --no-same-owner -xf -
+               tar -C ${D}/gopath/src/${GO_IMPORT} --no-same-owner -xf -
        tar -C ${B} -cf - pkg | tar -C ${D}${libdir}/go --no-same-owner -xf -
 
        if [ -n "`ls ${B}/${GO_BUILD_BINDIR}/`" ]; then

@@ -172,9 +172,21 @@ do_install_ptest_base() {
        chown -R root:root ${D}${PTEST_PATH}
 }
 
+do_fixupdeps() {
+       gopath="${WORKDIR}/recipe-sysroot/gopath"
+       if [ -d ${gopath} ]; then
+               tar -C ${gopath} -cf - src | tar -C ${B} --no-same-owner -k -xf -
+       else
+               echo "no dependencies to fixup"
+       fi
+}
+
 EXPORT_FUNCTIONS do_unpack do_configure do_compile do_install
 
-FILES_${PN}-dev = "${libdir}/go/src"
+addtask do_fixupdeps after do_configure before do_compile
+
+SYSROOT_DIRS_append = "/gopath"
+FILES_${PN}-dev = "/gopath/src"
 FILES_${PN}-staticdev = "${libdir}/go/pkg"
 
 INSANE_SKIP_${PN} += "ldflags"
--

2.19.2

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 32 lines · Damien Riegel

Quoted 17 lines · Matt Madison

Quoted 9 lines · Damien Riegel

To package a go application in Yocto, one needs to also package its dependencies to keep the build reproducible and under control. The default install task of the go class puts the source in ${libdir}/go/src.

${libdir}/go/src is where the standard go packages resides, aka $GOROOT. When a go package has dependencies on other go packages, they must be located under $GOPATH, otherwise errors like this one may happen:

Quoted 5 lines

Non-standard package <packagename> in standard package <packagename>

This point of this patch is to trigger a discussion on how this issue can be tackled in Yocto. Following code is working but should not be considered for inclusion.

Quoted 9 lines

Can you describe the context of this a bit more? The way go.bbclass is set up, all of your recipe's dependencies (and their dependencies) should already have been installed in the per-recipe sysroot (where GOROOT lives for the current recipe's build), and should get resolved from there. GOPATH gets set to ${B} for the current recipe's build. Nothing in the GOROOT should be importing packages that reside in GOPATH, which is what that particular error message is indicating. If you're getting that error, then there might be something wrong with recipe dependencies.

Sure. I think the issue is because I have the following chain of dependencies:

Quoted 7 lines

A --> B --> C
`-> C

Building C is fine, building B is fine, but building A triggers the error I mentioned, because the Go compiler complains about B using a non-standard package C. It thinks B is a standard package because it's located in GOROOT.

Are the three packages in separate source repositories? If so, the go tool should not be finding any C files in GOPATH.

If A and C are part of the same source repository, but B is separate, that would make the situation a bit trickier, but still solvable.

Quoted 2 lines

Regarding dependencies, I created recipes named foocorp.com-{A,B,C}.bb and used DEPENDS_append = " foocorp.com-A" for instance.

If A imports from B and C, you should have DEPENDS = "foocorp.com-B foocorp.com-C" in the recipe for A, and
DEPENDS = "foocorp.com-C" in the recipe for B.

Quoted 5 lines

I understand it's convenient to install dependencies in the GOROOT directory of recipes' sysroot, but we should keep in mind that when one calls go get, dependencies are installed in GOPATH, not GOROOT. I think that if we want to avoid (a bit obscure) build failures and cover all use cases, we should try to mimic that behaviour as close as we can.

go.bbclass does not use go get. It only uses go install.

-Matt

Quoted 56 lines

Quoted 56 lines

-Matt

Quoted 53 lines

Signed-off-by Damien Riegel <[email protected]>

---
 meta/classes/go.bbclass | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

meta/classes/go.bbclass +15 -4

index 167d02e3fa..c0ed3211cd 100644
@@ -111,9 +111,9 @@ do_compile_ptest_base() {
 do_compile_ptest_base[dirs] =+ "${GOTMPDIR}"
 
 go_do_install() {
-       install -d ${D}${libdir}/go/src/${GO_IMPORT}
+       install -d ${D}/gopath/src/${GO_IMPORT} ${D}${libdir}/go
        tar -C ${S}/src/${GO_IMPORT} -cf - --exclude-vcs --exclude '*.test' --exclude 'testdata' . | \
-               tar -C ${D}${libdir}/go/src/${GO_IMPORT} --no-same-owner -xf -
+               tar -C ${D}/gopath/src/${GO_IMPORT} --no-same-owner -xf -
        tar -C ${B} -cf - pkg | tar -C ${D}${libdir}/go --no-same-owner -xf -
 
        if [ -n "`ls ${B}/${GO_BUILD_BINDIR}/`" ]; then

@@ -172,9 +172,21 @@ do_install_ptest_base() {
        chown -R root:root ${D}${PTEST_PATH}
 }
 
+do_fixupdeps() {
+       gopath="${WORKDIR}/recipe-sysroot/gopath"
+       if [ -d ${gopath} ]; then
+               tar -C ${gopath} -cf - src | tar -C ${B} --no-same-owner -k -xf -
+       else
+               echo "no dependencies to fixup"
+       fi
+}
+
 EXPORT_FUNCTIONS do_unpack do_configure do_compile do_install
 
-FILES_${PN}-dev = "${libdir}/go/src"
+addtask do_fixupdeps after do_configure before do_compile
+
+SYSROOT_DIRS_append = "/gopath"
+FILES_${PN}-dev = "/gopath/src"
 FILES_${PN}-staticdev = "${libdir}/go/pkg"
 
 INSANE_SKIP_${PN} += "ldflags"
--

2.19.2

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 45 lines · Matt Madison

Quoted 32 lines · Damien Riegel

Quoted 17 lines · Matt Madison

Quoted 9 lines · Damien Riegel

To package a go application in Yocto, one needs to also package its dependencies to keep the build reproducible and under control. The default install task of the go class puts the source in ${libdir}/go/src.

${libdir}/go/src is where the standard go packages resides, aka $GOROOT. When a go package has dependencies on other go packages, they must be located under $GOPATH, otherwise errors like this one may happen:

Quoted 5 lines

Non-standard package <packagename> in standard package <packagename>

This point of this patch is to trigger a discussion on how this issue can be tackled in Yocto. Following code is working but should not be considered for inclusion.

Quoted 9 lines

Can you describe the context of this a bit more? The way go.bbclass is set up, all of your recipe's dependencies (and their dependencies) should already have been installed in the per-recipe sysroot (where GOROOT lives for the current recipe's build), and should get resolved from there. GOPATH gets set to ${B} for the current recipe's build. Nothing in the GOROOT should be importing packages that reside in GOPATH, which is what that particular error message is indicating. If you're getting that error, then there might be something wrong with recipe dependencies.

Sure. I think the issue is because I have the following chain of dependencies:

Quoted 7 lines

A --> B --> C
`-> C

Building C is fine, building B is fine, but building A triggers the error I mentioned, because the Go compiler complains about B using a non-standard package C. It thinks B is a standard package because it's located in GOROOT.

Are the three packages in separate source repositories? If so, the go tool should not be finding any C files in GOPATH.

Quoted 2 lines

If A and C are part of the same source repository, but B is separate, that would make the situation a bit trickier, but still solvable.

They are three separate repositories. The issue is precisely that C files are not in GOPATH but in GOROOT. C files _should be_ in GOPATH or that will break Go tools.

Quoted 6 lines

Quoted 2 lines

Regarding dependencies, I created recipes named foocorp.com-{A,B,C}.bb and used DEPENDS_append = " foocorp.com-A" for instance.

If A imports from B and C, you should have DEPENDS = "foocorp.com-B foocorp.com-C" in the recipe for A, and
DEPENDS = "foocorp.com-C" in the recipe for B.

Okay, that part I got right then!

Quoted 7 lines

Quoted 5 lines

I understand it's convenient to install dependencies in the GOROOT directory of recipes' sysroot, but we should keep in mind that when one calls go get, dependencies are installed in GOPATH, not GOROOT. I think that if we want to avoid (a bit obscure) build failures and cover all use cases, we should try to mimic that behaviour as close as we can.

go.bbclass does not use go get. It only uses go install.

Sorry, that was not clear, I was talking about Go development outside of Yocto. So outside of Yocto, I would use go fetch to get the dependencies and they would end up somewhere in GOPATH. When packaging these dependencies in Yocto, they will end up in the GOROOT of recipes' sysroots, and that breaks builds.

My point was that if we want to be able to write recipe for any valid Go package, dependencies should be found under GOPATH, not GOROOT.

Quoted 58 lines

-Matt

Quoted 56 lines

Quoted 56 lines

-Matt

Quoted 53 lines

Signed-off-by Damien Riegel <[email protected]>

---
 meta/classes/go.bbclass | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

meta/classes/go.bbclass +15 -4

index 167d02e3fa..c0ed3211cd 100644
@@ -111,9 +111,9 @@ do_compile_ptest_base() {
 do_compile_ptest_base[dirs] =+ "${GOTMPDIR}"
 
 go_do_install() {
-       install -d ${D}${libdir}/go/src/${GO_IMPORT}
+       install -d ${D}/gopath/src/${GO_IMPORT} ${D}${libdir}/go
        tar -C ${S}/src/${GO_IMPORT} -cf - --exclude-vcs --exclude '*.test' --exclude 'testdata' . | \
-               tar -C ${D}${libdir}/go/src/${GO_IMPORT} --no-same-owner -xf -
+               tar -C ${D}/gopath/src/${GO_IMPORT} --no-same-owner -xf -
        tar -C ${B} -cf - pkg | tar -C ${D}${libdir}/go --no-same-owner -xf -
 
        if [ -n "`ls ${B}/${GO_BUILD_BINDIR}/`" ]; then

@@ -172,9 +172,21 @@ do_install_ptest_base() {
        chown -R root:root ${D}${PTEST_PATH}
 }
 
+do_fixupdeps() {
+       gopath="${WORKDIR}/recipe-sysroot/gopath"
+       if [ -d ${gopath} ]; then
+               tar -C ${gopath} -cf - src | tar -C ${B} --no-same-owner -k -xf -
+       else
+               echo "no dependencies to fixup"
+       fi
+}
+
 EXPORT_FUNCTIONS do_unpack do_configure do_compile do_install
 
-FILES_${PN}-dev = "${libdir}/go/src"
+addtask do_fixupdeps after do_configure before do_compile
+
+SYSROOT_DIRS_append = "/gopath"
+FILES_${PN}-dev = "/gopath/src"
 FILES_${PN}-staticdev = "${libdir}/go/pkg"
 
 INSANE_SKIP_${PN} += "ldflags"
--

2.19.2

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 47 lines · Damien Riegel

Quoted 45 lines · Matt Madison

Quoted 32 lines · Damien Riegel

Quoted 17 lines · Matt Madison

Quoted 9 lines · Damien Riegel

To package a go application in Yocto, one needs to also package its dependencies to keep the build reproducible and under control. The default install task of the go class puts the source in ${libdir}/go/src.

${libdir}/go/src is where the standard go packages resides, aka $GOROOT. When a go package has dependencies on other go packages, they must be located under $GOPATH, otherwise errors like this one may happen:

Quoted 5 lines

Non-standard package <packagename> in standard package <packagename>

This point of this patch is to trigger a discussion on how this issue can be tackled in Yocto. Following code is working but should not be considered for inclusion.

Quoted 9 lines

Can you describe the context of this a bit more? The way go.bbclass is set up, all of your recipe's dependencies (and their dependencies) should already have been installed in the per-recipe sysroot (where GOROOT lives for the current recipe's build), and should get resolved from there. GOPATH gets set to ${B} for the current recipe's build. Nothing in the GOROOT should be importing packages that reside in GOPATH, which is what that particular error message is indicating. If you're getting that error, then there might be something wrong with recipe dependencies.

Sure. I think the issue is because I have the following chain of dependencies:

Quoted 7 lines

A --> B --> C
`-> C

Building C is fine, building B is fine, but building A triggers the error I mentioned, because the Go compiler complains about B using a non-standard package C. It thinks B is a standard package because it's located in GOROOT.

Are the three packages in separate source repositories? If so, the go tool should not be finding any C files in GOPATH.

Quoted 6 lines

Quoted 2 lines

If A and C are part of the same source repository, but B is separate, that would make the situation a bit trickier, but still solvable.

They are three separate repositories. The issue is precisely that C files are not in GOPATH but in GOROOT. C files _should be_ in GOPATH or that will break Go tools.

Go does allow for non-"standard" packages to be in GOROOT as well. The way the go tool marks a package as "standard" is if it resides in GOROOT _and_ starts with a prefix that does not look like a domain name (specifically, does not contain a dot '.'). The Go convention, though, is to use a domain name as the first element of the import path for anything that isn't part of the base runtime (std, cmd).

If you do have a situation where you have multiple interdependent packages that for some reason have import paths that do not start with a domain name, you could build them all using a single recipe, rather than splitting them up into separate recipes. I've had to do that for cases where there were circular dependencies between two (or more) packages; the same technique could work this, though, too.