Skip to contentSkip to conversation

pkg-golang: Allow per package/target CGO_ENABLED setting

PATCH New Cam Hutchison 4 participants 13 messages Jul 27, 2018 – Apr 12, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Allow the CGO_ENABLED variable to be controlled per package and per build target, instead of having the value determined by whether or not the toolchain has threads.

Some build targets may not build with CGO_ENABLED=1, so allowing a per package and build target override will allow those targets to be built with Buildroot.

---
 docs/manual/adding-packages-golang.txt | 14 ++++++++++++++
 package/pkg-golang.mk                  | 17 +++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
docs/manual/adding-packages-golang.txt +14 -0
index efcf696867..2683774861 100644
@@ -99,6 +99,20 @@ therefore only use a few of them, or none.
    +FOO_BUILD_TARGETS = cmd/docker cmd/dockerd+ the binaries produced
    are +docker+ and +dockerd+.
 
+* +FOO_CGO_ENABLED+ overrides the default value for the +CGO_ENABLED+
+  flag passed to the Go compiler. By default +CGO_ENABLED+ is set
+  to 1 if the toolchain supports threads, but you can force it to
+  0 for the package by setting this variable. If the toolchain does
+  not support threads and you set +FOO_CGO_ENABLED+ to 1, an error
+  will be generated.
+
+* +FOO_BUILD_TARGET_CGO_ENABLED_<target>+ can be used to override
+  the +CGO_ENABLED+ flag passed to the Go compiler for a specific
+  build target, similar to +FOO_CGO_ENABLED+. +<target>+ references
+  a target listed in +FOO_BUILD_TARGETS+. If the toolchain does
+  not support threads and you set this variable to 1, an error will
+  be generated.
+
 * +FOO_INSTALL_BINS+ can be used to pass the list of binaries that
   should be installed in +/usr/bin+ on the target. If
   +FOO_INSTALL_BINS+ is not specified, it defaults to the lower-case
package/pkg-golang.mk +15 -2
index bf178622b5..2be3bd4c14 100644
@@ -28,8 +28,20 @@ GO_BIN = $(HOST_DIR)/bin/go
 GO_TARGET_ENV = \
 	$(HOST_GO_TARGET_ENV) \
 	PATH=$(BR_PATH) \
-	GOBIN= \
-	CGO_ENABLED=$(HOST_GO_CGO_ENABLED)
+	GOBIN=
+
+# Allow CGO_ENABLED to be defined per package and per build target, but if
+# CGO is enabled for a package or a build target and the toolchain does not
+# support that, generate an error.
+GO_CGO_ENABLED = $(strip \
+	$(if $(filter 10,$($(1)_CGO_ENABLED)$(HOST_GO_CGO_ENABLED)),\
+		$(error Toolchain does not support CGO_ENABLED=1 for package $(1)),\
+		$(or $($(1)_CGO_ENABLED),$(HOST_GO_CGO_ENABLED))))
+
+GO_TARGET_CGO_ENABLED = $(strip \
+	$(if $(filter 10,$($(1)_BUILD_TARGET_CGO_ENABLED_$(2))$(HOST_GO_CGO_ENABLED)),\
+		$(error Toolchain does not support CGO_ENABLED=1 for target $(2)),\
+		$(or $($(1)_BUILD_TARGET_CGO_ENABLED_$(2)),$(call GO_CGO_ENABLED,$(1)))))
 
 ################################################################################
 # inner-golang-package -- defines how the configuration, compilation and
@@ -100,6 +112,7 @@ define $(2)_BUILD_CMDS
 	$$(foreach d,$$($(2)_BUILD_TARGETS),\
 		cd $$($(2)_SRC_PATH); \
 		$$(GO_TARGET_ENV) \
+			CGO_ENABLED=$$(call GO_TARGET_CGO_ENABLED,$(2),$$(d)) \
 			GOPATH="$$(@D)/$$($(2)_WORKSPACE)" \
 			$$($(2)_GO_ENV) \
 			$$(GO_BIN) build -v $$($(2)_BUILD_OPTS) \
Signature · 1 line
2.11.0
#1 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Hello Cam,

+Christian and Anisse in Cc.

Sorry for the very slow feedback.

Quoted 7 lines · On Fri, 27 Jul 2018 12:47:20 +1000 Cam Hutchison <[email protected]> wrote:

Allow the CGO_ENABLED variable to be controlled per package and per build target, instead of having the value determined by whether or not the toolchain has threads.

Some build targets may not build with CGO_ENABLED=1, so allowing a per package and build target override will allow those targets to be built with Buildroot.

Could you give some specific examples instead of the fuzzy "Some build targets" wording ?

Christian, Anisse, what do you think about this patch ? Do we need something like this in our Go support ?

I'm leaving the full patch below. You can also find it at
http://patchwork.ozlabs.org/patch/949972/.

Thanks!

Thomas

Quoted 66 lines
---
 docs/manual/adding-packages-golang.txt | 14 ++++++++++++++
 package/pkg-golang.mk                  | 17 +++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
docs/manual/adding-packages-golang.txt +14 -0
index efcf696867..2683774861 100644
@@ -99,6 +99,20 @@ therefore only use a few of them, or none.
    +FOO_BUILD_TARGETS = cmd/docker cmd/dockerd+ the binaries produced
    are +docker+ and +dockerd+.
 
+* +FOO_CGO_ENABLED+ overrides the default value for the +CGO_ENABLED+
+  flag passed to the Go compiler. By default +CGO_ENABLED+ is set
+  to 1 if the toolchain supports threads, but you can force it to
+  0 for the package by setting this variable. If the toolchain does
+  not support threads and you set +FOO_CGO_ENABLED+ to 1, an error
+  will be generated.
+
+* +FOO_BUILD_TARGET_CGO_ENABLED_<target>+ can be used to override
+  the +CGO_ENABLED+ flag passed to the Go compiler for a specific
+  build target, similar to +FOO_CGO_ENABLED+. +<target>+ references
+  a target listed in +FOO_BUILD_TARGETS+. If the toolchain does
+  not support threads and you set this variable to 1, an error will
+  be generated.
+
 * +FOO_INSTALL_BINS+ can be used to pass the list of binaries that
   should be installed in +/usr/bin+ on the target. If
   +FOO_INSTALL_BINS+ is not specified, it defaults to the lower-case
package/pkg-golang.mk +15 -2
index bf178622b5..2be3bd4c14 100644
@@ -28,8 +28,20 @@ GO_BIN = $(HOST_DIR)/bin/go
 GO_TARGET_ENV = \
 	$(HOST_GO_TARGET_ENV) \
 	PATH=$(BR_PATH) \
-	GOBIN= \
-	CGO_ENABLED=$(HOST_GO_CGO_ENABLED)
+	GOBIN=
+
+# Allow CGO_ENABLED to be defined per package and per build target, but if
+# CGO is enabled for a package or a build target and the toolchain does not
+# support that, generate an error.
+GO_CGO_ENABLED = $(strip \
+	$(if $(filter 10,$($(1)_CGO_ENABLED)$(HOST_GO_CGO_ENABLED)),\
+		$(error Toolchain does not support CGO_ENABLED=1 for package $(1)),\
+		$(or $($(1)_CGO_ENABLED),$(HOST_GO_CGO_ENABLED))))
+
+GO_TARGET_CGO_ENABLED = $(strip \
+	$(if $(filter 10,$($(1)_BUILD_TARGET_CGO_ENABLED_$(2))$(HOST_GO_CGO_ENABLED)),\
+		$(error Toolchain does not support CGO_ENABLED=1 for target $(2)),\
+		$(or $($(1)_BUILD_TARGET_CGO_ENABLED_$(2)),$(call GO_CGO_ENABLED,$(1)))))
 
 ################################################################################
 # inner-golang-package -- defines how the configuration, compilation and
@@ -100,6 +112,7 @@ define $(2)_BUILD_CMDS
 	$$(foreach d,$$($(2)_BUILD_TARGETS),\
 		cd $$($(2)_SRC_PATH); \
 		$$(GO_TARGET_ENV) \
+			CGO_ENABLED=$$(call GO_TARGET_CGO_ENABLED,$(2),$$(d)) \
 			GOPATH="$$(@D)/$$($(2)_WORKSPACE)" \
 			$$($(2)_GO_ENV) \
 			$$(GO_BIN) build -v $$($(2)_BUILD_OPTS) \
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Quoted 13 lines · Thomas Petazzoni

Hello Cam,

+Christian and Anisse in Cc.

Sorry for the very slow feedback.

Quoted 4 lines · On Fri, 27 Jul 2018 12:47:20 +1000 Cam Hutchison <[email protected]> wrote:

Allow the CGO_ENABLED variable to be controlled per package and per build target, instead of having the value determined by whether or not the toolchain has threads.

Quoted 7 lines
Quoted 3 lines

Some build targets may not build with CGO_ENABLED=1, so allowing a per package and build target override will allow those targets to be built with Buildroot.

Could you give some specific examples instead of the fuzzy "Some build targets" wording ?

Sorry for my lack of detail. Here are my notes that I made at the time:

* kube-apiserver is not linking, emitting the error direct call too far.
If I
override the go.mk to force CGO_ENABLED=0, it works. The kube build script builds a bunch of components with CGO_ENABLED=0, but the buildroot golang
infra does not support setting CGO_ENABLED - it is hard-coded based on
whether the toolchain has threads (BR2_TOOLCHAIN_HAS_THREADS). I will need
to somehow override that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190413/dffbe778/attachment.html>

#3 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

In reply to Cam Hutchison

Hi Cam,

Sorry for the delay,

(+Angelo in cc)

Quoted 7 lines · Cam Hutchison

Allow the CGO_ENABLED variable to be controlled per package and per build target, instead of having the value determined by whether or not the toolchain has threads.

Some build targets may not build with CGO_ENABLED=1, so allowing a per package and build target override will allow those targets to be built with Buildroot.

Just like Thomas, I'm wondering what specific use-case you have in mind; in particular, I'm interested why you needed build-target granularity in addition to package-level configuration.

To provide context:

Disabling CGO isn't an issue, until you try doing DNS requests, which are usually handled by your libc. If you have anything configured out of the ordinary /etc/resolv.conf, a cgo-disabled binary won't be able to do a DNS request. Same goes for user/group and user home dir resolution:
anything outside of /etc/{passwd,group} (pam, NSS, etc.), won't work.

Disabling CGO allows you to have static binaries; if you don't use any native libraries.

I don't know in which context it might be useful in buildroot.

Regards,

Anisse

#4 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Hi Anisse, Cam, all,

Overriding CGO_ENABLED when we know that the toolchain doesn't work properly with cgo doesn't seem correct.

Quoted 3 lines · Anisse Astier
Quoted 3 lines

Some build targets may not build with CGO_ENABLED=1, so allowing a per package and build target override will allow those targets to be built with Buildroot.

Why not just add the override in the existing build flag overrides?

Quoted 3 lines

Just like Thomas, I'm wondering what specific use-case you have in mind; in particular, I'm interested why you needed build-target granularity in addition to package-level configuration.

If you really need to do something this custom, I recommend just overriding the build stage entirely.

Quoted 5 lines

Disabling CGO isn't an issue, until you try doing DNS requests, which are usually handled by your libc. If you have anything configured out of the ordinary /etc/resolv.conf, a cgo-disabled binary won't be able to do a DNS request. Same goes for user/group and user home dir resolution:
anything outside of /etc/{passwd,group} (pam, NSS, etc.), won't work.

Are you referring to the case where a Go-implemented DNS is used? This should be the case with cgo disabled, and I don't know if the behavior differs from the regular libc dns.

Quoted 1 line

I don't know in which context it might be useful in buildroot.

I don't see a reason for it either.

Best,
Christian

#5 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Hello,

Quoted 3 lines · On Sun, 07 Apr 2019 19:00:20 -0700 Christian Stewart <[email protected]> wrote:
Quoted 1 line

I don't know in which context it might be useful in buildroot.

I don't see a reason for it either.

Thanks for the feedback, I'll mark the patch as Rejected for now. We can of course revisit if there's some further details about why this would be needed.

Anisse, Christian: nobody is listed in the DEVELOPERS file for the package/pkg-golang.mk file. Would you be willing to be listed as DEVELOPERS for this file ? It would be useful as you would be Cc'ed on patches touching this file, which would help in situations like this one. If you're fine with this, send a patch to add yourself :)

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Quoted 5 lines · Thomas Petazzoni

Hello,

On Sun, 07 Apr 2019 19:00:20 -0700
Christian Stewart <[email protected]> wrote:

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I don't know in which context it might be useful in buildroot.

I don't see a reason for it either.

Thanks for the feedback, I'll mark the patch as Rejected for now. We can of course revisit if there's some further details about why this would be needed.

Agreed, thanks.

Quoted 6 lines

Anisse, Christian: nobody is listed in the DEVELOPERS file for the package/pkg-golang.mk file. Would you be willing to be listed as DEVELOPERS for this file ? It would be useful as you would be Cc'ed on patches touching this file, which would help in situations like this one. If you're fine with this, send a patch to add yourself :)

Good catch, I was wondering why I didn't see the previous emails. I'll send a patch to add myself and add better email filtering on my side.

Regards,

Anisse

#7 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

In reply to Thomas Petazzoni
Quoted 11 lines · Thomas Petazzoni

Hello,

Quoted 3 lines · On Sun, 07 Apr 2019 19:00:20 -0700 Christian Stewart <[email protected]> wrote:
Quoted 1 line

I don't know in which context it might be useful in buildroot.

I don't see a reason for it either.

Thanks for the feedback, I'll mark the patch as Rejected for now.

Well that's rather demoralising - waiting 9 months for some feedback and given only 10 hours to respond before the patch is rejected.

My use case for this was for building Kubernetes, which builds some components with CGO_ENABLED=0 and others without it. If I tried to build some components with without it, I got linker errors. Since I didn't feel like trying to debug/understand the whole Kubernetes build process, I just went with building it the same way in buildroot as it builds outside it. But for that I needed to be able to selectively control CGO_ENABLED on a per-target basis.

But never mind. I can maintain this patch in my local buildroot patch queue. I just thought it might be useful for others.

#8 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Cam,

Quoted 3 lines · Cam Hutchison

But never mind. I can maintain this patch in my local buildroot patch queue. I just thought it might be useful for others.

Rejected doesn't mean rejected permanently, just rejected as is. This means that more information is needed in the commit message, or maybe something should be changed in the commit itself.

Fwiw, I'm also integrating kubernetes with buildroot at the moment. I don't see where cgo is required in this way.

I'd be happy to compare notes if you'd like?

Best,
Christian

Quoted 1 line

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190408/fd2b9731/attachment-0001.html>

#9 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Quoted 4 lines · Christian Stewart

Fwiw, I'm also integrating kubernetes with buildroot at the moment. I don't see where cgo is required in this way.

I'd be happy to compare notes if you'd like?

My local BR2_EXTERNAL kubernetes config is at
https://github.com/camh-/clusterpi/tree/master/package/kubernetes.
Feel free to take anything from there that is useful.

I'm going to look at k3s instead now (https://github.com/rancher/k3s)
for its smaller size - probably more appropriate for my targets.

Regards,
Cam

#10 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

Quoted 11 lines · Cam Hutchison
Quoted 2 lines · Christian Stewart

Fwiw, I'm also integrating kubernetes with buildroot at the moment. I

don't see where cgo is required in this way.

Quoted 2 lines

I'd be happy to compare notes if you'd like?

My local BR2_EXTERNAL kubernetes config is at
https://github.com/camh-/clusterpi/tree/master/package/kubernetes.
Feel free to take anything from there that is useful.

Also, here are my notes I made at the time. Perhaps they might be useful:

* Success building kubelet from buildroot
+ I think I've done that before, but I will next try other components
using
the same pattern
+ Kubernetes buildroot config directly uses go to build binaries, does not
use kube Makefile or build scripts.
* Success with kube-proxy and kubeadm
* kube-proxy, kubelet and kubeadm are the node binaries, all successfully
built
from a single buildroot invocation. All were built with CGO_ENABLED=1. kube-proxy and kubeadm would normally be build with CGO_ENABLED=0 using
the
kube build scripts (KUBE_STATIC_LIBRARIES)
* The buildroot golang support does not allow mixed CGO_ENABLED builds of different binaries. The value for CGO_ENABLED seems to be hard-coded at toolchain build time. I should investigate what it would take to build
both
CGO_ENABLED=0 and CGO_ENABLED=1 together in the toolchain.
* kube-apiserver build failed. It needs openapi generated files. I had
thought
that the openapi generation stuff was for docs - not so. I will have to
see
what needs to be done to hook in the generated files build too. Shouldn't
be
too hard as the go compiler can build both host and target binaries, so we don't need a second host-targetting go compiler.
* Extending the KUBERNETES_BUILD_CMDS to run make generated_files worked
just
fine.
* kube-apiserver is not linking, emitting the error direct call too far.
If I
override the go.mk to force CGO_ENABLED=0, it works. The kube build script builds a bunch of components with CGO_ENABLED=0, but the buildroot golang
infra does not support setting CGO_ENABLED - it is hard-coded based on
whether the toolchain has threads (BR2_TOOLCHAIN_HAS_THREADS). I will need
to somehow override that.
* Everything builds now (including hyperkube) using the right CGO settings
for
the right targets.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190413/48fa4586/attachment.html>

#11 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

In reply to Cam Hutchison

Hello Cam,

Quoted 4 lines · On Tue, 9 Apr 2019 10:22:19 +1000 Cam Hutchison <[email protected]> wrote:
Quoted 1 line

Thanks for the feedback, I'll mark the patch as Rejected for now.

Well that's rather demoralising - waiting 9 months for some feedback and given only 10 hours to respond before the patch is rejected.

Sorry for this. Taking a step back, I can understand how my very direct decision can be disappointing.

However, as Christian explained, it's definitely not a "reject forever", not at all. The status of patches in patchwork can always be flipped back to whatever state we want.

As you have seen, we have a huge backlog of patches, so when the feedback on a patch is fairly negative, I tend to immediately update the patch state to not leave it pending forever. Otherwise, I forget about it among the 200+ patches that are pending.

But clearly, if some additional explanation is given, either a new iteration of the patch is sent, or alternatively we could apply the original version after amending the commit log and/or adjusting the implementation.

Quoted 7 lines

My use case for this was for building Kubernetes, which builds some components with CGO_ENABLED=0 and others without it. If I tried to build some components with without it, I got linker errors. Since I didn't feel like trying to debug/understand the whole Kubernetes build process, I just went with building it the same way in buildroot as it builds outside it. But for that I needed to be able to selectively control CGO_ENABLED on a per-target basis.

So Christian said he would try to integrate Kubernetes, and see where it goes. Would that work for you ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#12 lore raw

PATCH 1/1 pkg-golang: Allow per package/target CGO_ENABLED setting

In reply to Christian Stewart
Quoted 25 lines · Christian Stewart

Hi Anisse, Cam, all,

Overriding CGO_ENABLED when we know that the toolchain doesn't work properly with cgo doesn't seem correct.

Quoted 3 lines · Anisse Astier
Quoted 3 lines

Some build targets may not build with CGO_ENABLED=1, so allowing a per package and build target override will allow those targets to be built with Buildroot.

Why not just add the override in the existing build flag overrides?

Quoted 3 lines

Just like Thomas, I'm wondering what specific use-case you have in mind; in particular, I'm interested why you needed build-target granularity in addition to package-level configuration.

If you really need to do something this custom, I recommend just overriding the build stage entirely.

Quoted 5 lines

Disabling CGO isn't an issue, until you try doing DNS requests, which are usually handled by your libc. If you have anything configured out of the ordinary /etc/resolv.conf, a cgo-disabled binary won't be able to do a DNS request. Same goes for user/group and user home dir resolution:
anything outside of /etc/{passwd,group} (pam, NSS, etc.), won't work.

Quoted 3 lines

Are you referring to the case where a Go-implemented DNS is used? This should be the case with cgo disabled, and I don't know if the behavior differs from the regular libc dns.

Yes, I'm talking about the go-implemented dns resolution when cgo is disabled.

On a glibc system you might not simply rely on /etc/{hosts,resolv.conf} since hostname resolution is pluggable and configurable with nsswitch.conf.

Regards,

Anisse