Skip to contentSkip to conversation

grpc-related fixes

PATCH New Thomas De Schampheleire 2 participants 7 messages Feb 3, 2021 – Feb 4, 2021

lore
Share & tools
lore
#0 lore raw

cover grpc-related fixes

From: Thomas De Schampheleire <[email protected]>

Commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e removed an obsolete patch on grpc, but this had the side effect that building collectd after grpc no longer succeeded, because grpc_cpp_plugin from the target was executed.

During investigation of this issue, a number of related problems were found. This series addresses these problems.

Thomas De Schampheleire (3):
package/collectd: stop passing '--with-libgrpc++' package/grpc: fix cross-compilation with
gRPC_BUILD_GRPC_CPP_PLUGIN=OFF
package/grpc: disable unnecessary build plugins

 package/collectd/collectd.mk                  |  3 --
 ...ix-cross-compilation-with-gRPC_BUILD.patch | 43 +++++++++++++++++++
 package/grpc/grpc.mk                          |  9 +++-
 3 files changed, 51 insertions(+), 4 deletions(-)
 create mode 100644 package/grpc/0001-CMakeLists.txt-fix-cross-compilation-with-gRPC_BUILD.patch
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/3 package/collectd: stop passing '--with-libgrpc++'

From: Thomas De Schampheleire <[email protected]>

Commit 903de16f5fd586a32a7aed7a792a43dce16365eb added passing '--with-libgrpc++' with the explanation:

"Use --with-libgrpc++ option as otherwise collectd will try to find grpc++.pc which is not available."

At the time of above commit, grpc version in Buildroot was 1.23.0. Since grpc 1.25.0, a grpc++.pc file _is_ generated from cmake builds. Hence, remove passing --with-libgrpc++.

This change fixes a problem introduced by commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e. As a side effect of that change, a target version of 'grpc_cpp_plugin' was now created. When collectd was built after grpc, even without grpc support in collectd enabled, the collectd configure script would find this target grpc_cpp_plugin and try to use it (which is not possible because it is built for target).
When not passing '--with-libgrpc++', collectd will instead find the host version of grpc_cpp_plugin, which works fine.

There are still two underlying problems:
1. the target version of grpc_cpp_plugin is not actually needed. This will be disabled in a subsequent commit.

2. collectd should not execute any grpc-related action if grpc support for collectd is disabled. This problem has been reported upstream:
https://github.com/collectd/collectd/issues/3836

---
 package/collectd/collectd.mk | 3 ---
 1 file changed, 3 deletions(-)
package/collectd/collectd.mk +0 -3
index ccca9bf388..3fcca12a45 100644
@@ -193,9 +193,6 @@ COLLECTD_DEPENDENCIES = \
 	$(if $(BR2_PACKAGE_COLLECTD_WRITEREDIS),hiredis)
 
 # include/library fixups
-ifeq ($(BR2_PACKAGE_GRPC),y)
-COLLECTD_CONF_OPTS += --with-libgrpc++=$(STAGING_DIR)/usr
-endif
 ifeq ($(BR2_PACKAGE_JANSSON),y)
 COLLECTD_CONF_OPTS += --with-libjansson=$(STAGING_DIR)/usr
 endif
Signature · 1 line
2.26.2
#2 lore raw

PATCH 2/3 package/grpc: fix cross-compilation with gRPC_BUILD_GRPC_CPP_PLUGIN=OFF

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

In commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e, an obsolete patch to support cross-compilation was removed, in favor of the upstream solution.

However, this caused a small change in behavior: for the target grpc, the tool 'grpc_cpp_plugin' is now also built, while before it was not.

This tool is only really needed on development machines. Since Buildroot does not support compilers and such on target itself, the tool is not needed.

There exists an option gRPC_BUILD_GRPC_CPP_PLUGIN which can be set to 'OFF', but disabling it in a cross-compilation context yields build failures.

Add a patch to fix that. This patch is intended to be upstreamed to grpc.

---
 ...ix-cross-compilation-with-gRPC_BUILD.patch | 43 +++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 package/grpc/0001-CMakeLists.txt-fix-cross-compilation-with-gRPC_BUILD.patch
package/grpc/0001-CMakeLists.txt-fix-cross-compilation-with-gRPC_BUILD.patch new file +43 -0
new file mode 100644
index 0000000000..553be8cb99
@@ -0,0 +1,43 @@
+From 268815d74ea34856c686dce9e9c23106b51f2782 Mon Sep 17 00:00:00 2001
+From: Thomas De Schampheleire <[email protected]>
+Date: Tue, 2 Feb 2021 21:46:29 +0100
+Subject: [PATCH] CMakeLists.txt: fix cross-compilation with
+ gRPC_BUILD_GRPC_CPP_PLUGIN=OFF
+
+When cross-compiling gRPC, a native version of 'grpc_cpp_plugin' is searched
+in the environment. For most use cases, a cross version of this file is not
+needed.
+
+However, when cross-building with -DgRPC_BUILD_GRPC_CPP_PLUGIN=OFF, there
+are some build errors, for example:
+
+make[3]: *** No rule to make target 'grpc_cpp_plugin', needed by 'gens/src/proto/grpc/reflection/v1alpha/reflection.grpc.pb.cc'.  Stop.
+
+This is because there is still a hard dependency on 'grpc_cpp_plugin' for
+these targets, not taking into account the cross-compilation case.
+
+Fix by depending on the variable gRPC_CPP_PLUGIN, which is set correctly for
+either cross or native case.
+
+Signed-off-by: Thomas De Schampheleire <[email protected]>
+
+---
+ CMakeLists.txt | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 6bdae22083..08aea609f7 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -317,7 +317,7 @@ function(protobuf_generate_grpc_cpp)
+            --plugin=protoc-gen-grpc=${_gRPC_CPP_PLUGIN}
+            ${_protobuf_include_path}
+            ${REL_FIL}
+-      DEPENDS ${ABS_FIL} ${_gRPC_PROTOBUF_PROTOC} grpc_cpp_plugin
++      DEPENDS ${ABS_FIL} ${_gRPC_PROTOBUF_PROTOC} ${_gRPC_CPP_PLUGIN}
+       WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+       COMMENT "Running gRPC C++ protocol buffer compiler on ${FIL}"
+       VERBATIM)
+-- 
+2.26.2
+
Signature · 1 line
2.26.2
#3 lore raw

PATCH 3/3 package/grpc: disable unnecessary build plugins

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

grpc has plugins for multiple programming languages, which are needed on development machines only. Examples are grpc_cpp_plugin, grpc_ruby_plugin, etc.

Even though before commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e, grpc_cpp_plugin was not installed for target, all other plugins still were. This causes additional build time and rootfs space.

As Buildroot does not support building a development environment for target, these tools can be disabled.

---
 package/grpc/grpc.mk | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
package/grpc/grpc.mk +8 -1
index a8a3af89ef..27b28deff3 100644
@@ -27,7 +27,14 @@ GRPC_CONF_OPTS = \
 	-DgRPC_PROTOBUF_PROVIDER=package \
 	-DgRPC_RE2_PROVIDER=package \
 	-DgRPC_SSL_PROVIDER=package \
-	-DgRPC_ZLIB_PROVIDER=package
+	-DgRPC_ZLIB_PROVIDER=package \
+	-DgRPC_BUILD_GRPC_CPP_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF
 
 # grpc can use __atomic builtins, so we need to link with
 # libatomic when available
Signature · 1 line
2.26.2
#4 lore raw

PATCH 3/3 package/grpc: disable unnecessary build plugins

Thomas, All,

On 2021-02-03 13:43 +0100, Thomas De Schampheleire spake thusly:

Quoted 25 lines

From: Thomas De Schampheleire <[email protected]>

grpc has plugins for multiple programming languages, which are needed on development machines only. Examples are grpc_cpp_plugin, grpc_ruby_plugin, etc.

Even though before commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e, grpc_cpp_plugin was not installed for target, all other plugins still were. This causes additional build time and rootfs space.

As Buildroot does not support building a development environment for target, these tools can be disabled.

---
 package/grpc/grpc.mk | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
package/grpc/grpc.mk +0 -0
index a8a3af89ef..27b28deff3 100644
@@ -27,7 +27,14 @@ GRPC_CONF_OPTS = \
 	-DgRPC_PROTOBUF_PROVIDER=package \
 	-DgRPC_RE2_PROVIDER=package \

package/grpc/grpc.mk GRPC_CONF_OPTS = \ · line 37

Quoted 10 lines
package/grpc/grpc.mk +8 -1
 	-DgRPC_SSL_PROVIDER=package \
-	-DgRPC_ZLIB_PROVIDER=package
+	-DgRPC_ZLIB_PROVIDER=package \
+	-DgRPC_BUILD_GRPC_CPP_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
+	-DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF

So I am OK with applying this patch in bulk, because it makes sense.

IIUC, the grpc_cpp_plugin will generate C++ code, and so the grpc_node_plugin will generate JS code, and so on... Right?

If so, would it make sense to also trim the host variant further, too?

I.e. if there is no nodejs on the target, then no need for the node plugin. Ditto, if there is no C++, then no need to build the cpp plugin, right?

Oh, except those may also be used to build host packages, too... And we do not have a good way to know what languages the host packages would need... So in the end, we do not want to disable any plugin for the host variant...

(Yeah, I'm thinking aloud, sorry for the noise...))

Damn, I already pushed, or I'd have added as a comment and to the commit.

`nyway... Thanks for the fixes! :-)

Regards,
Yann E. MORIN.

Quoted 9 lines
package/grpc/grpc.mk +0 -0
 # grpc can use __atomic builtins, so we need to link with
 # libatomic when available
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH 3/3 package/grpc: disable unnecessary build plugins

Hi Yann,

El mi?, 3 feb 2021 a las 23:09, Yann E. MORIN
(<[email protected]>) escribi?:

Quoted 42 lines

Thomas, All,

On 2021-02-03 13:43 +0100, Thomas De Schampheleire spake thusly:

Quoted 25 lines

From: Thomas De Schampheleire <[email protected]>

grpc has plugins for multiple programming languages, which are needed on development machines only. Examples are grpc_cpp_plugin, grpc_ruby_plugin, etc.

Even though before commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e, grpc_cpp_plugin was not installed for target, all other plugins still were. This causes additional build time and rootfs space.

As Buildroot does not support building a development environment for target, these tools can be disabled.

---
 package/grpc/grpc.mk | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
package/grpc/grpc.mk +0 -0
index a8a3af89ef..27b28deff3 100644
@@ -27,7 +27,14 @@ GRPC_CONF_OPTS = \
      -DgRPC_PROTOBUF_PROVIDER=package \
      -DgRPC_RE2_PROVIDER=package \

package/grpc/grpc.mk GRPC_CONF_OPTS = \ · line 37

Quoted 10 lines
package/grpc/grpc.mk +8 -1
      -DgRPC_SSL_PROVIDER=package \
-     -DgRPC_ZLIB_PROVIDER=package
+     -DgRPC_ZLIB_PROVIDER=package \
+     -DgRPC_BUILD_GRPC_CPP_PLUGIN=OFF \
+     -DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
+     -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
+     -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
+     -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+     -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
+     -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF

So I am OK with applying this patch in bulk, because it makes sense.

Quoted 2 lines

IIUC, the grpc_cpp_plugin will generate C++ code, and so the grpc_node_plugin will generate JS code, and so on... Right?

Yes, this is my understanding.

Quoted 11 lines

If so, would it make sense to also trim the host variant further, too?

I.e. if there is no nodejs on the target, then no need for the node plugin. Ditto, if there is no C++, then no need to build the cpp plugin, right?

Oh, except those may also be used to build host packages, too... And we do not have a good way to know what languages the host packages would need... So in the end, we do not want to disable any plugin for the host variant...

At this moment, the grpc target package in buildroot depends on the full host-grpc, which has (IMHO) excessive dependencies: it depends on host-libabseil-cpp, host-re2, etc.
Since libabseil-cpp requires a gcc >= 4.9, it cannot be built on CentOS 7 which uses 4.8.5.
But, the dependency grpc <- host-grpc is actualy only there to provide a native version of grpc_cpp_plugin. And that plugin does not need libabseil-cpp, re2, etc. Only host-protobuf.

So locally, I have prepared a patch to grpc CMakeLists.txt to add an option to only build grpc_cpp_plugin, and not all the rest, so that the Buildroot host-grpc can remove its dependencies on host-libabseil-cpp and host-re2.
I have opened an issue on the grpc project to discuss this:
https://github.com/grpc/grpc/issues/25322

In these changes, I disabled all but grpc_cpp_plugin. In Buildroot itself, there is no other package that currently has grpc support, except collectd, which only needs grpc_cpp_plugin.
While there could indeed be external uses for these additional language plugins like grpc_python_plugin, I'm not sure whether there is an immediate need to support them.
But as far as I can see, they don't include on libabseil-cpp and the others either, so it wouldn't be a big problem to retain them in host-grpc I think.

I am waiting a bit on feedback on the aformentioned issue before submitting these changes to buildroot.

Best regards,
Thomas

#6 lore raw

cover grpc-related fixes

In reply to Thomas De Schampheleire

Thomas, All,

On 2021-02-03 13:43 +0100, Thomas De Schampheleire spake thusly:

Quoted 8 lines

From: Thomas De Schampheleire <[email protected]>

Commit fedf3318e3fd3c9ba57389ed2b36472f1a772b9e removed an obsolete patch on grpc, but this had the side effect that building collectd after grpc no longer succeeded, because grpc_cpp_plugin from the target was executed.

During investigation of this issue, a number of related problems were found. This series addresses these problems.

Series applied to master, thanks.

Quoted 5 lines

Thomas De Schampheleire (3):
package/collectd: stop passing '--with-libgrpc++' package/grpc: fix cross-compilation with
gRPC_BUILD_GRPC_CPP_PLUGIN=OFF
package/grpc: disable unnecessary build plugins

I'll further reply to this last patch, though, because I have some questions and suggestions.

Regards,
Yann E. MORIN.

Quoted 13 lines
 package/collectd/collectd.mk                  |  3 --
 ...ix-cross-compilation-with-gRPC_BUILD.patch | 43 +++++++++++++++++++
 package/grpc/grpc.mk                          |  9 +++-
 3 files changed, 51 insertions(+), 4 deletions(-)
 create mode 100644 package/grpc/0001-CMakeLists.txt-fix-cross-compilation-with-gRPC_BUILD.patch
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'