Skip to contentSkip to conversation

protobuf: static link tools when generating sdk

PATCH In review Sinan Kaya 2 participants 11 messages Aug 17, 2021

lore
Share & tools
lore

Sinan Kaya

author
#0 lore raw

PATCH v3 1/5 protobuf: static link tools when generating sdk

Dynamically linked protoc.exe is failing as follows:

[libprotobuf ERROR google/protobuf/descriptor_database.cc:641] File already exists in database: google/protobuf/descriptor.proto [libprotobuf FATAL google/protobuf/descriptor.cc:1371] CHECK failed:
GeneratedDatabase()->Add(encoded_file_descriptor, size):

Switch to static linkage per upstream recommendation.

---
 recipes-devtools/protobuf/protobuf_%.bbappend | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 recipes-devtools/protobuf/protobuf_%.bbappend
recipes-devtools/protobuf/protobuf_%.bbappend new file +1 -0
new file mode 100644
index 0000000..b53b22d
@@ -0,0 +1 @@
+EXTRA_OECONF:append:sdkmingw32 = " --disable-shared"
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v3 2/5 protobuf-c: static link when generating sdk

[libprotobuf ERROR google/protobuf/descriptor_database.cc:641] File already exists in database: google/protobuf/descriptor.proto [libprotobuf FATAL google/protobuf/descriptor.cc:1371] CHECK failed:
GeneratedDatabase()->Add(encoded_file_descriptor, size):

Switch to static linkage per upstream recommendation.

---
 recipes-devtools/protobuf-c/protobuf-c_%.bbappend | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 recipes-devtools/protobuf-c/protobuf-c_%.bbappend
recipes-devtools/protobuf-c/protobuf-c_%.bbappend new file +1 -0
new file mode 100644
index 0000000..b53b22d
@@ -0,0 +1 @@
+EXTRA_OECONF:append:sdkmingw32 = " --disable-shared"
Signature · 1 line
2.17.1
author
#2 lore raw

PATCH v3 3/5 grpc: static link tools when generating SDK

In reply to Sinan Kaya

[libprotobuf ERROR google/protobuf/descriptor_database.cc:641] File already exists in database: google/protobuf/descriptor.proto [libprotobuf FATAL google/protobuf/descriptor.cc:1371] CHECK failed:
GeneratedDatabase()->Add(encoded_file_descriptor, size):

Switch to static linkage per upstream recommendation.

---
 recipes-devtools/grpc/grpc_%.bbappend | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 recipes-devtools/grpc/grpc_%.bbappend
recipes-devtools/grpc/grpc_%.bbappend new file +2 -0
new file mode 100644
index 0000000..4fcd8b9
@@ -0,0 +1,2 @@
+EXTRA_OECMAKE:remove:sdkmingw32 = "-DBUILD_SHARED_LIBS=ON"
+EXTRA_OECMAKE:append:sdkmingw32 = " -DBUILD_SHARED_LIBS=OFF"
Signature · 1 line
2.17.1
author
#3 lore raw

PATCH v3 4/5 abseil-cpp: disable shared build as it is broken

In reply to Sinan Kaya
---
 recipes-devtools/abseil-cpp/abseil-cpp_%.bbappend | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 recipes-devtools/abseil-cpp/abseil-cpp_%.bbappend
recipes-devtools/abseil-cpp/abseil-cpp_%.bbappend new file +1 -0
new file mode 100644
index 0000000..b73a8ea
@@ -0,0 +1 @@
+EXTRA_OECMAKE:remove:mingw32 = "-DBUILD_SHARED_LIBS=ON"
Signature · 1 line
2.17.1
author
#4 lore raw

PATCH v3 5/5 conf/layer.conf: use BBFILES_DYNAMIC for dynamic layers

In reply to Sinan Kaya

Add a dynamic BBFILES pattern so that patches for openembedded-layer are conditionally applied only if meta-oe is present.

---
 conf/layer.conf | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
conf/layer.conf +8 -1
index 5fefa73..d4823fc 100644
@@ -3,6 +3,12 @@ BBPATH := "${BBPATH}:${LAYERDIR}"
 
 # We have a packages directory, add to BBFILES
 BBFILES += "${LAYERDIR}/recipes*/*/*.bb ${LAYERDIR}/recipes*/*/*.bbappend"
+BBFILES_DYNAMIC += "\
+openembedded-layer:${LAYERDIR}/dynamic-layers/openembedded-layer/recipes-*/*/*.bb
+\
+openembedded-layer:${LAYERDIR}/dynamic-layers/openembedded-layer/recipes-*/*/*.bbappend
+\
+ "
 
 BBFILE_COLLECTIONS += "meta-mingw"
 BBFILE_PATTERN_meta-mingw := "^${LAYERDIR}/"
@@ -10,4 +16,5 @@ BBFILE_PRIORITY_meta-mingw = "8"
 
 LAYERDEPENDS_meta-mingw = "core"
 
-LAYERSERIES_COMPAT_meta-mingw = "honister"
\ No newline at end of file
+LAYERSERIES_COMPAT_meta-mingw = "honister"
+
Signature · 1 line
2.17.1
#5 lore raw
In reply to Sinan Kaya

I had to clean these up a little bit and pushed them to master-next. Please verify that they still work for you and if not used them as a starting point and send a V4 patchset.

Quoted 22 lines · Sinan Kaya

Dynamically linked protoc.exe is failing as follows:

[libprotobuf ERROR google/protobuf/descriptor_database.cc:641] File already exists in database: google/protobuf/descriptor.proto [libprotobuf FATAL google/protobuf/descriptor.cc:1371] CHECK failed:
GeneratedDatabase()->Add(encoded_file_descriptor, size):

Switch to static linkage per upstream recommendation.

---
  recipes-devtools/protobuf/protobuf_%.bbappend | 1 +
  1 file changed, 1 insertion(+)

create mode 100644 recipes-devtools/protobuf/protobuf_%.bbappend

recipes-devtools/protobuf/protobuf_%.bbappend new file +1 -0
new file mode 100644
index 0000000..b53b22d
@@ -0,0 +1 @@
+EXTRA_OECONF:append:sdkmingw32 = " --disable-shared"
author
#6 lore raw
Quoted 3 lines · Joshua Watt

I had to clean these up a little bit and pushed them to master-next. Please verify that they still work for you and if not used them as a starting point and send a V4 patchset.

Sounds good.

What do you think about this?

My reading of the problem says that the problem only happens on win32 platforms due to how DLLs initialize the heap.

We have use cases of both dynamic and static linkage for the target build that we have not seen any issues with.

I'd like to keep target builds same supporting both static and dynamic build but limit the tools to static link by using sdkmingw32 instead of mingw32.

Should I send a V4?

#7 lore raw
Quoted 6 lines · Sinan Kaya
Quoted 3 lines · Joshua Watt

I had to clean these up a little bit and pushed them to master-next. Please verify that they still work for you and if not used them as a starting point and send a V4 patchset.

Sounds good.

Quoted 7 lines

What do you think about this?

My reading of the problem says that the problem only happens on win32 platforms due to how DLLs initialize the heap.

We have use cases of both dynamic and static linkage for the target build that we have not seen any issues with.

Are you building with "mingw32" as the target (not as the SDK), and it works there? I wonder why it only fails for the SDK build and not the target build? If that is truly the case, then yes I suppose it should be the "sdkmingw32" override. The strange thing about that (and why I thought it was incorrect) is that we have a few recipes that disable shared libraries and/or enable static with just the "mingw32" override, which is why I assumed it was a general limitation of MinGW, not just the SDK. I looked through the recipes, and it does seem more apparent that it is inconsistent, with a few recipe using "mingw32", a few using "class-nativesdk:mingw32", and a few using "sdkmingw32".

Anyway, please send a V4 if it needs to be changed, and I apologize for changing it unnecessarily :)

Quoted 6 lines

I'd like to keep target builds same supporting both static and dynamic build but limit the tools to static link by using sdkmingw32 instead of mingw32.

Should I send a V4?

author
#8 lore raw
Quoted 14 lines · Joshua Watt
Quoted 2 lines

We have use cases of both dynamic and static linkage for the target build that we have not seen any issues with.

Are you building with "mingw32" as the target (not as the SDK), and it works there? I wonder why it only fails for the SDK build and not the target build? If that is truly the case, then yes I suppose it should be the "sdkmingw32" override. The strange thing about that (and why I thought it was incorrect) is that we have a few recipes that disable shared libraries and/or enable static with just the "mingw32" override, which is why I assumed it was a general limitation of MinGW, not just the SDK. I looked through the recipes, and it does seem more apparent that it is inconsistent, with a few recipe using "mingw32", a few using "class-nativesdk:mingw32", and a few using "sdkmingw32".

I see two classes of problems based on my engagement with the mingw32. Some recipes just don't build without using the static method for the SDK and the target both. (abseil-cpp is one of them)

The nativesdk problem for grpc and protobuf is a silent failure. Even though we are able to build the binary, we get a binary that just doesn't work on windows. (crashes upon execution)

I still would like to be able to static/dynamic link grpc/protobuf for my target using the SDK.

Quoted 3 lines

Anyway, please send a V4 if it needs to be changed, and I apologize for changing it unnecessarily :)

Will do.

#9 lore raw
Quoted 18 lines · Sinan Kaya
Quoted 13 lines · Joshua Watt
Quoted 2 lines

We have use cases of both dynamic and static linkage for the target build that we have not seen any issues with.

Are you building with "mingw32" as the target (not as the SDK), and it works there? I wonder why it only fails for the SDK build and not the target build? If that is truly the case, then yes I suppose it should be the "sdkmingw32" override. The strange thing about that (and why I thought it was incorrect) is that we have a few recipes that disable shared libraries and/or enable static with just the "mingw32" override, which is why I assumed it was a general limitation of MinGW, not just the SDK. I looked through the recipes, and it does seem more apparent that it is inconsistent, with a few recipe using "mingw32", a few using "class-nativesdk:mingw32", and a few using "sdkmingw32".

I see two classes of problems based on my engagement with the mingw32. Some recipes just don't build without using the static method for the SDK and the target both. (abseil-cpp is one of them)

Quoted 6 lines

The nativesdk problem for grpc and protobuf is a silent failure. Even though we are able to build the binary, we get a binary that just doesn't work on windows. (crashes upon execution)

I still would like to be able to static/dynamic link grpc/protobuf for my target using the SDK.

RIght, the "mingw32" override shouldn't affect what you do with the *target* unless your target is MinGW; I'm not sure if anyone is actually doing that, most SDKs target Linux for ARM, Linux for AArch64, Linux for x86, etc.

What is your target?

It's important to remember that the recipe can be built differently for the SDK vs the target :)

Quoted 4 lines
Quoted 3 lines

Anyway, please send a V4 if it needs to be changed, and I apologize for changing it unnecessarily :)

Will do.

author
#10 lore raw
Quoted 8 lines · Joshua Watt
Quoted 2 lines

I still would like to be able to static/dynamic link grpc/protobuf for my target using the SDK.

RIght, the "mingw32" override shouldn't affect what you do with the *target* unless your target is MinGW; I'm not sure if anyone is actually doing that, most SDKs target Linux for ARM, Linux for AArch64, Linux for x86, etc.

Quoted 5 lines

What is your target?

It's important to remember that the recipe can be built differently for the SDK vs the target :)

ah, mine is both arm and arm64. No need to touch the recipe then.