Skip to contentSkip to conversation

protobuf-c: fix race condition

PATCH In review Chen Qi 5 participants 9 messages Sep 29, 2019 – Oct 23, 2019

lore
Share & tools
lore

Chen Qi

author
#0 lore raw

PATCH 1/1 protobuf-c: fix race condition

---
 .../0001-avoid-race-condition.patch           | 36 +++++++++++++++++++
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644 meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +36 -0
new file mode 100644
index 000000000..4fc7703d8
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00 2001
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
+  DEBUG:	./t/test-full.pb.h:4:0: error: unterminated #ifndef
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+ 	t/generated-code2/cxx-generate-packed-data.cc \
+ 	t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS): t/test-full.pb.h
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+ 	$(AM_CXXFLAGS) \
+-- 
+2.17.1
+
meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -0
index 6d1ffc3f4..b92f82dec 100644
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
Signature · 1 line
2.17.1
#1 lore raw
Quoted 18 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36 +++++++++++++++++++
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644 meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
new file mode 100644
index 000000000..4fc7703d8
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00 2001
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch line 15

Quoted 10 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
+  DEBUG:	./t/test-full.pb.h:4:0: error: unterminated #ifndef
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:
https://www.mail-archive.com/[email protected]/msg64363.html

I think the upstream would be interested in this fix as well, see:
https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Cheers,

Quoted 41 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +21 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+ 	t/generated-code2/cxx-generate-packed-data.cc \
+ 	t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS): t/test-full.pb.h
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+ 	$(AM_CXXFLAGS) \
+-- 
+2.17.1
+
meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -0
index 6d1ffc3f4..b92f82dec 100644
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
Signature · 7 lines
2.17.1

-- 
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#2 lore raw

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’ is not a namespace-name

 using namespace foo;
                 ^~~
../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error: expected
namespace-name before ‘;’ token
 using namespace foo;
                    ^
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type
 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) }; ^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
};
^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of
macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =
{T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0) };
^~~~~~~~~~~~~~~~~~~~~~~~~
...
so it probably doesn't fix the issue completely.

Quoted 91 lines · Martin Jansa
Quoted 6 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36 +++++++++++++++++++
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 2 lines

diff --git

a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 4 lines

new file mode 100644
index 000000000..4fc7703d8
--- /dev/null
+++

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00 2001
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch line 15

Quoted 10 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
+  DEBUG:     ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:

https://www.mail-archive.com/[email protected]/msg64363.html

I think the upstream would be interested in this fix as well, see:

https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Cheers,

Review

Quoted 22 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +21 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+     t/generated-code2/cxx-generate-packed-data.cc \
+     t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS): t/test-full.pb.h
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+     $(AM_CXXFLAGS) \
+--
+2.17.1
+
(quoted diff) +0 -0

b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb

Quoted 14 lines

index 6d1ffc3f4..b92f82dec 100644

meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -1
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
--

2.17.1

Quoted 5 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Martin 'JaMa' Jansa jabber: [email protected]

#3 lore raw
Quoted 2 lines · Martin Jansa

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

Maybe, there are multiple issues as you mentioned. I'm monitoring our build pipeline for failures too.

I have always seen this:

+ DEBUG: ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+ ./t/test-full.pb.h:4:0: error: unterminated #ifndef

#4 lore raw
In reply to Martin Jansa

Yes I have seen same build failures on master even after this fix on machine with more cpus so the fix while seems to address the problem is not fixing is completely when I get a chance I will look into the logic myself too but I think we have to delve into it a bit More

Quoted 150 lines · Martin Jansa

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’ is not a namespace-name

 using namespace foo;
                 ^~~
../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error: expected
namespace-name before ‘;’ token
 using namespace foo;
                    ^
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type
 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) }; ^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
};
^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =

{T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0) };
^~~~~~~~~~~~~~~~~~~~~~~~~
...
so it probably doesn't fix the issue completely.

Quoted 8 lines · Martin Jansa
Quoted 6 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36 +++++++++++++++++++
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 3 lines
Quoted 2 lines

diff --git

a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 1 line

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 5 lines
Quoted 4 lines

new file mode 100644
index 000000000..4fc7703d8
--- /dev/null
+++

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 11 lines
Quoted 11 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00 2001
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
+  DEBUG:     ./t/test-full.pb.h:4:0: error: unterminated #ifndef
Quoted 10 lines

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch line 15

Quoted 5 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:

https://www.mail-archive.com/[email protected]/msg64363.html

Quoted 4 lines

I think the upstream would be interested in this fix as well, see:

https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Quoted 19 lines

Cheers,

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +6 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 10 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+     t/generated-code2/cxx-generate-packed-data.cc \
+     t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS):

t/test-full.pb.h

Quoted 21 lines

Review

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+     $(AM_CXXFLAGS) \
+--
+2.17.1
+
(quoted diff) +0 -0

b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb

Quoted 14 lines

index 6d1ffc3f4..b92f82dec 100644

meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -1
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
--

2.17.1

Quoted 9 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Martin 'JaMa' Jansa jabber: [email protected]

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#5 lore raw

One more data point is that it might be fixed completely with 1.3.2 version used in zeus/master and failing less often then without this patch in 1.3.1 used in thud/warrior.

I was rebuilding it 300 times in thud/warrior/zeus and got 5-6 failures in thud/warrior but none in zeus.

I've started another for loop to rebuilt it 1000 time with zeus and if it works then I'll retest warrior and thud with 1.3.2 upgrade backported to them - if it works then we can either backport whole upgrade or check the 1.3.1 1.3.2 diff to find out what else we need to backport from there.

Quoted 163 lines · Khem Raj

Yes I have seen same build failures on master even after this fix on machine with more cpus so the fix while seems to address the problem is not fixing is completely when I get a chance I will look into the logic myself too but I think we have to delve into it a bit More

Quoted 154 lines · Martin Jansa

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’ is
not a namespace-name

 using namespace foo;
                 ^~~
../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error:
expected
namespace-name before ‘;’ token
 using namespace foo;
                    ^
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type
 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) }; ^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
};
^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =

{T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0) };
^~~~~~~~~~~~~~~~~~~~~~~~~
...
so it probably doesn't fix the issue completely.

Quoted 4 lines · Martin Jansa
Quoted 3 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36

+++++++++++++++++++

Quoted 4 lines
Quoted 3 lines
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 3 lines
Quoted 2 lines

diff --git

a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 1 line

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 5 lines
Quoted 4 lines

new file mode 100644
index 000000000..4fc7703d8
--- /dev/null
+++

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 2 lines

Review line 1

Quoted 2 lines
(quoted diff) +1 -0
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00

2001

Quoted 9 lines
Quoted 9 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +9 -0
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
+  DEBUG:     ./t/test-full.pb.h:4:0: error: unterminated #ifndef
Quoted 10 lines

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch line 15

Quoted 5 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:

https://www.mail-archive.com/[email protected]/msg64363.html

Quoted 4 lines

I think the upstream would be interested in this fix as well, see:

https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Quoted 19 lines

Cheers,

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +6 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 10 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+     t/generated-code2/cxx-generate-packed-data.cc \
+     t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS):

t/test-full.pb.h

Quoted 21 lines

Review

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+     $(AM_CXXFLAGS) \
+--
+2.17.1
+
(quoted diff) +0 -0

b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb

Quoted 14 lines

index 6d1ffc3f4..b92f82dec 100644

meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -1
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
--

2.17.1

Quoted 9 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Martin 'JaMa' Jansa jabber: [email protected]

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#6 lore raw
Quoted 11 lines · Martin Jansa

One more data point is that it might be fixed completely with 1.3.2 version used in zeus/master and failing less often then without this patch in 1.3.1 used in thud/warrior.

I was rebuilding it 300 times in thud/warrior/zeus and got 5-6 failures in thud/warrior but none in zeus.

I've started another for loop to rebuilt it 1000 time with zeus and if it works then I'll retest warrior and thud with 1.3.2 upgrade backported to them - if it works then we can either backport whole upgrade or check the 1.3.1 1.3.2 diff to find out what else we need to backport from there.

1000 times in zeus = 0 failures
100 times in thud with 1.3.2 backported ~ 10 failures
1000 times in thud with 1.3.2 as well as protobuf-native 3.9.2 = 0 failures

The protobuf-c upgrade isn't need to be backported to thud, there aren't many changes:
https://github.com/protobuf-c/protobuf-c/compare/269771b4b45d3aba04e59569f53600003db8d9ff...1390409f4ee4e26d0635310995b516eb702c3f9e

protobuf on the other hand was upgraded from 3.6.1 to 3.9.2 in zeus and the diff is big (1028 commits):
https://github.com/protocolbuffers/protobuf/compare/48cb18e5c419ddd23d9badcfe4e9df7bde1979b2...52b2447247f535663ac1c292e088b4b27d2910ef

quick grep in git log didn't reveal what might be causing the race condition, upgrading whole protobuf in warrior and thud isn't an option as well.

Anyone interested in digging a bit deeper?

I don't want to block this change, because it seems to completely fix it for zeus/master and partially improve it also for warrior and thud.

We don't use protobuf much, so for our builds I think I'll just backport whole protobuf upgrade to thud once protobuf-c fix is in thud.

Regards,

Quoted 165 lines · Khem Raj
Quoted 164 lines

Yes I have seen same build failures on master even after this fix on machine with more cpus so the fix while seems to address the problem is not fixing is completely when I get a chance I will look into the logic myself too but I think we have to delve into it a bit More

Quoted 154 lines · Martin Jansa

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’ is
not a namespace-name

 using namespace foo;
                 ^~~
../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error:
expected
namespace-name before ‘;’ token
 using namespace foo;
                    ^
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type
 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) }; ^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
};
^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =

{T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0) };
^~~~~~~~~~~~~~~~~~~~~~~~~
...
so it probably doesn't fix the issue completely.

Quoted 4 lines · Martin Jansa
Quoted 3 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36

+++++++++++++++++++

Quoted 4 lines
Quoted 3 lines
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 3 lines
Quoted 2 lines

diff --git

a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 1 line

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 5 lines
Quoted 4 lines

new file mode 100644
index 000000000..4fc7703d8
--- /dev/null
+++

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 2 lines

Review line 1

Quoted 2 lines
(quoted diff) +1 -0
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00

2001

Quoted 9 lines
Quoted 9 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +9 -0
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
+  DEBUG:     ./t/test-full.pb.h:4:0: error: unterminated #ifndef
Quoted 10 lines

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch line 15

Quoted 5 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:

https://www.mail-archive.com/[email protected]/msg64363.html

Quoted 4 lines

I think the upstream would be interested in this fix as well, see:

https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Quoted 19 lines

Cheers,

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +6 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 10 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+     t/generated-code2/cxx-generate-packed-data.cc \
+     t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS):

t/test-full.pb.h

Quoted 21 lines

Review

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+     $(AM_CXXFLAGS) \
+--
+2.17.1
+
(quoted diff) +0 -0

b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb

Quoted 14 lines

index 6d1ffc3f4..b92f82dec 100644

meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -1
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
--

2.17.1

Quoted 9 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Martin 'JaMa' Jansa jabber: [email protected]

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#7 lore raw
Quoted 28 lines · Martin Jansa
Quoted 11 lines · Martin Jansa

One more data point is that it might be fixed completely with 1.3.2 version used in zeus/master and failing less often then without this patch in 1.3.1 used in thud/warrior.

I was rebuilding it 300 times in thud/warrior/zeus and got 5-6 failures in thud/warrior but none in zeus.

I've started another for loop to rebuilt it 1000 time with zeus and if it works then I'll retest warrior and thud with 1.3.2 upgrade backported to them - if it works then we can either backport whole upgrade or check the 1.3.1 1.3.2 diff to find out what else we need to backport from there.

1000 times in zeus = 0 failures
100 times in thud with 1.3.2 backported ~ 10 failures
1000 times in thud with 1.3.2 as well as protobuf-native 3.9.2 = 0 failures

The protobuf-c upgrade isn't need to be backported to thud, there aren't many changes:
https://github.com/protobuf-c/protobuf-c/compare/269771b4b45d3aba04e59569f53600003db8d9ff...1390409f4ee4e26d0635310995b516eb702c3f9e

protobuf on the other hand was upgraded from 3.6.1 to 3.9.2 in zeus and the diff is big (1028 commits):
https://github.com/protocolbuffers/protobuf/compare/48cb18e5c419ddd23d9badcfe4e9df7bde1979b2...52b2447247f535663ac1c292e088b4b27d2910ef

quick grep in git log didn't reveal what might be causing the race condition, upgrading whole protobuf in warrior and thud isn't an option as well.

Quoted 7 lines

Anyone interested in digging a bit deeper?

I don't want to block this change, because it seems to completely fix it for zeus/master and partially improve it also for warrior and thud.

We don't use protobuf much, so for our builds I think I'll just backport whole protobuf upgrade to thud once protobuf-c fix is in thud.

So is https://patchwork.openembedded.org/patch/165905/ still a go for warrior?

- armin

Quoted 158 lines

Regards,

Quoted 155 lines · Khem Raj
Quoted 154 lines

Yes I have seen same build failures on master even after this fix on machine with more cpus so the fix while seems to address the problem is not fixing is completely when I get a chance I will look into the logic myself too but I think we have to delve into it a bit More

Quoted 145 lines · Martin Jansa

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’ is
not a namespace-name

 using namespace foo;
                 ^~~
../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error:
expected
namespace-name before ‘;’ token
 using namespace foo;
                    ^
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type
 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) }; ^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
};
^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =

{T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0) };
^~~~~~~~~~~~~~~~~~~~~~~~~
...
so it probably doesn't fix the issue completely.

Quoted 4 lines · Martin Jansa
Quoted 3 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36

+++++++++++++++++++

Quoted 3 lines
Quoted 3 lines
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 1 line
Quoted 1 line

diff --git

a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 4 lines
Quoted 4 lines

new file mode 100644
index 000000000..4fc7703d8
--- /dev/null
+++

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 2 lines

Review line 1

Quoted 2 lines
(quoted diff) +1 -0
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00

2001

Quoted 8 lines
Quoted 8 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +8 -0
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
Quoted 10 lines

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch line 15

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +6 -0
+  DEBUG:     ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:

https://www.mail-archive.com/[email protected]/msg64363.html

Quoted 3 lines

I think the upstream would be interested in this fix as well, see:

https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Quoted 18 lines

Cheers,

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +6 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 10 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +10 -0
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+     t/generated-code2/cxx-generate-packed-data.cc \
+     t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS):

t/test-full.pb.h

Quoted 21 lines

Review

Quoted 6 lines
meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch new file +5 -0
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+     $(AM_CXXFLAGS) \
+--
+2.17.1
+
(quoted diff) +0 -0

b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb

Quoted 14 lines

index 6d1ffc3f4..b92f82dec 100644

meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -1
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
--

2.17.1

Quoted 8 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Martin 'JaMa' Jansa jabber: [email protected]

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#8 lore raw
Quoted 33 lines · On Wed, Oct 23, 2019 at 08:50:10AM -0700, akuster808 wrote:
Quoted 30 lines · Martin Jansa
Quoted 11 lines · Martin Jansa

One more data point is that it might be fixed completely with 1.3.2 version used in zeus/master and failing less often then without this patch in 1.3.1 used in thud/warrior.

I was rebuilding it 300 times in thud/warrior/zeus and got 5-6 failures in thud/warrior but none in zeus.

I've started another for loop to rebuilt it 1000 time with zeus and if it works then I'll retest warrior and thud with 1.3.2 upgrade backported to them - if it works then we can either backport whole upgrade or check the 1.3.1 1.3.2 diff to find out what else we need to backport from there.

1000 times in zeus = 0 failures
100 times in thud with 1.3.2 backported ~ 10 failures
1000 times in thud with 1.3.2 as well as protobuf-native 3.9.2 = 0 failures

The protobuf-c upgrade isn't need to be backported to thud, there aren't many changes:
https://github.com/protobuf-c/protobuf-c/compare/269771b4b45d3aba04e59569f53600003db8d9ff...1390409f4ee4e26d0635310995b516eb702c3f9e

protobuf on the other hand was upgraded from 3.6.1 to 3.9.2 in zeus and the diff is big (1028 commits):
https://github.com/protocolbuffers/protobuf/compare/48cb18e5c419ddd23d9badcfe4e9df7bde1979b2...52b2447247f535663ac1c292e088b4b27d2910ef

quick grep in git log didn't reveal what might be causing the race condition, upgrading whole protobuf in warrior and thud isn't an option as well.

Anyone interested in digging a bit deeper?

Quoted 8 lines
Quoted 5 lines

I don't want to block this change, because it seems to completely fix it for zeus/master and partially improve it also for warrior and thud.

We don't use protobuf much, so for our builds I think I'll just backport whole protobuf upgrade to thud once protobuf-c fix is in thud.

So is https://patchwork.openembedded.org/patch/165905/ still a go for warrior?

I think it is and for thud as well. Once there is a fix for protobuf as well, it should be in separate follow-up change anyway.

with protobuf-c fix included I got 28 failures in 1000 rebuilds without protobuf-c fix I got 5 failures in 308 rebuilds (still running)

so the rate seems similar if not the same (but there might be other aspects like load on the build machine from other builds), before I was seeing lower frequency of failures with this included.

Cheers,

Quoted 157 lines
Quoted 156 lines
Quoted 155 lines · Khem Raj
Quoted 154 lines

Yes I have seen same build failures on master even after this fix on machine with more cpus so the fix while seems to address the problem is not fixing is completely when I get a chance I will look into the logic myself too but I think we have to delve into it a bit More

Quoted 145 lines · Martin Jansa

Hmm I spoke too soon, at least rebuilding protobuf-c-native with this change backported to thud still failed once with:

../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’ is
not a namespace-name

 using namespace foo;
                 ^~~
../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error:
expected
namespace-name before ‘;’ token
 using namespace foo;
                    ^
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type
 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) }; ^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
};
^~~~~~~~~~~~~~~~~~~~~~~~~
../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
‘TestEnumSmall’ does not name a type

 #define TEST_ENUM_SMALL_TYPE_NAME  TestEnumSmall
                                    ^

../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of macro ‘TEST_ENUM_SMALL_TYPE_NAME’
TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =

{T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0) };
^~~~~~~~~~~~~~~~~~~~~~~~~
...
so it probably doesn't fix the issue completely.

Quoted 4 lines · Martin Jansa
Quoted 3 lines · Chen Qi
---
 .../0001-avoid-race-condition.patch           | 36

+++++++++++++++++++

Quoted 3 lines
Quoted 3 lines
 .../protobuf/protobuf-c_1.3.2.bb              |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644

meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 1 line
Quoted 1 line

diff --git

a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 4 lines
Quoted 4 lines

new file mode 100644
index 000000000..4fc7703d8
--- /dev/null
+++

b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch

Quoted 2 lines

Review line 1

Quoted 2 lines
(quoted diff) +1 -0
@@ -0,0 +1,36 @@
+From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00

2001

Quoted 8 lines
Quoted 8 lines
(quoted diff) +8 -0
+From: Chen Qi <[email protected]>
+Date: Sun, 29 Sep 2019 17:20:42 +0800
+Subject: [PATCH] avoid race condition
+
+It's possible that the cxx-generate-packed-data.cc is compiled
+while the t/test-full.pb.h is being generated. This will result
+the following error.
+
Quoted 10 lines

Review line 15

Quoted 6 lines
(quoted diff) +6 -0
+  DEBUG:     ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+  ./t/test-full.pb.h:4:0: error: unterminated #ifndef
+
+Add a dependency to avoid such problem.
+
+Upstream-Status: Pending

Thanks for this patch, it seems to indeed fix this issue, I've reported a while ago:

https://www.mail-archive.com/[email protected]/msg64363.html

Quoted 3 lines

I think the upstream would be interested in this fix as well, see:

https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c

Quoted 18 lines

Cheers,

Quoted 6 lines
(quoted diff) +6 -0
+
+Signed-off-by: Chen Qi <[email protected]>
+---
+ Makefile.am | 1 +
+ 1 file changed, 1 insertion(+)
+

Review line 31

Quoted 10 lines
(quoted diff) +10 -0
+diff --git a/Makefile.am b/Makefile.am
+index b0cb065..1608ae0 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
+ t_generated_code2_cxx_generate_packed_data_SOURCES = \
+     t/generated-code2/cxx-generate-packed-data.cc \
+     t/test-full.pb.cc
++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
+ $(t_generated_code2_cxx_generate_packed_data_OBJECTS):

t/test-full.pb.h

Quoted 21 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+ t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
+     $(AM_CXXFLAGS) \
+--
+2.17.1
+
(quoted diff) +0 -0

b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb

Quoted 14 lines

index 6d1ffc3f4..b92f82dec 100644

meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb +1 -1
@@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
 SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
 
 SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
+           file://0001-avoid-race-condition.patch \
           "
 
 S = "${WORKDIR}/git"
--

2.17.1

Quoted 8 lines
Quoted 5 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Martin 'JaMa' Jansa jabber: [email protected]

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]