Skip to contentSkip to conversation

bison: Reset load average settings if specified in environment

PATCH In review Khem Raj 6 participants 10 messages Mar 15, 2020 – Apr 29, 2020

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

PATCH 1/1 bison: Reset load average settings if specified in environment

In some cases, we run into parallel build failures where BUILT_SOURCES is skipped, as a result required header files are not generated and the build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore ensure that make is invoked with all target

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

[1] https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

---
 meta/recipes-devtools/bison/bison_3.5.3.bb | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-devtools/bison/bison_3.5.3.bb +6 -0
index 09e4b18f9e..27e09434f8 100644
@@ -18,6 +18,12 @@ SRC_URI[sha256sum] = "2bf85b5f88a5f2fa8069aed2a2dfc3a9f8d15a97e59c713e3906e5fdd9
 # No point in hardcoding path to m4, just use PATH
 EXTRA_OECONF += "M4=m4"
 
+# Reset any loadavg set via environment, it breaks parallel build
+# | ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No such file or directory
+# |  #include "uniwidth.h"
+# |           ^~~~~~~~~~~~
+EXTRA_OEMAKE_append = " -l"
+
 inherit autotools gettext texinfo
 
 # The automatic m4 path detection gets confused, so force the right value
Signature · 1 line
2.25.1
#1 lore raw

✗ patchtest: failure for bison: Reset load average settings if specified in environment

== Series Details ==

Series: bison: Reset load average settings if specified in environment
Revision: 1
URL : https://patchwork.openembedded.org/series/23244/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at b1ae2178fa)

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
#2 lore raw
In reply to Khem Raj

Shouldn’t the race be fixed? This might still happen because -j still parallelises the build,no?

Alex

Quoted 55 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES is skipped, as a result required header files are not generated and the build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore ensure that make is invoked with all target

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

[1]
https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

---
 meta/recipes-devtools/bison/bison_3.5.3.bb | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-devtools/bison/bison_3.5.3.bb +0 -0
index 09e4b18f9e..27e09434f8 100644
@@ -18,6 +18,12 @@ SRC_URI[sha256sum] =

"2bf85b5f88a5f2fa8069aed2a2dfc3a9f8d15a97e59c713e3906e5fdd9
# No point in hardcoding path to m4, just use PATH EXTRA_OECONF += "M4=m4"

meta/recipes-devtools/bison/bison_3.5.3.bb +2 -0
+# Reset any loadavg set via environment, it breaks parallel build
+# | ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:

No such file or directory

meta/recipes-devtools/bison/bison_3.5.3.bb +4 -1
+# |  #include "uniwidth.h"
+# |           ^~~~~~~~~~~~
+EXTRA_OEMAKE_append = " -l"
+
 inherit autotools gettext texinfo
 
 # The automatic m4 path detection gets confused, so force the right value
--

2.25.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

author
#3 lore raw
Quoted 3 lines · Alexander Kanavin

Shouldn’t the race be fixed? This might still happen because -j still parallelises the build,no?

It perhaps is a make bug which only triggers with -j and -l being both present it’s perhaps easily reproducible using these together on bison builds
As such I don’t see anything wrong with bison makery

Quoted 60 lines

Alex

Quoted 55 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES is skipped, as a result required header files are not generated and the build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore ensure that make is invoked with all target

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

[1]
https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

---
 meta/recipes-devtools/bison/bison_3.5.3.bb | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-devtools/bison/bison_3.5.3.bb +0 -0
index 09e4b18f9e..27e09434f8 100644
@@ -18,6 +18,12 @@ SRC_URI[sha256sum] =

"2bf85b5f88a5f2fa8069aed2a2dfc3a9f8d15a97e59c713e3906e5fdd9
# No point in hardcoding path to m4, just use PATH EXTRA_OECONF += "M4=m4"

meta/recipes-devtools/bison/bison_3.5.3.bb +2 -0
+# Reset any loadavg set via environment, it breaks parallel build
+# | ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:

No such file or directory

meta/recipes-devtools/bison/bison_3.5.3.bb +4 -1
+# |  #include "uniwidth.h"
+# |           ^~~~~~~~~~~~
+EXTRA_OEMAKE_append = " -l"
+
 inherit autotools gettext texinfo
 
 # The automatic m4 path detection gets confused, so force the right value
--

2.25.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#4 lore raw
Quoted 10 lines · Khem Raj
Quoted 2 lines · Alexander Kanavin

Shouldn’t the race be fixed? This might still happen because -j still parallelises the build,no?

It perhaps is a make bug which only triggers with -j and -l being both present it’s perhaps easily reproducible using these together on bison builds
As such I don’t see anything wrong with bison makery

This is the second bison-native problem in addition to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13825

It is possible that the gnulib issue was a red herring and it is actually is a parallel build issue, but both being in bison-native does point more towards a problem specific to bison.

cu
Adrian

#5 lore raw
In reply to Khem Raj
Quoted 16 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES
is skipped, as a result required header files are not generated and the
build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore
ensure that make is invoked with all target

Quoted 7 lines

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

[1]
https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

I'm not sure I understand this. We've seen a failure on our ubuntu1604 performance tester which I think is the same bug as this. The environment on that machine doesn't change, so how/where is this -l option coming from?

I suspect there is some issue we're not quite getting to still :(

Cheers,

Richard

author
#6 lore raw
Quoted 23 lines · Richard Purdie
Quoted 21 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES
is skipped, as a result required header files are not generated and the
build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore
ensure that make is invoked with all target

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

Quoted 8 lines
Quoted 2 lines

[1]
https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

I'm not sure I understand this. We've seen a failure on our ubuntu1604 performance tester which I think is the same bug as this. The environment on that machine doesn't change, so how/where is this -l option coming from?

From custom distro global settings for EXTRA_OEMAKE

Quoted 2 lines

I suspect there is some issue we're not quite getting to still :(

its well reproducible now with -j and -l together on our builders, From what
I see in makefiles bison is doing right thing in using BUILT_SOURCES so I suspect it could be make acting in a certain way when those options are present.

our custom settings use -l options globally in EXTRA_OEMAKE this is to let machines with certain configurations build when machines are shared, we don't want a given build to use all resources on box. Yes one could say why not do something else to limit the resources build uses but this is a global setting that works well

This patch just ensures that any -l setting passed from environment is cleaned up and reset. So for general case it will be a noop.

Quoted 4 lines

Cheers,

Richard

#7 lore raw
Quoted 42 lines · Khem Raj
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES
is skipped, as a result required header files are not generated and the
build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore
ensure that make is invoked with all target

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

Quoted 8 lines
Quoted 2 lines

[1]
https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

I'm not sure I understand this. We've seen a failure on our ubuntu1604 performance tester which I think is the same bug as this. The environment on that machine doesn't change, so how/where is this -l option coming from?

From custom distro global settings for EXTRA_OEMAKE

Quoted 2 lines

I suspect there is some issue we're not quite getting to still :(

its well reproducible now with -j and -l together on our builders, From what
I see in makefiles bison is doing right thing in using BUILT_SOURCES so I suspect it could be make acting in a certain way when those options are present.

Quoted 8 lines

our custom settings use -l options globally in EXTRA_OEMAKE this is to let machines with certain configurations build when machines are shared, we don't want a given build to use all resources on box. Yes one could say why not do something else to limit the resources build uses but this is a global setting that works well

This patch just ensures that any -l setting passed from environment is cleaned up and reset. So for general case it will be a noop.

Ok, that explains your failure but not the one on our performance testing worker :/.

Cheers,

Richard

author
#8 lore raw
Quoted 52 lines · Richard Purdie
Quoted 19 lines · Khem Raj
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES
is skipped, as a result required header files are not generated and the
build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore
ensure that make is invoked with all target

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

[1]

https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

Quoted 7 lines
Quoted 6 lines

I'm not sure I understand this. We've seen a failure on our ubuntu1604 performance tester which I think is the same bug as this. The environment on that machine doesn't change, so how/where is this -l option coming from?

Quoted 5 lines

From custom distro global settings for EXTRA_OEMAKE

Quoted 2 lines

I suspect there is some issue we're not quite getting to still :(

its well reproducible now with -j and -l together on our builders, From

what

Quoted 6 lines

I see in makefiles bison is doing right thing in using BUILT_SOURCES so I suspect it could be make acting in a certain way when those options are present.

our custom settings use -l options globally in EXTRA_OEMAKE this is to let machines with certain configurations build when machines

are

Quoted 4 lines

shared, we don't want a given build to use all resources on box. Yes one could say why not do something else to limit the resources build uses but this is a global setting that works well

Quoted 5 lines
Quoted 2 lines

This patch just ensures that any -l setting passed from environment is cleaned up and reset. So for general case it will be a noop.

Ok, that explains your failure but not the one on our performance testing worker :/.

What errors do you see on perf testing worker is it exactly same ?

Quoted 7 lines

Cheers,

Richard

#9 lore raw
Quoted 49 lines · Khem Raj
Quoted 48 lines · Richard Purdie
Quoted 47 lines · Khem Raj
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Khem Raj

In some cases, we run into parallel build failures where BUILT_SOURCES
is skipped, as a result required header files are not generated and the
build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
No such file or directory

 #include "uniwidth.h"
          ^~~~~~~~~~~~

compilation terminated.

BUILT_SOURCES should be built automatically with make all [1] therefore
ensure that make is invoked with all target

bison-native parallel build fails when -l<n> is passed globally from build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

Quoted 7 lines
Quoted 2 lines

[1]
https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

I'm not sure I understand this. We've seen a failure on our ubuntu1604 performance tester which I think is the same bug as this. The environment on that machine doesn't change, so how/where is this -l option coming from?

From custom distro global settings for EXTRA_OEMAKE

Quoted 2 lines

I suspect there is some issue we're not quite getting to still :(

its well reproducible now with -j and -l together on our builders, From what
I see in makefiles bison is doing right thing in using BUILT_SOURCES so I suspect it could be make acting in a certain way when those options are present.

our custom settings use -l options globally in EXTRA_OEMAKE this is to let machines with certain configurations build when machines are shared, we don't want a given build to use all resources on box. Yes one could say why not do something else to limit the resources build uses but this is a global setting that works well

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

This patch just ensures that any -l setting passed from environment is cleaned up and reset. So for general case it will be a noop.

Ok, that explains your failure but not the one on our performance testing worker :/.

I've heard reports of this still being an issue with latest OE 3.1 when -j is set high enough, without any -l options in EXTRA_OEMAKE (so Khem's workaround to over-ride -l doesn't help).

Disabling PARALLEL_MAKE completely might be the best workaround for now until we understand the real root cause.