qt5webengine: propagate parallel flag to ninja

PATCH New Gaël PORTAY 3 participants 3 messages Jun 29, 2018 – Jul 1, 2018

lore

Share & tools

lore

Gaël PORTAY

author

2018-06-29 14:35 UTC #0 lore raw

PATCH v2 1/1 qt5webengine: propagate parallel flag to ninja

WebEngine is a big project to build. It uses ninja which is a build system that does everything in parallel[1].

Builds are always run in parallel, based by default on the number of CPUs your system has. Underspecified build dependencies will result in incorrect builds.

QtWebEngine does not propagate the parallel flag from the running instance of make to ninja. Thus, all the machine cores are used to build this single project.

	make -j 10 -> ninja -j # all cores

This behavior disrupts powerful machines which are setup to build many projects in parallel. Even worse, the build fails on machines where the ratio CPU-cores / amount of RAM (including swap) is not appropriate. g++ may have not enough of memory to build all those files in parallel.

Unfortunatly, the target `run_ninja' is hardcoded in the Makefile src/core/Makefile.run_gn; there is no way to propagate flags using a variable through the make command-line.

	run_ninja:
	        /home/gportay/src/buildroot/output-qt5.11/build/qt5webengine-5.11.1/src/3rdparty/ninja/ninja

-C /home/gportay/src/buildroot/output-qt5.11/build/qt5webengine-5.11.1/src/core/release QtWebEngineCore

Luckily, the Qt gn_run.pro[2] can hardcode ninja extra flags through the variable NINJAFLAGS when the Makefile is generated (which is performed during the build and not during configure step).

This commit sets the NINJAFLAGS using the variable PARALLEL_JOBS to limit the number of cores used by ninja.

Fixes:
virtual memory exhausted: Cannot allocate memory

[1]: https://ninja-build.org/manual.html#_comparison_to_make
[2]: https://github.com/qt/qtwebengine/blob/v5.11.1/src/core/gn_run.pro#L49-L53

CC Damien Riegel <[email protected]>

CC Jean-Fran?ois T?tu <[email protected]>

Signed-off-by Ga?l PORTAY <[email protected]>

---
Changes since v1:
- Use PARALLEL_JOBS instead of BR2_JLEVEL
- Reword commit message

 package/qt5/qt5webengine/qt5webengine.mk | 2 ++
 1 file changed, 2 insertions(+)

package/qt5/qt5webengine/qt5webengine.mk +2 -0

index 08120421d6..2c622c91ba 100644
@@ -48,6 +48,8 @@ define QT5WEBENGINE_PYTHON2_SYMLINK
 endef
 QT5WEBENGINE_PRE_CONFIGURE_HOOKS += QT5WEBENGINE_PYTHON2_SYMLINK
 
+QT5WEBENGINE_ENV += NINJAFLAGS="-j$(PARALLEL_JOBS)"
+
 ifeq ($(BR2_PACKAGE_QT5_VERSION_LATEST),y)
 define QT5WEBENGINE_CREATE_HOST_PKG_CONFIG
 	sed s%@HOST_DIR@%$(HOST_DIR)%g $(QT5WEBENGINE_PKGDIR)/host-pkg-config.in > $(@D)/host-bin/host-pkg-config

Signature · 1 line

2.17.1

Yann E. MORIN

2018-06-30 17:07 UTC #1 lore raw

PATCH v2 1/1 qt5webengine: propagate parallel flag to ninja

Ga?l, All,

On 2018-06-29 10:35 -0400, Ga?l PORTAY spake thusly:

Quoted 36 lines

WebEngine is a big project to build. It uses ninja which is a build system that does everything in parallel[1].
Builds are always run in parallel, based by default on the number of CPUs your system has. Underspecified build dependencies will result in incorrect builds.
QtWebEngine does not propagate the parallel flag from the running instance of make to ninja. Thus, all the machine cores are used to build this single project.
	make -j 10 -> ninja -j # all cores
This behavior disrupts powerful machines which are setup to build many projects in parallel. Even worse, the build fails on machines where the ratio CPU-cores / amount of RAM (including swap) is not appropriate. g++ may have not enough of memory to build all those files in parallel.
Unfortunatly, the target `run_ninja' is hardcoded in the Makefile src/core/Makefile.run_gn; there is no way to propagate flags using a variable through the make command-line.
	run_ninja:
	        /home/gportay/src/buildroot/output-qt5.11/build/qt5webengine-5.11.1/src/3rdparty/ninja/ninja
-C /home/gportay/src/buildroot/output-qt5.11/build/qt5webengine-5.11.1/src/core/release QtWebEngineCore
Luckily, the Qt gn_run.pro[2] can hardcode ninja extra flags through the variable NINJAFLAGS when the Makefile is generated (which is performed during the build and not during configure step).
This commit sets the NINJAFLAGS using the variable PARALLEL_JOBS to limit the number of cores used by ninja.
Fixes:
virtual memory exhausted: Cannot allocate memory

Quoted 6 lines

[1]: https://ninja-build.org/manual.html#_comparison_to_make
[2]: https://github.com/qt/qtwebengine/blob/v5.11.1/src/core/gn_run.pro#L49-L53
CC Damien Riegel <[email protected]>
CC Jean-Fran?ois T?tu <[email protected]>
Signed-off-by Ga?l PORTAY <[email protected]>

Acked-by "Yann E. MORIN" <[email protected]>

Regards,
Yann E. MORIN.

Quoted 28 lines

---
Changes since v1:
- Use PARALLEL_JOBS instead of BR2_JLEVEL
- Reword commit message

 package/qt5/qt5webengine/qt5webengine.mk | 2 ++
 1 file changed, 2 insertions(+)

package/qt5/qt5webengine/qt5webengine.mk +2 -0

index 08120421d6..2c622c91ba 100644
@@ -48,6 +48,8 @@ define QT5WEBENGINE_PYTHON2_SYMLINK
 endef
 QT5WEBENGINE_PRE_CONFIGURE_HOOKS += QT5WEBENGINE_PYTHON2_SYMLINK
 
+QT5WEBENGINE_ENV += NINJAFLAGS="-j$(PARALLEL_JOBS)"
+
 ifeq ($(BR2_PACKAGE_QT5_VERSION_LATEST),y)
 define QT5WEBENGINE_CREATE_HOST_PKG_CONFIG
 	sed s%@HOST_DIR@%$(HOST_DIR)%g $(QT5WEBENGINE_PKGDIR)/host-pkg-config.in > $(@D)/host-bin/host-pkg-config

Signature · 6 lines

2.17.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Thomas Petazzoni

Applied

2018-07-01 09:05 UTC #2 lore raw

PATCH v2 1/1 qt5webengine: propagate parallel flag to ninja

In reply to Gaël PORTAY

Hello,

Quoted 39 lines · Ga?l PORTAY

WebEngine is a big project to build. It uses ninja which is a build system that does everything in parallel[1].
Builds are always run in parallel, based by default on the number of CPUs your system has. Underspecified build dependencies will result in incorrect builds.
QtWebEngine does not propagate the parallel flag from the running instance of make to ninja. Thus, all the machine cores are used to build this single project.
	make -j 10 -> ninja -j # all cores
This behavior disrupts powerful machines which are setup to build many projects in parallel. Even worse, the build fails on machines where the ratio CPU-cores / amount of RAM (including swap) is not appropriate. g++ may have not enough of memory to build all those files in parallel.
Unfortunatly, the target `run_ninja' is hardcoded in the Makefile src/core/Makefile.run_gn; there is no way to propagate flags using a variable through the make command-line.
	run_ninja:
	        /home/gportay/src/buildroot/output-qt5.11/build/qt5webengine-5.11.1/src/3rdparty/ninja/ninja
-C /home/gportay/src/buildroot/output-qt5.11/build/qt5webengine-5.11.1/src/core/release QtWebEngineCore
Luckily, the Qt gn_run.pro[2] can hardcode ninja extra flags through the variable NINJAFLAGS when the Makefile is generated (which is performed during the build and not during configure step).
This commit sets the NINJAFLAGS using the variable PARALLEL_JOBS to limit the number of cores used by ninja.
Fixes:
virtual memory exhausted: Cannot allocate memory
[1]: https://ninja-build.org/manual.html#_comparison_to_make
[2]: https://github.com/qt/qtwebengine/blob/v5.11.1/src/core/gn_run.pro#L49-L53

Quoted 7 lines

CC Damien Riegel <[email protected]>
CC Jean-Fran?ois T?tu <[email protected]>
Signed-off-by Ga?l PORTAY <[email protected]>
---
Changes since v1:
- Use PARALLEL_JOBS instead of BR2_JLEVEL
- Reword commit message

Applied to master, thanks.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com