Top-level parallel make improvements
v5 cover Top-level parallel make improvements
Hi all,
this patch set improve top-level parallel make support.
v5:
- improve the first patch commit message (Thomas P.)
v4:
- add a patch for luarocks packages
v3:
- add Acked-by (Arnout)
v2:
- use MAKEFLAGS to check if the "-j" option is used
- don't empty the PARALLEL_JOBS variable (Thomas P.)
Fabio Porcedda (2):
packages: fix and improve support for top-level parallel make
pkg-luarocks: fix top-level parallel makefile support
Makefile | 4 ++-- package/Makefile.in | 3 ++- package/luarocks/luarocks.mk | 6 ++++-- package/pkg-luarocks.mk | 4 ++-- 4 files changed, 10 insertions(+), 7 deletions(-)
Signature · 1 line
2.4.3
PATCH v5 1/2 packages: fix and improve support for top-level parallel make
The boost and jack2 packages fail to build when PARALLEL_JOBS is empty so instead of using an empty PARALLEL_JOBS don't use it in the MAKE variable when top-level parallel make is being used.
To simplify the use of top-level parallel make, check the MAKEFLAGS variable to know automatically if the -j option is being used, also use the "=" operator instead of the ":=" operator because the MAKEFLAGS variable can be checked only in a "recursively expanded variable". The "override" keyword must be used in order to change the automatic variable "MAKE".
When the top-parallel make is being used the sub-make are called without specifying the "-j" option in order to let GNU make share the job slots specified in the top make. This is done because GNU make is able to share the job slots available between each instance of make so if you want to increase the number of jobs you just need to increase the <jobs> value in the top make -j<jobs> command.
If we specify the -j<jobs> option in each instance of make, it is less efficient, e.g. in a processor with 8 cores we specify -j9 in each instance:
the number of processes goes up to 81 because each sub-make can execute 9 processes. The excessive number of processes is not a good thing because in my tests even -j16 is slower than -j9.
Instead if we don't specify the -j<jobs> option in the sub-make, the top make share the job slots automatically between each instance, so the number of process in this examples goes up to 9 that is faster than using up to 81 processes.
e.g. when the -j3 option is specified only in the top make:
possible state n. 1:
process 1 - <packagea>-build
process 2 - <packagea>-build
process 3 - <packagea>-build
possible state n. 2:
process 1 - <packagea>-extract
process 2 - <packageb>-configure
process 3 - <packagec>-build
possible state n. 3:
process 1 - <packagea>-build make -j1
process 2 - <packageb>-build make -j1
process 3 - <packagec>-build make -j1
--- Makefile | 4 ++-- package/Makefile.in | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
Makefile +2 -2
@@ -58,8 +58,8 @@ export HOSTARCH := $(shell uname -m | \ # # Taking into account the above considerations, if you still want to execute # this top-level Makefile in parallel comment the ".NOTPARALLEL" line and -# build using the following command: -# make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) +# use the -j<jobs> option when building, e.g: +# make -j$((`getconf _NPROCESSORS_ONLN`+1)) .NOTPARALLEL: # absolute path
package/Makefile.in +2 -1
@@ -18,7 +18,8 @@ PARALLEL_JOBS := $(BR2_JLEVEL) endif MAKE1 := $(HOSTMAKE) -j1 -MAKE := $(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS)) +override MAKE = $(HOSTMAKE) \ + $(if $(findstring j,$(filter-out --%,$(MAKEFLAGS))),,-j$(PARALLEL_JOBS)) ifeq ($(BR2_TOOLCHAIN_BUILDROOT),y) TARGET_VENDOR = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_VENDOR))
Signature · 1 line
2.4.3
Thomas Petazzoni
AppliedPATCH v5 1/2 packages: fix and improve support for top-level parallel make
Dear Fabio Porcedda,
Quoted 45 lines · Fabio Porcedda
The boost and jack2 packages fail to build when
PARALLEL_JOBSis empty so instead of using an emptyPARALLEL_JOBSdon't use it in the MAKE variable when top-level parallel make is being used.To simplify the use of top-level parallel make, check the MAKEFLAGS variable to know automatically if the -j option is being used, also use the "=" operator instead of the ":=" operator because the MAKEFLAGS variable can be checked only in a "recursively expanded variable". The "override" keyword must be used in order to change the automatic variable "MAKE".
When the top-parallel make is being used the sub-make are called without specifying the "-j" option in order to let GNU make share the job slots specified in the top make. This is done because GNU make is able to share the job slots available between each instance of make so if you want to increase the number of jobs you just need to increase the <jobs> value in the top make -j<jobs> command.
If we specify the -j<jobs> option in each instance of make, it is less efficient, e.g. in a processor with 8 cores we specify -j9 in each instance:
the number of processes goes up to 81 because each sub-make can execute 9 processes. The excessive number of processes is not a good thing because in my tests even -j16 is slower than -j9.
Instead if we don't specify the -j<jobs> option in the sub-make, the top make share the job slots automatically between each instance, so the number of process in this examples goes up to 9 that is faster than using up to 81 processes.e.g. when the -j3 option is specified only in the top make:
possible state n. 1:
process 1 - <packagea>-build
process 2 - <packagea>-build
process 3 - <packagea>-buildpossible state n. 2:
process 1 - <packagea>-extract
process 2 - <packageb>-configure
process 3 - <packagec>-buildpossible state n. 3:
process 1 - <packagea>-build make -j1
process 2 - <packageb>-build make -j1
process 3 - <packagec>-build make -j1
Quoted 6 lines
Signed-off-by Fabio Porcedda <[email protected]>--- Makefile | 4 ++-- package/Makefile.in | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
Applied, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v5 2/2 pkg-luarocks: fix top-level parallel makefile support
In reply to Fabio PorceddaIn the *-install-target phase the manifest file is being updated, if multiply packages try to update it they fail.
To avoid multiple access to the manifest file use flock to sync multiple luarocks packages.
e.g. installing three luarocks packages:
make lua-cjson-build lua-coat-build lua-coatpersistent-build make lua-cjson lua-coat lua-coatpersistent -j
Fix error:
Updating manifest for /home/tetsuya/buildroot/br2/output/target/usr/lib/luarocks/rocks No existing manifest. Attempting to rebuild...
Error: rock_manifest file not found for lua-coat 0.9.1-1 - not a LuaRocks 2 tree?
--- package/luarocks/luarocks.mk | 6 ++++-- package/pkg-luarocks.mk | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-)
package/luarocks/luarocks.mk +4 -2
@@ -53,8 +53,10 @@ endef $(eval $(host-generic-package)) -LUAROCKS_RUN = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua" \ - $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks +LUAROCKS_RUN_ENV = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua" +LUAROCKS_RUN_CMD = $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks + +LUAROCKS_RUN = $(LUAROCKS_RUN_ENV) $(LUAROCKS_RUN_CMD) define LUAROCKS_FINALIZE_TARGET rm -rf $(TARGET_DIR)/usr/lib/luarocks
package/pkg-luarocks.mk +2 -2
@@ -58,8 +58,8 @@ endif # ifndef $(2)_INSTALL_TARGET_CMDS define $(2)_INSTALL_TARGET_CMDS - cd $$($(2)_SRCDIR) && \ - $$(LUAROCKS_RUN) make --keep $$($(2)_ROCKSPEC) $$($(2)_BUILD_OPTS) + cd $$($(2)_SRCDIR) && $$(LUAROCKS_RUN_ENV) flock $$(TARGET_DIR) \ + $$(LUAROCKS_RUN_CMD) make --keep $$($(2)_ROCKSPEC) $$($(2)_BUILD_OPTS) endef endif
Signature · 1 line
2.4.3
PATCH v5 2/2 pkg-luarocks: fix top-level parallel makefile support
Dear Fabio Porcedda,
package/luarocks/luarocks.mk endef · line 59
Quoted 6 lines · Fabio Porcedda
package/luarocks/luarocks.mk +4
-2-LUAROCKS_RUN = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua" \ - $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks +LUAROCKS_RUN_ENV = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua" +LUAROCKS_RUN_CMD = $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks + +LUAROCKS_RUN = $(LUAROCKS_RUN_ENV) $(LUAROCKS_RUN_CMD)
I've removed completely LUAROCKS_RUN, and used LUAROCKS_RUN_ENV and LUAROCKS_RUN_CMD in the last place where LUAROCKS_RUN was used.
Applied with these changes, thanks!
However, won't we see some similar issues with other packages? Should we preventively do a flock on HOST_DIR/STAGING_DIR/TARGET_DIR when installing to host/staging/target respectively, so that we are sure that at any given time, we don't have two packages installing at the same time in one of the directories? Theoretically, this shouldn't be needed if all packages install their own files, or if they have the proper dependencies when they are overwriting files from another package.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v5 2/2 pkg-luarocks: fix top-level parallel makefile support
Quoted 8 lines · Thomas Petazzoni
However, won't we see some similar issues with other packages? Should we preventively do a flock on HOST_DIR/STAGING_DIR/TARGET_DIR when installing to host/staging/target respectively, so that we are sure that at any given time, we don't have two packages installing at the same time in one of the directories? Theoretically, this shouldn't be needed if all packages install their own files, or if they have the proper dependencies when they are overwriting files from another package.
There aren't many packages that would have this problem, but indeed a few. E.g.
packages installing kernel modules will typically call depmod. So yes, it would be a good idea to do that in general.
However, how are you going to do that in practice? flock works like fakeroot,
you have to run the commands from within the process. But all our INSTALL_*_CMDS are supposed to be called as separate shell scripts.
So we'd have to use something like dotlockfile, but I don't think we can assume
that's installed everywhere, can we? Or else we have to devise something similar around flock.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v5 2/2 pkg-luarocks: fix top-level parallel makefile support
Quoted 10 lines · Arnout Vandecappelle
Quoted 8 lines · Thomas Petazzoni
However, won't we see some similar issues with other packages? Should we preventively do a flock on HOST_DIR/STAGING_DIR/TARGET_DIR when installing to host/staging/target respectively, so that we are sure that at any given time, we don't have two packages installing at the same time in one of the directories? Theoretically, this shouldn't be needed if all packages install their own files, or if they have the proper dependencies when they are overwriting files from another package.
Quoted 3 lines
There aren't many packages that would have this problem, but indeed a few. E.g.
packages installing kernel modules will typically call depmod. So yes, it would be a good idea to do that in general.
Doing it in general is the simplest and safest solution, but to be the most efficient possible i was thinking to use the same solution of $(MAKE) and $(MAKE1) so we need to sync just a small part of packages.
Maybe we can try to test how much performance is lost if a global lock for all packages is used.
Quoted 7 lines
However, how are you going to do that in practice? flock works like fakeroot,
you have to run the commands from within the process. But all our INSTALL_*_CMDS are supposed to be called as separate shell scripts.So we'd have to use something like dotlockfile, but I don't think we can assume
that's installed everywhere, can we? Or else we have to devise something similar around flock.
I didn't know dotlockfile, it seems installed by default on Fedora 22 and Ubuntu 14.04.
Maybe the only downside of dotlockfile is that the sleep interval of 5 seconds is not configurable.
Anyway it seems a good solution for locking host/staging/target directories.
After merging the per-package staging feature we need to lock only host and target.
Finally after developing the per-package host we need to sync just the target directory.
Best regards
Signature · 1 line
Fabio Porcedda