Fix for top-level parallel make part 1
v3 cover Fix for top-level parallel make part 1
Hi all,
this is the first patch set for fixing top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefile is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
With this patch set the package building seems to work fine with top-level parallel make, example:
make clean toolchain
make BR2_JLEVEL= -j<n> <package-name list>
Before to try that you have to remove the ".NOTPARALLEL" line in the top-level Makefile.
The part that remain to fix for top-level parallel make is the toolchain building and targets after building all the packages.
Best regards
Fabio Porcedda
v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the first patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.
Fabio Porcedda (3):
package/Makefile.in: add a way to don't force jobs in sub-make
pkg-generic: fix rules for top-level parallel make
package: add toolchain dependency to inner-generic-package
Makefile | 2 +- package/Makefile.in | 2 +- package/pkg-autotools.mk | 3 ++- package/pkg-generic.mk | 51 ++++++++++++++++++++++++------------------------ package/uclibc/uclibc.mk | 2 +- 5 files changed, 31 insertions(+), 29 deletions(-)
Signature · 1 line
1.8.4
PATCH v3 1/3 package/Makefile.in: add a way to don't force jobs in sub-make
When the "BR2_JLEVEL" variable is empty use "make" without the "-j" option.
To be able to use top-level parallel make we must don't force the number of jobs in sub-make.
Example:
make BR2_JLEVEL= -j8
--- package/Makefile.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/Makefile.in +1 -1
@@ -18,7 +18,7 @@ PARALLEL_JOBS:=$(BR2_JLEVEL) endif MAKE1:=$(HOSTMAKE) -j1 -MAKE:=$(HOSTMAKE) -j$(PARALLEL_JOBS) +MAKE:=$(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS)) # Compute GNU_TARGET_NAME GNU_TARGET_NAME=$(ARCH)-buildroot-$(TARGET_OS)-$(LIBC)$(ABI)
Signature · 1 line
1.8.4
PATCH v3 1/3 package/Makefile.in: add a way to don't force jobs in sub-make
Dear Fabio Porcedda,
Quoted 10 lines · Fabio Porcedda
When the "
BR2_JLEVEL" variable is empty use "make" without the "-j" option.
To be able to use top-level parallel make we must don't force the number of jobs in sub-make.Example:
make BR2_JLEVEL= -j8Signed-off-by Fabio Porcedda <[email protected]>
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH v3 2/3 pkg-generic: fix rules for top-level parallel make
In reply to Fabio PorceddaTo be able to use top-level parallel make we must don't depend in a rule on the order of evaluation of the prerequisites, so instead of reling on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:
%/.stamp_extracted
%/.stamp_patched
%/.stamp_configured
%/.stamp_built
%/.stamp_host_installed %/.stamp_staging_installed %/.stamp_images_installed %/.stamp_target_installed
Because the %-build target is not anymore part of the dependcy chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by the uclibc package for building the toolchain.
--- package/pkg-generic.mk | 45 +++++++++++++++++++++------------------------ package/uclibc/uclibc.mk | 2 +- 2 files changed, 22 insertions(+), 25 deletions(-)
package/pkg-generic.mk +21 -24
@@ -53,7 +53,7 @@ ifeq ($(DL_MODE),DOWNLOAD) endif # Unpack the archive -$(BUILD_DIR)/%/.stamp_extracted: +$(BUILD_DIR)/%/.stamp_extracted: $(BUILD_DIR)/%/.stamp_downloaded @$(call MESSAGE,"Extracting") $(Q)mkdir -p $(@D) $($(PKG)_EXTRACT_CMDS)
@@ -88,7 +88,7 @@ endif # prefix of the patches $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION) $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME) -$(BUILD_DIR)/%/.stamp_patched: +$(BUILD_DIR)/%/.stamp_patched: $(BUILD_DIR)/%/.stamp_extracted @$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)") $(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep)) $(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -115,21 +115,21 @@ $(BUILD_DIR)/%/.stamp_configured: $(Q)touch $@ # Build -$(BUILD_DIR)/%/.stamp_built:: +$(BUILD_DIR)/%/.stamp_built: $(BUILD_DIR)/%/.stamp_configured @$(call MESSAGE,"Building") $($(PKG)_BUILD_CMDS) $(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@ # Install to host dir -$(BUILD_DIR)/%/.stamp_host_installed: +$(BUILD_DIR)/%/.stamp_host_installed: $(BUILD_DIR)/%/.stamp_built @$(call MESSAGE,"Installing to host directory") $($(PKG)_INSTALL_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@ # Install to staging dir -$(BUILD_DIR)/%/.stamp_staging_installed: +$(BUILD_DIR)/%/.stamp_staging_installed: $(BUILD_DIR)/%/.stamp_built @$(call MESSAGE,"Installing to staging directory") $($(PKG)_INSTALL_STAGING_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -143,14 +143,14 @@ $(BUILD_DIR)/%/.stamp_staging_installed: $(Q)touch $@ # Install to images dir -$(BUILD_DIR)/%/.stamp_images_installed: +$(BUILD_DIR)/%/.stamp_images_installed: $(BUILD_DIR)/%/.stamp_built @$(call MESSAGE,"Installing to images directory") $($(PKG)_INSTALL_IMAGES_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@ # Install to target dir -$(BUILD_DIR)/%/.stamp_target_installed: +$(BUILD_DIR)/%/.stamp_target_installed: $(BUILD_DIR)/%/.stamp_built @$(call MESSAGE,"Installing to target") $(if $(BR2_INIT_SYSTEMD),\ $($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -308,6 +308,7 @@ $(2)_REDISTRIBUTE ?= YES $(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +$(2)_BUILD_DEPENDENCIES ?= $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO
@@ -359,30 +360,29 @@ $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images endif ifeq ($$($(2)_INSTALL_TARGET),YES) -$(1)-install-target: $(1)-build \ - $$($(2)_TARGET_INSTALL_TARGET) +$(1)-install-target: $$($(2)_TARGET_INSTALL_TARGET) else $(1)-install-target: endif ifeq ($$($(2)_INSTALL_STAGING),YES) -$(1)-install-staging: $(1)-build \ - $$($(2)_TARGET_INSTALL_STAGING) +$(1)-install-staging: $$($(2)_TARGET_INSTALL_STAGING) else $(1)-install-staging: endif ifeq ($$($(2)_INSTALL_IMAGES),YES) -$(1)-install-images: $(1)-build \ - $$($(2)_TARGET_INSTALL_IMAGES) +$(1)-install-images: $$($(2)_TARGET_INSTALL_IMAGES) else $(1)-install-images: endif -$(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) +$(1)-install-host: $$($(2)_TARGET_INSTALL_HOST) -$(1)-build: $(1)-configure \ - $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_BUILD): | $$($(2)_BUILD_DEPENDENCIES) +$(1)-build: $$($(2)_TARGET_BUILD) + +$(1)-configure: $$($(2)_TARGET_CONFIGURE) ifeq ($$($(2)_OVERRIDE_SRCDIR),) # In the normal case (no package override), the sequence of steps is
@@ -391,13 +391,11 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # extract # patch # configure -$(1)-configure: $(1)-patch $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): | $$($(2)_DEPENDENCIES) $$($(2)_TARGET_PATCH) -$(1)-patch: $(1)-extract $$($(2)_TARGET_PATCH) +$(1)-patch: $$($(2)_TARGET_PATCH) -$(1)-extract: $(1)-source \ - $$($(2)_TARGET_EXTRACT) +$(1)-extract: $$($(2)_TARGET_EXTRACT) $(1)-depends: $$($(2)_DEPENDENCIES)
@@ -407,10 +405,9 @@ else # source, by rsyncing # depends # configure -$(1)-configure: $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): | $$($(2)_DEPENDENCIES) $$($(2)_TARGET_RSYNC) -$(1)-depends: $(1)-rsync $$($(2)_DEPENDENCIES) +$(1)-depends: $$($(2)_DEPENDENCIES) $(1)-patch: $(1)-rsync $(1)-extract: $(1)-rsync
package/uclibc/uclibc.mk +1 -1
@@ -23,7 +23,7 @@ UCLIBC_INSTALL_STAGING = YES UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers # Before uClibc is built, we must have the second stage cross-compiler -uclibc-build: host-gcc-intermediate +UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config # setting.
Signature · 1 line
1.8.4
PATCH v3 3/3 package: add toolchain dependency to inner-generic-package
In reply to Fabio PorceddaThis commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:
make clean <package-name>
This is a step forward top-level parallel make.
--- Makefile | 2 +- package/pkg-autotools.mk | 3 ++- package/pkg-generic.mk | 6 +++++- 3 files changed, 8 insertions(+), 3 deletions(-)
Makefile +1 -1
@@ -397,7 +397,7 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf toolchain: prepare dirs dependencies $(BASE_TARGETS) -world: toolchain $(TARGETS_ALL) +world: $(TARGETS_ALL) .PHONY: all world toolchain dirs clean distclean source outputmakefile \ legal-info legal-info-prepare legal-info-clean printvars \
package/pkg-autotools.mk +2 -1
@@ -202,7 +202,8 @@ endef # This must be repeated from inner-generic-package, otherwise we get an empty # _DEPENDENCIES if _AUTORECONF is YES. Also filter the result of _AUTORECONF # away from the non-host rule -$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\ +$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \ + host-toolchain $(1),\ $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
package/pkg-generic.mk +5 -1
@@ -307,7 +307,11 @@ endif $(2)_REDISTRIBUTE ?= YES -$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +$(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ + $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +ifeq ($$($(2)_TYPE),target) +$(2)_DEPENDENCIES += toolchain +endif $(2)_BUILD_DEPENDENCIES ?= $(2)_INSTALL_STAGING ?= NO
Signature · 1 line
1.8.4