Skip to contentSkip to conversation

Fix for top-level parallel make part 1

PATCH Superseded Fabio Porcedda 2 participants 5 messages Sep 10, 2013 – Sep 12, 2013 Kbuild

v3 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v3 cover Fix for top-level parallel make part 1

Hi all,
this is the first patch set for fixing top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefile is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

With this patch set the package building seems to work fine with top-level parallel make, example:

make clean toolchain
make BR2_JLEVEL= -j<n> <package-name list>

Before to try that you have to remove the ".NOTPARALLEL" line in the top-level Makefile.

The part that remain to fix for top-level parallel make is the toolchain building and targets after building all the packages.

Best regards
Fabio Porcedda

v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the first patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.

Fabio Porcedda (3):
package/Makefile.in: add a way to don't force jobs in sub-make
pkg-generic: fix rules for top-level parallel make
package: add toolchain dependency to inner-generic-package

 Makefile                 |  2 +-
 package/Makefile.in      |  2 +-
 package/pkg-autotools.mk |  3 ++-
 package/pkg-generic.mk   | 51 ++++++++++++++++++++++++------------------------
 package/uclibc/uclibc.mk |  2 +-
 5 files changed, 31 insertions(+), 29 deletions(-)
Signature · 1 line
1.8.4
#1 lore raw

PATCH v3 1/3 package/Makefile.in: add a way to don't force jobs in sub-make

When the "BR2_JLEVEL" variable is empty use "make" without the "-j" option.
To be able to use top-level parallel make we must don't force the number of jobs in sub-make.

Example:
make BR2_JLEVEL= -j8

---
 package/Makefile.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/Makefile.in +1 -1
index 91ae1b9..6702df4 100644
@@ -18,7 +18,7 @@ PARALLEL_JOBS:=$(BR2_JLEVEL)
 endif
 
 MAKE1:=$(HOSTMAKE) -j1
-MAKE:=$(HOSTMAKE) -j$(PARALLEL_JOBS)
+MAKE:=$(HOSTMAKE) $(if $(PARALLEL_JOBS),-j$(PARALLEL_JOBS))
 
 # Compute GNU_TARGET_NAME
 GNU_TARGET_NAME=$(ARCH)-buildroot-$(TARGET_OS)-$(LIBC)$(ABI)
Signature · 1 line
1.8.4
#2 lore raw

PATCH v3 1/3 package/Makefile.in: add a way to don't force jobs in sub-make

Dear Fabio Porcedda,

Quoted 10 lines · Fabio Porcedda

When the "BR2_JLEVEL" variable is empty use "make" without the "-j" option.
To be able to use top-level parallel make we must don't force the number of jobs in sub-make.

Example:
make BR2_JLEVEL= -j8

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#3 lore raw

PATCH v3 2/3 pkg-generic: fix rules for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must don't depend in a rule on the order of evaluation of the prerequisites, so instead of reling on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

So add explicit dependencies for the following stamp files:
%/.stamp_extracted
%/.stamp_patched
%/.stamp_configured
%/.stamp_built
%/.stamp_host_installed %/.stamp_staging_installed %/.stamp_images_installed %/.stamp_target_installed

Because the %-build target is not anymore part of the dependcy chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by the uclibc package for building the toolchain.

---
 package/pkg-generic.mk   | 45 +++++++++++++++++++++------------------------
 package/uclibc/uclibc.mk |  2 +-
 2 files changed, 22 insertions(+), 25 deletions(-)
package/pkg-generic.mk +21 -24
index 4dd395b..5ea41ff 100644
@@ -53,7 +53,7 @@ ifeq ($(DL_MODE),DOWNLOAD)
 endif
 
 # Unpack the archive
-$(BUILD_DIR)/%/.stamp_extracted:
+$(BUILD_DIR)/%/.stamp_extracted: $(BUILD_DIR)/%/.stamp_downloaded
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -88,7 +88,7 @@ endif
 # prefix of the patches
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
-$(BUILD_DIR)/%/.stamp_patched:
+$(BUILD_DIR)/%/.stamp_patched: $(BUILD_DIR)/%/.stamp_extracted
 	@$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -115,21 +115,21 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(Q)touch $@
 
 # Build
-$(BUILD_DIR)/%/.stamp_built::
+$(BUILD_DIR)/%/.stamp_built:	$(BUILD_DIR)/%/.stamp_configured
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to host dir
-$(BUILD_DIR)/%/.stamp_host_installed:
+$(BUILD_DIR)/%/.stamp_host_installed:		$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to staging dir
-$(BUILD_DIR)/%/.stamp_staging_installed:
+$(BUILD_DIR)/%/.stamp_staging_installed:	$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -143,14 +143,14 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 	$(Q)touch $@
 
 # Install to images dir
-$(BUILD_DIR)/%/.stamp_images_installed:
+$(BUILD_DIR)/%/.stamp_images_installed:		$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to target dir
-$(BUILD_DIR)/%/.stamp_target_installed:
+$(BUILD_DIR)/%/.stamp_target_installed:		$(BUILD_DIR)/%/.stamp_built
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -308,6 +308,7 @@ $(2)_REDISTRIBUTE		?= YES
 
 
 $(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_BUILD_DEPENDENCIES		?=
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
@@ -359,30 +360,29 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
 endif
 
 ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
 else
 $(1)-install-target:
 endif
 
 ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
 else
 $(1)-install-staging:
 endif
 
 ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
 else
 $(1)-install-images:
 endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:		$(1)-configure \
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	| $$($(2)_BUILD_DEPENDENCIES)
+$(1)-build:		$$($(2)_TARGET_BUILD)
+
+$(1)-configure:		$$($(2)_TARGET_CONFIGURE)
 
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
@@ -391,13 +391,11 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES) $$($(2)_TARGET_PATCH)
 
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:		$$($(2)_TARGET_EXTRACT)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
@@ -407,10 +405,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES) $$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
 $(1)-patch:		$(1)-rsync
 $(1)-extract:		$(1)-rsync
package/uclibc/uclibc.mk +1 -1
index 39877bb..b960f59 100644
@@ -23,7 +23,7 @@ UCLIBC_INSTALL_STAGING = YES
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
 # Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
+UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
Signature · 1 line
1.8.4
#4 lore raw

PATCH v3 3/3 package: add toolchain dependency to inner-generic-package

In reply to Fabio Porcedda

This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:

make clean <package-name>

This is a step forward top-level parallel make.

---
 Makefile                 | 2 +-
 package/pkg-autotools.mk | 3 ++-
 package/pkg-generic.mk   | 6 +++++-
 3 files changed, 8 insertions(+), 3 deletions(-)
Makefile +1 -1
index 485d0a4..5ed3d60 100644
@@ -397,7 +397,7 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
 toolchain: prepare dirs dependencies $(BASE_TARGETS)
 
-world: toolchain $(TARGETS_ALL)
+world: $(TARGETS_ALL)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
package/pkg-autotools.mk +2 -1
index b0eddde..aafa396 100644
@@ -202,7 +202,8 @@ endef
 # This must be repeated from inner-generic-package, otherwise we get an empty
 # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
 # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\
     $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
package/pkg-generic.mk +5 -1
index 5ea41ff..e511833 100644
@@ -307,7 +307,11 @@ endif
 $(2)_REDISTRIBUTE		?= YES
 
 
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($$($(2)_TYPE),target)
+$(2)_DEPENDENCIES += toolchain
+endif
 $(2)_BUILD_DEPENDENCIES		?=
 
 $(2)_INSTALL_STAGING		?= NO
Signature · 1 line
1.8.4