Skip to contentSkip to conversation

Add support for top-level parallel make

PATCH Superseded Fabio Porcedda 3 participants 12 messages Feb 12, 2014 – Feb 14, 2014

Subsystems (2)

v12 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v12 cover Add support for top-level parallel make

Hi all,
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

With this patch set the top-level parallel make seems to works fine, example:

	make clean
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

Before to try it remeber to remove the ".NOTPARALLEL" line.

On my quad core system the building time for the first 220 defconfig lines of the option "allpackageyesconfig" is 45m vs 24m.

I've tested the qemu_x86_defconfig.

I will test futrher the top-level parallel make feature for building all the packages so expect some other patches to improve this feature.

Thank you for all reviews and in particular Arnout.

Best regards
Fabio Porcedda

v12:
- Fixed a typo in the 1th patch
- Fixed a typo in the 2th patch
- Improved 3patch adding:
$$($(2)_TARGET_INSTALL_TARGET): $$($(2)_TARGET_INSTALL_STAGING) so removed patch for uclibc and glibc because now they are useless
- Improved the 4th patch adding an explicit dependency:
$(TARGETS_ROOTFS): target-finalize
and

	 $(TARGETS): dirs prepare dependencies

- Added a new patch for the sdl package
v11:
- Rebased over master
- Added support for pythone framework in the second patch.
- Changed comment on the latest patch.
v10:
- Added Acked-by: Arnout Vandecappelle on the first two patches
- Improved descption and text of the last patch
- Rebased over master
v9:
- Added Acked-by: Arnout Vandecappelle on some patches.
- Changed the patches contents and descriptions following advices of Arnout
- 1th patch: fixed to support ccache & sstrip
- 2th patch: added pkg-cmake.mk
- 2th patch: splitted ifeq
- 3th patch: improved description
- 3th patch: does not use %_BUILD_DEPENDENCIES anymore
- 3th patch: added a comment about "|"
- 4th patch: added a comment
- 5th patch: added a comment
- 6th patch:
- 2th patch: fixed a comment
- 8th patch: now it change only the comment
v8:
- rebased over master
- added patche for for base dependency
- added patche for for glibc package
- added patche for for uclibc package
- removed patch already merged
- changed some descriptions
- modified the patch for toolchain dependency to prevent circular dependency
v7:
- add the latest patch
- add to the first patch the <pkgname>_TOOLCHAIN variable
- improve the fifth patch
v6:
- added the fifth patch
- updated the fourth patch adding the install targets
- updated the second patch to remove TARGETS_ALL
v5:
- added the fourth patch
- fixed some typos
- rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
- add support for top-level parallel make for the glibc package
v4:
- rebased over master
- add Acked-by: Thomas Petazzoni on the third patch
- changed the orderd of the patches
v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the third patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.

Fabio Porcedda (7):
package: add base dependency to every package
package: add toolchain dependency to every target package
package: add support for top-level parallel make
Makefile: add support for top-level parallel make
package: enable jobserver for recursive make
Makefile: update comment about top-level parallel Makefile
sdl: add support for top-level parallel make

 Makefile                                           | 40 ++++++++-----
 fs/common.mk                                       |  4 +-
 package/glibc/glibc.mk                             |  9 ++-
 package/linux-headers/linux-headers.mk             |  3 +
 package/pkg-autotools.mk                           |  3 +-
 package/pkg-cmake.mk                               |  2 +-
 package/pkg-generic.mk                             | 69 ++++++++++++++--------
 package/pkg-python.mk                              |  2 +-
 package/sdl/sdl.mk                                 |  3 +
 package/uclibc/uclibc.mk                           |  9 ++-
 .../toolchain-buildroot/toolchain-buildroot.mk     |  2 +
 toolchain/toolchain-external/toolchain-external.mk |  2 +
 toolchain/toolchain/toolchain.mk                   |  4 +-
 13 files changed, 103 insertions(+), 49 deletions(-)
Signature · 1 line
1.8.5.4
#1 lore raw

PATCH v12 1/7 package: add base dependency to every package

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

This is also a step forward supporting top-level parallel make.

---
 package/pkg-generic.mk           | 5 +++++
 toolchain/toolchain/toolchain.mk | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
package/pkg-generic.mk +5 -0
index 555c3c2..e5b04ff 100644
@@ -427,6 +427,11 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
+ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
+endif
+
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
 #  source, by downloading
toolchain/toolchain/toolchain.mk +1 -1
index 1152e25..8559ac9 100644
@@ -14,4 +14,4 @@ endif
 
 $(eval $(generic-package))
 
-toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
+toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
1.8.5.4
#2 lore raw

PATCH v12 2/7 package: add toolchain dependency to every target package

In reply to Fabio Porcedda

This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:

make clean <package-name>

Also remove TARGETS_ALL because the only purpose was to add toolchain dependency so it's superseded by this commit.

To prevent circular dependency add the new variable <pkgname>_ADD_TOOLCHAIN_DEPENDENCY to avoid adding the toolchain dependency for toolchain packages.

This is also a step forward supporting top-level parallel make.

---
 Makefile                                             |  8 ++------
 package/glibc/glibc.mk                               |  3 +++
 package/linux-headers/linux-headers.mk               |  3 +++
 package/pkg-autotools.mk                             |  3 ++-
 package/pkg-cmake.mk                                 |  2 +-
 package/pkg-generic.mk                               | 14 ++++++++++++--
 package/pkg-python.mk                                |  2 +-
 package/uclibc/uclibc.mk                             |  3 +++
 toolchain/toolchain-buildroot/toolchain-buildroot.mk |  2 ++
 toolchain/toolchain-external/toolchain-external.mk   |  2 ++
 toolchain/toolchain/toolchain.mk                     |  2 ++
 11 files changed, 33 insertions(+), 11 deletions(-)
Makefile +2 -6
index 74022a5..8e65bc7 100644
@@ -399,7 +399,6 @@ TARGETS+=target-post-image
 
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
-TARGETS_ALL:=$(patsubst %,__real_tgt_%,$(TARGETS))
 
 # host-* dependencies have to be handled specially, as those aren't
 # visible in Kconfig and hence not added to a variable like TARGETS.
@@ -422,9 +421,6 @@ HOST_SOURCE += $(addsuffix -source,$(sort $(TARGETS_HOST_DEPS) $(HOST_DEPS)))
 TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 		$(TARGETS) $(BASE_TARGETS) $(TARGETS_HOST_DEPS) $(HOST_DEPS))))
 
-# all targets depend on the crosscompiler and it's prerequisites
-$(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
-
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
@@ -433,11 +429,11 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(BASE_TARGETS) $(TARGETS_ALL)
+world: $(TARGETS)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
+	$(BASE_TARGETS) $(TARGETS) \
 	$(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
package/glibc/glibc.mk +3 -0
index 96de02a..0968f67 100644
@@ -31,6 +31,9 @@ endif
 GLIBC_LICENSE = GPLv2+ (programs), LGPLv2.1+, BSD-3c, MIT (library)
 GLIBC_LICENSE_FILES = $(addprefix $(GLIBC_SRC_SUBDIR)/,COPYING COPYING.LIB LICENSES)
 
+# glibc is part of the toolchain so disable the toolchain dependency
+GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before (e)glibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
package/linux-headers/linux-headers.mk +3 -0
index 30d3076..6dac9e3 100644
@@ -17,6 +17,9 @@ LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 
 LINUX_HEADERS_INSTALL_STAGING = YES
 
+# linux-headers is part of the toolchain so disable the toolchain dependency
+LINUX_HEADERS_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 define LINUX_HEADERS_INSTALL_STAGING_CMDS
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) $(MAKE) \
package/pkg-autotools.mk +2 -1
index 5f6c31d..32ceef2 100644
@@ -210,7 +210,8 @@ endef
 # This must be repeated from inner-generic-package, otherwise we get an empty
 # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
 # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\
     $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
package/pkg-cmake.mk +1 -1
index dbe0d97..f7e454b 100644
@@ -89,7 +89,7 @@ endif
 
 # This must be repeated from inner-generic-package, otherwise we only get
 # host-cmake in _DEPENDENCIES because of the following line
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 $(2)_DEPENDENCIES += host-cmake
 
package/pkg-generic.mk +12 -2
index e5b04ff..b135b14 100644
@@ -351,8 +351,18 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
-
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+# When a target package is a toolchain dependency set this variable to
+# 'NO' so the 'toolchain' dependency is not added to prevent a circular
+# dependency
+$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($(4),target)
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+$(2)_DEPENDENCIES += toolchain
+endif
+endif
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
package/pkg-python.mk +1 -1
index 9b3c97a..f33980a 100644
@@ -142,7 +142,7 @@ endif
 # be derived automatically from the dependencies of the corresponding
 # target package. For example, target packages need
 # host-python-distutilscross, but not host packages.
-$(2)_DEPENDENCIES ?= $(filter-out host-python host-python-setuptools host-python-distutilscross $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+$(2)_DEPENDENCIES ?= $(filter-out host-python host-python-setuptools host-python-distutilscross host-toolchain $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 # Target packages need both the python interpreter on the target (for
 # runtime) and the python interpreter on the host (for
package/uclibc/uclibc.mk +3 -0
index 7df9721..ea1c694 100644
@@ -22,6 +22,9 @@ endif
 
 UCLIBC_INSTALL_STAGING = YES
 
+# uclibc is part of the toolchain so disable the toolchain dependency
+UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before uClibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
toolchain/toolchain-buildroot/toolchain-buildroot.mk +2 -0
index 2b5028c..853baf8 100644
@@ -14,4 +14,6 @@ BR_LIBC = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_LIBC))
 
 TOOLCHAIN_BUILDROOT_DEPENDENCIES = host-gcc-final
 
+TOOLCHAIN_BUILDROOT_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
toolchain/toolchain-external/toolchain-external.mk +2 -0
index a7bfafc..51da480 100644
@@ -366,6 +366,8 @@ TOOLCHAIN_EXTERNAL_SITE =
 TOOLCHAIN_EXTERNAL_SOURCE =
 endif
 
+TOOLCHAIN_EXTERNAL_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 TOOLCHAIN_EXTERNAL_INSTALL_STAGING = YES
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R1)$(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R2)$(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2013R1),y)
toolchain/toolchain/toolchain.mk +2 -0
index 8559ac9..7241fe7 100644
@@ -12,6 +12,8 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
1.8.5.4
#3 lore raw

PATCH v12 3/7 package: add support for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relying on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

We cannot use the pattern rules because they must have the same dependency for every package, but we need to change the dependencies depending on $(2)_OVERRIDE_SRCDIR variable value, so we must use a more flexible way like $(2)_TARGET_% variables.

So add explicit dependencies for the following stamp files:
$(2)_TARGET_EXTRACT
$(2)_TARGET_PATCH
$(2)_TARGET_CONFIGURE
$(2)_TARGET_BUILD
$(2)_TARGET_INSTALL_STAGING $(2)_TARGET_INSTALL_TARGET $(2)_TARGET_INSTALL_IMAGES $(2)_TARGET_INSTALL_HOST

---
 package/glibc/glibc.mk   |  6 +++---
 package/pkg-generic.mk   | 42 +++++++++++++++++++++++++-----------------
 package/uclibc/uclibc.mk |  6 +++---
 3 files changed, 31 insertions(+), 23 deletions(-)
package/glibc/glibc.mk +3 -3
index 0968f67..3013df3 100644
@@ -38,9 +38,6 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 # cross-compiler and the kernel headers
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
 
-# Before (e)glibc is built, we must have the second stage cross-compiler
-glibc-build: host-gcc-intermediate
-
 GLIBC_SUBDIR = build
 
 GLIBC_INSTALL_STAGING = YES
@@ -142,3 +139,6 @@ define GLIBC_INSTALL_TARGET_CMDS
 endef
 
 $(eval $(autotools-package))
+
+# Before (e)glibc is built, we must have the second stage cross-compiler
+$(GLIBC_TARGET_BUILD): | host-gcc-intermediate
package/pkg-generic.mk +25 -17

78 lines not on this page · raw message

package/uclibc/uclibc.mk +3 -3
index ea1c694..4860c58 100644
@@ -29,9 +29,6 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
-# Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
-
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
 ifndef UCLIBC_CONFIG_FILE
@@ -556,3 +553,6 @@ uclibc-menuconfig: dirs uclibc-patch
 	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
 $(eval $(generic-package))
+
+# Before uClibc is built, we must have the second stage cross-compiler
+$(UCLIBC_TARGET_BUILD): | host-gcc-intermediate
Signature · 1 line
1.8.5.4
#4 lore raw

PATCH v12 3/7 package: add support for top-level parallel make

Quoted 22 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relying on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

We cannot use the pattern rules because they must have the same dependency for every package, but we need to change the dependencies depending on $(2)_OVERRIDE_SRCDIR variable value, so we must use a more flexible way like $(2)_TARGET_% variables.

So add explicit dependencies for the following stamp files:
$(2)_TARGET_EXTRACT
$(2)_TARGET_PATCH
$(2)_TARGET_CONFIGURE
$(2)_TARGET_BUILD
$(2)_TARGET_INSTALL_STAGING $(2)_TARGET_INSTALL_TARGET $(2)_TARGET_INSTALL_IMAGES $(2)_TARGET_INSTALL_HOST

---

[snip]

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 438

Quoted 5 lines

-$(1)-build: $(1)-configure \

package/pkg-generic.mk +3 -1
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+	$$($(2)_TARGET_BUILD)

I'd prefer to see these on separate lines:

$$($(2)_TARGET_INSTALL_TARGET): $$($(2)_TARGET_BUILD) $$($(2)_TARGET_INSTALL_STAGING): $$($(2)_TARGET_BUILD)
...

And also I'd prefer them close to the corresponding $(1)-install target.

By the way, for these it would actually be possible to put them in the
pattern rule, because they don't depend on OVERRIDE_SRCDIR, right? But I guess we said a long time ago that it would be better to do it symmetrical for all targets.

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 444

Quoted 6 lines
package/pkg-generic.mk +6 -0
+
+$(1)-build:		$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE)
+
+# The symbol "|" specify a order-only-prerequisite, this is needed for
+# phony requisites to avoid rebuilding every time the target.

Possible improvement of the explanation:

Since $(2)_DEPENDENCIES are phony targets, they are always "newer" than $(2)_TARGET_CONFIGURE. This would force the configure step (and therefore the other steps as well) to be re-executed with every invocation of make. Therefore, make $(2)_DEPENDENCIES an order-only dependency by using |.

Quoted 14 lines
package/pkg-generic.mk +4 -2
+
+$(1)-configure:			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES)
 
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
 ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
@@ -449,13 +458,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
 

package/pkg-generic.mk ifeq ($$($(2)_OVERRIDE_SRCDIR),) · line 469

Quoted 10 lines
package/pkg-generic.mk +4 -3
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:			$$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)

Since _CONFIGURE now explicitly depends on _DEPENDENCIES, the -depends
target has become redundant, so you can remove it.

package/pkg-generic.mk else · line 479

Quoted 11 lines
package/pkg-generic.mk +2 -3
@@ -465,10 +474,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)

Here as well.

Regards,
Arnout

Quoted 25 lines
package/pkg-generic.mk +0 -0
 $(1)-patch:		$(1)-rsync
 $(1)-extract:		$(1)-rsync
package/uclibc/uclibc.mk +3 -3
index ea1c694..4860c58 100644
@@ -29,9 +29,6 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
-# Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
-
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
 ifndef UCLIBC_CONFIG_FILE
@@ -556,3 +553,6 @@ uclibc-menuconfig: dirs uclibc-patch
 	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
 $(eval $(generic-package))
+
+# Before uClibc is built, we must have the second stage cross-compiler
+$(UCLIBC_TARGET_BUILD): | host-gcc-intermediate
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#5 lore raw

PATCH v12 3/7 package: add support for top-level parallel make

Quoted 30 lines · Arnout Vandecappelle
Quoted 22 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relying on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

We cannot use the pattern rules because they must have the same dependency for every package, but we need to change the dependencies depending on $(2)_OVERRIDE_SRCDIR variable value, so we must use a more flexible way like $(2)_TARGET_% variables.

So add explicit dependencies for the following stamp files:
$(2)_TARGET_EXTRACT
$(2)_TARGET_PATCH
$(2)_TARGET_CONFIGURE
$(2)_TARGET_BUILD
$(2)_TARGET_INSTALL_STAGING $(2)_TARGET_INSTALL_TARGET $(2)_TARGET_INSTALL_IMAGES $(2)_TARGET_INSTALL_HOST

---

[snip]

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 438

Quoted 5 lines

-$(1)-build: $(1)-configure \

package/pkg-generic.mk +3 -1
-                     $$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+     $$($(2)_TARGET_BUILD)
Quoted 7 lines

I'd prefer to see these on separate lines:

$$($(2)_TARGET_INSTALL_TARGET): $$($(2)_TARGET_BUILD) $$($(2)_TARGET_INSTALL_STAGING): $$($(2)_TARGET_BUILD)
...

And also I'd prefer them close to the corresponding $(1)-install target.

Ok, like this?

ifeq ($$($(2)_INSTALL_TARGET),YES)
$(1)-install-target: $$($(2)_TARGET_INSTALL_TARGET)
else
$(1)-install-target:
endif
$$($(2)_TARGET_INSTALL_TARGET): $$($(2)_TARGET_BUILD)

Quoted 4 lines

By the way, for these it would actually be possible to put them in the
pattern rule, because they don't depend on OVERRIDE_SRCDIR, right? But I guess we said a long time ago that it would be better to do it symmetrical for all targets.

Yes we said exactly that, i've already sent a patch with asymmetrical targets that was rejected.

Quoted 10 lines

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 444

Quoted 7 lines
package/pkg-generic.mk +6 -0
+
+$(1)-build:          $$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):       $$($(2)_TARGET_CONFIGURE)
+
+# The symbol "|" specify a order-only-prerequisite, this is needed for
+# phony requisites to avoid rebuilding every time the target.

Possible improvement of the explanation:

Quoted 4 lines

Since $(2)_DEPENDENCIES are phony targets, they are always "newer" than $(2)_TARGET_CONFIGURE. This would force the configure step (and therefore the other steps as well) to be re-executed with every invocation of make. Therefore, make $(2)_DEPENDENCIES an order-only dependency by using |.

Ok

Quoted 23 lines
Quoted 23 lines
package/pkg-generic.mk +8 -5
+
+$(1)-configure:                      $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):   | $$($(2)_DEPENDENCIES)
 
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
 ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
@@ -449,13 +458,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:              $(1)-patch $(1)-depends \
-                     $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):   $$($(2)_TARGET_PATCH)
 
-$(1)-patch:          $(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:          $$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH):       $$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:                $(1)-source \
-                     $$($(2)_TARGET_EXTRACT)
+$(1)-extract:                        $$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):     $$($(2)_TARGET_SOURCE)
 
Quoted 4 lines

package/pkg-generic.mk ifeq ($$($(2)_OVERRIDE_SRCDIR),) · line 469

Quoted 1 line
package/pkg-generic.mk +0 -0
 $(1)-depends:                $$($(2)_DEPENDENCIES)

Since _CONFIGURE now explicitly depends on _DEPENDENCIES, the -depends
target has become redundant, so you can remove it.

Well, it's fine for me, but what if somebody want just to be sure that all the dependencies are satisfied without downloading/patching/configuring the package?
Do you think it's not a real or important enough use case?

Quoted 9 lines
Quoted 9 lines
package/pkg-generic.mk +1 -2
@@ -465,10 +474,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:              $(1)-depends \
-                     $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):   $$($(2)_TARGET_RSYNC)
 
Quoted 5 lines

package/pkg-generic.mk else · line 479

Quoted 2 lines
package/pkg-generic.mk +1 -1
-$(1)-depends:                $(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:                $$($(2)_DEPENDENCIES)

Here as well.

<snip>

Thanks

Signature · 1 line
Fabio Porcedda
#6 lore raw

PATCH v12 4/7 Makefile: add support for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image

---
 Makefile     | 21 ++++++++++++---------
 fs/common.mk |  4 ++--
 2 files changed, 14 insertions(+), 11 deletions(-)
Makefile +12 -9
index 8e65bc7..c89107a 100644
@@ -377,8 +377,6 @@ include system/system.mk
 
 include $(BR2_EXTERNAL)/external.mk
 
-TARGETS+=target-finalize
-
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 TARGETS+=target-purgelocales
 endif
@@ -395,8 +393,6 @@ endif
 
 include fs/common.mk
 
-TARGETS+=target-post-image
-
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
 
@@ -429,11 +425,16 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(TARGETS)
+# Add base dependencies to all targets even on those not based on the
+# package framework.
+$(TARGETS): dirs prepare dependencies
+
+world: target-post-image
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) \
+	target-finalize target-post-image \
+	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
 	$(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -493,7 +494,7 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
 
-target-finalize:
+target-finalize: $(TARGETS)
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
 		$(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
@@ -520,6 +521,8 @@ endif
 		find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
 		xargs -r $(KSTRIPCMD); fi
 
+$(TARGETS_ROOTFS): target-finalize
+
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"
@@ -598,7 +601,7 @@ target-generatelocales: host-localedef
 	done
 endif
 
-target-post-image:
+target-post-image: $(TARGETS_ROOTFS)
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
 		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
@@ -635,7 +638,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(TARGETS_LEGAL_INFO) \
 	@rm -f $(LEGAL_WARNINGS)
 
 show-targets:
-	@echo $(TARGETS)
+	@echo $(TARGETS) $(TARGETS_ROOTFS)
 
 graph-build: $(O)/build/build-time.log
 	@install -d $(O)/graphs
fs/common.mk +2 -2
index 30e7524..d95c26b 100644
@@ -64,7 +64,7 @@ ROOTFS_$(2)_COMPRESS_EXT = .xz
 ROOTFS_$(2)_COMPRESS_CMD = $$(XZ) -9 -C crc32 -c
 endif
 
-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
+$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
 	rm -f $$(FAKEROOT_SCRIPT)
@@ -95,7 +95,7 @@ rootfs-$(1)-show-depends:
 rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
 endif
 endef
 
Signature · 1 line
1.8.5.4
#7 lore raw

PATCH v12 5/7 package: enable jobserver for recursive make

In reply to Fabio Porcedda

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character at the beginning of the recipe line. This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ?+? token to get these special effects.

---
 package/pkg-generic.mk | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
package/pkg-generic.mk +5 -5
index 4c84188..8d09bf5 100644
@@ -173,7 +173,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 $(BUILD_DIR)/%/.stamp_built::
 	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
-	$($(PKG)_BUILD_CMDS)
+	+$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 	@$(call step_end,build)
@@ -182,7 +182,7 @@ $(BUILD_DIR)/%/.stamp_built::
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
-	$($(PKG)_INSTALL_CMDS)
+	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 	@$(call step_end,install-host)
@@ -191,7 +191,7 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 $(BUILD_DIR)/%/.stamp_staging_installed:
 	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
-	$($(PKG)_INSTALL_STAGING_CMDS)
+	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(call MESSAGE,"Fixing package configuration files") ;\
@@ -207,7 +207,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
-	$($(PKG)_INSTALL_IMAGES_CMDS)
+	+$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 	@$(call step_end,install-image)
@@ -220,7 +220,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
 	$(if $(BR2_INIT_SYSV)$(BR2_INIT_BUSYBOX),\
 		$($(PKG)_INSTALL_INIT_SYSV))
-	$($(PKG)_INSTALL_TARGET_CMDS)
+	+$($(PKG)_INSTALL_TARGET_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
Signature · 1 line
1.8.5.4
#8 lore raw

PATCH v12 6/7 Makefile: update comment about top-level parallel Makefile

In reply to Fabio Porcedda

After the latest patches top-level parallel Makefile is working but there is still an issue when a package has an unspecified optional dependency so change the comment to explain that.

---
 Makefile | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
Makefile +14 -1
index c89107a..ddb1f6a 100644
@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
 	    -e s/macppc/powerpc/\
 	    -e s/sh.*/sh/)
 
-# This top-level Makefile can *not* be executed in parallel
+# Parallel execution of this Makefile is disabled because it changes
+# the packages building order, that can be a problem for two reasons:
+# - If a package has an unspecified optional dependency and that
+#   dependency is present when the package is built, it is used,
+#   otherwise it isn't (but compilation happily proceeds) so the end
+#   result will differ if the order is swapped due to parallel
+#   building.
+# - Also changing the building order can be a problem if two packages
+#   manipulate the same file in the target directory.
+#
+# Taking into account the above considerations, if you still want to execute
+# this top-level Makefile in parallel comment the ".NOTPARALLEL" line and
+# build using the following command:
+#	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
 .NOTPARALLEL:
 
 # absolute path
Signature · 1 line
1.8.5.4
#9 lore raw

PATCH v12 7/7 sdl: add support for top-level parallel make

In reply to Fabio Porcedda

Add host-autoconf as a dependency of the sdl-patch target because is used by it.

---
 package/sdl/sdl.mk | 3 +++
 1 file changed, 3 insertions(+)
package/sdl/sdl.mk +3 -0
index ba5fbca..701022b 100644
@@ -80,3 +80,6 @@ endef
 SDL_POST_INSTALL_STAGING_HOOKS += SDL_FIXUP_SDL_CONFIG
 
 $(eval $(autotools-package))
+
+# autoconf is used in the sdl-patch target
+$(SDL_TARGET_PATCH): | host-autoconf
Signature · 1 line
1.8.5.4
#10 lore raw

PATCH v12 7/7 sdl: add support for top-level parallel make

Dear Fabio Porcedda,

Quoted 13 lines · Fabio Porcedda

Add host-autoconf as a dependency of the sdl-patch target because is used by it.

---
 package/sdl/sdl.mk | 3 +++
 1 file changed, 3 insertions(+)
package/sdl/sdl.mk +0 -0
index ba5fbca..701022b 100644

package/sdl/sdl.mk endef · line 85

Quoted 6 lines
package/sdl/sdl.mk +3 -0
@@ -80,3 +80,6 @@ endef
 SDL_POST_INSTALL_STAGING_HOOKS += SDL_FIXUP_SDL_CONFIG
 
 $(eval $(autotools-package))
+
+# autoconf is used in the sdl-patch target
+$(SDL_TARGET_PATCH): | host-autoconf

I believe I'd prefer if the SDL autogen stuff was done as a pre-configure hook, since this is the hook that is used in the package infrastructure to run autoreconf in the normal case.

Cc'ing Maxime Hadjinlian, who has recently worked on SDL :)

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#11 lore raw

PATCH v12 7/7 sdl: add support for top-level parallel make

Quoted 18 lines · Thomas Petazzoni

Dear Fabio Porcedda,

Quoted 15 lines · Fabio Porcedda

Add host-autoconf as a dependency of the sdl-patch target because is used by it.

---
 package/sdl/sdl.mk | 3 +++
 1 file changed, 3 insertions(+)
package/sdl/sdl.mk +0 -0
index ba5fbca..701022b 100644
@@ -80,3 +80,6 @@ endef
 SDL_POST_INSTALL_STAGING_HOOKS += SDL_FIXUP_SDL_CONFIG
 
Quoted 8 lines

package/sdl/sdl.mk endef · line 85

Quoted 4 lines
package/sdl/sdl.mk +3 -0
 $(eval $(autotools-package))
+
+# autoconf is used in the sdl-patch target
+$(SDL_TARGET_PATCH): | host-autoconf

I believe I'd prefer if the SDL autogen stuff was done as a pre-configure hook, since this is the hook that is used in the package infrastructure to run autoreconf in the normal case.

It's a nice idea, so i will do just this:

SDL_PRE_CONFIGURE_HOOKS += SDL_RUN_AUTOGEN

Quoted 1 line

Cc'ing Maxime Hadjinlian, who has recently worked on SDL :)

Thanks and best regards

Signature · 1 line
Fabio Porcedda