Skip to contentSkip to conversation

Add support for top-level parallel make

PATCH RESEND Superseded Fabio Porcedda 4 participants 40 messages Oct 18, 2013 – Nov 14, 2013

Subsystems (2)

v8 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v8 cover Add support for top-level parallel make

Hi all,
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

With this patch set the top-level parallel make seems to work fine, example:

	make clean
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

On my quad core system the building time for qemu_x86_defconfig is 13m versus 11m.

I've tested the qemu_x86_defconfig uclibc/eglibc/glibc.

Best regards
Fabio Porcedda

v8:
- rebased over master
- added patche for for base dependency
- added patche for for glibc package
- added patche for for uclibc package
- removed patch already merged
- changed some descriptions
- modified the patch for toolchain dependency to prevent circular dependency
v7:
- add the latest patch
- add to the first patch the <pkgname>_TOOLCHAIN variable
- improve the fifth patch
v6:
- added the fifth patch
- updated the fourth patch adding the install targets
- updated the second patch to remove TARGETS_ALL
v5:
- added the fourth patch
- fixed some typos
- rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
- add support for top-level parallel make for the glibc package
v4:
- rebased over master
- add Acked-by: Thomas Petazzoni on the third patch
- changed the orderd of the patches
v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the third patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.

Fabio Porcedda (8):
package: add base dependency to every package
package: add toolchain dependency to every target package
package: add support for top-level parallel make
Makefile: add support for top-level parallel make
glibc: add support for top-level parallel make
uclibc: add support for top-level parallel make
package: enable jobserver for recursive make
Makefile: enable top-level parallel make

 Makefile                                           | 22 +++-----
 fs/common.mk                                       |  4 +-
 package/glibc/glibc.mk                             |  7 ++-
 package/linux-headers/linux-headers.mk             |  3 ++
 package/pkg-autotools.mk                           |  3 +-
 package/pkg-generic.mk                             | 60 +++++++++++++---------
 package/uclibc/uclibc.mk                           |  7 ++-
 .../toolchain-buildroot/toolchain-buildroot.mk     |  2 +
 toolchain/toolchain-external/toolchain-external.mk |  2 +
 toolchain/toolchain/toolchain.mk                   |  5 +-
 10 files changed, 69 insertions(+), 46 deletions(-)
Signature · 1 line
1.8.4
#1 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

This is also a step forward supporting top-level parallel make.

---
 package/pkg-generic.mk           | 2 ++
 toolchain/toolchain/toolchain.mk | 3 +--
 2 files changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +2 -0
index 4bba4b5..1e7154e 100644
@@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
+
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
 #  source, by downloading
toolchain/toolchain/toolchain.mk +1 -2
index 44ed629..8559ac9 100644
@@ -14,5 +14,4 @@ endif
 
 $(eval $(generic-package))
 
-toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-
+toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
1.8.4
#2 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 11 lines · Fabio Porcedda

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

This is also a step forward supporting top-level parallel make.

Although this is one of the most useful patches of the series, it is
also the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.

Quoted 10 lines
---
  package/pkg-generic.mk           | 2 ++
  toolchain/toolchain/toolchain.mk | 3 +--
  2 files changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 4bba4b5..1e7154e 100644

package/pkg-generic.mk $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) · line 388

Quoted 4 lines
package/pkg-generic.mk +1 -0
@@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
  $(1)-build:		$(1)-configure \
  			$$($(2)_TARGET_BUILD)
 
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare

Is there any reason why you changed the order here?

Regards,
Arnout

Quoted 16 lines
package/pkg-generic.mk +1 -0
+
  ifeq ($$($(2)_OVERRIDE_SRCDIR),)
  # In the normal case (no package override), the sequence of steps is
  #  source, by downloading
toolchain/toolchain/toolchain.mk +1 -2
index 44ed629..8559ac9 100644
@@ -14,5 +14,4 @@ endif
 
  $(eval $(generic-package))
 
-toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-
+toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#3 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Hi Arnout,
thanks for reviewing the patch set.

Quoted 10 lines · Arnout Vandecappelle
Quoted 9 lines · Fabio Porcedda

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

Quoted 8 lines
Quoted 3 lines

This is also a step forward supporting top-level parallel make.

Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

Quoted 15 lines
Quoted 15 lines
---
  package/pkg-generic.mk           | 2 ++
  toolchain/toolchain/toolchain.mk | 3 +--
  2 files changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 4bba4b5..1e7154e 100644
@@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build
 $$($(2)_TARGET_INSTALL_HOST)
  $(1)-build:           $(1)-configure \
                        $$($(2)_TARGET_BUILD)
 
Quoted 4 lines

package/pkg-generic.mk $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) · line 388

Quoted 1 line
package/pkg-generic.mk +1 -0
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare

Is there any reason why you changed the order here?

The order whit top-level parallel make does not matter, it's just that i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directories

But now i understand that is "dirs prepare dependencies" or "dirs dependencies prepare" because dirs can require host-ccache and so directories,
nevertheless this does not matter anymore.

Quoted 26 lines

Regards,
Arnout

Quoted 16 lines
package/pkg-generic.mk +1 -0
+
  ifeq ($$($(2)_OVERRIDE_SRCDIR),)
  # In the normal case (no package override), the sequence of steps is

# source, by downloading

toolchain/toolchain/toolchain.mk +0 -1
index 44ed629..8559ac9 100644
@@ -14,5 +14,4 @@ endif
 
  $(eval $(generic-package))
 
-toolchain-source: prepare dirs dependencies

$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake

toolchain/toolchain/toolchain.mk endif · line 17

Quoted 2 lines
toolchain/toolchain/toolchain.mk +1 -1
-
+toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake

--

Quoted 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Best regards

Signature · 1 line
Fabio Porcedda
#4 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Hi,

Quoted 8 lines · Fabio Porcedda
Quoted 5 lines

Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.

I've found a work around:

Quoted 9 lines

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

I don't think this will work if there are more than one prerequisite:
$(1) will be one of them, and DEPENDENCIES_HOST_PREREQ will be the entire list, so that 'ifneq' will always be true.
I think you need 'filter' here.
http://www.gnu.org/software/make/manual/make.html#index-filter-587

Best regards,
Thomas

#5 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 19 lines · Thomas De Schampheleire

Hi,

Quoted 15 lines · Fabio Porcedda
Quoted 5 lines

Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif
Quoted 8 lines
Quoted 2 lines

What do you think about that?

I don't think this will work if there are more than one prerequisite:
$(1) will be one of them, and DEPENDENCIES_HOST_PREREQ will be the entire list, so that 'ifneq' will always be true.
I think you need 'filter' here.
http://www.gnu.org/software/make/manual/make.html#index-filter-587

Yes it works fine with filter, thanks for suggesting it.

Quoted 2 lines

Best regards,
Thomas

Thanks and regards

Signature · 1 line
Fabio Porcedda
#6 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

In reply to Fabio Porcedda
Quoted 25 lines · Fabio Porcedda

Hi Arnout,
thanks for reviewing the patch set.

Quoted 10 lines · Arnout Vandecappelle
Quoted 9 lines · Fabio Porcedda

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

Quoted 8 lines
Quoted 3 lines

This is also a step forward supporting top-level parallel make.

Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.

I've found a work around:

Quoted 8 lines

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

Looks good except that it's incorrect :-) If HOST_PREREQ contains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You can try something like:

ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)

[Just noticed now that Thomas gave the same answer.]

Quoted 21 lines
Quoted 16 lines
Quoted 16 lines

---
   package/pkg-generic.mk           | 2 ++
   toolchain/toolchain/toolchain.mk | 3 +--
   2 files changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 4bba4b5..1e7154e 100644
@@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build
 $$($(2)_TARGET_INSTALL_HOST)
   $(1)-build:           $(1)-configure \
                         $$($(2)_TARGET_BUILD)
 
Quoted 4 lines

package/pkg-generic.mk $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) · line 388

Quoted 1 line
package/pkg-generic.mk +1 -0
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare

Is there any reason why you changed the order here?

Quoted 8 lines

The order whit top-level parallel make does not matter, it's just that i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directories

But now i understand that is "dirs prepare dependencies" or "dirs dependencies prepare" because dirs can require host-ccache and so

You mean dependencies can require host-ccache, right? And host-ccache
depends on both dirs and prepare itself... So I do think the original order is more logical.

Regards,
Arnout

Quoted 2 lines

directories,
nevertheless this does not matter anymore.

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#7 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 41 lines · Arnout Vandecappelle
Quoted 40 lines · Fabio Porcedda

Hi Arnout,
thanks for reviewing the patch set.

Quoted 18 lines · Arnout Vandecappelle
Quoted 10 lines · Fabio Porcedda

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

Quoted 3 lines

This is also a step forward supporting top-level parallel make.

Quoted 5 lines

Although this is one of the most useful patches of the series, it is
also
the one that introduces the circular dependency. So although it looks good,
I'm not ready to ack it.

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif
Quoted 8 lines
Quoted 1 line

What do you think about that?

Looks good except that it's incorrect :-) If HOST_PREREQ contains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You can try something like:

ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)

Ok good, I've updated the patch according.

Quoted 34 lines

[Just noticed now that Thomas gave the same answer.]

Quoted 30 lines
Quoted 16 lines
Quoted 16 lines

---
   package/pkg-generic.mk           | 2 ++
   toolchain/toolchain/toolchain.mk | 3 +--
   2 files changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 4bba4b5..1e7154e 100644
@@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build
 $$($(2)_TARGET_INSTALL_HOST)
   $(1)-build:           $(1)-configure \
                         $$($(2)_TARGET_BUILD)
 
Quoted 6 lines
Quoted 2 lines
package/pkg-generic.mk +1 -0
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs

prepare

Is there any reason why you changed the order here?

The order whit top-level parallel make does not matter, it's just that i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directories

Quoted 7 lines
Quoted 2 lines

But now i understand that is "dirs prepare dependencies" or "dirs dependencies prepare" because dirs can require host-ccache and so

You mean dependencies can require host-ccache, right? And host-ccache
depends on both dirs and prepare itself... So I do think the original order is more logical.

Yes the final order is this: dirs prepare dependencies.

I moved dirs in front because i think creating directories it's the first thing to do.

Quoted 12 lines

Regards,
Arnout

Quoted 2 lines

directories,
nevertheless this does not matter anymore.

[snip]

Quoted 7 lines

--

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Thanks and regards

Signature · 1 line
Fabio Porcedda
#8 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 3 lines · Fabio Porcedda
Quoted 2 lines · Arnout Vandecappelle
Quoted 2 lines · Fabio Porcedda
Quoted 1 line
Quoted 1 line

[snip]

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 10 lines
Quoted 10 lines

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Looks good except that it's incorrect:-) If HOST_PREREQ contains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You can try something like:

ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)

Ok good, I've updated the patch according.

Have you tested it with various combinations of dependencies?

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#9 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 29 lines · Arnout Vandecappelle
Quoted 7 lines · Fabio Porcedda
Quoted 4 lines · Arnout Vandecappelle
Quoted 4 lines · Fabio Porcedda
Quoted 2 lines
Quoted 2 lines

[snip]

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

Looks good except that it's incorrect:-) If HOST_PREREQ contains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You
can try something like:

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)

Ok good, I've updated the patch according.

Have you tested it with various combinations of dependencies?

I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with qemu_x86_defconfig and seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.

I'm testing the whole patch set with top-level parallel make and enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".

Thanks and regards

Signature · 1 line
Fabio Porcedda
#10 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 37 lines · Fabio Porcedda
Quoted 29 lines · Arnout Vandecappelle
Quoted 7 lines · Fabio Porcedda
Quoted 4 lines · Arnout Vandecappelle
Quoted 4 lines · Fabio Porcedda
Quoted 2 lines
Quoted 2 lines

[snip]

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

Looks good except that it's incorrect:-) If HOST_PREREQ contains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You
can try something like:

Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)

Ok good, I've updated the patch according.

Have you tested it with various combinations of dependencies?

Quoted 7 lines

I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with qemu_x86_defconfig and seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.

I'm testing the whole patch set with top-level parallel make and enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".

The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and disabling /usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.

Regards

Signature · 1 line
Fabio Porcedda
#11 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Hi Fabio,

On Tue, Oct 29, 2013 at 9:36 AM, Fabio Porcedda
<[email protected]> wrote:
[..]

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Have you tested it with various combinations of dependencies?

I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with qemu_x86_defconfig and seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.

Quoted 8 lines
Quoted 2 lines

I'm testing the whole patch set with top-level parallel make and enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".

The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and disabling /usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.

Can you elaborate on this? What is the problem you're seeing exactly, and how did you 'disable /usr/bin/xz' ?
When buildroot cannot find a working xzcat version, it builds one itself through host-xz. This should still work in case of parallel builds.
The same principle applies when the 'tar' version on host is non-existing or too old.

Thanks,
Thomas

#12 lore raw

PATCH RESEND v8 1/8 package: add base dependency to every package

Quoted 15 lines · Thomas De Schampheleire

Hi Fabio,

On Tue, Oct 29, 2013 at 9:36 AM, Fabio Porcedda
<[email protected]> wrote:
[..]

Quoted 10 lines
Quoted 9 lines
Quoted 1 line

Have you tested it with various combinations of dependencies?

I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with qemu_x86_defconfig and seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.

I'm testing the whole patch set with top-level parallel make and enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".

Quoted 8 lines
Quoted 5 lines

The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and disabling /usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.

Can you elaborate on this? What is the problem you're seeing exactly, and how did you 'disable /usr/bin/xz' ?

To disable /usr/bin/xz i've just renamed it so buildroot doesn't find a suitable xz.
The host-ccache source file is a tar.xz so the host-xz is a dependency of host-ccache.
I need to add a rule to ensure the right order.

Quoted 5 lines

When buildroot cannot find a working xzcat version, it builds one itself through host-xz. This should still work in case of parallel builds.
The same principle applies when the 'tar' version on host is non-existing or too old.

Yes i think is affected by the same problem.

Thanks

Signature · 1 line
Fabio Porcedda
#13 lore raw

PATCH RESEND v8 2/8 package: add toolchain dependency to every target package

In reply to Fabio Porcedda

This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:

make clean <package-name>

Also remove TARGETS_ALL because the only purpose was to add toolchain dependency so it's superseded by this commit.

To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN for not adding the toolchain dependency for toolchain packages.

This is also a step forward supporting top-level parallel make.

---
 Makefile                                             |  8 ++------
 package/glibc/glibc.mk                               |  3 +++
 package/linux-headers/linux-headers.mk               |  3 +++
 package/pkg-autotools.mk                             |  3 ++-
 package/pkg-generic.mk                               | 12 ++++++++++--
 package/uclibc/uclibc.mk                             |  3 +++
 toolchain/toolchain-buildroot/toolchain-buildroot.mk |  2 ++
 toolchain/toolchain-external/toolchain-external.mk   |  2 ++
 toolchain/toolchain/toolchain.mk                     |  2 ++
 9 files changed, 29 insertions(+), 9 deletions(-)
Makefile +2 -6
index f266e2d..ebecec4 100644
@@ -352,7 +352,6 @@ TARGETS+=target-post-image
 TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
-TARGETS_ALL:=$(patsubst %,__real_tgt_%,$(TARGETS))
 
 # host-* dependencies have to be handled specially, as those aren't
 # visible in Kconfig and hence not added to a variable like TARGETS.
@@ -375,9 +374,6 @@ HOST_SOURCE += $(addsuffix -source,$(sort $(TARGETS_HOST_DEPS) $(HOST_DEPS)))
 TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 		$(TARGETS) $(BASE_TARGETS) $(TARGETS_HOST_DEPS) $(HOST_DEPS))))
 
-# all targets depend on the crosscompiler and it's prerequisites
-$(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
-
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
@@ -386,11 +382,11 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(BASE_TARGETS) $(TARGETS_ALL)
+world: $(TARGETS)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
+	$(BASE_TARGETS) $(TARGETS) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
package/glibc/glibc.mk +3 -0
index 0dab492..68b98a9 100644
@@ -19,6 +19,9 @@ endif
 GLIBC_LICENSE = GPLv2+ (programs), LGPLv2.1+, BSD-3c, MIT (library)
 GLIBC_LICENSE_FILES = $(addprefix $(GLIBC_SRC_SUBDIR)/,COPYING COPYING.LIB LICENSES)
 
+# glibc is part of the toolchain so disable the toolchain dependency
+GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before (e)glibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
package/linux-headers/linux-headers.mk +3 -0
index 30d3076..6dac9e3 100644
@@ -17,6 +17,9 @@ LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 
 LINUX_HEADERS_INSTALL_STAGING = YES
 
+# linux-headers is part of the toolchain so disable the toolchain dependency
+LINUX_HEADERS_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 define LINUX_HEADERS_INSTALL_STAGING_CMDS
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) $(MAKE) \
package/pkg-autotools.mk +2 -1
index 9523529..f58570d 100644
@@ -209,7 +209,8 @@ endef
 # This must be repeated from inner-generic-package, otherwise we get an empty
 # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
 # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\
     $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
package/pkg-generic.mk +10 -2
index 1e7154e..3f19aea 100644
@@ -307,8 +307,16 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
-
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+# When a target package is a toolchain dependency set this variable to
+# 'NO' so the 'toolchain' dependency is not added to prevent a circular
+# dependency
+$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
+$(2)_DEPENDENCIES += toolchain
+endif
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
package/uclibc/uclibc.mk +3 -0
index 20d3bb6..0d12afd 100644
@@ -18,6 +18,9 @@ endif
 
 UCLIBC_INSTALL_STAGING = YES
 
+# uclibc is part of the toolchain so disable the toolchain dependency
+UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before uClibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
toolchain/toolchain-buildroot/toolchain-buildroot.mk +2 -0
index f1f07a9..d941854 100644
@@ -14,4 +14,6 @@ BUILDROOT_LIBC = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_LIBC))
 
 TOOLCHAIN_BUILDROOT_DEPENDENCIES = host-gcc-final
 
+TOOLCHAIN_BUILDROOT_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
toolchain/toolchain-external/toolchain-external.mk +2 -0
index d41cc7c..2076cf7 100644
@@ -340,6 +340,8 @@ TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 endif
 
+TOOLCHAIN_EXTERNAL_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 TOOLCHAIN_EXTERNAL_INSTALL_STAGING = YES
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R1)$(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R2),y)
toolchain/toolchain/toolchain.mk +2 -0
index 8559ac9..7241fe7 100644
@@ -12,6 +12,8 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
1.8.4
#14 lore raw

PATCH RESEND v8 2/8 package: add toolchain dependency to every target package

Quoted 9 lines · Fabio Porcedda

This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:

make clean <package-name>

Also remove TARGETS_ALL because the only purpose was to add toolchain dependency so it's superseded by this commit.

Quoted 6 lines

To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN for not adding the toolchain dependency for toolchain packages.

This is also a step forward supporting top-level parallel make.

I'm not happy with this _ADD_TOOLCHAIN_DEPENDENCY variable.

What I would prefer is to move glibc, linux-headers and uclibc back
under the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)

ifeq ($(5),target)
ifneq ($(pkgparentdir),toolchain)
$(2)_DEPENDENCIES += toolchain
endif
endif

[snip]

package/pkg-autotools.mk endef · line 213

Quoted 11 lines
package/pkg-autotools.mk +2 -1
index 9523529..f58570d 100644
@@ -209,7 +209,8 @@ endef
  # This must be repeated from inner-generic-package, otherwise we get an empty
  # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
  # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\

You'll also need to do this in pkg-cmake.mk.

Quoted 11 lines
package/pkg-autotools.mk +0 -0
      $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
package/pkg-generic.mk +0 -0
index 1e7154e..3f19aea 100644
@@ -307,8 +307,16 @@ endif
 
  $(2)_REDISTRIBUTE		?= YES
 

package/pkg-generic.mk endif · line 317

Quoted 10 lines
package/pkg-generic.mk +8 -2
-
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+# When a target package is a toolchain dependency set this variable to
+# 'NO' so the 'toolchain' dependency is not added to prevent a circular
+# dependency
+$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)

It's more readable if you split this and-ing in two separate conditions.

For $(2)_TYPE, you can directly use $(5) because you're anyway inside
inner-generic-package here.

Regards,
Arnout

Quoted 5 lines
package/pkg-generic.mk +2 -0
+$(2)_DEPENDENCIES += toolchain
+endif
 
  $(2)_INSTALL_STAGING		?= NO
  $(2)_INSTALL_IMAGES		?= NO

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#15 lore raw

PATCH RESEND v8 2/8 package: add toolchain dependency to every target package

Dear Arnout Vandecappelle,

Quoted 3 lines · Arnout Vandecappelle

What I would prefer is to move glibc, linux-headers and uclibc back
under the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)

Well, the idea is that gcc, uclibc, glibc, linux-headers and so on are now real, normal, packages. The only thing we keep in toolchain/ are the "entry" point of toolchain backends: toolchain-buildroot and toolchain-external.

Moreover, it would be strange to have glibc, uclibc and linux-headers in toolchain/, but not gcc and binutils, for example. I do understand that these two packages are host packages, so they don't get the automatic dependency on the toolchain added, but that's an internal detail. From an external point of view, all those components are part of the toolchain, so it would be weird to have some of them in toolchain/, and some of them in package/.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#16 lore raw

PATCH RESEND v8 2/8 package: add toolchain dependency to every target package

Quoted 13 lines · Thomas Petazzoni

Dear Arnout Vandecappelle,

Quoted 3 lines · Arnout Vandecappelle

What I would prefer is to move glibc, linux-headers and uclibc back
under the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)

Well, the idea is that gcc, uclibc, glibc, linux-headers and so on are now real, normal, packages. The only thing we keep in toolchain/ are the "entry" point of toolchain backends: toolchain-buildroot and toolchain-external.

Quoted 7 lines

Moreover, it would be strange to have glibc, uclibc and linux-headers in toolchain/, but not gcc and binutils, for example. I do understand that these two packages are host packages, so they don't get the automatic dependency on the toolchain added, but that's an internal detail. From an external point of view, all those components are part of the toolchain, so it would be weird to have some of them in toolchain/, and some of them in package/.

So what's your conclusion now? Move all toolchain-related packages back
to the toolchain directory, or find another solution for circumventing the toolchain circular dependency?

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#17 lore raw

PATCH RESEND v8 2/8 package: add toolchain dependency to every target package

In reply to Arnout Vandecappelle
Quoted 44 lines · Arnout Vandecappelle
Quoted 10 lines · Fabio Porcedda

This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:

make clean <package-name>

Also remove TARGETS_ALL because the only purpose was to add toolchain dependency so it's superseded by this commit.

Quoted 6 lines

To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN for not adding the toolchain dependency for toolchain packages.

This is also a step forward supporting top-level parallel make.

I'm not happy with this _ADD_TOOLCHAIN_DEPENDENCY variable.

What I would prefer is to move glibc, linux-headers and uclibc back under
the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)

ifeq ($(5),target)
ifneq ($(pkgparentdir),toolchain)
$(2)_DEPENDENCIES += toolchain
endif
endif

[snip]

Quoted 10 lines
package/pkg-autotools.mk +0 -0
index 9523529..f58570d 100644
@@ -209,7 +209,8 @@ endef
  # This must be repeated from inner-generic-package, otherwise we get an

empty
# _DEPENDENCIES if _AUTORECONF is YES. Also filter the result of
_AUTORECONF

  # away from the non-host rule
Quoted 8 lines
Quoted 5 lines

-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+ host-toolchain $(1),\

You'll also need to do this in pkg-cmake.mk.

Ok, done.

Quoted 29 lines
Quoted 15 lines
      $(patsubst host-host-%,host-%,$(addprefix

host-,$($(3)_DEPENDENCIES))))

package/pkg-generic.mk +0 -2
index 1e7154e..3f19aea 100644
@@ -307,8 +307,16 @@ endif
 
  $(2)_REDISTRIBUTE             ?= YES
 
-
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst

package/pkg-generic.mk endif · line 319

Quoted 10 lines
package/pkg-generic.mk +8 -0
 host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+# When a target package is a toolchain dependency set this variable to
+# 'NO' so the 'toolchain' dependency is not added to prevent a circular
+# dependency
+$(2)_ADD_TOOLCHAIN_DEPENDENCY  ?= YES
+
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+       $(patsubst host-host-%,host-%,$(addprefix
 host-,$($(3)_DEPENDENCIES))))
+ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)

It's more readable if you split this and-ing in two separate conditions.

Quoted 2 lines

For $(2)_TYPE, you can directly use $(5) because you're anyway inside
inner-generic-package here.

Ok, done.

Best regards

Signature · 1 line
Fabio Porcedda
#18 lore raw

PATCH RESEND v8 3/8 package: add support for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:
%/.stamp_extracted
%/.stamp_patched
%/.stamp_configured
%/.stamp_built
%/.stamp_host_installed
%/.stamp_staging_installed
%/.stamp_images_installed
%/.stamp_target_installed

Because the %-build target is not anymore part of the dependency chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by uclibc and glibc packages.

---
 package/glibc/glibc.mk   |  2 +-
 package/pkg-generic.mk   | 38 +++++++++++++++++++++-----------------
 package/uclibc/uclibc.mk |  2 +-
 3 files changed, 23 insertions(+), 19 deletions(-)
package/glibc/glibc.mk +1 -1
index 68b98a9..e89c12a 100644
@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
 
 # Before (e)glibc is built, we must have the second stage cross-compiler
-glibc-build: host-gcc-intermediate
+GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
 GLIBC_SUBDIR = build
 
package/pkg-generic.mk +21 -17
index 3f19aea..094868c 100644
@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
 ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
 $(2)_DEPENDENCIES += toolchain
 endif
+$(2)_BUILD_DEPENDENCIES		?=
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
@@ -368,30 +369,34 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
 endif
 
 ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
 else
 $(1)-install-target:
 endif
 
 ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
 else
 $(1)-install-staging:
 endif
 
 ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
 else
 $(1)-install-images:
 endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:		$(1)-configure \
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+	$$($(2)_TARGET_BUILD)
+
+$(1)-build:		$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES)
+
+$(1)-configure:			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES)
 
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
 
@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
 
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:			$$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
@@ -418,10 +423,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
 $(1)-patch:		$(1)-rsync
 $(1)-extract:		$(1)-rsync
package/uclibc/uclibc.mk +1 -1
index 0d12afd..dea36e7 100644
@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
 # Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
+UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
Signature · 1 line
1.8.4
#19 lore raw

PATCH RESEND v8 3/8 package: add support for top-level parallel make

Quoted 2 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on

nitpick: relying

Quoted 11 lines

the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:

    %/.stamp_extracted
    %/.stamp_patched
    %/.stamp_configured
    %/.stamp_built
    %/.stamp_host_installed
    %/.stamp_staging_installed
    %/.stamp_images_installed
    %/.stamp_target_installed

Your description here is not entirely accurate, because this makes it
look as if the dependencies are added for the pattern rules.

Quoted 17 lines

Because the %-build target is not anymore part of the dependency chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by uclibc and glibc packages.

---
  package/glibc/glibc.mk   |  2 +-
  package/pkg-generic.mk   | 38 +++++++++++++++++++++-----------------
  package/uclibc/uclibc.mk |  2 +-
  3 files changed, 23 insertions(+), 19 deletions(-)
package/glibc/glibc.mk +0 -0
index 68b98a9..e89c12a 100644

package/glibc/glibc.mk GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO · line 30

Quoted 5 lines
package/glibc/glibc.mk +1 -1
@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
  GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
 
  # Before (e)glibc is built, we must have the second stage cross-compiler
-glibc-build: host-gcc-intermediate
+GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate

I don't like this. (You'll notice I generally don't like adding new
variables :-). This build-dependency is really an exceptional situation, so adding generic package infrastructure for it feels wrong to me. Instead, you can keep the explicit dependency that was there already, only now it is even more explicit:

$(GLIBC_TARGET_BUILD): host-gcc-intermediate

Note that GLIBC_TARGET_BUILD is only defined after the
$(eval $(autotools-package)) so it will have to be moved down as well.

Quoted 16 lines
package/glibc/glibc.mk +0 -0
  GLIBC_SUBDIR = build
 
package/pkg-generic.mk +1 -0
index 3f19aea..094868c 100644
@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
  ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
  $(2)_DEPENDENCIES += toolchain
  endif
+$(2)_BUILD_DEPENDENCIES		?=
 
  $(2)_INSTALL_STAGING		?= NO
  $(2)_INSTALL_IMAGES		?= NO

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 370

Quoted 2 lines
package/pkg-generic.mk +0 -0
@@ -368,30 +369,34 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
  endif
 

I would like to see somewhere a comment explaining how the dependencies
between the steps are handled. In particular, explaining that this is not done in the pattern rules because then make would remove the stamp files and do too much rebuilding.

Quoted 22 lines
package/pkg-generic.mk +3 -6
  ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
  else
  $(1)-install-target:
  endif
 
  ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
  else
  $(1)-install-staging:
  endif
 
  ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
  else
  $(1)-install-images:

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 393

Quoted 10 lines
package/pkg-generic.mk +4 -3
  endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:		$(1)-configure \
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+	$$($(2)_TARGET_BUILD)

I think it would also be possible to turn this into a dependency on
$(1)-build instead of $$($(2)_TARGET_BUILD). That would also avoid the need for _BUILD_DEPENDENCIES and maybe some other changes.

Of course, to be able to do that, you have to add
.PHONY: $(1)-build
otherwise everything will always be rebuilt.

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 399

Quoted 6 lines
package/pkg-generic.mk +6 -0
+
+$(1)-build:		$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES)
+
+$(1)-configure:			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES)

This warrants a comment explaining why you use | here.

Quoted 11 lines
package/pkg-generic.mk +1 -2
  $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
 
@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
  #  extract
  #  patch
  #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
 

package/pkg-generic.mk ifeq ($$($(2)_OVERRIDE_SRCDIR),) · line 418

Quoted 10 lines
package/pkg-generic.mk +4 -3
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:			$$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
 
  $(1)-depends:		$$($(2)_DEPENDENCIES)

Is the -depends target still useful? Does anybody ever run 'make
foo-depends'?

Regards,
Arnout

Quoted 28 lines
package/pkg-generic.mk +2 -3
@@ -418,10 +423,9 @@ else
  #  source, by rsyncing
  #  depends
  #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
  $(1)-patch:		$(1)-rsync
  $(1)-extract:		$(1)-rsync
package/uclibc/uclibc.mk +1 -1
index 0d12afd..dea36e7 100644
@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
  UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
  # Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
+UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
  # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
  # setting.
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#20 lore raw

PATCH RESEND v8 3/8 package: add support for top-level parallel make

Quoted 7 lines · Arnout Vandecappelle
Quoted 3 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on

nitpick: relying

Ok, done.

Quoted 14 lines
Quoted 4 lines

the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:

Quoted 8 lines
    %/.stamp_extracted
    %/.stamp_patched
    %/.stamp_configured
    %/.stamp_built
    %/.stamp_host_installed
    %/.stamp_staging_installed
    %/.stamp_images_installed
    %/.stamp_target_installed
Quoted 2 lines

Your description here is not entirely accurate, because this makes it look
as if the dependencies are added for the pattern rules.

Ok, I'll use $(2)_TARGET_%

Quoted 30 lines
Quoted 17 lines

Because the %-build target is not anymore part of the dependency chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by uclibc and glibc packages.

---
  package/glibc/glibc.mk   |  2 +-
  package/pkg-generic.mk   | 38 +++++++++++++++++++++-----------------
  package/uclibc/uclibc.mk |  2 +-
  3 files changed, 23 insertions(+), 19 deletions(-)
package/glibc/glibc.mk +0 -0
index 68b98a9..e89c12a 100644

package/glibc/glibc.mk GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO · line 30

Quoted 5 lines
package/glibc/glibc.mk +1 -1
@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
  GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
 
  # Before (e)glibc is built, we must have the second stage cross-compiler
-glibc-build: host-gcc-intermediate
+GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate

I don't like this. (You'll notice I generally don't like adding new
variables :-). This build-dependency is really an exceptional situation, so adding generic package infrastructure for it feels wrong to me. Instead, you can keep the explicit dependency that was there already, only now it is even more explicit:

Quoted 5 lines

$(GLIBC_TARGET_BUILD): host-gcc-intermediate

Note that GLIBC_TARGET_BUILD is only defined after the
$(eval $(autotools-package)) so it will have to be moved down as well.

Ok, done.

Quoted 14 lines
Quoted 14 lines
package/glibc/glibc.mk +0 -0
  GLIBC_SUBDIR = build
 
package/pkg-generic.mk +1 -0
index 3f19aea..094868c 100644
@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out  host-toolchain
 $(1),\
  ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
  $(2)_DEPENDENCIES += toolchain
  endif
+$(2)_BUILD_DEPENDENCIES                ?=
 
Quoted 11 lines
Quoted 6 lines
package/pkg-generic.mk +0 -0
  $(2)_INSTALL_STAGING          ?= NO
  $(2)_INSTALL_IMAGES           ?= NO
package/pkg-generic.mk +0 -0
@@ -368,30 +369,34 @@ $(1)-install:             $(1)-install-staging

$(1)-install-target $(1)-install-images
endif

I would like to see somewhere a comment explaining how the dependencies
between the steps are handled. In particular, explaining that this is not done in the pattern rules because then make would remove the stamp files and do too much rebuilding.

Do you mean to add a comment in the description of the commit? My explanation is:

We cannot use the pattern rules because they must have the same dependency for every package,
but we need to change the dependencies depending on $(2)_OVERRIDE_SRCDIR variable value,
so we must use a more flexible way like $(2)_TARGET_% variables.

Quoted 35 lines
Quoted 23 lines

ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target: $(1)-build \

package/pkg-generic.mk +3 -5
-                       $$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:   $$($(2)_TARGET_INSTALL_TARGET)
  else
  $(1)-install-target:
  endif
 
  ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:  $(1)-build \
-                       $$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:  $$($(2)_TARGET_INSTALL_STAGING)
  else
  $(1)-install-staging:
  endif
 
  ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:   $(1)-build \
-                       $$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:   $$($(2)_TARGET_INSTALL_IMAGES)
  else
  $(1)-install-images:

package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 393

Quoted 10 lines
package/pkg-generic.mk +4 -3
  endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:            $(1)-configure \
-                       $$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+       $$($(2)_TARGET_BUILD)
Quoted 3 lines

I think it would also be possible to turn this into a dependency on
$(1)-build instead of $$($(2)_TARGET_BUILD). That would also avoid the need for _BUILD_DEPENDENCIES and maybe some other changes.

That's not possible because the $$($(2)_TARGET_BUILD) must depends on $$($(2)_BUILD_DEPENDENCIES) because
if build depends on $$($(2)_TARGET_BUILD) and the contents of $$($(2)_BUILD_DEPENDENCIES) they can be built in parallel and that's is wrong.

Quoted 6 lines

Of course, to be able to do that, you have to add
.PHONY: $(1)-build
otherwise everything will always be rebuilt.

Quoted 10 lines
Quoted 7 lines

+
+$(1)-build: $$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES)
+
+$(1)-configure: $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE): | $$($(2)_DEPENDENCIES)

This warrants a comment explaining why you use | here.

Ok.

Quoted 22 lines
Quoted 22 lines

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs
prepare

package/pkg-generic.mk +5 -5
@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
  #  extract
  #  patch
  #  configure
-$(1)-configure:                $(1)-patch $(1)-depends \
-                       $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):     $$($(2)_TARGET_PATCH)
 
-$(1)-patch:            $(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:            $$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:          $(1)-source \
-                       $$($(2)_TARGET_EXTRACT)
+$(1)-extract:                  $$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):       $$($(2)_TARGET_SOURCE)
 
Quoted 5 lines

package/pkg-generic.mk ifeq ($$($(2)_OVERRIDE_SRCDIR),) · line 418

Quoted 1 line
package/pkg-generic.mk +0 -0
  $(1)-depends:         $$($(2)_DEPENDENCIES)

Is the -depends target still useful? Does anybody ever run 'make
foo-depends'?

I don't know.

Quoted 37 lines

Regards,
Arnout

Quoted 27 lines
package/pkg-generic.mk +2 -3
@@ -418,10 +423,9 @@ else
  #  source, by rsyncing
  #  depends
  #  configure
-$(1)-configure:                $(1)-depends \
-                       $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):     $$($(2)_TARGET_RSYNC)
 
-$(1)-depends:          $(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:          $$($(2)_DEPENDENCIES)
 
  $(1)-patch:           $(1)-rsync
  $(1)-extract:         $(1)-rsync
package/uclibc/uclibc.mk +1 -1
index 0d12afd..dea36e7 100644
@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
  UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
  # Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
+UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
  # specifying UCLIBC_CONFIG_FILE on the command-line overrides the

.config

Quoted 2 lines
  # setting.

--

Quoted 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Best regards

Signature · 1 line
Fabio Porcedda
#21 lore raw

PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image

---
 Makefile     | 15 ++++++---------
 fs/common.mk |  4 ++--
 2 files changed, 8 insertions(+), 11 deletions(-)
Makefile +6 -9
index ebecec4..299d34b 100644
@@ -329,8 +329,6 @@ include boot/common.mk
 include linux/linux.mk
 include system/system.mk
 
-TARGETS+=target-finalize
-
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 TARGETS+=target-purgelocales
 endif
@@ -347,8 +345,6 @@ endif
 
 include fs/common.mk
 
-TARGETS+=target-post-image
-
 TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(TARGETS)
+world: target-post-image
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) \
+	target-finalize target-post-image \
+	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -443,7 +440,7 @@ endif
 STRIP_FIND_CMD += -type f -perm /111
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
 
-target-finalize:
+target-finalize: $(TARGETS)
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
 		$(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
@@ -550,7 +547,7 @@ target-generatelocales: host-localedef
 	done
 endif
 
-target-post-image:
+target-post-image: target-finalize $(TARGETS_ROOTFS)
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
 		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \
 	@rm -f $(LEGAL_WARNINGS)
 
 show-targets:
-	@echo $(TARGETS)
+	@echo $(TARGETS) $(TARGETS_ROOTFS)
 
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
fs/common.mk +2 -2
index 4dab7ea..7698c53 100644
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
 # extra deps
 ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
 
-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
+$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
 	rm -f $$(FAKEROOT_SCRIPT)
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
 rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
 endif
 endef
 
Signature · 1 line
1.8.4
#22 lore raw

PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make

Quoted 9 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image

I think this one is ready to go in, so perhaps you can reorder it as
one of the first ones of the series. Although it serves no function except for the top-level parallel make, so perhaps Peter will object to committing it on its own.

Regards,
Arnout

Quoted 92 lines
---
  Makefile     | 15 ++++++---------
  fs/common.mk |  4 ++--
  2 files changed, 8 insertions(+), 11 deletions(-)
Makefile +6 -9
index ebecec4..299d34b 100644
@@ -329,8 +329,6 @@ include boot/common.mk
  include linux/linux.mk
  include system/system.mk
 
-TARGETS+=target-finalize
-
  ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
  TARGETS+=target-purgelocales
  endif
@@ -347,8 +345,6 @@ endif
 
  include fs/common.mk
 
-TARGETS+=target-post-image
-
  TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
  TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
  TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
  prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(TARGETS)
+world: target-post-image
 
  .PHONY: all world toolchain dirs clean distclean source outputmakefile \
  	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) \
+	target-finalize target-post-image \
+	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
  	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
  	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
  	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -443,7 +440,7 @@ endif
  STRIP_FIND_CMD += -type f -perm /111
  STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
 
-target-finalize:
+target-finalize: $(TARGETS)
  	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
  		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
  		$(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
@@ -550,7 +547,7 @@ target-generatelocales: host-localedef
  	done
  endif
 
-target-post-image:
+target-post-image: target-finalize $(TARGETS_ROOTFS)
  	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
  		$(call MESSAGE,"Executing post-image script $(s)"); \
  		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \
  	@rm -f $(LEGAL_WARNINGS)
 
  show-targets:
-	@echo $(TARGETS)
+	@echo $(TARGETS) $(TARGETS_ROOTFS)
 
  else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
fs/common.mk +2 -2
index 4dab7ea..7698c53 100644
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
  # extra deps
  ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
 
-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
+$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
  	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
  	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
  	rm -f $$(FAKEROOT_SCRIPT)
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
  rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
  ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
  endif
  endef
 
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#23 lore raw

PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make

Quoted 13 lines · Arnout Vandecappelle
Quoted 10 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image

Quoted 6 lines

I think this one is ready to go in, so perhaps you can reorder it as one of
the first ones of the series. Although it serves no function except for the top-level parallel make, so perhaps Peter will object to committing it on its own.

Is a little cumbersome to change the order so i prefer at least for the next iteration not to change it.

Quoted 110 lines

Regards,
Arnout

Quoted 95 lines
---
  Makefile     | 15 ++++++---------
  fs/common.mk |  4 ++--
  2 files changed, 8 insertions(+), 11 deletions(-)
Makefile +3 -6
index ebecec4..299d34b 100644
@@ -329,8 +329,6 @@ include boot/common.mk
  include linux/linux.mk
  include system/system.mk
 
-TARGETS+=target-finalize
-
  ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
  TARGETS+=target-purgelocales
  endif
@@ -347,8 +345,6 @@ endif
 
  include fs/common.mk
 
-TARGETS+=target-post-image
-
  TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
  TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
  TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf:
 $(BUILDROOT_CONFIG)
 
  prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(TARGETS)
+world: target-post-image
 
  .PHONY: all world toolchain dirs clean distclean source outputmakefile \
        legal-info legal-info-prepare legal-info-clean printvars \
-       $(BASE_TARGETS) $(TARGETS) \
+       target-finalize target-post-image \
+       $(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
        $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE)
 $(TARGETS_LEGAL_INFO) \
        $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
        $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
Makefile +0 -0
@@ -443,7 +440,7 @@ endif
  STRIP_FIND_CMD += -type f -perm /111
  STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call

qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

Makefile +1 -1
-target-finalize:
+target-finalize: $(TARGETS)
        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig
$(TARGET_DIR)/usr/share/pkgconfig \
                $(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
Makefile +1 -1
@@ -550,7 +547,7 @@ target-generatelocales: host-localedef
        done
  endif
 
-target-post-image:
+target-post-image: target-finalize $(TARGETS_ROOTFS)
        @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
                $(call MESSAGE,"Executing post-image script $(s)"); \
                $(s) $(BINARIES_DIR) $(call

qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))

Makefile +1 -1
@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare
 $(REDIST_SOURCES_DIR) \
        @rm -f $(LEGAL_WARNINGS)
 
  show-targets:
-       @echo $(TARGETS)
+       @echo $(TARGETS) $(TARGETS_ROOTFS)
 
  else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
fs/common.mk +0 -0
index 4dab7ea..7698c53 100644
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
  # extra deps
  ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if

$$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)

-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)

        @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
        $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call
$$(hook))$$(sep))
        rm -f $$(FAKEROOT_SCRIPT)

fs/common.mk rootfs-$(1)-show-depends: · line 92

Quoted 8 lines
fs/common.mk +1 -1
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
  rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
  ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
  endif
  endef
 

--

Quoted 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Thanks and best regards

Signature · 1 line
Fabio Porcedda
#24 lore raw

PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make

In reply to Fabio Porcedda
Quoted 14 lines · Fabio Porcedda
fs/common.mk +1 -1
index 4dab7ea..7698c53 100644
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
  # extra deps
  ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
 
-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
+$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
  	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
  	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
  	rm -f $$(FAKEROOT_SCRIPT)

fs/common.mk rootfs-$(1)-show-depends: · line 91

Quoted 7 lines
fs/common.mk +1 -1
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
  rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
  ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
  endif
  endef

Oh I forgot: you should also fix up initramfs and iso9660. They add
themselves to TARGETS manually.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#25 lore raw

PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make

Quoted 23 lines · Arnout Vandecappelle
Quoted 22 lines · Fabio Porcedda
fs/common.mk +0 -0
index 4dab7ea..7698c53 100644
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
  # extra deps
  ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if

$$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)

-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)

        @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
        $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call
$$(hook))$$(sep))
        rm -f $$(FAKEROOT_SCRIPT)
fs/common.mk +0 -0
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
  rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
Quoted 9 lines

fs/common.mk rootfs-$(1)-show-depends: · line 91

Quoted 5 lines
fs/common.mk +1 -1
  ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
  endif
  endef

Oh I forgot: you should also fix up initramfs and iso9660. They add
themselves to TARGETS manually.

Done, thanks.

Best regards

Signature · 1 line
Fabio Porcedda
#26 lore raw

PATCH RESEND v8 5/8 glibc: add support for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

The glibc-install-target depends on glibc-install-staging so add a rule for it.

---
 package/glibc/glibc.mk | 2 ++
 1 file changed, 2 insertions(+)
package/glibc/glibc.mk +2 -0
index e89c12a..838fb8c 100644
@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS
 endef
 
 $(eval $(autotools-package))
+
+$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
Signature · 1 line
1.8.4
#27 lore raw

PATCH RESEND v8 5/8 glibc: add support for top-level parallel make

Quoted 9 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

The glibc-install-target depends on glibc-install-staging so add a rule for it.

Quoted 10 lines
---
  package/glibc/glibc.mk | 2 ++
  1 file changed, 2 insertions(+)
package/glibc/glibc.mk +0 -0
index e89c12a..838fb8c 100644

package/glibc/glibc.mk define GLIBC_INSTALL_TARGET_CMDS · line 134

Quoted 5 lines
package/glibc/glibc.mk +2 -0
@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS
  endef
 
  $(eval $(autotools-package))
+
+$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)

Nitpick: I would have added a comment why this is needed.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#28 lore raw

PATCH RESEND v8 5/8 glibc: add support for top-level parallel make

Quoted 7 lines · Arnout Vandecappelle
Quoted 6 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

Quoted 7 lines
Quoted 4 lines

The glibc-install-target depends on glibc-install-staging so add a rule for it.

Added, thanks.

Quoted 12 lines
Quoted 12 lines
---
  package/glibc/glibc.mk | 2 ++
  1 file changed, 2 insertions(+)
package/glibc/glibc.mk +0 -0
index e89c12a..838fb8c 100644
@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS
  endef
 
Quoted 8 lines

package/glibc/glibc.mk define GLIBC_INSTALL_TARGET_CMDS · line 134

Quoted 4 lines
package/glibc/glibc.mk +2 -0
  $(eval $(autotools-package))
+
+$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)

Nitpick: I would have added a comment why this is needed. Regards,
Arnout

I've added:
# The glic-install-target use files from glibc-install-staging

Thanks and best regards

Signature · 1 line
Fabio Porcedda
#29 lore raw

PATCH RESEND v8 6/8 uclibc: add support for top-level parallel make

In reply to Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

The uclibc-install-target depends on uclibc-install-staging so add a rule for it.

---
 package/uclibc/uclibc.mk | 2 ++
 1 file changed, 2 insertions(+)
package/uclibc/uclibc.mk +2 -0
index dea36e7..c8eeb3d 100644
@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch
 	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
 $(eval $(generic-package))
+
+$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
Signature · 1 line
1.8.4
#30 lore raw

PATCH RESEND v8 6/8 uclibc: add support for top-level parallel make

Quoted 9 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

The uclibc-install-target depends on uclibc-install-staging so add a rule for it.

Quoted 10 lines
---
  package/uclibc/uclibc.mk | 2 ++
  1 file changed, 2 insertions(+)
package/uclibc/uclibc.mk +0 -0
index dea36e7..c8eeb3d 100644

package/uclibc/uclibc.mk uclibc-menuconfig: dirs uclibc-patch · line 563

Quoted 5 lines
package/uclibc/uclibc.mk +2 -0
@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch
  	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
  $(eval $(generic-package))
+
+$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)

Same nitpick :-)

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#31 lore raw

PATCH RESEND v8 6/8 uclibc: add support for top-level parallel make

Quoted 7 lines · Arnout Vandecappelle
Quoted 6 lines · Fabio Porcedda

To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.

Quoted 7 lines
Quoted 4 lines

The uclibc-install-target depends on uclibc-install-staging so add a rule for it.

Added, thanks.

Quoted 13 lines
Quoted 13 lines
---
  package/uclibc/uclibc.mk | 2 ++
  1 file changed, 2 insertions(+)
package/uclibc/uclibc.mk +0 -0
index dea36e7..c8eeb3d 100644
@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch
        rm -f
 $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
Quoted 6 lines
Quoted 4 lines

$(eval $(generic-package))
+
+$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)

Same nitpick :-)

Same comment:
# The uclibc-install-target use files from uclibc-install-staging

Thanks and best regards

Signature · 1 line
Fabio Porcedda
#32 lore raw

PATCH RESEND v8 7/8 package: enable jobserver for recursive make

In reply to Fabio Porcedda

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character at the beginning of the recipe line. This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ?+? token to get these special effects.

---
 package/pkg-generic.mk | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
package/pkg-generic.mk +5 -5
index 094868c..122b4f9 100644
@@ -118,21 +118,21 @@ $(BUILD_DIR)/%/.stamp_configured:
 # Build
 $(BUILD_DIR)/%/.stamp_built::
 	@$(call MESSAGE,"Building")
-	$($(PKG)_BUILD_CMDS)
+	+$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call MESSAGE,"Installing to host directory")
-	$($(PKG)_INSTALL_CMDS)
+	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
 	@$(call MESSAGE,"Installing to staging directory")
-	$($(PKG)_INSTALL_STAGING_CMDS)
+	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(call MESSAGE,"Fixing package configuration files") ;\
@@ -146,7 +146,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call MESSAGE,"Installing to images directory")
-	$($(PKG)_INSTALL_IMAGES_CMDS)
+	+$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
@@ -157,7 +157,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
 	$(if $(BR2_INIT_SYSV)$(BR2_INIT_BUSYBOX),\
 		$($(PKG)_INSTALL_INIT_SYSV))
-	$($(PKG)_INSTALL_TARGET_CMDS)
+	+$($(PKG)_INSTALL_TARGET_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
Signature · 1 line
1.8.4
#33 lore raw

PATCH RESEND v8 8/8 Makefile: enable top-level parallel make

In reply to Fabio Porcedda

Enable top-level parallel make because the previous patches have fixed all the know issue about top-level parallel make.

Example:

	make <board>_defconfig
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
---
 Makefile | 3 ---
 1 file changed, 3 deletions(-)
Makefile +0 -3
index 299d34b..9752f96 100644
@@ -42,9 +42,6 @@ export HOSTARCH := $(shell uname -m | \
 	    -e s/macppc/powerpc/\
 	    -e s/sh.*/sh/)
 
-# This top-level Makefile can *not* be executed in parallel
-.NOTPARALLEL:
-
 # absolute path
 TOPDIR:=$(shell pwd)
 CONFIG_CONFIG_IN=Config.in
Signature · 1 line
1.8.4
#34 lore raw

v8 cover Add support for top-level parallel make

In reply to Fabio Porcedda
Quoted 14 lines · Fabio Porcedda

Hi all,
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

With this patch set the top-level parallel make seems to work fine, example:

	make clean
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))
Quoted 7 lines

On my quad core system the building time for qemu_x86_defconfig is 13m versus 11m.

I've tested the qemu_x86_defconfig uclibc/eglibc/glibc.

Best regards
Fabio Porcedda

Hi Fabio,

I've tested this patch series now. I'll give acks and comments for the
individual patches later. But first some overall observations.

When ccache is enabled, make complains about a circular dependency
$(BUILD_DIR)/host-ccache-3.1.8/.stamp_downloaded <- dependencies You'll find something similar for sstrip, and when the host doesn't have a suitable tar or xz. It's not critical because make will break the dependency cycle and still build these packages before the rest, but it's not nice. Unfortunately, I don't see a simple way to avoid it.

In system/system.mk, there are still a few different targets
manipulating inittab; these have to be serialized.

The initramfs rootfs will rebuild the kernel in parallel with building
the cpio archive. This should be fixed, which is probably a bit invasive.

legal-info doesn't work in parallel. Not a critical concern, but
ideally that should be fixed as well.

Regards,
Arnout

Quoted 57 lines

v8:
- rebased over master
- added patche for for base dependency
- added patche for for glibc package
- added patche for for uclibc package
- removed patch already merged
- changed some descriptions
- modified the patch for toolchain dependency to prevent circular dependency
v7:
- add the latest patch
- add to the first patch the <pkgname>_TOOLCHAIN variable
- improve the fifth patch
v6:
- added the fifth patch
- updated the fourth patch adding the install targets
- updated the second patch to remove TARGETS_ALL
v5:
- added the fourth patch
- fixed some typos
- rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
- add support for top-level parallel make for the glibc package
v4:
- rebased over master
- add Acked-by: Thomas Petazzoni on the third patch
- changed the orderd of the patches
v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the third patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.

Fabio Porcedda (8):
package: add base dependency to every package
package: add toolchain dependency to every target package
package: add support for top-level parallel make
Makefile: add support for top-level parallel make
glibc: add support for top-level parallel make
uclibc: add support for top-level parallel make
package: enable jobserver for recursive make
Makefile: enable top-level parallel make

  Makefile                                           | 22 +++-----
  fs/common.mk                                       |  4 +-
  package/glibc/glibc.mk                             |  7 ++-
  package/linux-headers/linux-headers.mk             |  3 ++
  package/pkg-autotools.mk                           |  3 +-
  package/pkg-generic.mk                             | 60 +++++++++++++---------
  package/uclibc/uclibc.mk                           |  7 ++-
  .../toolchain-buildroot/toolchain-buildroot.mk     |  2 +
  toolchain/toolchain-external/toolchain-external.mk |  2 +
  toolchain/toolchain/toolchain.mk                   |  5 +-
  10 files changed, 69 insertions(+), 46 deletions(-)
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#35 lore raw

v8 cover Add support for top-level parallel make

In reply to Fabio Porcedda

Fabio, Arnout,

Quoted 7 lines · Fabio Porcedda

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:

(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.

(2) That being said, Fabio's patches have anyway a merit on their own since they make things cleaner, and that therefore, patches 1 to 6 can be merged. Maybe patch 7 even if I'm not sure it makes sense to have it without the top-level parallel build.

What do you think?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#36 lore raw

v8 cover Add support for top-level parallel make

Quoted 12 lines · Thomas Petazzoni

Fabio, Arnout,

Quoted 7 lines · Fabio Porcedda

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

Quoted 5 lines

I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:

(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.

What do you think if we provide an easy way to use top-level parallel make that warns about the problems:

ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!) BR2_JLEVEL=
else
# This top-level Makefile can *not* be executed in parallel .NOTPARALLEL:
endif

make P=1 -j5

Quoted 7 lines

(2) That being said, Fabio's patches have anyway a merit on their own since they make things cleaner, and that therefore, patches 1 to 6 can be merged. Maybe patch 7 even if I'm not sure it makes sense to have it without the top-level parallel build.

What do you think?

Quoted 7 lines

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Best regards

Signature · 1 line
Fabio Porcedda
#37 lore raw

v8 cover Add support for top-level parallel make

Quoted 32 lines · Fabio Porcedda
Quoted 12 lines · Thomas Petazzoni

Fabio, Arnout,

Quoted 7 lines · Fabio Porcedda

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

Quoted 5 lines

I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:

(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.

What do you think if we provide an easy way to use top-level parallel make that warns about the problems:

ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!) BR2_JLEVEL=
else
# This top-level Makefile can *not* be executed in parallel .NOTPARALLEL:
endif

Quoted 7 lines

make P=1 -j5

Quoted 4 lines

(2) That being said, Fabio's patches have anyway a merit on their own since they make things cleaner, and that therefore, patches 1 to 6 can be merged. Maybe patch 7 even if I'm not sure it makes sense to have it without the top-level parallel build.

The patch 7 make sense only for top-level parallel build but i suggest to accept it because does not harm the serial building and is needed for someone that want to try to use top-level parallel make.

Quoted 1 line
Quoted 1 line

What do you think?

Is great to have most of the patch accepted, i think it's a step in the right direction.
I will send the patch set v9 soon.

Quoted 10 lines
Quoted 8 lines

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Quoted 3 lines

Best regards
--
Fabio Porcedda

Thanks and best regards

Signature · 1 line
Fabio Porcedda
#38 lore raw

v8 cover Add support for top-level parallel make

In reply to Fabio Porcedda
Quoted 17 lines · Fabio Porcedda
Quoted 15 lines · Thomas Petazzoni

Fabio, Arnout,

Quoted 7 lines · Fabio Porcedda

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:

Quoted 8 lines
Quoted 2 lines

(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.

What do you think if we provide an easy way to use top-level parallel make that warns about the problems:

ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!)

Such a warning will not really be visible, because it appears at the
very beginning of the build and is immediately followed by all of the build output...

Quoted 5 lines

BR2_JLEVEL=
else
# This top-level Makefile can *not* be executed in parallel .NOTPARALLEL:
endif

Since you only need to make this simple change in the Makefile (remove
.NOTPARALLEL), I think it's a lot more effective for people who want top-level parallel builds to remove this line in their copy of buildroot. We also agreed at the BR developer meeting that we would add an explanatory comment at that location to make it clear why we disable parallel builds. That way, someone who wants to enable it will be aware of the risks.

Regards,
Arnout

Quoted 3 lines

   make P=1 -j5

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#39 lore raw

v8 cover Add support for top-level parallel make

Quoted 44 lines · Arnout Vandecappelle
Quoted 24 lines · Fabio Porcedda
Quoted 14 lines · Thomas Petazzoni

Fabio, Arnout,

Quoted 7 lines · Fabio Porcedda

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

Quoted 5 lines

I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:

(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.

Quoted 5 lines

What do you think if we provide an easy way to use top-level parallel make that warns about the problems:

ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!)

Such a warning will not really be visible, because it appears at the very
beginning of the build and is immediately followed by all of the build output...

Quoted 5 lines

BR2_JLEVEL=
else
# This top-level Makefile can *not* be executed in parallel .NOTPARALLEL:
endif

Quoted 6 lines

Since you only need to make this simple change in the Makefile (remove
.NOTPARALLEL), I think it's a lot more effective for people who want top-level parallel builds to remove this line in their copy of buildroot. We also agreed at the BR developer meeting that we would add an explanatory comment at that location to make it clear why we disable parallel builds. That way, someone who wants to enable it will be aware of the risks.

Ok, I will just change the comment in the latest commit, i will add this comment:
-# This top-level Makefile can *not* be executed in parallel +# The parallel execution of this top-level Makefile is disabled +# because it would make builds non-reproducible also some packages +# have unspecified optional dependencies so if those dependencies are +# present when the package is built, they are used, otherwise they +# aren't (but compilation happily proceeds). This means that the +# package order is relevant in this case, and the end result will +# differ if the order is swapped due to parallel building. +# Taking in account the above warnings, if you still want to execute +# this top-level Makefile in parrallel remove/comment the following +# line and execute:
+# make BR2_JLEVEL= -j$((getconf _NPROCESSORS_ONLN+1))

I've used a slightly modified description give by Thomas De Schampheleire. What do you think about that?

I think that there two problems:
- first: non reproducible build, that can be resolved just recording the building steps and replaying it.
- second: the unspecified optional dependency that is harder to fix.

Best regards

Signature · 1 line
Fabio Porcedda