Add support for top-level parallel make

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

This is also a step forward supporting top-level parallel make.

Signed-off-by Fabio Porcedda<[email protected]>

Acked-by Arnout Vandecappelle (Essensium/Mind)<[email protected]>

Quoted 11 lines · Fabio Porcedda

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc

This is also a step forward supporting top-level parallel make.

Quoted 2 lines

Signed-off-by Fabio Porcedda<[email protected]>

Acked-by Arnout Vandecappelle (Essensium/Mind)<[email protected]>

Peter,

This one is fixing a problem reported by a user and has already gone
through 10 iterations. Can it be committed?

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character at the beginning of the recipe line. This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ?+? token to get these special effects.

Signed-off-by Fabio Porcedda<[email protected]>

Quoted 6 lines · Fabio Porcedda

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

$(PKG)_BUILD_CMDS is a multiline define, and the recursive make call may
not be the first command. Can you check if this still works if the build commands are something like:

define FOO_BUILD_CMDS
echo "Hello world!"
$(MAKE) -C $(@D)
endef

Quoted 1 line

Quoted 1 line

wget 1.14 Building

Quoted 1 line

Quoted 1 line

wget 1.14 Building

Quoted 13 lines · Arnout Vandecappelle

Quoted 6 lines · Fabio Porcedda

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

$(PKG)_BUILD_CMDS is a multiline define, and the recursive make call may
not be the first command. Can you check if this still works if the build commands are something like:

Quoted 4 lines

define FOO_BUILD_CMDS
echo "Hello world!"
$(MAKE) -C $(@D)
endef

The following test seems to work fine:

define test

     echo test
     $($(PKG)_BUILD_CMDS)

endef

# Build
$(BUILD_DIR)/%/.stamp_built::

     @$(call step_start,build)
     @$(call MESSAGE,"Building")
     +$(test)
     ...

Test output:
-----

$ make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) -s wget-build

Quoted 1 line

Quoted 1 line

Quoted 1 line

wget 1.14 Building

test
Making all in lib
Making all in src
Making all in doc
Making all in po
Making all in tests
Making all in util
----

If i remove the '+' charter:
----

$ make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1)) -s wget-build

Quoted 1 line

Quoted 1 line

Quoted 1 line

wget 1.14 Building

test
make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Making all in lib
Making all in src
Making all in doc
Making all in po
Making all in tests
Making all in util
----

Best regards

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Fabio" == Fabio Porcedda <[email protected]> writes:

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character at the beginning of the recipe line. This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ?+? token to get these special effects.

Quoted 8 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Fabio" == Fabio Porcedda <[email protected]> writes:

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.

Quoted 7 lines

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character at the beginning of the recipe line. This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ?+? token to get these special effects.

Wouldn't it make more sense to add the '+' to MAKE / MAKE1?

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Fabio" == Fabio Porcedda <[email protected]> writes:

Quoted 1 line

Wouldn't it make more sense to add the '+' to MAKE / MAKE1?

Unfortunately that does not work, the '+' must be the first character of the command line, e.g.:

After the latest patches top-level parallel Makefile is working but there is still an issue when a package has an unspecified optional dependency so change the comment to explain that.

Signed-off-by Fabio Porcedda <[email protected]>

---
  Makefile | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)

Makefile +0 -0

index ef2582e..27a6a65 100644

Makefile +2 -0

+# execute:
+#	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
  .NOTPARALLEL:
 
  # absolute path

Quoted 18 lines · Fabio Porcedda

After the latest patches top-level parallel Makefile is working but there is still an issue when a package has an unspecified optional dependency so change the comment to explain that.

Signed-off-by Fabio Porcedda <[email protected]>

---
  Makefile | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)

Makefile +0 -0

index ef2582e..27a6a65 100644
@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
            -e s/macppc/powerpc/\
            -e s/sh.*/sh/)
 

Makefile export HOSTARCH := $(shell uname -m | \ · line 47

Quoted 4 lines

Makefile +3 -1

-# This top-level Makefile can *not* be executed in parallel
+# Parallel execution of this Makefile is disabled because it could
+# change the build result if a package has an unspecified optional
+# dependency.

... or if two packages manipulate the same file in the target directory.

Makefile export HOSTARCH := $(shell uname -m | \ · line 51

Quoted 5 lines

Makefile +4 -0

+# Parallel execution changes the packages building order, that can be
+# a problem when a package has an unspecified optional dependency,
+# because if that dependency is present when the package is built, it
+# is used, otherwise it isn't (but compilation happily proceeds). This

This is a repeat of the first sentence.

Makefile export HOSTARCH := $(shell uname -m | \ · line 55

Quoted 5 lines

Makefile +4 -0

+# means that the packages building order is relevant in that case, and
+# the end result will differ if the order is swapped due to parallel
+# building.
+# Taking in account the above warning, if you still want to execute

Taking into acount

Makefile export HOSTARCH := $(shell uname -m | \ · line 56

Quoted 2 lines

Makefile +1 -0

+# this top-level Makefile in parrallel comment the following line and

parallel

Quoted 19 lines · Arnout Vandecappelle

Quoted 19 lines · Fabio Porcedda

Quoted 18 lines

Quoted 18 lines

After the latest patches top-level parallel Makefile is working but there is still an issue when a package has an unspecified optional dependency so change the comment to explain that.

Signed-off-by Fabio Porcedda<[email protected]>

---
  Makefile | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)

Makefile +0 -0

index ef2582e..27a6a65 100644
@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
            -e s/macppc/powerpc/\
            -e s/sh.*/sh/)
 

Quoted 7 lines

Quoted 7 lines

Quoted 4 lines

Makefile export HOSTARCH := $(shell uname -m | \ · line 47

Quoted 4 lines

Makefile +3 -1

-# This top-level Makefile can*not*  be executed in parallel
+# Parallel execution of this Makefile is disabled because it could
+# change the build result if a package has an unspecified optional
+# dependency.

... or if two packages manipulate the same file in the target directory.

Ok, added.

Quoted 1 line

Quoted 1 line

Quoted 7 lines

Quoted 7 lines

Quoted 4 lines

Makefile export HOSTARCH := $(shell uname -m | \ · line 51

Quoted 4 lines

Makefile +4 -0

+# Parallel execution changes the packages building order, that can be
+# a problem when a package has an unspecified optional dependency,
+# because if that dependency is present when the package is built, it
+# is used, otherwise it isn't (but compilation happily proceeds). This

This is a repeat of the first sentence.

So i just remove it?

Quoted 47 lines · Fabio Porcedda

Quoted 23 lines · Arnout Vandecappelle

Quoted 23 lines · Fabio Porcedda

Quoted 20 lines

Quoted 20 lines

After the latest patches top-level parallel Makefile is working but there is still an issue when a package has an unspecified optional dependency so change the comment to explain that.

Signed-off-by Fabio Porcedda<[email protected]>

---
  Makefile | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)

Makefile +0 -1

index ef2582e..27a6a65 100644
@@ -42,7 +42,20 @@ export HOSTARCH := $(shell uname -m | \
            -e s/macppc/powerpc/\
            -e s/sh.*/sh/)
 
-# This top-level Makefile can*not*  be executed in parallel

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Makefile export HOSTARCH := $(shell uname -m | \ · line 47

Quoted 3 lines

Makefile +3 -0

+# Parallel execution of this Makefile is disabled because it could
+# change the build result if a package has an unspecified optional
+# dependency.

... or if two packages manipulate the same file in the target
directory.

Ok, added.

Quoted 10 lines

Quoted 10 lines

Quoted 7 lines

Quoted 7 lines

Makefile +4 -0

+# Parallel execution changes the packages building order, that can be
+# a problem when a package has an unspecified optional dependency,
+# because if that dependency is present when the package is built, it
+# is used, otherwise it isn't (but compilation happily proceeds).

This

Quoted 4 lines

Quoted 1 line

Quoted 1 line

This is a repeat of the first sentence.

So i just remove it?

Probably better to remove the first sentence, the second one is more
descriptive.

This is the fixed comment:

# Parallel execution of this Makefile is disabled because it changes

# the packages building order, that can be a problem when a package
# has an unspecified optional dependency, because if that dependency
# is present when the package is built, it is used, otherwise it isn't
# (but compilation happily proceeds). This means that the packages
# building order is relevant in that case, and the end result will
# differ if the order is swapped due to parallel building.  Also
# changing the building order can be a problem if two packages
# manipulate the same file in the target directory.
# Taking into account the above warnings, if you still want to execute
# this top-level Makefile in parallel comment the following line and
# execute:
#    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Regards

Quoted 16 lines · On Tue, 7 Jan 2014 11:05:52 +0100 Fabio Porcedda <[email protected]>…

This is the fixed comment:

# Parallel execution of this Makefile is disabled because it changes

# the packages building order, that can be a problem when a package
# has an unspecified optional dependency, because if that dependency
# is present when the package is built, it is used, otherwise it isn't
# (but compilation happily proceeds). This means that the packages
# building order is relevant in that case, and the end result will
# differ if the order is swapped due to parallel building.  Also
# changing the building order can be a problem if two packages
# manipulate the same file in the target directory.
# Taking into account the above warnings, if you still want to execute
# this top-level Makefile in parallel comment the following line and
# execute:
#    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Quoted 2 lines

Regards

Been a silent reader for years here now, but I just must make a suggestion:

# Parallel execution of this top-level Makefile is disabled. #
# Single execution is forced here for two reasons:
#
# Maintaining reproducible builds:
# Packages with an unspecified, optional, dependency could

# otherwise change the build order among multiple builds.
#
# Prevent file modification collisions:
# This can happen if two packages manipulate the same file.
#
# After taking into account the above considerations;
# If you still want to execute this top-level Makefile in
# parallel, comment out the following line and then execute:
#    make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Just clarify which line is "the following line" in the context of this comment.

Add styling to first line of each paragraph to match the context.

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

With this patch set the top-level parallel make seems to works fine, example:

	make clean
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

Before to try it remeber to remove the ".NOTPARALLEL" line.

On my quad core system the building time for qemu_x86_defconfig is 14m versus 11m and with ccache enabled 7m vs 5m.

Quoted 9 lines · Fabio Porcedda

this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.

Quoted 4 lines

With this patch set the top-level parallel make seems to works fine, example:

        make clean
        make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

Would it be possible to disable BR2_JLEVEL automatically when -j appears on
the command line?

Quoted 4 lines

Before to try it remeber to remove the ".NOTPARALLEL" line.

Quoted 2 lines

On my quad core system the building time for qemu_x86_defconfig is 14m versus 11m and with ccache enabled 7m vs 5m.

These are not very relevant benchmarks because there's not a lot that can
be done in parallel there. It would be much more interesting with an allpackageyesconfig (which you'll have to modify by hand BTW to exclude some packages that don't build...).

The autobuild does not try to test allpackageyesconfig, does it?

Dear Fabio Porcedda,

Quoted 1 line · Fabio Porcedda

The autobuild does not try to test allpackageyesconfig, does it?

But, the autobuild is not designed for that. However, an allyespackageconfig configuration is something we could add to the Jenkins if you want.

Quoted 3 lines · Thomas Petazzoni

Dear Fabio Porcedda,

Quoted 8 lines · Fabio Porcedda

Quoted 2 lines

The autobuild does not try to test allpackageyesconfig, does it?

But, the autobuild is not designed for that. However, an allyespackageconfig configuration is something we could add to the Jenkins if you want.

If you do that, expect it to fail :-)

It will certainly fail if you include a kernel in that config: some
packages (Xenomai, rtnet, some modules) only work with certain kernel versions and AFAIK there is no single kernel that works for all these packages.

Not building a kernel excludes many packages so may give you more luck.

On 18/12/13 11:36, Fabio Porcedda wrote:

Quoted 2 lines

On my quad core system the building time for qemu_x86_defconfig is 14m versus 11m and with ccache enabled 7m vs 5m.

These are not very relevant benchmarks because there's not a lot that can
be done in parallel there. It would be much more interesting with an allpackageyesconfig (which you'll have to modify by hand BTW to exclude some packages that don't build...).