Add support for top-level parallel make
v8 cover Add support for top-level parallel make
Hi all,
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
With this patch set the top-level parallel make seems to work fine, example:
make clean make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))
On my quad core system the building time for qemu_x86_defconfig is 13m versus 11m.
I've tested the qemu_x86_defconfig uclibc/eglibc/glibc.
Best regards
Fabio Porcedda
v8:
- rebased over master
- added patche for for base dependency
- added patche for for glibc package
- added patche for for uclibc package
- removed patch already merged
- changed some descriptions
- modified the patch for toolchain dependency to prevent circular dependency
v7:
- add the latest patch
- add to the first patch the <pkgname>_TOOLCHAIN variable
- improve the fifth patch
v6:
- added the fifth patch
- updated the fourth patch adding the install targets
- updated the second patch to remove TARGETS_ALL
v5:
- added the fourth patch
- fixed some typos
- rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
- add support for top-level parallel make for the glibc package
v4:
- rebased over master
- add Acked-by: Thomas Petazzoni on the third patch
- changed the orderd of the patches
v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the third patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.
Fabio Porcedda (8):
package: add base dependency to every package
package: add toolchain dependency to every target package
package: add support for top-level parallel make
Makefile: add support for top-level parallel make
glibc: add support for top-level parallel make
uclibc: add support for top-level parallel make
package: enable jobserver for recursive make
Makefile: enable top-level parallel make
Makefile | 22 +++----- fs/common.mk | 4 +- package/glibc/glibc.mk | 7 ++- package/linux-headers/linux-headers.mk | 3 ++ package/pkg-autotools.mk | 3 +- package/pkg-generic.mk | 60 +++++++++++++--------- package/uclibc/uclibc.mk | 7 ++- .../toolchain-buildroot/toolchain-buildroot.mk | 2 + toolchain/toolchain-external/toolchain-external.mk | 2 + toolchain/toolchain/toolchain.mk | 5 +- 10 files changed, 69 insertions(+), 46 deletions(-)
Signature · 1 line
1.8.4
PATCH RESEND v8 1/8 package: add base dependency to every package
Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc
This is also a step forward supporting top-level parallel make.
--- package/pkg-generic.mk | 2 ++ toolchain/toolchain/toolchain.mk | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +2 -0
@@ -385,6 +385,8 @@ $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) $(1)-build: $(1)-configure \ $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare + ifeq ($$($(2)_OVERRIDE_SRCDIR),) # In the normal case (no package override), the sequence of steps is # source, by downloading
toolchain/toolchain/toolchain.mk +1 -2
@@ -14,5 +14,4 @@ endif $(eval $(generic-package)) -toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake - +toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
1.8.4
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 11 lines · Fabio Porcedda
Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibcThis is also a step forward supporting top-level parallel make.
Signed-off-by Fabio Porcedda <[email protected]>
Although this is one of the most useful patches of the series, it is
also the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.
Quoted 10 lines
--- package/pkg-generic.mk | 2 ++ toolchain/toolchain/toolchain.mk | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)package/pkg-generic.mk +0
-0
package/pkg-generic.mk $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) · line 388
Quoted 4 lines
package/pkg-generic.mk +1
-0@@ -385,6 +385,8 @@ $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) $(1)-build: $(1)-configure \ $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
Is there any reason why you changed the order here?
Regards,
Arnout
Quoted 16 lines
package/pkg-generic.mk +1
-0+ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # In the normal case (no package override), the sequence of steps is # source, by downloadingtoolchain/toolchain/toolchain.mk +1
-2@@ -14,5 +14,4 @@ endif $(eval $(generic-package)) -toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake - +toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 1/8 package: add base dependency to every package
Hi Arnout,
thanks for reviewing the patch set.
Quoted 10 lines · Arnout Vandecappelle
Quoted 9 lines · Fabio Porcedda
Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibc
Quoted 8 lines
Quoted 3 lines
This is also a step forward supporting top-level parallel make.
Signed-off-by Fabio Porcedda <[email protected]>Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.
I've found a work around:
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endif
What do you think about that?
Quoted 15 lines
Quoted 15 lines
--- package/pkg-generic.mk | 2 ++ toolchain/toolchain/toolchain.mk | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)package/pkg-generic.mk +0
-0@@ -385,6 +385,8 @@ $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) $(1)-build: $(1)-configure \ $$($(2)_TARGET_BUILD)
Quoted 4 lines
package/pkg-generic.mk $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) · line 388
Quoted 1 line
package/pkg-generic.mk +1
-0+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepareIs there any reason why you changed the order here?
The order whit top-level parallel make does not matter, it's just that i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directories
But now i understand that is "dirs prepare dependencies" or "dirs dependencies prepare" because dirs can require host-ccache and so directories,
nevertheless this does not matter anymore.
Quoted 26 lines
Regards,
ArnoutQuoted 16 lines
package/pkg-generic.mk +1
-0+ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # In the normal case (no package override), the sequence of steps is# source, by downloading
toolchain/toolchain/toolchain.mk +0
-1@@ -14,5 +14,4 @@ endif $(eval $(generic-package)) -toolchain-source: prepare dirs dependencies$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
toolchain/toolchain/toolchain.mk endif · line 17
Quoted 2 lines
toolchain/toolchain/toolchain.mk +1
-1- +toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake--
Quoted 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 1/8 package: add base dependency to every package
Hi,
Quoted 8 lines · Fabio Porcedda
Quoted 5 lines
Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.I've found a work around:
Quoted 9 lines
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endifWhat do you think about that?
I don't think this will work if there are more than one prerequisite:
$(1) will be one of them, and DEPENDENCIES_HOST_PREREQ will be the entire list, so that 'ifneq' will always be true.
I think you need 'filter' here.
http://www.gnu.org/software/make/manual/make.html#index-filter-587
Best regards,
Thomas
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 19 lines · Thomas De Schampheleire
Hi,
Quoted 15 lines · Fabio Porcedda
Quoted 5 lines
Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.I've found a work around:
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endif
Quoted 8 lines
Quoted 2 lines
What do you think about that?
I don't think this will work if there are more than one prerequisite:
$(1) will be one of them, andDEPENDENCIES_HOST_PREREQwill be the entire list, so that 'ifneq' will always be true.
I think you need 'filter' here.
http://www.gnu.org/software/make/manual/make.html#index-filter-587
Yes it works fine with filter, thanks for suggesting it.
Quoted 2 lines
Best regards,
Thomas
Thanks and regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 1/8 package: add base dependency to every package
In reply to Fabio PorceddaQuoted 25 lines · Fabio Porcedda
Hi Arnout,
thanks for reviewing the patch set.Quoted 10 lines · Arnout Vandecappelle
Quoted 9 lines · Fabio Porcedda
Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibcQuoted 8 lines
Quoted 3 lines
This is also a step forward supporting top-level parallel make.
Signed-off-by Fabio Porcedda <[email protected]>Although this is one of the most useful patches of the series, it is also
the one that introduces the circular dependency. So although it looks good, I'm not ready to ack it.I've found a work around:
Quoted 8 lines
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endifWhat do you think about that?
Looks good except that it's incorrect :-) If HOST_PREREQ contains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You can try something like:
ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
[Just noticed now that Thomas gave the same answer.]
Quoted 21 lines
Quoted 16 lines
Quoted 16 lines
--- package/pkg-generic.mk | 2 ++ toolchain/toolchain/toolchain.mk | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)package/pkg-generic.mk +0
-0@@ -385,6 +385,8 @@ $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) $(1)-build: $(1)-configure \ $$($(2)_TARGET_BUILD)Quoted 4 lines
package/pkg-generic.mk $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) · line 388
Quoted 1 line
package/pkg-generic.mk +1
-0+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepareIs there any reason why you changed the order here?
Quoted 8 lines
The order whit top-level parallel make does not matter, it's just that i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directoriesBut now i understand that is "dirs prepare dependencies" or "dirs dependencies prepare" because dirs can require host-ccache and so
You mean dependencies can require host-ccache, right? And host-ccache
depends on both dirs and prepare itself... So I do think the original order is more logical.
Regards,
Arnout
Quoted 2 lines
directories,
nevertheless this does not matter anymore.
[snip]
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 41 lines · Arnout Vandecappelle
Quoted 40 lines · Fabio Porcedda
Hi Arnout,
thanks for reviewing the patch set.Quoted 18 lines · Arnout Vandecappelle
Quoted 10 lines · Fabio Porcedda
Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to every package.
This way we can build correctly every package right after the clean stage.
As example with this commit we can build successfully the glibc right after the clean stage:
make clean glibcQuoted 3 lines
This is also a step forward supporting top-level parallel make.
Signed-off-by Fabio Porcedda <[email protected]>Quoted 5 lines
Although this is one of the most useful patches of the series, it is
also
the one that introduces the circular dependency. So although it looks good,
I'm not ready to ack it.I've found a work around:
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endif
Quoted 8 lines
Quoted 1 line
What do you think about that?
Looks good except that it's incorrect :-) If
HOST_PREREQcontains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You can try something like:ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
Ok good, I've updated the patch according.
Quoted 34 lines
[Just noticed now that Thomas gave the same answer.]
Quoted 30 lines
Quoted 16 lines
Quoted 16 lines
--- package/pkg-generic.mk | 2 ++ toolchain/toolchain/toolchain.mk | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-)package/pkg-generic.mk +0
-0@@ -385,6 +385,8 @@ $(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) $(1)-build: $(1)-configure \ $$($(2)_TARGET_BUILD)Quoted 6 lines
Quoted 2 lines
package/pkg-generic.mk +1
-0+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirsprepare
Is there any reason why you changed the order here?
The order whit top-level parallel make does not matter, it's just that i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directories
Quoted 7 lines
Quoted 2 lines
But now i understand that is "dirs prepare dependencies" or "dirs dependencies prepare" because dirs can require host-ccache and so
You mean dependencies can require host-ccache, right? And host-ccache
depends on both dirs and prepare itself... So I do think the original order is more logical.
Yes the final order is this: dirs prepare dependencies.
I moved dirs in front because i think creating directories it's the first thing to do.
Quoted 12 lines
Regards,
ArnoutQuoted 2 lines
directories,
nevertheless this does not matter anymore.[snip]
Quoted 7 lines
--
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Thanks and regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 3 lines · Fabio Porcedda
Quoted 2 lines · Arnout Vandecappelle
Quoted 2 lines · Fabio Porcedda
Quoted 1 line
Quoted 1 line
[snip]
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 10 lines
Quoted 10 lines
I've found a work around:
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endifWhat do you think about that?
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Looks good except that it's incorrect:-) If
HOST_PREREQcontains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You can try something like:ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
Ok good, I've updated the patch according.
Have you tested it with various combinations of dependencies?
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 29 lines · Arnout Vandecappelle
Quoted 7 lines · Fabio Porcedda
Quoted 4 lines · Arnout Vandecappelle
Quoted 4 lines · Fabio Porcedda
Quoted 2 lines
Quoted 2 lines
[snip]
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
I've found a work around:
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endifWhat do you think about that?
Looks good except that it's incorrect:-) If
HOST_PREREQcontains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You
can try something like:
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
Ok good, I've updated the patch according.
Have you tested it with various combinations of dependencies?
I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with qemu_x86_defconfig and seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.
I'm testing the whole patch set with top-level parallel make and enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".
Thanks and regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 37 lines · Fabio Porcedda
Quoted 29 lines · Arnout Vandecappelle
Quoted 7 lines · Fabio Porcedda
Quoted 4 lines · Arnout Vandecappelle
Quoted 4 lines · Fabio Porcedda
Quoted 2 lines
Quoted 2 lines
[snip]
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
I've found a work around:
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
# to prevent circular dependency ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ)) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies endifWhat do you think about that?
Looks good except that it's incorrect:-) If
HOST_PREREQcontains both
ccache and sstrip (not to mention tar or xz), the condition won't match. You
can try something like:Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
Ok good, I've updated the patch according.
Have you tested it with various combinations of dependencies?
Quoted 7 lines
I've tested enabling "
BR2_CCACHE" and "BR2_STRIP_sstrip" withqemu_x86_defconfigand seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.I'm testing the whole patch set with top-level parallel make and enabling "
BR2_CCACHE" and "BR2_STRIP_sstrip".
The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and disabling /usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.
Regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 1/8 package: add base dependency to every package
Hi Fabio,
On Tue, Oct 29, 2013 at 9:36 AM, Fabio Porcedda
<[email protected]> wrote:
[..]
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Have you tested it with various combinations of dependencies?
I've tested enabling "
BR2_CCACHE" and "BR2_STRIP_sstrip" withqemu_x86_defconfigand seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.
Quoted 8 lines
Quoted 2 lines
I'm testing the whole patch set with top-level parallel make and enabling "
BR2_CCACHE" and "BR2_STRIP_sstrip".The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and disabling/usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.
Can you elaborate on this? What is the problem you're seeing exactly, and how did you 'disable /usr/bin/xz' ?
When buildroot cannot find a working xzcat version, it builds one itself through host-xz. This should still work in case of parallel builds.
The same principle applies when the 'tar' version on host is non-existing or too old.
Thanks,
Thomas
PATCH RESEND v8 1/8 package: add base dependency to every package
Quoted 15 lines · Thomas De Schampheleire
Hi Fabio,
On Tue, Oct 29, 2013 at 9:36 AM, Fabio Porcedda
<[email protected]> wrote:
[..]Quoted 10 lines
Quoted 9 lines
Quoted 1 line
Have you tested it with various combinations of dependencies?
I've tested enabling "
BR2_CCACHE" and "BR2_STRIP_sstrip" withqemu_x86_defconfigand seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.I'm testing the whole patch set with top-level parallel make and enabling "
BR2_CCACHE" and "BR2_STRIP_sstrip".
Quoted 8 lines
Quoted 5 lines
The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and disabling/usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.Can you elaborate on this? What is the problem you're seeing exactly, and how did you 'disable
/usr/bin/xz' ?
To disable /usr/bin/xz i've just renamed it so buildroot doesn't find a suitable xz.
The host-ccache source file is a tar.xz so the host-xz is a dependency of host-ccache.
I need to add a rule to ensure the right order.
Quoted 5 lines
When buildroot cannot find a working xzcat version, it builds one itself through host-xz. This should still work in case of parallel builds.
The same principle applies when the 'tar' version on host is non-existing or too old.
Yes i think is affected by the same problem.
Thanks
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 2/8 package: add toolchain dependency to every target package
In reply to Fabio PorceddaThis commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:
make clean <package-name>
Also remove TARGETS_ALL because the only purpose was to add toolchain dependency so it's superseded by this commit.
To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN for not adding the toolchain dependency for toolchain packages.
This is also a step forward supporting top-level parallel make.
--- Makefile | 8 ++------ package/glibc/glibc.mk | 3 +++ package/linux-headers/linux-headers.mk | 3 +++ package/pkg-autotools.mk | 3 ++- package/pkg-generic.mk | 12 ++++++++++-- package/uclibc/uclibc.mk | 3 +++ toolchain/toolchain-buildroot/toolchain-buildroot.mk | 2 ++ toolchain/toolchain-external/toolchain-external.mk | 2 ++ toolchain/toolchain/toolchain.mk | 2 ++ 9 files changed, 29 insertions(+), 9 deletions(-)
Makefile +2 -6
@@ -352,7 +352,6 @@ TARGETS+=target-post-image TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS)) TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS)) TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS)) -TARGETS_ALL:=$(patsubst %,__real_tgt_%,$(TARGETS)) # host-* dependencies have to be handled specially, as those aren't # visible in Kconfig and hence not added to a variable like TARGETS.
@@ -375,9 +374,6 @@ HOST_SOURCE += $(addsuffix -source,$(sort $(TARGETS_HOST_DEPS) $(HOST_DEPS))) TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\ $(TARGETS) $(BASE_TARGETS) $(TARGETS_HOST_DEPS) $(HOST_DEPS)))) -# all targets depend on the crosscompiler and it's prerequisites -$(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) % - dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \ $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -386,11 +382,11 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG) prepare: $(BUILD_DIR)/buildroot-config/auto.conf -world: $(BASE_TARGETS) $(TARGETS_ALL) +world: $(TARGETS) .PHONY: all world toolchain dirs clean distclean source outputmakefile \ legal-info legal-info-prepare legal-info-clean printvars \ - $(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \ + $(BASE_TARGETS) $(TARGETS) \ $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \ $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \ $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
package/glibc/glibc.mk +3 -0
@@ -19,6 +19,9 @@ endif GLIBC_LICENSE = GPLv2+ (programs), LGPLv2.1+, BSD-3c, MIT (library) GLIBC_LICENSE_FILES = $(addprefix $(GLIBC_SRC_SUBDIR)/,COPYING COPYING.LIB LICENSES) +# glibc is part of the toolchain so disable the toolchain dependency +GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO + # Before (e)glibc is configured, we must have the first stage # cross-compiler and the kernel headers GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
package/linux-headers/linux-headers.mk +3 -0
@@ -17,6 +17,9 @@ LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz LINUX_HEADERS_INSTALL_STAGING = YES +# linux-headers is part of the toolchain so disable the toolchain dependency +LINUX_HEADERS_ADD_TOOLCHAIN_DEPENDENCY = NO + define LINUX_HEADERS_INSTALL_STAGING_CMDS (cd $(@D); \ $(TARGET_MAKE_ENV) $(MAKE) \
package/pkg-autotools.mk +2 -1
@@ -209,7 +209,8 @@ endef # This must be repeated from inner-generic-package, otherwise we get an empty # _DEPENDENCIES if _AUTORECONF is YES. Also filter the result of _AUTORECONF # away from the non-host rule -$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\ +$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \ + host-toolchain $(1),\ $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
package/pkg-generic.mk +10 -2
@@ -307,8 +307,16 @@ endif $(2)_REDISTRIBUTE ?= YES - -$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +# When a target package is a toolchain dependency set this variable to +# 'NO' so the 'toolchain' dependency is not added to prevent a circular +# dependency +$(2)_ADD_TOOLCHAIN_DEPENDENCY ?= YES + +$(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ + $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES) +$(2)_DEPENDENCIES += toolchain +endif $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO
package/uclibc/uclibc.mk +3 -0
@@ -18,6 +18,9 @@ endif UCLIBC_INSTALL_STAGING = YES +# uclibc is part of the toolchain so disable the toolchain dependency +UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO + # Before uClibc is configured, we must have the first stage # cross-compiler and the kernel headers UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
toolchain/toolchain-buildroot/toolchain-buildroot.mk +2 -0
@@ -14,4 +14,6 @@ BUILDROOT_LIBC = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_LIBC)) TOOLCHAIN_BUILDROOT_DEPENDENCIES = host-gcc-final +TOOLCHAIN_BUILDROOT_ADD_TOOLCHAIN_DEPENDENCY = NO + $(eval $(generic-package))
toolchain/toolchain-external/toolchain-external.mk +2 -0
@@ -340,6 +340,8 @@ TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL))) TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL))) endif +TOOLCHAIN_EXTERNAL_ADD_TOOLCHAIN_DEPENDENCY = NO + TOOLCHAIN_EXTERNAL_INSTALL_STAGING = YES ifeq ($(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R1)$(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R2),y)
toolchain/toolchain/toolchain.mk +2 -0
@@ -12,6 +12,8 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y) TOOLCHAIN_DEPENDENCIES += toolchain-external endif +TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO + $(eval $(generic-package)) toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
Signature · 1 line
1.8.4
PATCH RESEND v8 2/8 package: add toolchain dependency to every target package
Quoted 9 lines · Fabio Porcedda
This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:
make clean <package-name>
Also remove
TARGETS_ALLbecause the only purpose was to add toolchain dependency so it's superseded by this commit.
Quoted 6 lines
To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN for not adding the toolchain dependency for toolchain packages.
This is also a step forward supporting top-level parallel make.
Signed-off-by Fabio Porcedda <[email protected]>
I'm not happy with this _ADD_TOOLCHAIN_DEPENDENCY variable.
What I would prefer is to move glibc, linux-headers and uclibc back
under the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)
ifeq ($(5),target)
ifneq ($(pkgparentdir),toolchain)
$(2)_DEPENDENCIES += toolchain
endif
endif
[snip]
package/pkg-autotools.mk endef · line 213
Quoted 11 lines
package/pkg-autotools.mk +2
-1@@ -209,7 +209,8 @@ endef # This must be repeated from inner-generic-package, otherwise we get an empty # _DEPENDENCIES if _AUTORECONF is YES. Also filter the result of _AUTORECONF # away from the non-host rule -$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\ +$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \ + host-toolchain $(1),\
You'll also need to do this in pkg-cmake.mk.
Quoted 11 lines
package/pkg-autotools.mk +0
-0$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))package/pkg-generic.mk +0
-0@@ -307,8 +307,16 @@ endif $(2)_REDISTRIBUTE ?= YES
package/pkg-generic.mk endif · line 317
Quoted 10 lines
package/pkg-generic.mk +8
-2- -$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +# When a target package is a toolchain dependency set this variable to +# 'NO' so the 'toolchain' dependency is not added to prevent a circular +# dependency +$(2)_ADD_TOOLCHAIN_DEPENDENCY ?= YES + +$(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ + $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
It's more readable if you split this and-ing in two separate conditions.
For $(2)_TYPE, you can directly use $(5) because you're anyway inside
inner-generic-package here.
Regards,
Arnout
Quoted 5 lines
package/pkg-generic.mk +2
-0+$(2)_DEPENDENCIES += toolchain +endif $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO
[snip]
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 2/8 package: add toolchain dependency to every target package
Dear Arnout Vandecappelle,
Quoted 3 lines · Arnout Vandecappelle
What I would prefer is to move glibc, linux-headers and uclibc back
under the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)
Well, the idea is that gcc, uclibc, glibc, linux-headers and so on are now real, normal, packages. The only thing we keep in toolchain/ are the "entry" point of toolchain backends: toolchain-buildroot and toolchain-external.
Moreover, it would be strange to have glibc, uclibc and linux-headers in toolchain/, but not gcc and binutils, for example. I do understand that these two packages are host packages, so they don't get the automatic dependency on the toolchain added, but that's an internal detail. From an external point of view, all those components are part of the toolchain, so it would be weird to have some of them in toolchain/, and some of them in package/.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH RESEND v8 2/8 package: add toolchain dependency to every target package
Quoted 13 lines · Thomas Petazzoni
Dear Arnout Vandecappelle,
Quoted 3 lines · Arnout Vandecappelle
What I would prefer is to move glibc, linux-headers and uclibc back
under the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)Well, the idea is that gcc, uclibc, glibc, linux-headers and so on are now real, normal, packages. The only thing we keep in toolchain/ are the "entry" point of toolchain backends: toolchain-buildroot and toolchain-external.
Quoted 7 lines
Moreover, it would be strange to have glibc, uclibc and linux-headers in toolchain/, but not gcc and binutils, for example. I do understand that these two packages are host packages, so they don't get the automatic dependency on the toolchain added, but that's an internal detail. From an external point of view, all those components are part of the toolchain, so it would be weird to have some of them in toolchain/, and some of them in package/.
So what's your conclusion now? Move all toolchain-related packages back
to the toolchain directory, or find another solution for circumventing the toolchain circular dependency?
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 2/8 package: add toolchain dependency to every target package
In reply to Arnout VandecappelleQuoted 44 lines · Arnout Vandecappelle
Quoted 10 lines · Fabio Porcedda
This commit makes the dependency from the target toolchain explicit. This way we can buid from command line a package that use innger-generic-package right after the configuration phase, example:
make clean <package-name>
Also remove
TARGETS_ALLbecause the only purpose was to add toolchain dependency so it's superseded by this commit.Quoted 6 lines
To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN for not adding the toolchain dependency for toolchain packages.
This is also a step forward supporting top-level parallel make.
Signed-off-by Fabio Porcedda <[email protected]>I'm not happy with this
_ADD_TOOLCHAIN_DEPENDENCYvariable.What I would prefer is to move glibc, linux-headers and uclibc back under
the toolchain directory (why did they ever move out, Thomas?) and to filter on $(pkgparentdir)ifeq ($(5),target)
ifneq ($(pkgparentdir),toolchain)
$(2)_DEPENDENCIES += toolchain
endif
endif[snip]
Quoted 10 lines
package/pkg-autotools.mk +0
-0@@ -209,7 +209,8 @@ endef # This must be repeated from inner-generic-package, otherwise we get anempty
#_DEPENDENCIESif_AUTORECONFis YES. Also filter the result of_AUTORECONF# away from the non-host rule
Quoted 8 lines
Quoted 5 lines
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+ host-toolchain $(1),\You'll also need to do this in pkg-cmake.mk.
Ok, done.
Quoted 29 lines
Quoted 15 lines
$(patsubst host-host-%,host-%,$(addprefixhost-,$($(3)_DEPENDENCIES))))
package/pkg-generic.mk +0
-2@@ -307,8 +307,16 @@ endif $(2)_REDISTRIBUTE ?= YES - -$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubstpackage/pkg-generic.mk endif · line 319
Quoted 10 lines
package/pkg-generic.mk +8
-0host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +# When a target package is a toolchain dependency set this variable to +# 'NO' so the 'toolchain' dependency is not added to prevent a circular +# dependency +$(2)_ADD_TOOLCHAIN_DEPENDENCY ?= YES + +$(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ + $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES)))) +ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)It's more readable if you split this and-ing in two separate conditions.
Quoted 2 lines
For $(2)_TYPE, you can directly use $(5) because you're anyway inside
inner-generic-package here.
Ok, done.
Best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 3/8 package: add support for top-level parallel make
In reply to Fabio PorceddaTo be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:
%/.stamp_extracted
%/.stamp_patched
%/.stamp_configured
%/.stamp_built
%/.stamp_host_installed
%/.stamp_staging_installed
%/.stamp_images_installed
%/.stamp_target_installed
Because the %-build target is not anymore part of the dependency chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by uclibc and glibc packages.
--- package/glibc/glibc.mk | 2 +- package/pkg-generic.mk | 38 +++++++++++++++++++++----------------- package/uclibc/uclibc.mk | 2 +- 3 files changed, 23 insertions(+), 19 deletions(-)
package/glibc/glibc.mk +1 -1
@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk # Before (e)glibc is built, we must have the second stage cross-compiler -glibc-build: host-gcc-intermediate +GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate GLIBC_SUBDIR = build
package/pkg-generic.mk +21 -17
@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES) $(2)_DEPENDENCIES += toolchain endif +$(2)_BUILD_DEPENDENCIES ?= $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO
@@ -368,30 +369,34 @@ $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images endif ifeq ($$($(2)_INSTALL_TARGET),YES) -$(1)-install-target: $(1)-build \ - $$($(2)_TARGET_INSTALL_TARGET) +$(1)-install-target: $$($(2)_TARGET_INSTALL_TARGET) else $(1)-install-target: endif ifeq ($$($(2)_INSTALL_STAGING),YES) -$(1)-install-staging: $(1)-build \ - $$($(2)_TARGET_INSTALL_STAGING) +$(1)-install-staging: $$($(2)_TARGET_INSTALL_STAGING) else $(1)-install-staging: endif ifeq ($$($(2)_INSTALL_IMAGES),YES) -$(1)-install-images: $(1)-build \ - $$($(2)_TARGET_INSTALL_IMAGES) +$(1)-install-images: $$($(2)_TARGET_INSTALL_IMAGES) else $(1)-install-images: endif -$(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) +$(1)-install-host: $$($(2)_TARGET_INSTALL_HOST) -$(1)-build: $(1)-configure \ - $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \ +$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \ + $$($(2)_TARGET_BUILD) + +$(1)-build: $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES) + +$(1)-configure: $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): | $$($(2)_DEPENDENCIES) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # extract # patch # configure -$(1)-configure: $(1)-patch $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_PATCH) -$(1)-patch: $(1)-extract $$($(2)_TARGET_PATCH) +$(1)-patch: $$($(2)_TARGET_PATCH) +$$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT) -$(1)-extract: $(1)-source \ - $$($(2)_TARGET_EXTRACT) +$(1)-extract: $$($(2)_TARGET_EXTRACT) +$$($(2)_TARGET_EXTRACT): $$($(2)_TARGET_SOURCE) $(1)-depends: $$($(2)_DEPENDENCIES)
@@ -418,10 +423,9 @@ else # source, by rsyncing # depends # configure -$(1)-configure: $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_RSYNC) -$(1)-depends: $(1)-rsync $$($(2)_DEPENDENCIES) +$(1)-depends: $$($(2)_DEPENDENCIES) $(1)-patch: $(1)-rsync $(1)-extract: $(1)-rsync
package/uclibc/uclibc.mk +1 -1
@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers # Before uClibc is built, we must have the second stage cross-compiler -uclibc-build: host-gcc-intermediate +UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config # setting.
Signature · 1 line
1.8.4
PATCH RESEND v8 3/8 package: add support for top-level parallel make
Quoted 2 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on
nitpick: relying
Quoted 11 lines
the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:%/.stamp_extracted %/.stamp_patched %/.stamp_configured %/.stamp_built %/.stamp_host_installed %/.stamp_staging_installed %/.stamp_images_installed %/.stamp_target_installed
Your description here is not entirely accurate, because this makes it
look as if the dependencies are added for the pattern rules.
Quoted 17 lines
Because the %-build target is not anymore part of the dependency chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by uclibc and glibc packages.
Signed-off-by Fabio Porcedda <[email protected]>--- package/glibc/glibc.mk | 2 +- package/pkg-generic.mk | 38 +++++++++++++++++++++----------------- package/uclibc/uclibc.mk | 2 +- 3 files changed, 23 insertions(+), 19 deletions(-)package/glibc/glibc.mk +0
-0
package/glibc/glibc.mk GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO · line 30
Quoted 5 lines
package/glibc/glibc.mk +1
-1@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk # Before (e)glibc is built, we must have the second stage cross-compiler -glibc-build: host-gcc-intermediate +GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
I don't like this. (You'll notice I generally don't like adding new
variables :-). This build-dependency is really an exceptional situation, so adding generic package infrastructure for it feels wrong to me. Instead, you can keep the explicit dependency that was there already, only now it is even more explicit:
$(GLIBC_TARGET_BUILD): host-gcc-intermediate
Note that GLIBC_TARGET_BUILD is only defined after the
$(eval $(autotools-package)) so it will have to be moved down as well.
Quoted 16 lines
package/glibc/glibc.mk +0
-0GLIBC_SUBDIR = buildpackage/pkg-generic.mk +1
-0@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES) $(2)_DEPENDENCIES += toolchain endif +$(2)_BUILD_DEPENDENCIES ?= $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO
package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 370
Quoted 2 lines
package/pkg-generic.mk +0
-0@@ -368,30 +369,34 @@ $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images endif
I would like to see somewhere a comment explaining how the dependencies
between the steps are handled. In particular, explaining that this is not done in the pattern rules because then make would remove the stamp files and do too much rebuilding.
Quoted 22 lines
package/pkg-generic.mk +3
-6ifeq ($$($(2)_INSTALL_TARGET),YES) -$(1)-install-target: $(1)-build \ - $$($(2)_TARGET_INSTALL_TARGET) +$(1)-install-target: $$($(2)_TARGET_INSTALL_TARGET) else $(1)-install-target: endif ifeq ($$($(2)_INSTALL_STAGING),YES) -$(1)-install-staging: $(1)-build \ - $$($(2)_TARGET_INSTALL_STAGING) +$(1)-install-staging: $$($(2)_TARGET_INSTALL_STAGING) else $(1)-install-staging: endif ifeq ($$($(2)_INSTALL_IMAGES),YES) -$(1)-install-images: $(1)-build \ - $$($(2)_TARGET_INSTALL_IMAGES) +$(1)-install-images: $$($(2)_TARGET_INSTALL_IMAGES) else $(1)-install-images:
package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 393
Quoted 10 lines
package/pkg-generic.mk +4
-3endif -$(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) +$(1)-install-host: $$($(2)_TARGET_INSTALL_HOST) -$(1)-build: $(1)-configure \ - $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \ +$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \ + $$($(2)_TARGET_BUILD)
I think it would also be possible to turn this into a dependency on
$(1)-build instead of $$($(2)_TARGET_BUILD). That would also avoid the need for _BUILD_DEPENDENCIES and maybe some other changes.
Of course, to be able to do that, you have to add
.PHONY: $(1)-build
otherwise everything will always be rebuilt.
package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 399
Quoted 6 lines
package/pkg-generic.mk +6
-0+ +$(1)-build: $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES) + +$(1)-configure: $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): | $$($(2)_DEPENDENCIES)
This warrants a comment explaining why you use | here.
Quoted 11 lines
package/pkg-generic.mk +1
-2$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # extract # patch # configure -$(1)-configure: $(1)-patch $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_PATCH)
package/pkg-generic.mk ifeq ($$($(2)_OVERRIDE_SRCDIR),) · line 418
Quoted 10 lines
package/pkg-generic.mk +4
-3-$(1)-patch: $(1)-extract $$($(2)_TARGET_PATCH) +$(1)-patch: $$($(2)_TARGET_PATCH) +$$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT) -$(1)-extract: $(1)-source \ - $$($(2)_TARGET_EXTRACT) +$(1)-extract: $$($(2)_TARGET_EXTRACT) +$$($(2)_TARGET_EXTRACT): $$($(2)_TARGET_SOURCE) $(1)-depends: $$($(2)_DEPENDENCIES)
Is the -depends target still useful? Does anybody ever run 'make
foo-depends'?
Regards,
Arnout
Quoted 28 lines
package/pkg-generic.mk +2
-3@@ -418,10 +423,9 @@ else # source, by rsyncing # depends # configure -$(1)-configure: $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_RSYNC) -$(1)-depends: $(1)-rsync $$($(2)_DEPENDENCIES) +$(1)-depends: $$($(2)_DEPENDENCIES) $(1)-patch: $(1)-rsync $(1)-extract: $(1)-rsyncpackage/uclibc/uclibc.mk +1
-1@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers # Before uClibc is built, we must have the second stage cross-compiler -uclibc-build: host-gcc-intermediate +UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config # setting.
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 3/8 package: add support for top-level parallel make
Quoted 7 lines · Arnout Vandecappelle
Quoted 3 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on
nitpick: relying
Ok, done.
Quoted 14 lines
Quoted 4 lines
the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:Quoted 8 lines
%/.stamp_extracted %/.stamp_patched %/.stamp_configured %/.stamp_built %/.stamp_host_installed %/.stamp_staging_installed %/.stamp_images_installed %/.stamp_target_installed
Quoted 2 lines
Your description here is not entirely accurate, because this makes it look
as if the dependencies are added for the pattern rules.
Ok, I'll use $(2)_TARGET_%
Quoted 30 lines
Quoted 17 lines
Because the %-build target is not anymore part of the dependency chain, add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead. This new variable is used only by uclibc and glibc packages.
Signed-off-by Fabio Porcedda <[email protected]>--- package/glibc/glibc.mk | 2 +- package/pkg-generic.mk | 38 +++++++++++++++++++++----------------- package/uclibc/uclibc.mk | 2 +- 3 files changed, 23 insertions(+), 19 deletions(-)package/glibc/glibc.mk +0
-0package/glibc/glibc.mk GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO · line 30
Quoted 5 lines
package/glibc/glibc.mk +1
-1@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk # Before (e)glibc is built, we must have the second stage cross-compiler -glibc-build: host-gcc-intermediate +GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediateI don't like this. (You'll notice I generally don't like adding new
variables :-). This build-dependency is really an exceptional situation, so adding generic package infrastructure for it feels wrong to me. Instead, you can keep the explicit dependency that was there already, only now it is even more explicit:
Quoted 5 lines
$(GLIBC_TARGET_BUILD): host-gcc-intermediate
Note that
GLIBC_TARGET_BUILDis only defined after the
$(eval $(autotools-package)) so it will have to be moved down as well.
Ok, done.
Quoted 14 lines
Quoted 14 lines
package/glibc/glibc.mk +0
-0GLIBC_SUBDIR = buildpackage/pkg-generic.mk +1
-0@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out host-toolchain $(1),\ ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES) $(2)_DEPENDENCIES += toolchain endif +$(2)_BUILD_DEPENDENCIES ?=
Quoted 11 lines
Quoted 6 lines
package/pkg-generic.mk +0
-0$(2)_INSTALL_STAGING ?= NO$(2)_INSTALL_IMAGES ?= NOpackage/pkg-generic.mk +0
-0@@ -368,30 +369,34 @@ $(1)-install: $(1)-install-staging$(1)-install-target $(1)-install-images
endifI would like to see somewhere a comment explaining how the dependencies
between the steps are handled. In particular, explaining that this is not done in the pattern rules because then make would remove the stamp files and do too much rebuilding.
Do you mean to add a comment in the description of the commit? My explanation is:
We cannot use the pattern rules because they must have the same dependency for every package,
but we need to change the dependencies depending on $(2)_OVERRIDE_SRCDIR variable value,
so we must use a more flexible way like $(2)_TARGET_% variables.
Quoted 35 lines
Quoted 23 lines
ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target: $(1)-build \package/pkg-generic.mk +3
-5- $$($(2)_TARGET_INSTALL_TARGET) +$(1)-install-target: $$($(2)_TARGET_INSTALL_TARGET) else $(1)-install-target: endif ifeq ($$($(2)_INSTALL_STAGING),YES) -$(1)-install-staging: $(1)-build \ - $$($(2)_TARGET_INSTALL_STAGING) +$(1)-install-staging: $$($(2)_TARGET_INSTALL_STAGING) else $(1)-install-staging: endif ifeq ($$($(2)_INSTALL_IMAGES),YES) -$(1)-install-images: $(1)-build \ - $$($(2)_TARGET_INSTALL_IMAGES) +$(1)-install-images: $$($(2)_TARGET_INSTALL_IMAGES) else $(1)-install-images:package/pkg-generic.mk $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images · line 393
Quoted 10 lines
package/pkg-generic.mk +4
-3endif -$(1)-install-host: $(1)-build $$($(2)_TARGET_INSTALL_HOST) +$(1)-install-host: $$($(2)_TARGET_INSTALL_HOST) -$(1)-build: $(1)-configure \ - $$($(2)_TARGET_BUILD) +$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \ +$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \ + $$($(2)_TARGET_BUILD)
Quoted 3 lines
I think it would also be possible to turn this into a dependency on
$(1)-build instead of $$($(2)_TARGET_BUILD). That would also avoid the need for_BUILD_DEPENDENCIESand maybe some other changes.
That's not possible because the $$($(2)_TARGET_BUILD) must depends on $$($(2)_BUILD_DEPENDENCIES) because
if build depends on $$($(2)_TARGET_BUILD) and the contents of $$($(2)_BUILD_DEPENDENCIES) they can be built in parallel and that's is wrong.
Quoted 6 lines
Of course, to be able to do that, you have to add
.PHONY: $(1)-build
otherwise everything will always be rebuilt.
Quoted 10 lines
Quoted 7 lines
+
+$(1)-build: $$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES)
+
+$(1)-configure: $$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE): | $$($(2)_DEPENDENCIES)This warrants a comment explaining why you use | here.
Ok.
Quoted 22 lines
Quoted 22 lines
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs
preparepackage/pkg-generic.mk +5
-5@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # extract # patch # configure -$(1)-configure: $(1)-patch $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_PATCH) -$(1)-patch: $(1)-extract $$($(2)_TARGET_PATCH) +$(1)-patch: $$($(2)_TARGET_PATCH) +$$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT) -$(1)-extract: $(1)-source \ - $$($(2)_TARGET_EXTRACT) +$(1)-extract: $$($(2)_TARGET_EXTRACT) +$$($(2)_TARGET_EXTRACT): $$($(2)_TARGET_SOURCE)
Quoted 5 lines
package/pkg-generic.mk ifeq ($$($(2)_OVERRIDE_SRCDIR),) · line 418
Quoted 1 line
package/pkg-generic.mk +0
-0$(1)-depends: $$($(2)_DEPENDENCIES)Is the -depends target still useful? Does anybody ever run 'make
foo-depends'?
I don't know.
Quoted 37 lines
Regards,
ArnoutQuoted 27 lines
package/pkg-generic.mk +2
-3@@ -418,10 +423,9 @@ else # source, by rsyncing # depends # configure -$(1)-configure: $(1)-depends \ - $$($(2)_TARGET_CONFIGURE) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_RSYNC) -$(1)-depends: $(1)-rsync $$($(2)_DEPENDENCIES) +$(1)-depends: $$($(2)_DEPENDENCIES) $(1)-patch: $(1)-rsync $(1)-extract: $(1)-rsyncpackage/uclibc/uclibc.mk +1
-1@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers # Before uClibc is built, we must have the second stage cross-compiler -uclibc-build: host-gcc-intermediate +UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate # specifying UCLIBC_CONFIG_FILE on the command-line overrides the.config
Quoted 2 lines
# setting.--
Quoted 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make
In reply to Fabio PorceddaTo be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image
--- Makefile | 15 ++++++--------- fs/common.mk | 4 ++-- 2 files changed, 8 insertions(+), 11 deletions(-)
Makefile +6 -9
@@ -329,8 +329,6 @@ include boot/common.mk include linux/linux.mk include system/system.mk -TARGETS+=target-finalize - ifeq ($(BR2_ENABLE_LOCALE_PURGE),y) TARGETS+=target-purgelocales endif
@@ -347,8 +345,6 @@ endif include fs/common.mk -TARGETS+=target-post-image - TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS)) TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS)) TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG) prepare: $(BUILD_DIR)/buildroot-config/auto.conf -world: $(TARGETS) +world: target-post-image .PHONY: all world toolchain dirs clean distclean source outputmakefile \ legal-info legal-info-prepare legal-info-clean printvars \ - $(BASE_TARGETS) $(TARGETS) \ + target-finalize target-post-image \ + $(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \ $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \ $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \ $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -443,7 +440,7 @@ endif STRIP_FIND_CMD += -type f -perm /111 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print -target-finalize: +target-finalize: $(TARGETS) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \ $(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
@@ -550,7 +547,7 @@ target-generatelocales: host-localedef done endif -target-post-image: +target-post-image: target-finalize $(TARGETS_ROOTFS) @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \ @rm -f $(LEGAL_WARNINGS) show-targets: - @echo $(TARGETS) + @echo $(TARGETS) $(TARGETS_ROOTFS) else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
fs/common.mk +2 -2
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz) -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep)) rm -f $$(FAKEROOT_SCRIPT)
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends: rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS) ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) -TARGETS += rootfs-$(1) +TARGETS_ROOTFS += rootfs-$(1) endif endef
Signature · 1 line
1.8.4
PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make
Quoted 9 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-imageSigned-off-by Fabio Porcedda <[email protected]>
I think this one is ready to go in, so perhaps you can reorder it as
one of the first ones of the series. Although it serves no function except for the top-level parallel make, so perhaps Peter will object to committing it on its own.
Regards,
Arnout
Quoted 92 lines
--- Makefile | 15 ++++++--------- fs/common.mk | 4 ++-- 2 files changed, 8 insertions(+), 11 deletions(-)Makefile +6
-9@@ -329,8 +329,6 @@ include boot/common.mk include linux/linux.mk include system/system.mk -TARGETS+=target-finalize - ifeq ($(BR2_ENABLE_LOCALE_PURGE),y) TARGETS+=target-purgelocales endif@@ -347,8 +345,6 @@ endif include fs/common.mk -TARGETS+=target-post-image - TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS)) TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS)) TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG) prepare: $(BUILD_DIR)/buildroot-config/auto.conf -world: $(TARGETS) +world: target-post-image .PHONY: all world toolchain dirs clean distclean source outputmakefile \ legal-info legal-info-prepare legal-info-clean printvars \ - $(BASE_TARGETS) $(TARGETS) \ + target-finalize target-post-image \ + $(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \ $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \ $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \ $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)@@ -443,7 +440,7 @@ endif STRIP_FIND_CMD += -type f -perm /111 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print -target-finalize: +target-finalize: $(TARGETS) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \ $(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake@@ -550,7 +547,7 @@ target-generatelocales: host-localedef done endif -target-post-image: +target-post-image: target-finalize $(TARGETS_ROOTFS) @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \ @rm -f $(LEGAL_WARNINGS) show-targets: - @echo $(TARGETS) + @echo $(TARGETS) $(TARGETS_ROOTFS) else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)fs/common.mk +2
-2@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz) -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep)) rm -f $$(FAKEROOT_SCRIPT)@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends: rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS) ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) -TARGETS += rootfs-$(1) +TARGETS_ROOTFS += rootfs-$(1) endif endef
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make
Quoted 13 lines · Arnout Vandecappelle
Quoted 10 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-imageSigned-off-by Fabio Porcedda <[email protected]>
Quoted 6 lines
I think this one is ready to go in, so perhaps you can reorder it as one of
the first ones of the series. Although it serves no function except for the top-level parallel make, so perhaps Peter will object to committing it on its own.
Is a little cumbersome to change the order so i prefer at least for the next iteration not to change it.
Quoted 110 lines
Regards,
ArnoutQuoted 95 lines
--- Makefile | 15 ++++++--------- fs/common.mk | 4 ++-- 2 files changed, 8 insertions(+), 11 deletions(-)Makefile +3
-6@@ -329,8 +329,6 @@ include boot/common.mk include linux/linux.mk include system/system.mk -TARGETS+=target-finalize - ifeq ($(BR2_ENABLE_LOCALE_PURGE),y) TARGETS+=target-purgelocales endif@@ -347,8 +345,6 @@ endif include fs/common.mk -TARGETS+=target-post-image - TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS)) TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS)) TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG) prepare: $(BUILD_DIR)/buildroot-config/auto.conf -world: $(TARGETS) +world: target-post-image .PHONY: all world toolchain dirs clean distclean source outputmakefile \ legal-info legal-info-prepare legal-info-clean printvars \ - $(BASE_TARGETS) $(TARGETS) \ + target-finalize target-post-image \ + $(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \ $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \ $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)Makefile +0
-0@@ -443,7 +440,7 @@ endif STRIP_FIND_CMD += -type f -perm /111 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(callqstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
Makefile +1
-1-target-finalize: +target-finalize: $(TARGETS) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig$(TARGET_DIR)/usr/share/pkgconfig \ $(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmakeMakefile +1
-1@@ -550,7 +547,7 @@ target-generatelocales: host-localedef done endif -target-post-image: +target-post-image: target-finalize $(TARGETS_ROOTFS) @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ $(s) $(BINARIES_DIR) $(callqstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
Makefile +1
-1@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \ @rm -f $(LEGAL_WARNINGS) show-targets: - @echo $(TARGETS) + @echo $(TARGETS) $(TARGETS_ROOTFS) else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)fs/common.mk +0
-0@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if$$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep)) rm -f $$(FAKEROOT_SCRIPT)fs/common.mk rootfs-$(1)-show-depends: · line 92
Quoted 8 lines
fs/common.mk +1
-1@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends: rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS) ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) -TARGETS += rootfs-$(1) +TARGETS_ROOTFS += rootfs-$(1) endif endef--
Quoted 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Thanks and best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make
In reply to Fabio PorceddaQuoted 14 lines · Fabio Porcedda
fs/common.mk +1
-1@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz) -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep)) rm -f $$(FAKEROOT_SCRIPT)
fs/common.mk rootfs-$(1)-show-depends: · line 91
Quoted 7 lines
fs/common.mk +1
-1@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends: rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS) ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) -TARGETS += rootfs-$(1) +TARGETS_ROOTFS += rootfs-$(1) endif endef
Oh I forgot: you should also fix up initramfs and iso9660. They add
themselves to TARGETS manually.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 4/8 Makefile: add support for top-level parallel make
Quoted 23 lines · Arnout Vandecappelle
Quoted 22 lines · Fabio Porcedda
fs/common.mk +0
-0@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if$$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if
$$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep)) rm -f $$(FAKEROOT_SCRIPT)fs/common.mk +0
-0@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends: rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
Quoted 9 lines
fs/common.mk rootfs-$(1)-show-depends: · line 91
Quoted 5 lines
fs/common.mk +1
-1ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) -TARGETS += rootfs-$(1) +TARGETS_ROOTFS += rootfs-$(1) endif endefOh I forgot: you should also fix up initramfs and iso9660. They add
themselves to TARGETS manually.
Done, thanks.
Best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 5/8 glibc: add support for top-level parallel make
In reply to Fabio PorceddaTo be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
The glibc-install-target depends on glibc-install-staging so add a rule for it.
--- package/glibc/glibc.mk | 2 ++ 1 file changed, 2 insertions(+)
package/glibc/glibc.mk +2 -0
@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS endef $(eval $(autotools-package)) + +$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
Signature · 1 line
1.8.4
PATCH RESEND v8 5/8 glibc: add support for top-level parallel make
Quoted 9 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
The glibc-install-target depends on glibc-install-staging so add a rule for it.
Signed-off-by Fabio Porcedda <[email protected]>
Quoted 10 lines
--- package/glibc/glibc.mk | 2 ++ 1 file changed, 2 insertions(+)package/glibc/glibc.mk +0
-0
package/glibc/glibc.mk define GLIBC_INSTALL_TARGET_CMDS · line 134
Quoted 5 lines
package/glibc/glibc.mk +2
-0@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS endef $(eval $(autotools-package)) + +$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
Nitpick: I would have added a comment why this is needed.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 5/8 glibc: add support for top-level parallel make
Quoted 7 lines · Arnout Vandecappelle
Quoted 6 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
Quoted 7 lines
Quoted 4 lines
The glibc-install-target depends on glibc-install-staging so add a rule for it.
Signed-off-by Fabio Porcedda <[email protected]>
Added, thanks.
Quoted 12 lines
Quoted 12 lines
--- package/glibc/glibc.mk | 2 ++ 1 file changed, 2 insertions(+)package/glibc/glibc.mk +0
-0@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS endef
Quoted 8 lines
package/glibc/glibc.mk define GLIBC_INSTALL_TARGET_CMDS · line 134
Quoted 4 lines
package/glibc/glibc.mk +2
-0$(eval $(autotools-package)) + +$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)Nitpick: I would have added a comment why this is needed. Regards,
Arnout
I've added:
# The glic-install-target use files from glibc-install-staging
Thanks and best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 6/8 uclibc: add support for top-level parallel make
In reply to Fabio PorceddaTo be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
The uclibc-install-target depends on uclibc-install-staging so add a rule for it.
--- package/uclibc/uclibc.mk | 2 ++ 1 file changed, 2 insertions(+)
package/uclibc/uclibc.mk +2 -0
@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed} $(eval $(generic-package)) + +$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
Signature · 1 line
1.8.4
PATCH RESEND v8 6/8 uclibc: add support for top-level parallel make
Quoted 9 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
The uclibc-install-target depends on uclibc-install-staging so add a rule for it.
Signed-off-by Fabio Porcedda <[email protected]>
Quoted 10 lines
--- package/uclibc/uclibc.mk | 2 ++ 1 file changed, 2 insertions(+)package/uclibc/uclibc.mk +0
-0
package/uclibc/uclibc.mk uclibc-menuconfig: dirs uclibc-patch · line 563
Quoted 5 lines
package/uclibc/uclibc.mk +2
-0@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed} $(eval $(generic-package)) + +$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
Same nitpick :-)
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH RESEND v8 6/8 uclibc: add support for top-level parallel make
Quoted 7 lines · Arnout Vandecappelle
Quoted 6 lines · Fabio Porcedda
To be able to use top-level parallel make we must not depend in a rule on the order of evaluation of the prerequisites, so instead of relyng on the left to right ordering of evaluation of the prerequisites add an explicit rule to describe the dependencies.
Quoted 7 lines
Quoted 4 lines
The uclibc-install-target depends on uclibc-install-staging so add a rule for it.
Signed-off-by Fabio Porcedda <[email protected]>
Added, thanks.
Quoted 13 lines
Quoted 13 lines
--- package/uclibc/uclibc.mk | 2 ++ 1 file changed, 2 insertions(+)package/uclibc/uclibc.mk +0
-0@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
Quoted 6 lines
Quoted 4 lines
$(eval $(generic-package))
+
+$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)Same nitpick :-)
Same comment:
# The uclibc-install-target use files from uclibc-install-staging
Thanks and best regards
Signature · 1 line
Fabio Porcedda
PATCH RESEND v8 7/8 package: enable jobserver for recursive make
In reply to Fabio PorceddaAdd '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS) commands to enable jobserver for the sub-make.
Without the '+' prefix GNU make does not detect the sub-make so it disable the jobserver for the sub-make.
From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character at the beginning of the recipe line. This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ?+? token to get these special effects.
--- package/pkg-generic.mk | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
package/pkg-generic.mk +5 -5
@@ -118,21 +118,21 @@ $(BUILD_DIR)/%/.stamp_configured: # Build $(BUILD_DIR)/%/.stamp_built:: @$(call MESSAGE,"Building") - $($(PKG)_BUILD_CMDS) + +$($(PKG)_BUILD_CMDS) $(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@ # Install to host dir $(BUILD_DIR)/%/.stamp_host_installed: @$(call MESSAGE,"Installing to host directory") - $($(PKG)_INSTALL_CMDS) + +$($(PKG)_INSTALL_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@ # Install to staging dir $(BUILD_DIR)/%/.stamp_staging_installed: @$(call MESSAGE,"Installing to staging directory") - $($(PKG)_INSTALL_STAGING_CMDS) + +$($(PKG)_INSTALL_STAGING_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) $(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \ $(call MESSAGE,"Fixing package configuration files") ;\
@@ -146,7 +146,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed: # Install to images dir $(BUILD_DIR)/%/.stamp_images_installed: @$(call MESSAGE,"Installing to images directory") - $($(PKG)_INSTALL_IMAGES_CMDS) + +$($(PKG)_INSTALL_IMAGES_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep)) $(Q)touch $@
@@ -157,7 +157,7 @@ $(BUILD_DIR)/%/.stamp_target_installed: $($(PKG)_INSTALL_INIT_SYSTEMD)) $(if $(BR2_INIT_SYSV)$(BR2_INIT_BUSYBOX),\ $($(PKG)_INSTALL_INIT_SYSV)) - $($(PKG)_INSTALL_TARGET_CMDS) + +$($(PKG)_INSTALL_TARGET_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep)) $(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \ $(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
Signature · 1 line
1.8.4
PATCH RESEND v8 8/8 Makefile: enable top-level parallel make
In reply to Fabio PorceddaEnable top-level parallel make because the previous patches have fixed all the know issue about top-level parallel make.
Example:
make <board>_defconfig make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))
--- Makefile | 3 --- 1 file changed, 3 deletions(-)
Makefile +0 -3
@@ -42,9 +42,6 @@ export HOSTARCH := $(shell uname -m | \ -e s/macppc/powerpc/\ -e s/sh.*/sh/) -# This top-level Makefile can *not* be executed in parallel -.NOTPARALLEL: - # absolute path TOPDIR:=$(shell pwd) CONFIG_CONFIG_IN=Config.in
Signature · 1 line
1.8.4
v8 cover Add support for top-level parallel make
In reply to Fabio PorceddaQuoted 14 lines · Fabio Porcedda
Hi all,
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.With this patch set the top-level parallel make seems to work fine, example:
make clean make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))
Quoted 7 lines
On my quad core system the building time for
qemu_x86_defconfigis 13m versus 11m.I've tested the
qemu_x86_defconfiguclibc/eglibc/glibc.Best regards
Fabio Porcedda
Hi Fabio,
I've tested this patch series now. I'll give acks and comments for the
individual patches later. But first some overall observations.
When ccache is enabled, make complains about a circular dependency
$(BUILD_DIR)/host-ccache-3.1.8/.stamp_downloaded <- dependencies You'll find something similar for sstrip, and when the host doesn't have a suitable tar or xz. It's not critical because make will break the dependency cycle and still build these packages before the rest, but it's not nice. Unfortunately, I don't see a simple way to avoid it.
In system/system.mk, there are still a few different targets
manipulating inittab; these have to be serialized.
The initramfs rootfs will rebuild the kernel in parallel with building
the cpio archive. This should be fixed, which is probably a bit invasive.
legal-info doesn't work in parallel. Not a critical concern, but
ideally that should be fixed as well.
Regards,
Arnout
Quoted 57 lines
v8:
- rebased over master
- added patche for for base dependency
- added patche for for glibc package
- added patche for for uclibc package
- removed patch already merged
- changed some descriptions
- modified the patch for toolchain dependency to prevent circular dependency
v7:
- add the latest patch
- add to the first patch the <pkgname>_TOOLCHAIN variable
- improve the fifth patch
v6:
- added the fifth patch
- updated the fourth patch adding the install targets
- updated the second patch to removeTARGETS_ALL
v5:
- added the fourth patch
- fixed some typos
- rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
- add support for top-level parallel make for the glibc package
v4:
- rebased over master
- add Acked-by: Thomas Petazzoni on the third patch
- changed the orderd of the patches
v3:
- add back the patch "package: add toolchain dependency to inner-generic-package" because now is working fine.
- add Acked-by: Arnout Vandecappelle to the third patch.
- reworked the second patch following Arnout suggestions.
v2:
- remove patch "package: add toolchain dependency to inner-generic-package" because was not working fine against recent toolchain changes.Fabio Porcedda (8):
package: add base dependency to every package
package: add toolchain dependency to every target package
package: add support for top-level parallel make
Makefile: add support for top-level parallel make
glibc: add support for top-level parallel make
uclibc: add support for top-level parallel make
package: enable jobserver for recursive make
Makefile: enable top-level parallel makeMakefile | 22 +++----- fs/common.mk | 4 +- package/glibc/glibc.mk | 7 ++- package/linux-headers/linux-headers.mk | 3 ++ package/pkg-autotools.mk | 3 +- package/pkg-generic.mk | 60 +++++++++++++--------- package/uclibc/uclibc.mk | 7 ++- .../toolchain-buildroot/toolchain-buildroot.mk | 2 + toolchain/toolchain-external/toolchain-external.mk | 2 + toolchain/toolchain/toolchain.mk | 5 +- 10 files changed, 69 insertions(+), 46 deletions(-)
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
v8 cover Add support for top-level parallel make
In reply to Fabio PorceddaFabio, Arnout,
Quoted 7 lines · Fabio Porcedda
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:
(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.
(2) That being said, Fabio's patches have anyway a merit on their own since they make things cleaner, and that therefore, patches 1 to 6 can be merged. Maybe patch 7 even if I'm not sure it makes sense to have it without the top-level parallel build.
What do you think?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
v8 cover Add support for top-level parallel make
Quoted 12 lines · Thomas Petazzoni
Fabio, Arnout,
Quoted 7 lines · Fabio Porcedda
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
Quoted 5 lines
I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:
(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.
What do you think if we provide an easy way to use top-level parallel make that warns about the problems:
ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!) BR2_JLEVEL=
else
# This top-level Makefile can *not* be executed in parallel .NOTPARALLEL:
endif
make P=1 -j5
Quoted 7 lines
(2) That being said, Fabio's patches have anyway a merit on their own since they make things cleaner, and that therefore, patches 1 to 6 can be merged. Maybe patch 7 even if I'm not sure it makes sense to have it without the top-level parallel build.
What do you think?
Quoted 7 lines
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
Best regards
Signature · 1 line
Fabio Porcedda
v8 cover Add support for top-level parallel make
Quoted 32 lines · Fabio Porcedda
Quoted 12 lines · Thomas Petazzoni
Fabio, Arnout,
Quoted 7 lines · Fabio Porcedda
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
Quoted 5 lines
I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:
(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.
What do you think if we provide an easy way to use top-level parallel make that warns about the problems:
ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!) BR2_JLEVEL=
else
# This top-levelMakefilecan *not* be executed in parallel .NOTPARALLEL:
endif
Quoted 7 lines
make P=1 -j5
Quoted 4 lines
(2) That being said, Fabio's patches have anyway a merit on their own since they make things cleaner, and that therefore, patches 1 to 6 can be merged. Maybe patch 7 even if I'm not sure it makes sense to have it without the top-level parallel build.
The patch 7 make sense only for top-level parallel build but i suggest to accept it because does not harm the serial building and is needed for someone that want to try to use top-level parallel make.
Quoted 1 line
Quoted 1 line
What do you think?
Is great to have most of the patch accepted, i think it's a step in the right direction.
I will send the patch set v9 soon.
Quoted 10 lines
Quoted 8 lines
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
Quoted 3 lines
Best regards
--
Fabio Porcedda
Thanks and best regards
Signature · 1 line
Fabio Porcedda
v8 cover Add support for top-level parallel make
In reply to Fabio PorceddaQuoted 17 lines · Fabio Porcedda
Quoted 15 lines · Thomas Petazzoni
Fabio, Arnout,
Quoted 7 lines · Fabio Porcedda
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:
Quoted 8 lines
Quoted 2 lines
(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.
What do you think if we provide an easy way to use top-level parallel make that warns about the problems:
ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!)
Such a warning will not really be visible, because it appears at the
very beginning of the build and is immediately followed by all of the build output...
Quoted 5 lines
BR2_JLEVEL=
else
# This top-levelMakefilecan *not* be executed in parallel .NOTPARALLEL:
endif
Since you only need to make this simple change in the Makefile (remove
.NOTPARALLEL), I think it's a lot more effective for people who want top-level parallel builds to remove this line in their copy of buildroot. We also agreed at the BR developer meeting that we would add an explanatory comment at that location to make it clear why we disable parallel builds. That way, someone who wants to enable it will be aware of the risks.
Regards,
Arnout
Quoted 3 lines
make P=1 -j5
[snip]
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
v8 cover Add support for top-level parallel make
Quoted 44 lines · Arnout Vandecappelle
Quoted 24 lines · Fabio Porcedda
Quoted 14 lines · Thomas Petazzoni
Fabio, Arnout,
Quoted 7 lines · Fabio Porcedda
this is a patch set for adding support for top-level parallel make in buildroot, the common problem scattered in buildroot's top-level makefiles is that in the rules it relies on the order of evaluation of the prerequisites, to be able to use top-level parallel make instead of reling on the left to right ordering of evaluation of the prerequisites we must add an explicit rule to describe the dependencies.
Quoted 5 lines
I'd like to agree on a plan with this patch series. At the latest Buildroot Developer Day in Edinburgh, I think we agreed that:
(1) Enabling top-level parallel without having per-package sysroot was not desirable, as it would make the builds non-reproducible.
Quoted 5 lines
What do you think if we provide an easy way to use top-level parallel make that warns about the problems:
ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!)Such a warning will not really be visible, because it appears at the very
beginning of the build and is immediately followed by all of the build output...Quoted 5 lines
BR2_JLEVEL=
else
# This top-levelMakefilecan *not* be executed in parallel .NOTPARALLEL:
endif
Quoted 6 lines
Since you only need to make this simple change in the
Makefile(remove
.NOTPARALLEL), I think it's a lot more effective for people who want top-level parallel builds to remove this line in their copy of buildroot. We also agreed at the BR developer meeting that we would add an explanatory comment at that location to make it clear why we disable parallel builds. That way, someone who wants to enable it will be aware of the risks.
Ok, I will just change the comment in the latest commit, i will add this comment:
-# This top-level Makefile can *not* be executed in parallel +# The parallel execution of this top-level Makefile is disabled +# because it would make builds non-reproducible also some packages +# have unspecified optional dependencies so if those dependencies are +# present when the package is built, they are used, otherwise they +# aren't (but compilation happily proceeds). This means that the +# package order is relevant in this case, and the end result will +# differ if the order is swapped due to parallel building. +# Taking in account the above warnings, if you still want to execute +# this top-level Makefile in parrallel remove/comment the following +# line and execute:
+# make BR2_JLEVEL= -j$((getconf _NPROCESSORS_ONLN+1))
I've used a slightly modified description give by Thomas De Schampheleire. What do you think about that?
I think that there two problems:
- first: non reproducible build, that can be resolved just recording the building steps and replaying it.
- second: the unspecified optional dependency that is harder to fix.
Best regards
Signature · 1 line
Fabio Porcedda