build: Parallelize subdirectory recursion in generic_trunk_target
PATCH 1/1 build: Parallelize subdirectory recursion in generic_trunk_target
From: Andrea Cervesato <[email protected]>
Currently, generic_trunk_target traverses $(SUBDIRS) using a serial shell 'for' loop. This serializes directory traversal and prevents 'make -jN' from building sibling directories in parallel.
Replace the shell loop with target-based recursion:
1. Map each subdirectory to a phony target: <dir>-all, <dir>-install, and <dir>-check.
2. Make 'all', 'install', and 'check' depend on these targets so GNU Make can dispatch them concurrently across available job slots.
3. In each recipe, strip the suffix with $(patsubst %-<action>,%,$@) to extract the directory name and invoke $(MAKE) -C <dir>.
4. Add explicit ordering dependencies where subdirectories produce libraries required by sibling directories (tools before testcases, kernel/lib before syscalls and mem, rpc-tirpc before basic_tests).
This cuts full build time by over 20% on multi-core systems.
---
The following results show how this optimization moves compile times really far from what we have now. With a single threaded build, LTP would take approximately 12 minutes to compile. With a multi-threaded build on 16 CPUs, this drops down to ~46 seconds.
time make -j16
Executed in 45.96 secs fish external usr time 519.11 secs 0.24 millis 519.11 secs sys time 167.13 secs 1.08 millis 167.13 secs --- Makefile | 2 ++ include/mk/generic_trunk_target.inc | 35 ++++++++++++++++++++++++++++------- testcases/kernel/Makefile | 3 +++ testcases/network/rpc/Makefile | 3 +++ 4 files changed, 36 insertions(+), 7 deletions(-)
Makefile +2 -0
@@ -63,6 +63,8 @@ CHECK_TARGETS := $(addsuffix -check,testcases lib) all: $(addsuffix -all,$(COMMON_TARGETS)) Version $(MAKE_TARGETS): lib-all libs-all +# tools contains apicmds/tst_kvercmp required by some testcases +testcases-all: tools-all .PHONY: include-all include-install include-install: $(top_builddir)/include/config.h include/mk/config.mk include-all
include/mk/generic_trunk_target.inc +28 -7
@@ -83,15 +83,36 @@ ifeq ($(strip $(SUBDIRS)),) $(warning CURDIR is: $(CURDIR)) $(error SUBDIRS empty -- did you want generic_leaf_target instead?) else -$(RECURSIVE_TARGETS): %: | $(SUBDIRS) +# Recurse into subdirectories in parallel instead of using a serial shell loop. +# Each subdirectory is mapped to a phony target (<dir>-all, <dir>-install, etc.). +# GNU Make can then schedule sibling directories concurrently under -jN. +all: $(addsuffix -all,$(SUBDIRS)) +install: $(addsuffix -install,$(SUBDIRS)) +check: $(addsuffix -check,$(SUBDIRS)) + +.PHONY: $(addsuffix -all,$(SUBDIRS)) $(addsuffix -install,$(SUBDIRS)) $(addsuffix -check,$(SUBDIRS)) + +# For each target (e.g. 'open-all'), strip the suffix to get the directory name, +# build trunk-local targets first, and ensure the subdirectory exists. +$(addsuffix -all,$(SUBDIRS)): trunk-all | $(SUBDIRS) $(MAKE_DEPS) ifdef VERBOSE - @set -e; for dir in $(SUBDIRS); do \ - $(MAKE) -C $$dir -f "$(abs_srcdir)/$$dir/Makefile" $@; \ - done + $(MAKE) -C "$(patsubst %-all,%,$@)" -f "$(abs_srcdir)/$(patsubst %-all,%,$@)/Makefile" all else - @set -e; for dir in $(SUBDIRS); do \ - $(MAKE) --no-print-directory -C $$dir -f "$(abs_srcdir)/$$dir/Makefile" $@; \ - done + @$(MAKE) --no-print-directory -C "$(patsubst %-all,%,$@)" -f "$(abs_srcdir)/$(patsubst %-all,%,$@)/Makefile" all +endif + +$(addsuffix -install,$(SUBDIRS)): trunk-install | $(SUBDIRS) $(MAKE_DEPS) +ifdef VERBOSE + $(MAKE) -C "$(patsubst %-install,%,$@)" -f "$(abs_srcdir)/$(patsubst %-install,%,$@)/Makefile" install +else + @$(MAKE) --no-print-directory -C "$(patsubst %-install,%,$@)" -f "$(abs_srcdir)/$(patsubst %-install,%,$@)/Makefile" install +endif + +$(addsuffix -check,$(SUBDIRS)): trunk-check | $(SUBDIRS) $(MAKE_DEPS) +ifdef VERBOSE + $(MAKE) -C "$(patsubst %-check,%,$@)" -f "$(abs_srcdir)/$(patsubst %-check,%,$@)/Makefile" check +else + @$(MAKE) --no-print-directory -C "$(patsubst %-check,%,$@)" -f "$(abs_srcdir)/$(patsubst %-check,%,$@)/Makefile" check endif endif
testcases/kernel/Makefile +3 -0
@@ -54,3 +54,6 @@ FILTER_OUT_DIRS += containers controllers device-drivers fs io ipc mem \ endif include $(top_srcdir)/include/mk/generic_trunk_target.mk + +# libkerntest.a in lib/ must be built before syscalls and mem consumers +syscalls-all mem-all: lib-all
testcases/network/rpc/Makefile +3 -0
@@ -9,3 +9,6 @@ include $(top_srcdir)/include/mk/env_pre.mk INSTALL_TARGETS := *.sh include $(top_srcdir)/include/mk/generic_trunk_target.mk + +# librpc-tirpc.a in rpc-tirpc/ must be built before basic_tests +basic_tests-all: rpc-tirpc-all
---
base-commit: 2ae4cfe69b9010018f2e5df6fa4bbba21810d4f2
change-id: 20261020-build_system_optimize-6cd0343f08e1
Best regards,
Signature · 5 lines
Andrea Cervesato <[email protected]> -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Andrea,
testcases/kernel/Makefile FILTER_OUT_DIRS += containers controllers device-drivers fs io ipc mem \ · line 59
Quoted 4 lines · Andrea Cervesato
build: Parallelize subdirectory recursion in
generic_trunk_targettestcases/kernel/Makefile +1
-0+syscalls-all mem-all: lib-all
Medium: running make -j8 directly in testcases/kernel/syscalls or testcases/kernel/mem bypasses this parent dependency, allowing sibling leaves to rebuild the same missing libkerntest.a concurrently. This races object/archive writes and can cause intermittent link failures. Build shared prerequisites once at these subtree entry points before launching their children.
Verdict - Needs revision
---
Note:
The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.
Regards,
LTP AI Reviewer
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Andrea,
thanks for the speedup.
...
Quoted 6 lines
include/mk/generic_trunk_target.inc +6
-0+# Recurse into subdirectories in parallel instead of using a serial shell loop. +# Each subdirectory is mapped to a phony target (<dir>-all, <dir>-install, etc.). +# GNU Make can then schedule sibling directories concurrently under -jN. +all: $(addsuffix -all,$(SUBDIRS)) +install: $(addsuffix -install,$(SUBDIRS)) +check: $(addsuffix -check,$(SUBDIRS))
include/mk/generic_trunk_target.inc ifeq ($(strip $(SUBDIRS)),) · line 90
Quoted 10 lines
include/mk/generic_trunk_target.inc +6
-3+ +.PHONY: $(addsuffix -all,$(SUBDIRS)) $(addsuffix -install,$(SUBDIRS)) $(addsuffix -check,$(SUBDIRS)) + +# For each target (e.g. 'open-all'), strip the suffix to get the directory name, +# build trunk-local targets first, and ensure the subdirectory exists. +$(addsuffix -all,$(SUBDIRS)): trunk-all | $(SUBDIRS) $(MAKE_DEPS) ifdef VERBOSE - @set -e; for dir in $(SUBDIRS); do \ - $(MAKE) -C $$dir -f "$(abs_srcdir)/$$dir/Makefile" $@; \ - done
Yeah, replacing shell for loop which is single process only to make processing makes sense.
Quoted 28 lines
include/mk/generic_trunk_target.inc +16
-3+ $(MAKE) -C "$(patsubst %-all,%,$@)" -f "$(abs_srcdir)/$(patsubst %-all,%,$@)/Makefile" all else - @set -e; for dir in $(SUBDIRS); do \ - $(MAKE) --no-print-directory -C $$dir -f "$(abs_srcdir)/$$dir/Makefile" $@; \ - done + @$(MAKE) --no-print-directory -C "$(patsubst %-all,%,$@)" -f "$(abs_srcdir)/$(patsubst %-all,%,$@)/Makefile" all +endif + +$(addsuffix -install,$(SUBDIRS)): trunk-install | $(SUBDIRS) $(MAKE_DEPS) +ifdef VERBOSE + $(MAKE) -C "$(patsubst %-install,%,$@)" -f "$(abs_srcdir)/$(patsubst %-install,%,$@)/Makefile" install +else + @$(MAKE) --no-print-directory -C "$(patsubst %-install,%,$@)" -f "$(abs_srcdir)/$(patsubst %-install,%,$@)/Makefile" install +endif + +$(addsuffix -check,$(SUBDIRS)): trunk-check | $(SUBDIRS) $(MAKE_DEPS) +ifdef VERBOSE + $(MAKE) -C "$(patsubst %-check,%,$@)" -f "$(abs_srcdir)/$(patsubst %-check,%,$@)/Makefile" check +else + @$(MAKE) --no-print-directory -C "$(patsubst %-check,%,$@)" -f "$(abs_srcdir)/$(patsubst %-check,%,$@)/Makefile" check endif endiftestcases/kernel/Makefile +0
-0
testcases/kernel/Makefile FILTER_OUT_DIRS += containers controllers device-drivers fs io ipc mem \ · line 59
Quoted 6 lines
testcases/kernel/Makefile +3
-0@@ -54,3 +54,6 @@ FILTER_OUT_DIRS += containers controllers device-drivers fs io ipc mem \ endif include $(top_srcdir)/include/mk/generic_trunk_target.mk + +# libkerntest.a in lib/ must be built before syscalls and mem consumers +syscalls-all mem-all: lib-all
I wonder if this can be autogenerated, because this was not needed before.
I'm ok with this explicit definition. Probably in the future we should move it to libs/ (using it as LTPLIBS), which will solve the dependency.
testcases/network/rpc/Makefile include $(top_srcdir)/include/mk/env_pre.mk · line 14
Quoted 11 lines
testcases/network/rpc/Makefile +3
-0@@ -9,3 +9,6 @@ include $(top_srcdir)/include/mk/env_pre.mk INSTALL_TARGETS := *.sh include $(top_srcdir)/include/mk/generic_trunk_target.mk + +# librpc-tirpc.a in rpc-tirpc/ must be built before basic_tests +basic_tests-all: rpc-tirpc-all
Also librpc-tirpc.a should be moved into libs/ (if we still keep this code and not move it to libtirpc project where it'd make more sense).
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Petr,
Quoted 17 lines
testcases/kernel/Makefile FILTER_OUT_DIRS += containers controllers device-drivers fs io ipc mem \ · line 59
Quoted 4 lines
include $(top_srcdir)/include/mk/generic_trunk_target.mk
testcases/kernel/Makefile +3
-0+ +# libkerntest.a in lib/ must be built before syscalls and mem consumers +syscalls-all mem-all: lib-allI wonder if this can be autogenerated, because this was not needed before.
I'm ok with this explicit definition. Probably in the future we should move it to libs/ (using it as LTPLIBS), which will solve the dependency.
Quoted 6 lines
testcases/network/rpc/Makefile +0
-0@@ -9,3 +9,6 @@ include $(top_srcdir)/include/mk/env_pre.mk INSTALL_TARGETS := *.sh
Quoted 7 lines
testcases/network/rpc/Makefile include $(top_srcdir)/include/mk/env_pre.mk · line 14
Quoted 4 lines
testcases/network/rpc/Makefile +3
-0include $(top_srcdir)/include/mk/generic_trunk_target.mk + +# librpc-tirpc.a in rpc-tirpc/ must be built before basic_tests +basic_tests-all: rpc-tirpc-allAlso librpc-tirpc.a should be moved into libs/ (if we still keep this code and not move it to libtirpc project where it'd make more sense).
I'm also thinking about it. We should move all libraries in there. Let's see what Cyril says.
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp