utils/test-pkg: add concurrency parameter
PATCH v3 1/1 utils/test-pkg: add concurrency parameter
Builds run by test-pkg are often not CPU limited, and so running multiple builds concurrently has the potential to speed things up quite a lot.
Add a concurrency parameter to the script that allows for multiple builds to run concurrently. The default is the current behavior, running a single build at a time. If -C|--concurrent=0 is specified, the number of concurrent builds is set to the output of $(nproc) to match the number of logical CPUs.
$ time bash utils/test-pkg -c sdl2.config -p sdl2 -C1
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: SKIPPED
bootlin-x86-64-musl [4/6]: OK
br-arm-full-static [5/6]: SKIPPED
arm-aarch64 [6/6]: OK6 builds, 2 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 2m32.979s user 6m15.356s sys 0m30.878s
$ time bash utils/test-pkg -c sdl2.config -p sdl2 -C0
bootlin-armv5-uclibc [1/6]: OK
bootlin-armv7-glibc [2/6]: OK
bootlin-armv7m-uclibc [3/6]: SKIPPED
bootlin-x86-64-musl [4/6]: OK
br-arm-full-static [5/6]: SKIPPED
arm-aarch64 [6/6]: OK6 builds, 2 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 0m41.704s user 6m40.802s sys 0m29.976s
---
The patch appears to behave as intended in my testing, and it speeds up package testing quite a lot. The commit description shows the results testing sdl2, and I've pasted another result below from building libpng, which illustrates the time saved with a large number of builds.
$ time utils/test-pkg -c libpng.config -p libpng -a
arm-aarch64 [ 1/35]: OK
bootlin-aarch64-glibc [ 2/35]: OK
bootlin-arcle-hs38-uclibc [ 3/35]: OK
bootlin-armv5-uclibc [ 4/35]: OK
bootlin-armv7-glibc [ 5/35]: OK
bootlin-armv7-musl [ 6/35]: OK
bootlin-armv7m-uclibc [ 7/35]: OK
bootlin-m68k-5208-uclibc [ 8/35]: OK
bootlin-m68k-68040-uclibc [ 9/35]: OK
bootlin-microblazeel-uclibc [10/35]: OK
bootlin-mipsel-uclibc [11/35]: OK
bootlin-mipsel32r6-glibc [12/35]: OK
bootlin-openrisc-uclibc [13/35]: OK
bootlin-powerpc-e500mc-uclibc [14/35]: OK
bootlin-powerpc64le-power8-glibc [15/35]: OK
bootlin-riscv32-glibc [16/35]: OK
bootlin-riscv64-glibc [17/35]: OK
bootlin-riscv64-musl [18/35]: OK
bootlin-s390x-z13-glibc [19/35]: OK
bootlin-sh4-uclibc [20/35]: OK
bootlin-sparc-uclibc [21/35]: OK
bootlin-sparc64-glibc [22/35]: OK
bootlin-x86-64-glibc [23/35]: OK
bootlin-x86-64-musl [24/35]: OK
bootlin-x86-64-uclibc [25/35]: OK
bootlin-x86-i686-musl [26/35]: OK
bootlin-xtensa-uclibc [27/35]: OK
br-arm-basic [28/35]: OK
br-arm-full-nothread [29/35]: OK
br-arm-full-static [30/35]: OK
br-i386-pentium4-full [31/35]: OK
br-mips64-n64-full [32/35]: OK
br-mips64r6-el-hf-glibc [33/35]: OK
br-powerpc-603e-basic-cpp [34/35]: OK
br-powerpc64-power7-glibc [35/35]: OK35 builds, 0 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 23m34.593s user 25m30.113s sys 2m10.823s
$ time utils/test-pkg -c libpng.config -p libpng -a -C0
arm-aarch64 [ 1/35]: OK
bootlin-aarch64-glibc [ 2/35]: OK
bootlin-arcle-hs38-uclibc [ 3/35]: OK
bootlin-armv5-uclibc [ 4/35]: OK
bootlin-armv7-glibc [ 5/35]: OK
bootlin-armv7-musl [ 6/35]: OK
bootlin-armv7m-uclibc [ 7/35]: OK
bootlin-m68k-5208-uclibc [ 8/35]: OK
bootlin-m68k-68040-uclibc [ 9/35]: OK
bootlin-microblazeel-uclibc [10/35]: OK
bootlin-mipsel-uclibc [11/35]: OK
bootlin-mipsel32r6-glibc [12/35]: OK
bootlin-openrisc-uclibc [13/35]: OK
bootlin-powerpc-e500mc-uclibc [14/35]: OK
bootlin-powerpc64le-power8-glibc [15/35]: OK
bootlin-riscv32-glibc [16/35]: OK
bootlin-riscv64-glibc [17/35]: OK
bootlin-riscv64-musl [18/35]: OK
bootlin-s390x-z13-glibc [19/35]: OK
bootlin-sh4-uclibc [20/35]: OK
bootlin-sparc-uclibc [21/35]: OK
bootlin-sparc64-glibc [22/35]: OK
bootlin-x86-64-glibc [23/35]: OK
bootlin-x86-64-musl [24/35]: OK
bootlin-x86-64-uclibc [25/35]: OK
bootlin-x86-i686-musl [26/35]: OK
bootlin-xtensa-uclibc [27/35]: OK
br-arm-basic [28/35]: OK
br-arm-full-nothread [29/35]: OK
br-arm-full-static [30/35]: OK
br-i386-pentium4-full [31/35]: OK
br-mips64-n64-full [32/35]: OK
br-mips64r6-el-hf-glibc [33/35]: OK
br-powerpc-603e-basic-cpp [34/35]: OK
br-powerpc64-power7-glibc [35/35]: OK35 builds, 0 skipped, 0 build failed, 0 legal-info failed, 0 show-info failed
real 2m6.510s user 41m24.460s sys 3m31.484s
This test was performed on a 16-core Ryzen 9 9950X, and the all-toolchain build was ~11x faster compared to the same serial run.
As always, feedback is very welcome.
---
Changes in v3:
- Fix improper removal of '-T' short option during rebase
- Minor tweak to remove newline between prompt and output
- Simplify make job termination
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Rebase on origin/master
- Properly restore cursor on interrupt
- Properly terminate running jobs on interrupt
- Add animated spinner for running builds
- Simplify status updates
- Link to v1: https://lore.kernel.org/r/[email protected]
--- utils/test-pkg | 142 ++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 121 insertions(+), 21 deletions(-)
utils/test-pkg +121 -21
@@ -3,27 +3,45 @@ set -e TOOLCHAINS_CSV='support/config-fragments/autobuild/toolchain-configs.csv' TEMP_CONF="" -abort=0 + +# associative array tracking running build jobs by PID +declare -a running + +# offet for end of output +end_offs=0 + +restore_cursor() { + tput cnorm + tput rc + tput cud "$((end_offs + 1))" + printf '\n' +} do_abort() { - abort=1 + restore_cursor + do_clean } do_clean() { if [ -n "${TEMP_CONF}" ]; then rm -f "${TEMP_CONF}" fi + + # Terminate any running jobs + pkill -TERM --parent $$ 2>/dev/null || true + wait } main() { local o O opts - local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only + local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only \ + concurrency local ret nb nb_skip nb_fail nb_legal nb_show nb_tc build_dir keep local -a toolchains local pkg_br_name - o='hakc:d:n:p:r:t:T:' - O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:,toolchain-name:' + o='hakc:d:n:p:r:t:T:C:' + O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:,toolchain-name:,concurrency:' opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")" eval set -- "${opts}"
@@ -33,6 +51,7 @@ main() { number=0 mode=0 prepare_only=0 + concurrency=1 toolchains_csv="${TOOLCHAINS_CSV}" while [ ${#} -gt 0 ]; do case "${1}" in
@@ -69,6 +88,9 @@ main() { (-T|--toolchain-name) toolchain_name="${2}"; shift 2 ;; + (-C|--concurrency) + concurrency="${2}"; shift 2 + ;; (--) shift; break ;;
@@ -88,6 +110,13 @@ main() { if [ ! -e "${cfg}" ]; then printf "error: %s: no such file\n" "${cfg}" >&2; exit 1 fi + if [ "${concurrency}" -eq 0 ] 2>/dev/null; then + concurrency=$(nproc) + fi + + if ! [ "${concurrency}" -gt 0 ] 2>/dev/null; then + printf "error: concurrency must be an integer\n" >&2; exit 1 + fi if [ -z "${dir}" ]; then dir="${HOME}/br-test-pkg" fi
@@ -137,31 +166,99 @@ main() { nb_fail=0 nb_legal=0 nb_show=0 - for toolchainconfig in "${toolchains[@]}"; do - : $((nb++)) - toolchain="$(basename "${toolchainconfig}" .config)" - build_dir="${dir}/${toolchain}" - printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} "${nb}" "${nb_tc}" - build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${prepare_only}" && ret=0 || ret=${?} - case ${ret} in - (0) printf "OK\n";; - (1) : $((nb_skip++)); printf "SKIPPED\n";; - (2) : $((nb_fail++)); printf "FAILED\n";; - (3) : $((nb_legal++)); printf "FAILED\n";; - (4) : $((nb_show++)); printf "FAILED\n";; - esac - if [ "${abort}" -eq 1 ]; then - return 1 - fi + tput civis + + # Allocate lines for all toolchains up front to avoid scroll-invalidating + # the saved cursor + for ((i = 0; i < nb_tc; i++)); do printf '\n'; done + tput cuu "$((nb_tc + 1))" + tput sc + + declare -A pid_to_idx + declare -a display_order + + spinc='/-\|' + spini=0 + while (( nb < nb_tc || ${#running[@]} > 0)); do + while (( nb < nb_tc && ${#running[@]} < concurrency )); do + toolchainconfig=${toolchains[$nb]} + toolchain="$(basename "${toolchainconfig}" .config)" + build_dir="${dir}/${toolchain}" + + build_one \ + "${build_dir}" \ + "${toolchainconfig}" \ + "${cfg}" \ + "${pkg}" \ + "${prepare_only}" & + + pid=$!; pid_to_idx[${pid}]=${nb} + running+=( "$pid" ) + slot=${#display_order[@]} + display_order+=( "$nb" ) + end_offs=${#display_order[@]} + nb=$((nb + 1)) + done + + for i in "${!running[@]}"; do + pid="${running[${i}]}" + idx="${pid_to_idx[$pid]}" + toolchainconfig=${toolchains[$idx]} + toolchain="$(basename "${toolchainconfig}" .config)" + + if ! kill -0 "${pid}" 2>/dev/null; then + wait "${pid}" && ret=0 || ret=${?} + case "${ret}" in + (0) stat="OK";; + (1) : $((nb_skip++)); stat="SKIPPED";; + (2) : $((nb_fail++)); stat="FAILED";; + (3) : $((nb_legal++)); stat="FAILED";; + (4) : $((nb_show++)); stat="FAILED";; + esac + + unset 'running[i]' + else + stat="${spinc:$spini:1}" + fi + + # Find the line to print the status on for this PID + for slot in "${!display_order[@]}"; do + if [[ ${display_order[$slot]} -eq ${idx} ]]; then + break + fi + done + + update_line "$slot" "%40s [%*d/%d]: %s" \ + "${toolchain}" \ + "${#nb_tc}" \ + "$((idx + 1))" \ + "${nb_tc}" \ + "${stat}" + done + + running=( "${running[@]}" ) + spini=$(((spini+1) % ${#spinc})) + sleep 0.1 done + restore_cursor printf "%d builds, %d skipped, %d build failed, %d legal-info failed, %d show-info failed\n" \ "${nb}" "${nb_skip}" "${nb_fail}" "${nb_legal}" "${nb_show}" return $((nb_fail + nb_legal)) } +update_line() { + local slot=$1; shift + tput rc + tput cud "$((slot + 1))" + fmt=$1; shift + + # shellcheck disable=SC2059 + printf -- "\033[K${fmt}" "$@" +} + build_one() { local dir="${1}" local toolchainconfig="${2}"
@@ -288,6 +385,9 @@ Options: -r N, --random N Limit the tests to the N randomly selected toolchains. + -C N, --concurrency N + Run N builds concurrently. If N is 0, match the number of logical CPUs. + -t CSVFILE, --toolchains-csv CSVFILE CSV file containing the paths to config fragments of toolchains to try. If not specified, the toolchains in ${TOOLCHAINS_CSV} will be
---
base-commit: 6144b0f4b73bea810809f09d23bbe76b4979bc13
change-id: 20250619-concurrent-test-pkg-f3ad6d3c01b4
Best regards,
Signature · 6 lines
Joseph Kogut <[email protected]> _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Joseph,
Quoted 3 lines · Joseph Kogut
Builds run by test-pkg are often not CPU limited, and so running multiple builds concurrently has the potential to speed things up quite a lot.
Thanks for your patch, and sorry for the delay in getting back to you. We took advantage of the on-going Buildroot Developers Meeting to (finally) have a discussion around your contribution. We believe the feature is definitely useful, however the hand-written implementation of the parallelization seems a bit tricky. Can you instead use something like GNU Parallel which is designed for that?
Thanks a lot!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Thomas,
Quoted 7 lines · Thomas Petazzoni
Thanks for your patch, and sorry for the delay in getting back to you. We took advantage of the on-going Buildroot Developers Meeting to (finally) have a discussion around your contribution. We believe the feature is definitely useful, however the hand-written implementation of the parallelization seems a bit tricky. Can you instead use something like GNU Parallel which is designed for that?
Thanks for taking the time to discuss this! I originally sent it as an RFC because I had similar concerns regarding the implementation, and wanted to start a discussion about something more suitable.
I can spin a GNU parallel version and resubmit.
Thanks,
Joseph
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot