gitlab-ci: allow running test-pkg (branch yem/test-pkg-in-gitlab-ci)
v5 cover gitlab-ci: allow running test-pkg (branch yem/test-pkg-in-gitlab-ci)
Hello All!
This series is a respin of the previous work by Romain:
http://lists.busybox.net/pipermail/buildroot/2021-June/thread.html#313369
Yann has done quite some extensive rework, so much so that it is not sane to apply without further review.
So, the big changes from Romain's work, are:
- introduce new mode for utils/test-pkg, to only prepare the configuration but niot build them,
- support/scripts/generate-gitlab-ci-yml generates the yaml blurb from the valid configurations
- an empty config fragment is now an error
- a config fragment that yields no test is also an error
- some eye-candy and code-styles in the shell scripts...
See a pipeline generated with this branch:
https://gitlab.com/ymorin/buildroot/-/pipelines/328074297
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Romain Naour (2):
support/misc/gitlab-ci.yml.in: templatise the make command utils/test-pkg: add gitlab-ci support
Yann E. MORIN (2):
utils/test-pkg: remove configurations that are skipped utils/test-pkg: add mode to only prepare .config files
.gitlab-ci.yml | 5 +++++ support/misc/gitlab-ci.yml.in | 38 ++++++++++++++++++++++++++++------ support/scripts/generate-gitlab-ci-yml | 27 +++++++++++++++++++++++- utils/test-pkg | 26 +++++++++++++++++++---- 4 files changed, 85 insertions(+), 11 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v5 1/4 support/misc/gitlab-ci.yml.in: templatise the make command
From: Romain Naour <[email protected]>
In a followup commit, the make command used to log and display the last lines on error will be used in another job.
Factorize it by introducing .run_make template.
--- support/misc/gitlab-ci.yml.in | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
support/misc/gitlab-ci.yml.in +10 -6
@@ -26,20 +26,24 @@ paths: - .config +.run_make: &run_make + | + make O=${OUTPUT_DIR} > >(tee build.log |grep '>>>') 2>&1 || { + echo 'Failed build last output' + tail -200 build.log + exit 1 + } + .defconfig_base: before_script: - DEFCONFIG_NAME=${CI_JOB_NAME} + - OUTPUT_DIR=output script: - echo "Configure Buildroot for ${DEFCONFIG_NAME}" - make ${DEFCONFIG_NAME} - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME} - echo 'Build buildroot' - - | - make > >(tee build.log |grep '>>>') 2>&1 || { - echo 'Failed build last output' - tail -200 build.log - exit 1 - } + - *run_make - | ./support/scripts/boot-qemu-image.py "${DEFCONFIG_NAME}" > >(tee runtime-test.log) 2>&1 || { echo 'Failed runtime test last output'
Signature · 1 line
2.25.1
PATCH v5 1/4 support/misc/gitlab-ci.yml.in: templatise the make command
Quoted 9 lines · On Mon, 28 Jun 2021 22:15:12 +0200 "Yann E. MORIN" <[email protected]>…
From: Romain Naour <[email protected]>
In a followup commit, the make command used to log and display the last lines on error will be used in another job.
Factorize it by introducing .run_make template.Signed-off-by Romain Naour <[email protected]>Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Arnout Vandecappelle
AppliedRe: [Buildroot] [PATCH 1/4 v5] support/misc/gitlab-ci.yml.in: templatise the make command
In reply to Yann E. MORINQuoted 9 lines · Yann E. MORIN
From: Romain Naour <[email protected]>
In a followup commit, the make command used to log and display the last lines on error will be used in another job.
Factorize it by introducing .run_make template.Signed-off-by Romain Naour <[email protected]>Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>Signed-off-by Yann E. MORIN <[email protected]>
Applied to next, thanks, and the second one as well.
Regards,
Arnout
Quoted 30 lines
--- support/misc/gitlab-ci.yml.in | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)support/misc/gitlab-ci.yml.in +9
-0@@ -26,20 +26,24 @@ paths: - .config +.run_make: &run_make + | + make O=${OUTPUT_DIR} > >(tee build.log |grep '>>>') 2>&1 || { + echo 'Failed build last output' + tail -200 build.log + exit 1 + } + .defconfig_base: before_script: - DEFCONFIG_NAME=${CI_JOB_NAME} + - OUTPUT_DIR=output script: - echo "Configure Buildroot for ${DEFCONFIG_NAME}" - make ${DEFCONFIG_NAME} - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME} - echo 'Build buildroot'
support/misc/gitlab-ci.yml.in line 49
Quoted 10 lines
support/misc/gitlab-ci.yml.in +1
-6- - | - make > >(tee build.log |grep '>>>') 2>&1 || { - echo 'Failed build last output' - tail -200 build.log - exit 1 - } + - *run_make - | ./support/scripts/boot-qemu-image.py "${DEFCONFIG_NAME}" > >(tee runtime-test.log) 2>&1 || { echo 'Failed runtime test last output'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v5 2/4 utils/test-pkg: remove configurations that are skipped
In reply to Yann E. MORINWhen the config fragment provided by the user is not usable with a specific toolchain configuration, the resulting .config file was kept around.
In a follow up commit, we'll need to know, from outside test-pkg, if a specific configuration was indeed usable or not.
So, unless if the user actually requested to keep the build directories, remove the .config file when it contains a configration that would be skipped.
---
v5: new patch
--- utils/test-pkg | 4 ++++ 1 file changed, 4 insertions(+)
utils/test-pkg +4 -0
@@ -161,6 +161,10 @@ build_one() { # done in the same locale. comm -23 <(sort "${cfg}") <(sort "${dir}/.config") >"${dir}/missing.config" if [ -s "${dir}/missing.config" ]; then + if [ ${keep} -ne 1 ]; then + # Invalid configuration, drop it + rm -f "${dir}/.config" + fi return 1 fi # Remove file, it's empty anyway.
Signature · 1 line
2.25.1
PATCH v5 3/4 utils/test-pkg: add mode to only prepare .config files
In reply to Yann E. MORINCurrently, running test-pkg is only done locally on the developpers machine.
In a follow up commit, we'll add the possibility to run test-pkg in a gitlab-ci pipeline and, to speed up things, with one job per buildable configuration.
As such, we will need that test-pkg only ever prepares the configuration, and that it does not build them.
Add such a mode, with a new option, --prepare-only
---
Note: naming is hard; naming options is harder; naming options with a terse term is even harder; naming options with a terse term that is still meaningful and explains what the option does, is even harder yet.
---
v5: split off from the next patch into this patch
--- utils/test-pkg | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
utils/test-pkg +18 -4
@@ -12,13 +12,13 @@ do_clean() { main() { local o O opts - local cfg dir pkg random toolchains_csv toolchain all number mode + local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only local ret nb nb_skip nb_fail nb_legal nb_tc build_dir keep local -a toolchains local pkg_br_name - o='hakc:d:n:p:r:t:' - O='help,all,keep,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:' + o='hakgc:d:n:p:r:t:' + O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:' opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")" eval set -- "${opts}"
@@ -27,6 +27,7 @@ main() { keep=0 number=0 mode=0 + prepare_only=0 toolchains_csv="${TOOLCHAINS_CSV}" while [ ${#} -gt 0 ]; do case "${1}" in
@@ -39,6 +40,9 @@ main() { (-k|--keep) keep=1; shift 1 ;; + (-l|--prepare-only) + prepare_only=1; shift 1 + ;; (-c|--config-snippet) cfg="${2}"; shift 2 ;;
@@ -127,7 +131,7 @@ main() { toolchain="$(basename "${toolchainconfig}" .config)" build_dir="${dir}/${toolchain}" printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc} - build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?} + build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${prepare_only}" && ret=0 || ret=${?} case ${ret} in (0) printf "OK\n";; (1) : $((nb_skip++)); printf "SKIPPED\n";;
@@ -147,6 +151,7 @@ build_one() { local toolchainconfig="${2}" local cfg="${3}" local pkg="${4}" + local defer="${5}" mkdir -p "${dir}"
@@ -170,6 +175,11 @@ build_one() { # Remove file, it's empty anyway. rm -f "${dir}/missing.config" + # Defer building the job to the caller (e.g. a gitlab pipeline) + if [ ${defer} -eq 1 ]; then + return 0 + fi + if [ -n "${pkg}" ]; then if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then return 2
@@ -257,6 +267,10 @@ Options: Note: the logfile and configuration is always retained, even without this option. + --prepare-only + Only prepare the .config files, but do not build them. Output the + list of build directories to stdout, and the status on stderr. + Example: Testing libcec would require a config snippet that contains:
Signature · 1 line
2.25.1
Re: [Buildroot] [PATCH 3/4 v5] utils/test-pkg: add mode to only prepare .config files
Quoted 17 lines · Yann E. MORIN
Currently, running test-pkg is only done locally on the developpers machine.
In a follow up commit, we'll add the possibility to run test-pkg in a gitlab-ci pipeline and, to speed up things, with one job per buildable configuration.
As such, we will need that test-pkg only ever prepares the configuration, and that it does not build them.
Add such a mode, with a new option, --prepare-only
Signed-off-by Yann E. MORIN <[email protected]>Cc Romain Naour <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Quoted 4 lines
---
Note: naming is hard; naming options is harder; naming options with a terse term is even harder; naming options with a terse term that is still meaningful and explains what the option does, is even harder yet.
But doing it inconsistently is easy (see below). :-)
Anyway, there's an easy solution to that (which I believe we should apply
here): don't define a terse option.
I think terse options should only be defined for stuff that a human has to
type. In scripts, terse options shouldn't be used, because it makes it harder for the programmer to understand what the command does.
Since prepare-only is meant ot be used by script, I don't think a terse option
is needed.
Quoted 13 lines
---
v5: split off from the next patch into this patch--- utils/test-pkg | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)utils/test-pkg +0
-0@@ -12,13 +12,13 @@ do_clean() {
utils/test-pkg do_clean() { · line 21
Quoted 10 lines
utils/test-pkg +1
-3main() { local o O opts - local cfg dir pkg random toolchains_csv toolchain all number mode + local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only local ret nb nb_skip nb_fail nb_legal nb_tc build_dir keep local -a toolchains local pkg_br_name - o='hakc:d:n:p:r:t:' - O='help,all,keep,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:> + o='hakgc:d:n:p:r:t:'
^ AFAICS this is a 'g'...
Quoted 13 lines
utils/test-pkg +2
-0+ O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:' opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")" eval set -- "${opts}"@@ -27,6 +27,7 @@ main() { keep=0 number=0 mode=0 + prepare_only=0 toolchains_csv="${TOOLCHAINS_CSV}" while [ ${#} -gt 0 ]; do case "${1}" in
utils/test-pkg main() { · line 43
Quoted 4 lines
utils/test-pkg +1
-0@@ -39,6 +40,9 @@ main() { (-k|--keep) keep=1; shift 1 ;; + (-l|--prepare-only)
^ ... but this is an 'l'!
Clearly someone didn't test with the terse option. :-)
Quoted 15 lines
utils/test-pkg +3
-1+ prepare_only=1; shift 1 + ;; (-c|--config-snippet) cfg="${2}"; shift 2 ;;@@ -127,7 +131,7 @@ main() { toolchain="$(basename "${toolchainconfig}" .config)" build_dir="${dir}/${toolchain}" printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc} - build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?} + build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${prepare_only}" && ret=0 || ret=${?} case ${ret} in (0) printf "OK\n";; (1) : $((nb_skip++)); printf "SKIPPED\n";;
utils/test-pkg build_one() { · line 154
Quoted 4 lines
utils/test-pkg +1
-0@@ -147,6 +151,7 @@ build_one() { local toolchainconfig="${2}" local cfg="${3}" local pkg="${4}" + local defer="${5}"
I like it if variables that are the same are called the same. I.e., useprepare_only here as well. Perhaps you can even skip passing it at all since it's a global variable.
Quoted 16 lines
utils/test-pkg +5
-0mkdir -p "${dir}"@@ -170,6 +175,11 @@ build_one() { # Remove file, it's empty anyway. rm -f "${dir}/missing.config" + # Defer building the job to the caller (e.g. a gitlab pipeline) + if [ ${defer} -eq 1 ]; then + return 0 + fi + if [ -n "${pkg}" ]; then if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then return 2
utils/test-pkg Options: · line 270
Quoted 4 lines
utils/test-pkg +1
-0@@ -257,6 +267,10 @@ Options: Note: the logfile and configuration is always retained, even without this option. + --prepare-only
And the terse option isn't even in the help! So nobody would use it anyway...
Regards,
Arnout
utils/test-pkg Options: · line 276
Quoted 7 lines
utils/test-pkg +3
-0+ Only prepare the .config files, but do not build them. Output the + list of build directories to stdout, and the status on stderr. + Example: Testing libcec would require a config snippet that contains:
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 3/4 v5] utils/test-pkg: add mode to only prepare .config files
Hello Arnout, Yann,
Le 05/08/2021 à 22:45, Arnout Vandecappelle a écrit :
Quoted 30 lines · Yann E. MORIN
Quoted 17 lines
Currently, running test-pkg is only done locally on the developpers machine.
In a follow up commit, we'll add the possibility to run test-pkg in a gitlab-ci pipeline and, to speed up things, with one job per buildable configuration.
As such, we will need that test-pkg only ever prepares the configuration, and that it does not build them.
Add such a mode, with a new option, --prepare-only
Signed-off-by Yann E. MORIN <[email protected]>Cc Romain Naour <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>Quoted 4 lines
---
Note: naming is hard; naming options is harder; naming options with a terse term is even harder; naming options with a terse term that is still meaningful and explains what the option does, is even harder yet.But doing it inconsistently is easy (see below). :-)
Anyway, there's an easy solution to that (which I believe we should apply
here): don't define a terse option.
Quoted 6 lines
I think terse options should only be defined for stuff that a human has to
type. In scripts, terse options shouldn't be used, because it makes it harder for the programmer to understand what the command does.Since prepare-only is meant ot be used by script, I don't think a terse option
is needed.
Thanks for your advice. Indeed I don't think we need a terse option.
Best regards,
Romain
Quoted 91 lines
Quoted 15 lines
---
v5: split off from the next patch into this patch--- utils/test-pkg | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)utils/test-pkg +0
-0@@ -12,13 +12,13 @@ do_clean() {utils/test-pkg do_clean() { · line 21
Quoted 10 lines
utils/test-pkg +1
-3main() { local o O opts - local cfg dir pkg random toolchains_csv toolchain all number mode + local cfg dir pkg random toolchains_csv toolchain all number mode prepare_only local ret nb nb_skip nb_fail nb_legal nb_tc build_dir keep local -a toolchains local pkg_br_name - o='hakc:d:n:p:r:t:' - O='help,all,keep,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:> + o='hakgc:d:n:p:r:t:'^ AFAICS this is a 'g'...
Quoted 13 lines
utils/test-pkg +2
-0+ O='help,all,keep,prepare-only,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:' opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")" eval set -- "${opts}"@@ -27,6 +27,7 @@ main() { keep=0 number=0 mode=0 + prepare_only=0 toolchains_csv="${TOOLCHAINS_CSV}" while [ ${#} -gt 0 ]; do case "${1}" inutils/test-pkg main() { · line 43
Quoted 4 lines
utils/test-pkg +1
-0@@ -39,6 +40,9 @@ main() { (-k|--keep) keep=1; shift 1 ;; + (-l|--prepare-only)^ ... but this is an 'l'!Clearly someone didn't test with the terse option. :-)
Quoted 15 lines
utils/test-pkg +3
-1+ prepare_only=1; shift 1 + ;; (-c|--config-snippet) cfg="${2}"; shift 2 ;;@@ -127,7 +131,7 @@ main() { toolchain="$(basename "${toolchainconfig}" .config)" build_dir="${dir}/${toolchain}" printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc} - build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?} + build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${prepare_only}" && ret=0 || ret=${?} case ${ret} in (0) printf "OK\n";; (1) : $((nb_skip++)); printf "SKIPPED\n";;utils/test-pkg build_one() { · line 154
Quoted 4 lines
utils/test-pkg +1
-0@@ -147,6 +151,7 @@ build_one() { local toolchainconfig="${2}" local cfg="${3}" local pkg="${4}" + local defer="${5}"I like it if variables that are the same are called the same. I.e., use
prepare_onlyhere as well. Perhaps you can even skip passing it at all since it's a global variable.Quoted 19 lines
utils/test-pkg +5
-0mkdir -p "${dir}"@@ -170,6 +175,11 @@ build_one() { # Remove file, it's empty anyway. rm -f "${dir}/missing.config" + # Defer building the job to the caller (e.g. a gitlab pipeline) + if [ ${defer} -eq 1 ]; then + return 0 + fi + if [ -n "${pkg}" ]; then if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then return 2@@ -257,6 +267,10 @@ Options: Note: the logfile and configuration is always retained, even without this option.
Quoted 15 lines
utils/test-pkg Options: · line 270
Quoted 1 line
utils/test-pkg +1
-0+ --prepare-onlyAnd the terse option isn't even in the help! So nobody would use it anyway...
Regards,
ArnoutQuoted 7 lines
utils/test-pkg +3
-0+ Only prepare the .config files, but do not build them. Output the + list of build directories to stdout, and the status on stderr. + Example: Testing libcec would require a config snippet that contains:
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 3/4 v5] utils/test-pkg: add mode to only prepare .config files
Romain, Arnout, All,
On 2021-08-21 15:38 +0200, Romain Naour spake thusly:
Quoted 28 lines
Le 05/08/2021 à 22:45, Arnout Vandecappelle a écrit :
Quoted 27 lines · Yann E. MORIN
Quoted 17 lines
Currently, running test-pkg is only done locally on the developpers machine.
In a follow up commit, we'll add the possibility to run test-pkg in a gitlab-ci pipeline and, to speed up things, with one job per buildable configuration.
As such, we will need that test-pkg only ever prepares the configuration, and that it does not build them.
Add such a mode, with a new option, --prepare-only
Signed-off-by Yann E. MORIN <[email protected]>Cc Romain Naour <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>Quoted 4 lines
---
Note: naming is hard; naming options is harder; naming options with a terse term is even harder; naming options with a terse term that is still meaningful and explains what the option does, is even harder yet.But doing it inconsistently is easy (see below). :-)
Anyway, there's an easy solution to that (which I believe we should apply
here): don't define a terse option.
Ah, but there was a misunderstanding: I was refering to the "long option" that I tried to keep terse. I.e. I started off with:
--just-generate-config-for-later-use-in-gitlab-CI-or-anyother-such-CI
and eventually tried to shorten it as much as possible, while still keeping the meaning, so I ended up with just:
--prepare-only
As for the short, one-char option, indeed., we don't really need one.
Quoted 7 lines
Quoted 6 lines
I think terse options should only be defined for stuff that a human has to
type. In scripts, terse options shouldn't be used, because it makes it harder for the programmer to understand what the command does.Since prepare-only is meant ot be used by script, I don't think a terse option
is needed.Thanks for your advice. Indeed I don't think we need a terse option.
We need a terse "long option", but we don;t need a one-char "short option". ;-)
Yes, this is confusing... ;-)
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
+ (-l|--prepare-only)
^ ... but this is an 'l'!Clearly someone didn't test with the terse option. :-)
Yes, I did test with terse "long option", but not with the one-char "short option". Indeed. ;-]
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v5 4/4 utils/test-pkg: add gitlab-ci support
In reply to Yann E. MORINFrom: Romain Naour <[email protected]>
The gitlab-ci support in test-pkg allows to parallelize the test-pkg work into several gitlab jobs. It's much faster than local serialized testing.
To triger this, a developper will have to add, to the latest commit of their branch, a token on its own line, followd by a configuration fragment, e.g.:
test-pkg config:
SOME_OPTION=y
# OTHER_OPTION is not set
SOME_VARIABLE="some value"
This configuration fragment is used as input to test-pkg.
To be able to generate one job per test to run, we need the list of tests in the parent pipeline, and the individual .config files (one per test) in the child pipeline. We use the newly-introduce --prepare-only mode to test-pkg, and collect all the generated .config files as artefacts; those are inherited in the child pipeline via the "needs::pipeline" and "needs::job" directives. This is a bit tricky, and is best described by the Gitlab-CI documentation [0].
We also list those .config files to generate the actual list of jobs to run in the child pipeline.
Notes:
- if the user provides an empty fragment, this is considered an error:
indeed, without a fragment (and the package name), there is no way to know what to test;
- if that fragment yields an empty list of tests, then there is nothing to test either, so that is also considered an error.
[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
[yann.morin.1998 at free.fr:
- split the change to test-pkg to its own patch
- generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml, listing the .config files created by test-pkg
- some code-style-candies...
]
---
v5: Split the change to test-pkg to its own patch
List the .config files
Also store the missing.config files as artefacts
v4: reworked by Yann:
tst-pkg only lists, does not generate the yml code
The yml code is generated by support/scripts/generate-gitlab-ci-yml Empty fragment is an error
Fragment that yierlds no test is also an error
v3: Implement Arnout's review: http://lists.busybox.net/pipermail/buildroot/2021-May/310656.html
Enable artifacts download from child-pipeline
v2: Rework this patch following Arnout review
use CI_COMMIT_DESCRIPTION
remove .config from artifacts but keep images directory since it can be useful for further issue investigation use the "br-test-pkg" prefix for test-pkg jobs
--- .gitlab-ci.yml | 5 +++++ support/misc/gitlab-ci.yml.in | 22 +++++++++++++++++++++ support/scripts/generate-gitlab-ci-yml | 27 +++++++++++++++++++++++++- 3 files changed, 53 insertions(+), 1 deletion(-)
.gitlab-ci.yml +5 -0
@@ -11,8 +11,11 @@ generate-gitlab-ci-yml: stage: generate-gitlab-ci script: ./support/scripts/generate-gitlab-ci-yml support/misc/gitlab-ci.yml.in > generated-gitlab-ci.yml artifacts: + when: always paths: - generated-gitlab-ci.yml + - br-test-pkg/*/.config + - br-test-pkg/*/missing.config buildroot-pipeline: stage: build
@@ -21,3 +24,5 @@ buildroot-pipeline: - artifact: generated-gitlab-ci.yml job: generate-gitlab-ci-yml strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID
support/misc/gitlab-ci.yml.in +22 -0
@@ -80,3 +80,25 @@ - test-output/*/.config - test-output/*/images/* +.test_pkg: + stage: build + before_script: + - OUTPUT_DIR=${CI_JOB_NAME} + script: + - echo "Configure Buildroot for ${OUTPUT_DIR}" + - make O=${OUTPUT_DIR} syncconfig + - make O=${OUTPUT_DIR} savedefconfig + - echo 'Build buildroot' + - *run_make + needs: + - pipeline: $PARENT_PIPELINE_ID + job: generate-gitlab-ci-yml + artifacts: + when: always + expire_in: 2 weeks + paths: + - build.log + - br-test-pkg/*/.config + - br-test-pkg/*/defconfig + - br-test-pkg/*/build/build-time.log + - br-test-pkg/*/build/packages-file-list*.txt
support/scripts/generate-gitlab-ci-yml +26 -1
@@ -23,7 +23,7 @@ _EOF_ gen_tests() { local -a basics defconfigs runtimes - local do_basics do_defconfigs do_runtime + local do_basics do_defconfigs do_runtime do_testpkg local defconfigs_ext cfg tst basics=( DEVELOPERS flake8 package )
@@ -77,9 +77,30 @@ gen_tests() { esac fi + # Retrieve defconfig for test-pkg from the git commit message (if any) + if grep -q -E '^test-pkg config:$' <<<"${CI_COMMIT_DESCRIPTION}"; then + sed -r -n -e '/^test-pkg config:$/{:a;n;p;ba;}' \ + <<<"${CI_COMMIT_DESCRIPTION}" \ + >defconfig.frag + if [ ! -s defconfig.frag ]; then + printf "Empty configuration fragment.\n" >&2; exit 1 + fi + # Use --all since we expect the user having already pre-tested the + # new package with the default subset of toolchains. + ./utils/test-pkg \ + --all --prepare-only \ + --config-snippet defconfig.frag \ + --build-dir br-test-pkg >&2 + do_testpkg=( $(ls -1 br-test-pkg/*/.config 2>/dev/null |xargs -r dirname ) ) + if [ "${#do_testpkg[@]}" -eq 0 ]; then + printf "Configuration fragment enables no test.\n" >&2; exit 1 + fi + fi + # If nothing else, at least do the basics to generate a valid pipeline if [ -z "${do_defconfigs}" \ -a -z "${do_runtime}" \ + -a -z "${do_testpkg}" \ ] then do_basics=true
@@ -101,6 +122,10 @@ gen_tests() { if ${do_runtime:-false}; then printf '%s: { extends: .runtime_test_base }\n' "${runtimes[@]}" fi + + if [ -n "${do_testpkg}" ]; then + printf '%s: { extends: .test_pkg }\n' "${do_testpkg[@]}" + fi } main "${@}"
Signature · 1 line
2.25.1
PATCH v5 4/4 utils/test-pkg: add gitlab-ci support
Hi Yann, all,
some typo fixes.
Quoted 8 lines · Yann E. MORIN
From: Romain Naour <[email protected]>
The gitlab-ci support in test-pkg allows to parallelize the test-pkg work into several gitlab jobs. It's much faster than local serialized testing.
To triger this, a developper will have to add, to the latest commit of
s/triger/trigger and s/developper/developer
Quoted 1 line
their branch, a token on its own line, followd by a configuration
s/followd/followed
Quoted 11 lines
fragment, e.g.:
test-pkg config:
SOME_OPTION=y
#OTHER_OPTIONis not set
SOME_VARIABLE="some value"This configuration fragment is used as input to test-pkg.
To be able to generate one job per test to run, we need the list of tests in the parent pipeline, and the individual .config files (one per
remove a comma after "pipeline"
Quoted 1 line
test) in the child pipeline. We use the newly-introduce --prepare-only
s/newly-introduce/newly-introduced
Quoted 3 lines
mode to test-pkg, and collect all the generated .config files as artefacts; those are inherited in the child pipeline via the "
needs::pipeline" and "needs::job" directives. This is a bit tricky,
remove a comma after tricky
Best regards,
Yegor
Quoted 163 lines
and is best described by the Gitlab-CI documentation [0].
We also list those .config files to generate the actual list of jobs to run in the child pipeline.
Notes:
- if the user provides an empty fragment, this is considered an error:
indeed, without a fragment (and the package name), there is no way to know what to test;
- if that fragment yields an empty list of tests, then there is nothing to test either, so that is also considered an error.[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
Signed-off-by Romain Naour <[email protected]>Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>[yann.morin.1998 at free.fr:
- split the change to test-pkg to its own patch
- generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml, listing the .config files created by test-pkg
- some code-style-candies...
]Signed-off-by Yann E. MORIN <[email protected]>---
v5: Split the change to test-pkg to its own patch
List the .config files
Also store the missing.config files as artefactsv4: reworked by Yann:
tst-pkg only lists, does not generate the yml code
The yml code is generated by support/scripts/generate-gitlab-ci-yml Empty fragment is an error
Fragment that yierlds no test is also an errorv3: Implement Arnout's review: http://lists.busybox.net/pipermail/buildroot/2021-May/310656.html
Enable artifacts download from child-pipelinev2: Rework this patch following Arnout review
useCI_COMMIT_DESCRIPTION
remove .config from artifacts but keep images directory since it can be useful for further issue investigation use the "br-test-pkg" prefix for test-pkg jobs--- .gitlab-ci.yml | 5 +++++ support/misc/gitlab-ci.yml.in | 22 +++++++++++++++++++++ support/scripts/generate-gitlab-ci-yml | 27 +++++++++++++++++++++++++- 3 files changed, 53 insertions(+), 1 deletion(-).gitlab-ci.yml +5
-0@@ -11,8 +11,11 @@ generate-gitlab-ci-yml: stage: generate-gitlab-ci script: ./support/scripts/generate-gitlab-ci-yml support/misc/gitlab-ci.yml.in > generated-gitlab-ci.yml artifacts: + when: always paths: - generated-gitlab-ci.yml + - br-test-pkg/*/.config + - br-test-pkg/*/missing.config buildroot-pipeline: stage: build@@ -21,3 +24,5 @@ buildroot-pipeline: - artifact: generated-gitlab-ci.yml job: generate-gitlab-ci-yml strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_IDsupport/misc/gitlab-ci.yml.in +22
-0@@ -80,3 +80,25 @@ - test-output/*/.config - test-output/*/images/* +.test_pkg: + stage: build + before_script: + - OUTPUT_DIR=${CI_JOB_NAME} + script: + - echo "Configure Buildroot for ${OUTPUT_DIR}" + - make O=${OUTPUT_DIR} syncconfig + - make O=${OUTPUT_DIR} savedefconfig + - echo 'Build buildroot' + - *run_make + needs: + - pipeline: $PARENT_PIPELINE_ID + job: generate-gitlab-ci-yml + artifacts: + when: always + expire_in: 2 weeks + paths: + - build.log + - br-test-pkg/*/.config + - br-test-pkg/*/defconfig + - br-test-pkg/*/build/build-time.log + - br-test-pkg/*/build/packages-file-list*.txtsupport/scripts/generate-gitlab-ci-yml +26
-2@@ -23,7 +23,7 @@ _EOF_ gen_tests() { local -a basics defconfigs runtimes - local do_basics do_defconfigs do_runtime + local do_basics do_defconfigs do_runtime do_testpkg local defconfigs_ext cfg tst basics=( DEVELOPERS flake8 package )@@ -77,9 +77,30 @@ gen_tests() { esac fi + # Retrieve defconfig for test-pkg from the git commit message (if any) + if grep -q -E '^test-pkg config:$' <<<"${CI_COMMIT_DESCRIPTION}"; then + sed -r -n -e '/^test-pkg config:$/{:a;n;p;ba;}' \ + <<<"${CI_COMMIT_DESCRIPTION}" \ + >defconfig.frag + if [ ! -s defconfig.frag ]; then + printf "Empty configuration fragment.\n" >&2; exit 1 + fi + # Use --all since we expect the user having already pre-tested the + # new package with the default subset of toolchains. + ./utils/test-pkg \ + --all --prepare-only \ + --config-snippet defconfig.frag \ + --build-dir br-test-pkg >&2 + do_testpkg=( $(ls -1 br-test-pkg/*/.config 2>/dev/null |xargs -r dirname ) ) + if [ "${#do_testpkg[@]}" -eq 0 ]; then + printf "Configuration fragment enables no test.\n" >&2; exit 1 + fi + fi + # If nothing else, at least do the basics to generate a valid pipeline if [ -z "${do_defconfigs}" \ -a -z "${do_runtime}" \ + -a -z "${do_testpkg}" \ ] then do_basics=true@@ -101,6 +122,10 @@ gen_tests() { if ${do_runtime:-false}; then printf '%s: { extends: .runtime_test_base }\n' "${runtimes[@]}" fi + + if [ -n "${do_testpkg}" ]; then + printf '%s: { extends: .test_pkg }\n' "${do_testpkg[@]}" + fi } main "${@}" --2.25.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot