gitlab-ci: fix and handle the weekly scheduled pipeline (branch yem/gitlab-ci-schedule)
cover gitlab-ci: fix and handle the weekly scheduled pipeline (branch yem/gitlab-ci-schedule)
Hello All!
Our weekly scheduled pipeline no longer builds the defconfigs (it only checks them for validity), and also no longer builds the runtime tests either.
This seems to have been caused by the fact that a scheduled pipeline is (no longer) recognised as a triggerred pipeline by gitlab-CI, and so does not match our current heurostocs.
This series fixes the priority of pipeline creation conditions, and adds the scheduled condition, both to the defconfigs and the runtime tests.
note however that the jobs limit if 500 on gitlab.com, and the current weekly schedule would created ~588 jobs. As a consequence, the pipeline *will* fail.
But until we find a better solution, let's at least fix the conditions under which it should fail! :-]
Here is a result of the new scheduled pipeline:
https://gitlab.com/ymorin/buildroot/-/pipelines/192690611 (parent) https://gitlab.com/ymorin/buildroot/-/pipelines/192690931 (child)
Regards,
Yann E. MORIN.
The following changes since commit bb216e3d8ade12442db794cbe5780e53bc082be8
package/sentry-native: fix double variable assignment (2020-09-21 21:39:36 +0200)
are available in the git repository at:
https://git.busybox.net/~ymorin/git/buildroot
for you to fetch changes up to d01c55362b599b6b41c5e914f46644ac4088e243
support/scripts: handle scheduled pipelines (2020-09-21 22:50:45 +0200)
----------------------------------------------------------------
Yann E. MORIN (2):
support/scripts: prioritize conditions for pipeline creation support/scripts: handle scheduled pipelines
support/scripts/generate-gitlab-ci-yml | 64 +++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 29 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/2 support/scripts: prioritize conditions for pipeline creation
When multiple conditions match simultaneously, even though that should not happen in practice, we want the more "important" one to win over the less "important" ones. For example, a tag is more important than a branch name or a trigger.
Currently, the latest condition to match takes precendence over any previous one, while we want the exact opposite.
Fix that with proper fallbacks in else-blocks.
--- support/scripts/generate-gitlab-ci-yml | 58 +++++++++++++------------- 1 file changed, 29 insertions(+), 29 deletions(-)
support/scripts/generate-gitlab-ci-yml +29 -29
@@ -49,26 +49,26 @@ gen_defconfigs() { if [ -n "${CI_COMMIT_TAG}" ]; then # For tags, create a pipeline. template=base - fi - if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then + elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then # For pipeline created by using a trigger token. template=base + else + case "${CI_COMMIT_REF_NAME}" in + # For master, next, and maintenance branches, only check the defconfigs + (master|next|????.??.x) + template=check + ext=_check + ;; + # For the branch or tag name named *-defconfigs, create a pipeline. + (*-defconfigs) + template=base + ;; + (*-*_defconfig) + defconfigs=( "${CI_COMMIT_REF_NAME##*-}" ) + template=base + ;; + esac fi - case "${CI_COMMIT_REF_NAME}" in - # For master, next, and maintenance branches, only check the defconfigs - (master|next|????.??.x) - template=check - ext=_check - ;; - # For the branch or tag name named *-defconfigs, create a pipeline. - (*-defconfigs) - template=base - ;; - (*-*_defconfig) - defconfigs=( "${CI_COMMIT_REF_NAME##*-}" ) - template=base - ;; - esac if [ -n "${template}" ]; then for cfg in "${defconfigs[@]}"; do
@@ -91,21 +91,21 @@ gen_tests() { if [ -n "${CI_COMMIT_TAG}" ]; then # For tags, create a pipeline. run_tests=true - fi - if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then + elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then # For pipeline created by using a trigger token. run_tests=true + else + case "${CI_COMMIT_REF_NAME}" in + # For the branch or tag name named *-runtime-tests, create a pipeline. + (*-runtime-tests) + run_tests=true + ;; + (*-tests.*) + tests=( "${CI_COMMIT_REF_NAME##*-}" ) + run_tests=true + ;; + esac fi - case "${CI_COMMIT_REF_NAME}" in - # For the branch or tag name named *-runtime-tests, create a pipeline. - (*-runtime-tests) - run_tests=true - ;; - (*-tests.*) - tests=( "${CI_COMMIT_REF_NAME##*-}" ) - run_tests=true - ;; - esac if ${run_tests}; then printf '%s: { extends: .runtime_test_base }\n' "${tests[@]}"
Signature · 1 line
2.20.1
Thomas Petazzoni
AppliedPATCH 1/2 support/scripts: prioritize conditions for pipeline creation
Quoted 8 lines · On Mon, 21 Sep 2020 23:08:41 +0200 "Yann E. MORIN" <[email protected]>…
When multiple conditions match simultaneously, even though that should not happen in practice, we want the more "important" one to win over the less "important" ones. For example, a tag is more important than a branch name or a trigger.
Currently, the latest condition to match takes precendence over any previous one, while we want the exact opposite.
Quoted 6 lines
Fix that with proper fallbacks in else-blocks.
Signed-off-by Yann E. MORIN <[email protected]>--- support/scripts/generate-gitlab-ci-yml | 58 +++++++++++++------------- 1 file changed, 29 insertions(+), 29 deletions(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/2 support/scripts: handle scheduled pipelines
In reply to Yann E. MORINWe currently only handle triggerred pipelines, i.e. pipelines that have been created via the PAI with a token.
However, the weekly trigger is not a trigger, it is a schedule.
Unbeknownst to us, it seems the behaviour of scheduled pipeline has changed, and they are no longer treated as a triggerred pipeline, and the corresponding variable is not set; instead, the scheduled property is reflected in the CI_PIPELINE_SOURCE variable, that is aptly set to 'schedule'.
Use that to recognise our weekly scheduled pipeline.
--- support/scripts/generate-gitlab-ci-yml | 6 ++++++ 1 file changed, 6 insertions(+)
support/scripts/generate-gitlab-ci-yml +6 -0
@@ -52,6 +52,9 @@ gen_defconfigs() { elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then # For pipeline created by using a trigger token. template=base + elif [ "${CI_PIPELINE_SOURCE}" = "schedule" ]; then + # For scheduled pipelines + template=base else case "${CI_COMMIT_REF_NAME}" in # For master, next, and maintenance branches, only check the defconfigs
@@ -94,6 +97,9 @@ gen_tests() { elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then # For pipeline created by using a trigger token. run_tests=true + elif [ "${CI_PIPELINE_SOURCE}" = "schedule" ]; then + # For scheduled pipelines + run_tests=true else case "${CI_COMMIT_REF_NAME}" in # For the branch or tag name named *-runtime-tests, create a pipeline.
Signature · 1 line
2.20.1