Skip to contentSkip to conversation

gitlab-ci: fix and handle the weekly scheduled pipeline (branch yem/gitlab-ci-schedule)

PATCH New Yann E. MORIN 2 participants 4 messages Sep 21, 2020 – Sep 22, 2020

lore
Share & tools
lore
#0 lore raw

cover gitlab-ci: fix and handle the weekly scheduled pipeline (branch yem/gitlab-ci-schedule)

Hello All!

Our weekly scheduled pipeline no longer builds the defconfigs (it only checks them for validity), and also no longer builds the runtime tests either.

This seems to have been caused by the fact that a scheduled pipeline is (no longer) recognised as a triggerred pipeline by gitlab-CI, and so does not match our current heurostocs.

This series fixes the priority of pipeline creation conditions, and adds the scheduled condition, both to the defconfigs and the runtime tests.

note however that the jobs limit if 500 on gitlab.com, and the current weekly schedule would created ~588 jobs. As a consequence, the pipeline *will* fail.

But until we find a better solution, let's at least fix the conditions under which it should fail! :-]

Here is a result of the new scheduled pipeline:

    https://gitlab.com/ymorin/buildroot/-/pipelines/192690611 (parent)
    https://gitlab.com/ymorin/buildroot/-/pipelines/192690931 (child)

Regards,
Yann E. MORIN.

The following changes since commit bb216e3d8ade12442db794cbe5780e53bc082be8

package/sentry-native: fix double variable assignment (2020-09-21 21:39:36 +0200)

are available in the git repository at:

https://git.busybox.net/~ymorin/git/buildroot

for you to fetch changes up to d01c55362b599b6b41c5e914f46644ac4088e243

support/scripts: handle scheduled pipelines (2020-09-21 22:50:45 +0200)

----------------------------------------------------------------
Yann E. MORIN (2):
support/scripts: prioritize conditions for pipeline creation support/scripts: handle scheduled pipelines

 support/scripts/generate-gitlab-ci-yml | 64 +++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 29 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/2 support/scripts: prioritize conditions for pipeline creation

When multiple conditions match simultaneously, even though that should not happen in practice, we want the more "important" one to win over the less "important" ones. For example, a tag is more important than a branch name or a trigger.

Currently, the latest condition to match takes precendence over any previous one, while we want the exact opposite.

Fix that with proper fallbacks in else-blocks.

---
 support/scripts/generate-gitlab-ci-yml | 58 +++++++++++++-------------
 1 file changed, 29 insertions(+), 29 deletions(-)
support/scripts/generate-gitlab-ci-yml +29 -29
index e42943953c..faa52e85b2 100755
@@ -49,26 +49,26 @@ gen_defconfigs() {
     if [ -n "${CI_COMMIT_TAG}" ]; then
         # For tags, create a pipeline.
         template=base
-    fi
-    if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
+    elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         template=base
+    else
+        case "${CI_COMMIT_REF_NAME}" in
+            # For master, next, and maintenance branches, only check the defconfigs
+            (master|next|????.??.x)
+                template=check
+                ext=_check
+            ;;
+            # For the branch or tag name named *-defconfigs, create a pipeline.
+            (*-defconfigs)
+                template=base
+            ;;
+            (*-*_defconfig)
+                defconfigs=( "${CI_COMMIT_REF_NAME##*-}" )
+                template=base
+            ;;
+        esac
     fi
-    case "${CI_COMMIT_REF_NAME}" in
-        # For master, next, and maintenance branches, only check the defconfigs
-        (master|next|????.??.x)
-            template=check
-            ext=_check
-        ;;
-        # For the branch or tag name named *-defconfigs, create a pipeline.
-        (*-defconfigs)
-            template=base
-        ;;
-        (*-*_defconfig)
-            defconfigs=( "${CI_COMMIT_REF_NAME##*-}" )
-            template=base
-        ;;
-    esac
 
     if [ -n "${template}" ]; then
         for cfg in "${defconfigs[@]}"; do
@@ -91,21 +91,21 @@ gen_tests() {
     if [ -n "${CI_COMMIT_TAG}" ]; then
         # For tags, create a pipeline.
         run_tests=true
-    fi
-    if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
+    elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         run_tests=true
+    else
+        case "${CI_COMMIT_REF_NAME}" in
+            # For the branch or tag name named *-runtime-tests, create a pipeline.
+            (*-runtime-tests)
+                run_tests=true
+            ;;
+            (*-tests.*)
+                tests=( "${CI_COMMIT_REF_NAME##*-}" )
+                run_tests=true
+            ;;
+        esac
     fi
-    case "${CI_COMMIT_REF_NAME}" in
-        # For the branch or tag name named *-runtime-tests, create a pipeline.
-        (*-runtime-tests)
-            run_tests=true
-        ;;
-        (*-tests.*)
-            tests=( "${CI_COMMIT_REF_NAME##*-}" )
-            run_tests=true
-        ;;
-    esac
 
     if ${run_tests}; then
         printf '%s: { extends: .runtime_test_base }\n' "${tests[@]}"
Signature · 1 line
2.20.1
#2 lore raw

PATCH 1/2 support/scripts: prioritize conditions for pipeline creation

Quoted 8 lines · On Mon, 21 Sep 2020 23:08:41 +0200 "Yann E. MORIN" <[email protected]>…

When multiple conditions match simultaneously, even though that should not happen in practice, we want the more "important" one to win over the less "important" ones. For example, a tag is more important than a branch name or a trigger.

Currently, the latest condition to match takes precendence over any previous one, while we want the exact opposite.

Quoted 6 lines

Fix that with proper fallbacks in else-blocks.

---
 support/scripts/generate-gitlab-ci-yml | 58 +++++++++++++-------------
 1 file changed, 29 insertions(+), 29 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 2/2 support/scripts: handle scheduled pipelines

In reply to Yann E. MORIN

We currently only handle triggerred pipelines, i.e. pipelines that have been created via the PAI with a token.

However, the weekly trigger is not a trigger, it is a schedule.

Unbeknownst to us, it seems the behaviour of scheduled pipeline has changed, and they are no longer treated as a triggerred pipeline, and the corresponding variable is not set; instead, the scheduled property is reflected in the CI_PIPELINE_SOURCE variable, that is aptly set to 'schedule'.

Use that to recognise our weekly scheduled pipeline.

---
 support/scripts/generate-gitlab-ci-yml | 6 ++++++
 1 file changed, 6 insertions(+)
support/scripts/generate-gitlab-ci-yml +6 -0
index faa52e85b2..01d850d09c 100755
@@ -52,6 +52,9 @@ gen_defconfigs() {
     elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         template=base
+    elif [ "${CI_PIPELINE_SOURCE}" = "schedule" ]; then
+        # For scheduled pipelines
+        template=base
     else
         case "${CI_COMMIT_REF_NAME}" in
             # For master, next, and maintenance branches, only check the defconfigs
@@ -94,6 +97,9 @@ gen_tests() {
     elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         run_tests=true
+    elif [ "${CI_PIPELINE_SOURCE}" = "schedule" ]; then
+        # For scheduled pipelines
+        run_tests=true
     else
         case "${CI_COMMIT_REF_NAME}" in
             # For the branch or tag name named *-runtime-tests, create a pipeline.
Signature · 1 line
2.20.1