Skip to contentSkip to conversation

gitlab-ci: Allow forks to skip some build jobs

RFC PATCH Superseded Philippe Mathieu-Daudé 1 participant 11 messages Nov 4, 2020 – Nov 5, 2020

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover gitlab-ci: Allow forks to skip some build jobs

Hi,

2 months ago GitLab added time limit to their free CI offer [1]. This series provide developers with the possibility to not run all jobs. By default all jobs are started, but we can disable a subset of them.

I think this should be the other way around (enable features one wants to test, with a default to the "all" keyword), but I didn't want to disrupt the current workflow.

I'm not sure supporting both ("SKIP_BUILD=3Dall INCLUDE_BUILD=3Dsystem") options is easy to implement, or to maintain (unlikely).

In the next iteration I'll add the possibility to use a project- wide environment variable [2] to set the default set of enabled / disabled features a fork is interested in. User will still be able to overload using git-push on the command line. A friendly way to use this feature is with git aliases [3]:

 $ git config alias.pushci_system \
    'push -o ci.variable=3D"SKIP_BUILD=3Duser"'
 $ git config alias.pushci_debian \
    'push -o ci.variable=3D"SKIP_BUILD=3Dcentos,fedora,ubuntu"'

Then you can run the jobs based on Debian images (only) using:

 $ git pushci_debian gitlab_repo my_branch_for_debian

Or run all system-mode emulation jobs only using:

  $ git pushci_system my_gitlab_repo branch_with_system_feature

Comments welcomed!

Regards,

Phil.

[1] https://about.gitlab.com/releases/2020/09/01/ci-minutes-update-free-users/
[2] https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-varia=
ble-in-the-ui
[3] https://docs.gitlab.com/ee/user/project/push_options.html#useful-git-alia=
ses

Philippe Mathieu-Daud=C3=A9 (9):
gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
gitlab-ci: Replace YAML anchors by extends (native_build_job)
gitlab-ci: Replace YAML anchors by extends (native_test_job)
gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
gitlab-ci: Rename acceptance_test_job -> integration_test_job
gitlab-ci: Extract common job definition as 'cross_common_job'
gitlab-ci: Extract common job definition as 'native_common_job'
gitlab-ci: Add rules to skip building cross-jobs
gitlab-ci: Add rules to skip building/testing native jobs

 .gitlab-ci.d/crossbuilds.yml |  57 +++++++++++--------
 .gitlab-ci.yml               | 104 +++++++++++++++++++++--------------
 2 files changed, 98 insertions(+), 63 deletions(-)

--=20
2.26.2

#1 lore raw

RFC PATCH 1/9 gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.d/crossbuilds.yml | 40 ++++++++++++++++++------------------
 1 file changed, 20 insertions(+), 20 deletions(-)
.gitlab-ci.d/crossbuilds.yml +20 -20
index 03ebfabb3fa..099949aaef3 100644
@@ -1,5 +1,5 @@
 
-.cross_system_build_job_template: &cross_system_build_job_definition
+.cross_system_build_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -13,7 +13,7 @@
           xtensa-softmmu"
     - make -j$(expr $(nproc) + 1) all check-build
 
-.cross_user_build_job_template: &cross_user_build_job_definition
+.cross_user_build_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -24,91 +24,91 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 cross-armel-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-armel-cross
 
 cross-armel-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-armel-cross
 
 cross-armhf-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-armhf-cross
 
 cross-armhf-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-armhf-cross
 
 cross-arm64-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-arm64-cross
 
 cross-arm64-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-arm64-cross
 
 cross-mips-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-mips-cross
 
 cross-mips-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-mips-cross
 
 cross-mipsel-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-mipsel-cross
 
 cross-mipsel-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-mipsel-cross
 
 cross-mips64el-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-mips64el-cross
 
 cross-mips64el-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-mips64el-cross
 
 cross-ppc64el-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-ppc64el-cross
 
 cross-ppc64el-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-ppc64el-cross
 
 cross-s390x-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-s390x-cross
 
 cross-s390x-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-s390x-cross
 
 cross-win32-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: fedora-win32-cross
 
 cross-win64-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: fedora-win64-cross
Signature · 1 line
2.26.2
#2 lore raw

RFC PATCH 2/9 gitlab-ci: Replace YAML anchors by extends (native_build_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.yml | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
.gitlab-ci.yml +16 -16
index 3b15ae5c302..ff3a8bd58cf 100644
@@ -19,7 +19,7 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job_template: &native_build_job_definition
+.native_build_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -74,7 +74,7 @@ include:
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
 build-system-ubuntu:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: ubuntu2004
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -105,7 +105,7 @@ acceptance-system-ubuntu:
   <<: *acceptance_definition
 
 build-system-debian:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-amd64
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -136,7 +136,7 @@ acceptance-system-debian:
   <<: *acceptance_definition
 
 build-system-fedora:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
@@ -168,7 +168,7 @@ acceptance-system-fedora:
   <<: *acceptance_definition
 
 build-system-centos:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
@@ -200,7 +200,7 @@ acceptance-system-centos:
   <<: *acceptance_definition
 
 build-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-attr --disable-avx2 --disable-bochs
@@ -225,7 +225,7 @@ build-disabled:
     MAKE_CHECK_ARGS: check-qtest SPEED=slow
 
 build-tcg-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos8
   script:
@@ -245,7 +245,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-tools --disable-system
@@ -255,7 +255,7 @@ build-user:
 # we skip sparc64-linux-user until it has been fixed somewhat
 # we skip cris-linux-user as it doesn't use the common run loop
 build-user-plugins:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
@@ -263,7 +263,7 @@ build-user-plugins:
   timeout: 1h 30m
 
 build-clang:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --cc=clang --cxx=clang++
@@ -273,7 +273,7 @@ build-clang:
 
 # These targets are on the way out
 build-deprecated:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-docs --disable-tools
@@ -298,7 +298,7 @@ check-deprecated:
   allow_failure: true
 
 build-oss-fuzz:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
   script:
@@ -316,7 +316,7 @@ build-oss-fuzz:
     - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
 build-tci:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
   script:
@@ -341,7 +341,7 @@ build-tci:
 # These jobs test old gcrypt and nettle from RHEL7
 # which had some API differences.
 build-crypto-old-nettle:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
@@ -362,7 +362,7 @@ check-crypto-old-nettle:
 
 
 build-crypto-old-gcrypt:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
@@ -383,7 +383,7 @@ check-crypto-old-gcrypt:
 
 
 build-crypto-only-gnutls:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
Signature · 1 line
2.26.2
#3 lore raw

RFC PATCH 3/9 gitlab-ci: Replace YAML anchors by extends (native_test_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.yml | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)
.gitlab-ci.yml +13 -13
index ff3a8bd58cf..83beefa2c85 100644
@@ -40,7 +40,7 @@ include:
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
 
-.native_test_job_template: &native_test_job_definition
+.native_test_job:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -86,7 +86,7 @@ build-system-ubuntu:
       - build
 
 check-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -95,7 +95,7 @@ check-system-ubuntu:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -117,7 +117,7 @@ build-system-debian:
       - build
 
 check-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -126,7 +126,7 @@ check-system-debian:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -149,7 +149,7 @@ build-system-fedora:
       - build
 
 check-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -158,7 +158,7 @@ check-system-fedora:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -181,7 +181,7 @@ build-system-centos:
       - build
 
 check-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-centos
       artifacts: true
@@ -190,7 +190,7 @@ check-system-centos:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-centos
       artifacts: true
@@ -288,7 +288,7 @@ build-deprecated:
 # We split the check-tcg step as test failures are expected but we still
 # want to catch the build breaking.
 check-deprecated:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-deprecated
       artifacts: true
@@ -352,7 +352,7 @@ build-crypto-old-nettle:
       - build
 
 check-crypto-old-nettle:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-crypto-old-nettle
       artifacts: true
@@ -373,7 +373,7 @@ build-crypto-old-gcrypt:
       - build
 
 check-crypto-old-gcrypt:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-crypto-old-gcrypt
       artifacts: true
@@ -394,7 +394,7 @@ build-crypto-only-gnutls:
       - build
 
 check-crypto-only-gnutls:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-crypto-only-gnutls
       artifacts: true
Signature · 1 line
2.26.2
#4 lore raw

RFC PATCH 4/9 gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.yml | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
.gitlab-ci.yml +6 -9
index 83beefa2c85..5c64e477c9d 100644
@@ -48,7 +48,8 @@ include:
     - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
-.acceptance_template: &acceptance_definition
+.acceptance_test_job:
+  extends: .native_test_job
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -95,14 +96,13 @@ check-system-ubuntu:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-ubuntu:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-system-debian:
   extends: .native_build_job
@@ -126,14 +126,13 @@ check-system-debian:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-debian:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-debian
       artifacts: true
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-system-fedora:
   extends: .native_build_job
@@ -158,14 +157,13 @@ check-system-fedora:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-fedora:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-system-centos:
   extends: .native_build_job
@@ -190,14 +188,13 @@ check-system-centos:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-centos:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-centos
       artifacts: true
   variables:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-disabled:
   extends: .native_build_job
Signature · 1 line
2.26.2
#5 lore raw

RFC PATCH 5/9 gitlab-ci: Rename acceptance_test_job -> integration_test_job

In reply to Philippe Mathieu-Daudé
---
 .gitlab-ci.yml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
.gitlab-ci.yml +9 -9
index 5c64e477c9d..0b77a90dd73 100644
@@ -48,7 +48,7 @@ include:
     - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
-.acceptance_test_job:
+.integration_test_job:
   extends: .native_test_job
   cache:
     key: "${CI_JOB_NAME}-cache"
@@ -95,8 +95,8 @@ check-system-ubuntu:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu:
-  extends: .acceptance_test_job
+integration-system-ubuntu:
+  extends: .integration_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -125,8 +125,8 @@ check-system-debian:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
-acceptance-system-debian:
-  extends: .acceptance_test_job
+integration-system-debian:
+  extends: .integration_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -156,8 +156,8 @@ check-system-fedora:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora:
-  extends: .acceptance_test_job
+integration-system-fedora:
+  extends: .integration_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -187,8 +187,8 @@ check-system-centos:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
-acceptance-system-centos:
-  extends: .acceptance_test_job
+integration-system-centos:
+  extends: .integration_test_job
   needs:
     - job: build-system-centos
       artifacts: true
Signature · 1 line
2.26.2
#6 lore raw

RFC PATCH 6/9 gitlab-ci: Extract common job definition as 'cross_common_job'

In reply to Philippe Mathieu-Daudé

Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.

---
 .gitlab-ci.d/crossbuilds.yml | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/crossbuilds.yml +5 -4
index 099949aaef3..701550f028c 100644
@@ -1,7 +1,9 @@
-
-.cross_system_build_job:
+.cross_common_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+
+.cross_system_build_job:
+  extends: .cross_common_job
   timeout: 80m
   script:
     - mkdir build
@@ -14,8 +16,7 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 .cross_user_build_job:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  extends: .cross_common_job
   script:
     - mkdir build
     - cd build
Signature · 1 line
2.26.2
#7 lore raw

RFC PATCH 7/9 gitlab-ci: Extract common job definition as 'native_common_job'

In reply to Philippe Mathieu-Daudé

Extract the common definitions shared by '.native_build_job' and '.native_test_job' to '.native_common_job'.

---
 .gitlab-ci.yml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
.gitlab-ci.yml +6 -3
index 0b77a90dd73..961070d2cbe 100644
@@ -19,9 +19,12 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job:
-  stage: build
+.native_common_job:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+
+.native_build_job:
+  extends: .native_common_job
+  stage: build
   before_script:
     - JOBS=$(expr $(nproc) + 1)
     - sed -i s,git.qemu.org/git,gitlab.com/qemu-project, .gitmodules
@@ -41,8 +44,8 @@ include:
       fi
 
 .native_test_job:
+  extends: .native_common_job
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
     - cd build
     - find . -type f -exec touch {} +
Signature · 1 line
2.26.2
#8 lore raw

RFC PATCH 8/9 gitlab-ci: Add rules to skip building cross-jobs

In reply to Philippe Mathieu-Daudé

Add rules to skip some crossbuild jobs.

The following tags are available to skip CI jobs:
- cross (skip all cross-jobs)
- system (skip all cross-system jobs)
- user (skip all cross-user jobs)

Developers can combine tags in the SKIP_BUILD variable when pushing a branch (or tag) to repositories. Examples:

  $ git push -o ci.variable="SKIP_BUILD=user"        myrepo mybranch
  $ git push -o ci.variable="SKIP_BUILD=user,system" myrepo mybranch

References:
- https://docs.gitlab.com/ee/ci/yaml/#rulesif
- https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd

---
 .gitlab-ci.d/crossbuilds.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)
.gitlab-ci.d/crossbuilds.yml +10 -0
index 701550f028c..08f27649eb2 100644
@@ -1,6 +1,16 @@
 .cross_common_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  rules:
+    # If the if statement is true, the job is excluded from a pipeline.
+    - if: $SKIP_BUILD =~ /cross/
+      when: never
+    - if: $CI_JOB_NAME =~ /system/ && $SKIP_BUILD =~ /system/
+      when: never
+    - if: $CI_JOB_NAME =~ /user/ && $SKIP_BUILD =~ /user/
+      when: never
+    # In all other cases, the job is added to the pipeline.
+    - when: on_success
 
 .cross_system_build_job:
   extends: .cross_common_job
Signature · 1 line
2.26.2
#9 lore raw

RFC PATCH 9/9 gitlab-ci: Add rules to skip building/testing native jobs

In reply to Philippe Mathieu-Daudé

Add rules to skip various build/test jobs.

The following tags are available to skip CI jobs:

- user        (user-mode jobs)
- system      (system-mode jobs)
- centos      (jobs based on CentOS distribution image)
- debian      (...           Debian)
- fedora      (...           Fedora)
- ubuntu      (...           Ubuntu)
- crypto      (jobs testing the crypto feature)
- tci         (jobs testing TCI feature)
- fuzz        (fuzzer job)

- integration (integration tests)

Developers can combine tags in the SKIP_BUILD variable when pushing a branch (or tag) to repositories. Examples:

  $ git push -o ci.variable="SKIP_BUILD=user"                    myrepo mybranch
  $ git push -o ci.variable="SKIP_BUILD=user,debian,crypto,fuzz" myrepo mybranch

References:
- https://docs.gitlab.com/ee/ci/yaml/#rulesif
- https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd

---
 .gitlab-ci.yml | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
.gitlab-ci.yml +24 -0
index 961070d2cbe..432daccf590 100644
@@ -21,6 +21,30 @@ include:
 
 .native_common_job:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  rules:
+    # If the if statement is true, the job is excluded from a pipeline.
+    - if: ($TARGETS =~ /softmmu/ || $CONFIGURE_ARGS =~ /disable-user/) && $SKIP_BUILD =~ /system/
+      when: never
+    - if: ($TARGETS =~ /user/ || $CONFIGURE_ARGS =~ /disable-system/) && $SKIP_BUILD =~ /user/
+      when: never
+    - if: $IMAGE =~ /^centos/ && $SKIP_BUILD =~ /centos/
+      when: never
+    - if: $IMAGE =~ /^debian/ && $SKIP_BUILD =~ /debian/
+      when: never
+    - if: $IMAGE =~ /^fedora/ && $SKIP_BUILD =~ /fedora/
+      when: never
+    - if: $IMAGE =~ /^ubuntu/ && $SKIP_BUILD =~ /ubuntu/
+      when: never
+    - if: $CI_JOB_NAME =~ /crypto/ && $SKIP_BUILD =~ /crypto/
+      when: never
+    - if: $CI_JOB_NAME =~ /tci/ && $SKIP_BUILD =~ /tci/
+      when: never
+    - if: $CI_JOB_NAME =~ /fuzz/ && $SKIP_BUILD =~ /fuzz/
+      when: never
+    - if: $CI_JOB_NAME =~ /^acceptance/ && $SKIP_BUILD =~ /integration/
+      when: never
+    # In all other cases, the job is added to the pipeline.
+    - when: on_success
 
 .native_build_job:
   extends: .native_common_job
Signature · 1 line
2.26.2
#10 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 10 lines · Philippe Mathieu-Daudé

Hi,

2 months ago GitLab added time limit to their free CI offer [1]. This series provide developers with the possibility to not run all jobs. By default all jobs are started, but we can disable a subset of them.

I think this should be the other way around (enable features one wants to test, with a default to the "all" keyword), but I didn't want to disrupt the current workflow.

I was a bit too focused on getting the YAML right to not realize it is simpler to do:

QEMU_JOBS=all # default, overwritable
QEMU_JOBS=system,fuzz # only build those jobs

                       # can be set in fork UI config (see [2])
                       # can be set with git-push option (see below)
Quoted 52 lines

I'm not sure supporting both ("SKIP_BUILD=3Dall INCLUDE_BUILD=3Dsystem") options is easy to implement, or to maintain (unlikely).

In the next iteration I'll add the possibility to use a project- wide environment variable [2] to set the default set of enabled / disabled features a fork is interested in. User will still be able to overload using git-push on the command line. A friendly way to use this feature is with git aliases [3]:

 $ git config alias.pushci_system \
    'push -o ci.variable=3D"SKIP_BUILD=3Duser"'
 $ git config alias.pushci_debian \
    'push -o ci.variable=3D"SKIP_BUILD=3Dcentos,fedora,ubuntu"'

Then you can run the jobs based on Debian images (only) using:

 $ git pushci_debian gitlab_repo my_branch_for_debian

Or run all system-mode emulation jobs only using:

  $ git pushci_system my_gitlab_repo branch_with_system_feature

Comments welcomed!

Regards,

Phil.

[1] https://about.gitlab.com/releases/2020/09/01/ci-minutes-update-free-users/
[2] https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-varia=
ble-in-the-ui
[3] https://docs.gitlab.com/ee/user/project/push_options.html#useful-git-alia=
ses

Philippe Mathieu-Daud=C3=A9 (9):
gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
gitlab-ci: Replace YAML anchors by extends (native_build_job)
gitlab-ci: Replace YAML anchors by extends (native_test_job)
gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
gitlab-ci: Rename acceptance_test_job -> integration_test_job
gitlab-ci: Extract common job definition as 'cross_common_job'
gitlab-ci: Extract common job definition as 'native_common_job'
gitlab-ci: Add rules to skip building cross-jobs
gitlab-ci: Add rules to skip building/testing native jobs

 .gitlab-ci.d/crossbuilds.yml |  57 +++++++++++--------
 .gitlab-ci.yml               | 104 +++++++++++++++++++++--------------
 2 files changed, 98 insertions(+), 63 deletions(-)

--=20
2.26.2