Skip to contentSkip to conversation

gitlab-ci: Allow forks to select & restrict build jobs

PATCH In review Philippe Mathieu-Daudé 4 participants 29 messages Nov 8, 2020 – Nov 23, 2020

lore
Share & tools
lore
#0 lore raw

v3 cover gitlab-ci: Allow forks to select & restrict build jobs

Hi,

2 months ago GitLab added time limit to their free CI offer [1]. This series provide developers with the possibility to not run all jobs. By default all jobs are started, but we can restrict by selecting a subset of them.

Since v2:
- no more RFC
- project-wide environment variable works
- do not remove the jobs from the pipeline, mark them 'manual' so user can still run them

Since v1:
- switch from "all but skip some" to "all or select some"

A friendly way to use this feature is with git aliases [3]:

 $ git config alias.pushci_system \
    'push -o ci.variable="QEMU_BUILD=system"'
 $ git config alias.pushci_debian \
    'push -o ci.variable="QEMU_BUILD=debian"'

Then you can run the jobs based on Debian images (only) using:

 $ git pushci_debian gitlab_repo my_branch_for_debian

Or run all system-mode emulation jobs only using:

  $ git pushci_system my_gitlab_repo branch_with_system_feature

You can also set the default QEMU_BUILD you want for your fork adding a project-wide environment variable [2]. Your pipelines will be restricted to this set, but you can overwrite it from the git-push command line (using QEMU_BUILD=all).

Comments welcomed!

Regards,

Phil.

[1] https://about.gitlab.com/releases/2020/09/01/ci-minutes-update-free-users/
[2] https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui
[3] https://docs.gitlab.com/ee/user/project/push_options.html#useful-git-aliases

Philippe Mathieu-Daudé (11):
gitlab-ci: Drop generic cache rule
gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
gitlab-ci: Replace YAML anchors by extends (native_build_job)
gitlab-ci: Replace YAML anchors by extends (native_test_job)
gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
gitlab-ci: Rename acceptance_test_job -> integration_test_job
gitlab-ci: Extract common job definition as 'cross_common_job'
gitlab-ci: Extract common job definition as 'native_common_job'
gitlab-ci: Add rules to select cross-jobs to build
gitlab-ci: Add rules to select building/testing native jobs
gitlab-ci: Move artifacts expiry rule to common 'native_build_job'

 .gitlab-ci.d/crossbuilds.yml |  82 +++++++++++++------
 .gitlab-ci.yml               | 153 ++++++++++++++++++++---------------
 2 files changed, 146 insertions(+), 89 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH v3 1/11 gitlab-ci: Drop generic cache rule

This cache rule is meant for Avocado artifacts, but affects all jobs. Moreover the 'acceptance_template' template already include a more detailled rule to cache artifacts.

---
 .gitlab-ci.yml | 6 ------
 1 file changed, 6 deletions(-)
.gitlab-ci.yml +0 -6
index 3b15ae5c302..5763318d375 100644
@@ -7,12 +7,6 @@ stages:
   - build
   - test
 
-# We assume GitLab has it's own caching set up for RPM/APT repositories so we
-# just take care of avocado assets here.
-cache:
-  paths:
-    - $HOME/avocado/data/cache
-
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
Signature · 1 line
2.26.2
#3 lore raw

PATCH v3 2/11 gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.d/crossbuilds.yml | 40 ++++++++++++++++++------------------
 1 file changed, 20 insertions(+), 20 deletions(-)
.gitlab-ci.d/crossbuilds.yml +20 -20
index 03ebfabb3fa..099949aaef3 100644
@@ -1,5 +1,5 @@
 
-.cross_system_build_job_template: &cross_system_build_job_definition
+.cross_system_build_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -13,7 +13,7 @@
           xtensa-softmmu"
     - make -j$(expr $(nproc) + 1) all check-build
 
-.cross_user_build_job_template: &cross_user_build_job_definition
+.cross_user_build_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -24,91 +24,91 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 cross-armel-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-armel-cross
 
 cross-armel-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-armel-cross
 
 cross-armhf-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-armhf-cross
 
 cross-armhf-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-armhf-cross
 
 cross-arm64-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-arm64-cross
 
 cross-arm64-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-arm64-cross
 
 cross-mips-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-mips-cross
 
 cross-mips-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-mips-cross
 
 cross-mipsel-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-mipsel-cross
 
 cross-mipsel-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-mipsel-cross
 
 cross-mips64el-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-mips64el-cross
 
 cross-mips64el-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-mips64el-cross
 
 cross-ppc64el-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-ppc64el-cross
 
 cross-ppc64el-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-ppc64el-cross
 
 cross-s390x-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: debian-s390x-cross
 
 cross-s390x-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
   variables:
     IMAGE: debian-s390x-cross
 
 cross-win32-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: fedora-win32-cross
 
 cross-win64-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
   variables:
     IMAGE: fedora-win64-cross
Signature · 1 line
2.26.2
#17 lore raw

PATCH v3 7/11 gitlab-ci: Extract common job definition as 'cross_common_job'

In reply to Philippe Mathieu-Daudé

Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.

---
 .gitlab-ci.d/crossbuilds.yml | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/crossbuilds.yml +5 -4
index 099949aaef3..701550f028c 100644
@@ -1,7 +1,9 @@
-
-.cross_system_build_job:
+.cross_common_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+
+.cross_system_build_job:
+  extends: .cross_common_job
   timeout: 80m
   script:
     - mkdir build
@@ -14,8 +16,7 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 .cross_user_build_job:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  extends: .cross_common_job
   script:
     - mkdir build
     - cd build
Signature · 1 line
2.26.2
#9 lore raw

PATCH v3 5/11 gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.yml | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
.gitlab-ci.yml +6 -9
index e11f80f6d65..0ef814764a0 100644
@@ -42,7 +42,8 @@ include:
     - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
-.acceptance_template: &acceptance_definition
+.acceptance_test_job:
+  extends: .native_test_job
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -89,14 +90,13 @@ check-system-ubuntu:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-ubuntu:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-system-debian:
   extends: .native_build_job
@@ -120,14 +120,13 @@ check-system-debian:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-debian:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-debian
       artifacts: true
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-system-fedora:
   extends: .native_build_job
@@ -152,14 +151,13 @@ check-system-fedora:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-fedora:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-system-centos:
   extends: .native_build_job
@@ -184,14 +182,13 @@ check-system-centos:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-centos:
-  extends: .native_test_job
+  extends: .acceptance_test_job
   needs:
     - job: build-system-centos
       artifacts: true
   variables:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
 build-disabled:
   extends: .native_build_job
Signature · 1 line
2.26.2
#5 lore raw

PATCH v3 3/11 gitlab-ci: Replace YAML anchors by extends (native_build_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.yml | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
.gitlab-ci.yml +16 -16
index 5763318d375..a96e7dd23e5 100644
@@ -13,7 +13,7 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job_template: &native_build_job_definition
+.native_build_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -68,7 +68,7 @@ include:
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
 build-system-ubuntu:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: ubuntu2004
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -99,7 +99,7 @@ acceptance-system-ubuntu:
   <<: *acceptance_definition
 
 build-system-debian:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-amd64
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -130,7 +130,7 @@ acceptance-system-debian:
   <<: *acceptance_definition
 
 build-system-fedora:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
@@ -162,7 +162,7 @@ acceptance-system-fedora:
   <<: *acceptance_definition
 
 build-system-centos:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
@@ -194,7 +194,7 @@ acceptance-system-centos:
   <<: *acceptance_definition
 
 build-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-attr --disable-avx2 --disable-bochs
@@ -219,7 +219,7 @@ build-disabled:
     MAKE_CHECK_ARGS: check-qtest SPEED=slow
 
 build-tcg-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos8
   script:
@@ -239,7 +239,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-tools --disable-system
@@ -249,7 +249,7 @@ build-user:
 # we skip sparc64-linux-user until it has been fixed somewhat
 # we skip cris-linux-user as it doesn't use the common run loop
 build-user-plugins:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
@@ -257,7 +257,7 @@ build-user-plugins:
   timeout: 1h 30m
 
 build-clang:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --cc=clang --cxx=clang++
@@ -267,7 +267,7 @@ build-clang:
 
 # These targets are on the way out
 build-deprecated:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-docs --disable-tools
@@ -292,7 +292,7 @@ check-deprecated:
   allow_failure: true
 
 build-oss-fuzz:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
   script:
@@ -310,7 +310,7 @@ build-oss-fuzz:
     - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
 build-tci:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: fedora
   script:
@@ -335,7 +335,7 @@ build-tci:
 # These jobs test old gcrypt and nettle from RHEL7
 # which had some API differences.
 build-crypto-old-nettle:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
@@ -356,7 +356,7 @@ check-crypto-old-nettle:
 
 
 build-crypto-old-gcrypt:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
@@ -377,7 +377,7 @@ check-crypto-old-gcrypt:
 
 
 build-crypto-only-gnutls:
-  <<: *native_build_job_definition
+  extends: .native_build_job
   variables:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
Signature · 1 line
2.26.2
#7 lore raw

PATCH v3 4/11 gitlab-ci: Replace YAML anchors by extends (native_test_job)

In reply to Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
 .gitlab-ci.yml | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)
.gitlab-ci.yml +13 -13
index a96e7dd23e5..e11f80f6d65 100644
@@ -34,7 +34,7 @@ include:
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
 
-.native_test_job_template: &native_test_job_definition
+.native_test_job:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -80,7 +80,7 @@ build-system-ubuntu:
       - build
 
 check-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -89,7 +89,7 @@ check-system-ubuntu:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -111,7 +111,7 @@ build-system-debian:
       - build
 
 check-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -120,7 +120,7 @@ check-system-debian:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -143,7 +143,7 @@ build-system-fedora:
       - build
 
 check-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -152,7 +152,7 @@ check-system-fedora:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -175,7 +175,7 @@ build-system-centos:
       - build
 
 check-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-centos
       artifacts: true
@@ -184,7 +184,7 @@ check-system-centos:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-system-centos
       artifacts: true
@@ -282,7 +282,7 @@ build-deprecated:
 # We split the check-tcg step as test failures are expected but we still
 # want to catch the build breaking.
 check-deprecated:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-deprecated
       artifacts: true
@@ -346,7 +346,7 @@ build-crypto-old-nettle:
       - build
 
 check-crypto-old-nettle:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-crypto-old-nettle
       artifacts: true
@@ -367,7 +367,7 @@ build-crypto-old-gcrypt:
       - build
 
 check-crypto-old-gcrypt:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-crypto-old-gcrypt
       artifacts: true
@@ -388,7 +388,7 @@ build-crypto-only-gnutls:
       - build
 
 check-crypto-only-gnutls:
-  <<: *native_test_job_definition
+  extends: .native_test_job
   needs:
     - job: build-crypto-only-gnutls
       artifacts: true
Signature · 1 line
2.26.2
#11 lore raw

PATCH v3 6/11 gitlab-ci: Rename acceptance_test_job -> integration_test_job

In reply to Philippe Mathieu-Daudé
---
 .gitlab-ci.yml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
.gitlab-ci.yml +9 -9
index 0ef814764a0..d4526323169 100644
@@ -42,7 +42,7 @@ include:
     - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
-.acceptance_test_job:
+.integration_test_job:
   extends: .native_test_job
   cache:
     key: "${CI_JOB_NAME}-cache"
@@ -89,8 +89,8 @@ check-system-ubuntu:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu:
-  extends: .acceptance_test_job
+integration-system-ubuntu:
+  extends: .integration_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -119,8 +119,8 @@ check-system-debian:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
-acceptance-system-debian:
-  extends: .acceptance_test_job
+integration-system-debian:
+  extends: .integration_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -150,8 +150,8 @@ check-system-fedora:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora:
-  extends: .acceptance_test_job
+integration-system-fedora:
+  extends: .integration_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -181,8 +181,8 @@ check-system-centos:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
-acceptance-system-centos:
-  extends: .acceptance_test_job
+integration-system-centos:
+  extends: .integration_test_job
   needs:
     - job: build-system-centos
       artifacts: true
Signature · 1 line
2.26.2
#21 lore raw

PATCH v3 8/11 gitlab-ci: Extract common job definition as 'native_common_job'

In reply to Philippe Mathieu-Daudé

Extract the common definitions shared by '.native_build_job' and '.native_test_job' to '.native_common_job'.

---
 .gitlab-ci.yml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
.gitlab-ci.yml +6 -3
index d4526323169..f708573884e 100644
@@ -13,9 +13,12 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job:
-  stage: build
+.native_common_job:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+
+.native_build_job:
+  extends: .native_common_job
+  stage: build
   before_script:
     - JOBS=$(expr $(nproc) + 1)
     - sed -i s,git.qemu.org/git,gitlab.com/qemu-project, .gitmodules
@@ -35,8 +38,8 @@ include:
       fi
 
 .native_test_job:
+  extends: .native_common_job
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
     - cd build
     - find . -type f -exec touch {} +
Signature · 1 line
2.26.2
#25 lore raw

PATCH v3 10/11 gitlab-ci: Add rules to select building/testing native jobs

In reply to Philippe Mathieu-Daudé

Add rules to select various build/test jobs.

The following tags are available to restrict CI jobs:

- user        (user-mode jobs)
- system      (system-mode jobs)
- centos      (jobs based on CentOS distribution image)
- debian      (...           Debian)
- fedora      (...           Fedora)
- ubuntu      (...           Ubuntu)
- crypto      (jobs testing the crypto feature)
- tci         (jobs testing TCI feature)
- fuzz        (fuzzer job)

- integration (integration tests)
- $ARCH (select an architecture: arm/mips/ppc/sparc/...)

Developers can combine tags in the QEMU_BUILD variable when pushing a branch (or tag) to repositories. Examples:

  $ git push -o ci.variable="QEMU_BUILD=user"                    myrepo mybranch
  $ git push -o ci.variable="QEMU_BUILD=user,debian,crypto,fuzz" myrepo mybranch

References:
- https://docs.gitlab.com/ee/ci/yaml/#rulesif
- https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd

---
 .gitlab-ci.yml | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
.gitlab-ci.yml +43 -0
index f708573884e..dd5f9a4c505 100644
@@ -15,6 +15,49 @@ include:
 
 .native_common_job:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  rules:
+    # If the if statement is true, the job is added to the pipeline.
+    # We only filter for push events
+    - if: '$CI_PIPELINE_SOURCE != "push"'
+    # Build all when no variable defined, or set to "all"
+    - if: $QEMU_BUILD == null || $QEMU_BUILD =~ /^all$/
+    # Build specific job name
+    - if: $QEMU_BUILD == $CI_JOB_NAME
+    # Build jobs using particular distribution image
+    - if: $QEMU_BUILD =~ /centos/ && $IMAGE =~ /^centos/
+    - if: $QEMU_BUILD =~ /debian/ && $IMAGE =~ /^debian/
+    - if: $QEMU_BUILD =~ /fedora/ && $IMAGE =~ /^fedora/
+    - if: $QEMU_BUILD =~ /ubuntu/ && $IMAGE =~ /^ubuntu/
+    # Build set of jobs by feature
+    - if: $QEMU_BUILD =~ /system/ && ($TARGETS =~ /softmmu/ || $CONFIGURE_ARGS =~ /disable-user/)
+    - if: $QEMU_BUILD =~ /user/ && ($TARGETS =~ /user/ || $CONFIGURE_ARGS =~ /disable-system/)
+    - if: $QEMU_BUILD =~ /integration/ && ($CI_JOB_NAME =~ /^integration/ || $CI_JOB_NAME =~ /^build-system/ || $MAKE_CHECK_ARGS =~ /check-acceptance/) # integration depends on build-system
+    - if: $QEMU_BUILD =~ /crypto/ && $CI_JOB_NAME =~ /crypto/
+    - if: $QEMU_BUILD =~ /tci/ && $CI_JOB_NAME =~ /tci$/
+    - if: $QEMU_BUILD =~ /fuzz/ && $CI_JOB_NAME =~ /fuzz$/
+    # Build set of jobs by arch
+    - if: $QEMU_BUILD =~ /aarch64/ && ($CI_JOB_NAME =~ /aarch64/ || $TARGETS =~ /aarch64/)
+    - if: $QEMU_BUILD =~ /alpha/ && ($CI_JOB_NAME =~ /alpha/ || $TARGETS =~ /alpha/)
+    - if: $QEMU_BUILD =~ /arm/ && ($CI_JOB_NAME =~ /arm/ || $TARGETS =~ /arm/)
+    - if: $QEMU_BUILD =~ /avr/ && ($CI_JOB_NAME =~ /avr/ || $TARGETS =~ /avr/)
+    - if: $QEMU_BUILD =~ /hppa/ && ($CI_JOB_NAME =~ /hppa/ || $TARGETS =~ /hppa/)
+    - if: $QEMU_BUILD =~ /i386/ && ($CI_JOB_NAME =~ /i386/ || $TARGETS =~ /i386/)
+    - if: $QEMU_BUILD =~ /lm32/ && ($CI_JOB_NAME =~ /lm32/ || $TARGETS =~ /lm32/)
+    - if: $QEMU_BUILD =~ /m68k/ && ($CI_JOB_NAME =~ /m68k/ || $TARGETS =~ /m68k/)
+    - if: $QEMU_BUILD =~ /mips/ && ($CI_JOB_NAME =~ /mips/ || $IMAGE =~ /mips/)
+    - if: $QEMU_BUILD =~ /ppc/ && ($CI_JOB_NAME =~ /ppc/ || $TARGETS =~ /ppc/)
+    - if: $QEMU_BUILD =~ /riscv/ && ($CI_JOB_NAME =~ /riscv/ || $TARGETS =~ /riscv/)
+    - if: $QEMU_BUILD =~ /s390x/ && ($CI_JOB_NAME =~ /s390x/ || $TARGETS =~ /s390x/)
+    - if: $QEMU_BUILD =~ /sparc/ && ($CI_JOB_NAME =~ /sparc/ || $TARGETS =~ /sparc/)
+    - if: $QEMU_BUILD =~ /tricore/ && ($CI_JOB_NAME =~ /tricore/ || $TARGETS =~ /tricore/)
+    - if: $QEMU_BUILD =~ /x86/ && ($CI_JOB_NAME =~ /x86/ || $TARGETS =~ /x86/)
+    - if: $QEMU_BUILD =~ /xtensa/ && ($CI_JOB_NAME =~ /xtensa/ || $TARGETS =~ /xtensa/)
+    # In all other cases, do not not execute the job automatically. Note the
+    # job is not excluded from the pipeline, user can still start it manually.
+    # As the job is optional, we have to mark it 'allow_failure' to not block
+    # the pipeline.
+    - when: manual
+      allow_failure: true
 
 .native_build_job:
   extends: .native_common_job
Signature · 1 line
2.26.2
#23 lore raw

PATCH v3 9/11 gitlab-ci: Add rules to select cross-jobs to build

In reply to Philippe Mathieu-Daudé

Add rules to select some crossbuild jobs.

The following tags are available to restrict the CI jobs:
- all (select all jobs, this is default)
- cross (select all cross-jobs)
- system (select all cross-system jobs)
- user (select all cross-user jobs)
- $ARCH (select an architecture: arm/mips/ppc/sparc/...)

Developers can combine tags in the QEMU_BUILD variable when pushing a branch (or tag) to repositories. Examples:

  $ git push -o ci.variable="QEMU_BUILD=user"        myrepo mybranch
  $ git push -o ci.variable="QEMU_BUILD=user,system" myrepo mybranch

References:
- https://docs.gitlab.com/ee/ci/yaml/#rulesif
- https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd

---
 .gitlab-ci.d/crossbuilds.yml | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
.gitlab-ci.d/crossbuilds.yml +35 -0
index 701550f028c..017bc706689 100644
@@ -1,6 +1,41 @@
 .cross_common_job:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  rules:
+    # If the if statement is true, the job is added to the pipeline.
+    # We only filter for push events
+    - if: '$CI_PIPELINE_SOURCE != "push"'
+    # Build all when no variable defined, or set to "all"
+    - if: $QEMU_BUILD == null || $QEMU_BUILD =~ /^all$/
+    # Build specific job name
+    - if: $QEMU_BUILD =~ /^$CI_JOB_NAME$/
+    # Build set of jobs by feature
+    - if: $QEMU_BUILD =~ /cross/
+    - if: $QEMU_BUILD =~ /system/ && $CI_JOB_NAME =~ /system/
+    - if: $QEMU_BUILD =~ /user/ && $CI_JOB_NAME =~ /user/
+    # Build set of jobs by arch
+    - if: $QEMU_BUILD =~ /aarch64/ && ($CI_JOB_NAME =~ /aarch64/ || $IMAGE =~ /aarch64/)
+    - if: $QEMU_BUILD =~ /alpha/ && ($CI_JOB_NAME =~ /alpha/ || $IMAGE =~ /alpha/)
+    - if: $QEMU_BUILD =~ /arm/ && ($CI_JOB_NAME =~ /arm/ || $IMAGE =~ /arm/)
+    - if: $QEMU_BUILD =~ /avr/ && ($CI_JOB_NAME =~ /avr/ || $IMAGE =~ /avr/)
+    - if: $QEMU_BUILD =~ /hppa/ && ($CI_JOB_NAME =~ /hppa/ || $IMAGE =~ /hppa/)
+    - if: $QEMU_BUILD =~ /i386/ && ($CI_JOB_NAME =~ /i386/ || $IMAGE =~ /i386/)
+    - if: $QEMU_BUILD =~ /lm32/ && ($CI_JOB_NAME =~ /lm32/ || $IMAGE =~ /lm32/)
+    - if: $QEMU_BUILD =~ /m68k/ && ($CI_JOB_NAME =~ /m68k/ || $IMAGE =~ /m68k/)
+    - if: $QEMU_BUILD =~ /mips/ && ($CI_JOB_NAME =~ /mips/ || $IMAGE =~ /mips/)
+    - if: $QEMU_BUILD =~ /ppc/ && ($CI_JOB_NAME =~ /ppc/ || $IMAGE =~ /ppc/)
+    - if: $QEMU_BUILD =~ /riscv/ && ($CI_JOB_NAME =~ /riscv/ || $IMAGE =~ /riscv/)
+    - if: $QEMU_BUILD =~ /s390x/ && ($CI_JOB_NAME =~ /s390x/ || $IMAGE =~ /s390x/)
+    - if: $QEMU_BUILD =~ /sparc/ && ($CI_JOB_NAME =~ /sparc/ || $IMAGE =~ /sparc/)
+    - if: $QEMU_BUILD =~ /tricore/ && ($CI_JOB_NAME =~ /tricore/ || $IMAGE =~ /tricore/)
+    - if: $QEMU_BUILD =~ /x86/ && ($CI_JOB_NAME =~ /x86/ || $IMAGE =~ /x86/)
+    - if: $QEMU_BUILD =~ /xtensa/ && ($CI_JOB_NAME =~ /xtensa/ || $IMAGE =~ /xtensa/)
+    # In all other cases, do not not execute the job automatically. Note the
+    # job is not excluded from the pipeline, user can still start it manually.
+    # As the job is optional, we have to mark it 'allow_failure' to not block
+    # the pipeline.
+    - when: manual
+      allow_failure: true
 
 .cross_system_build_job:
   extends: .cross_common_job
Signature · 1 line
2.26.2
#26 lore raw

PATCH v3 11/11 gitlab-ci: Move artifacts expiry rule to common 'native_build_job'

In reply to Philippe Mathieu-Daudé

Build jobs include the 'native_build_job' template. Move the 'artifacts expiry' rule there. Now all build jobs benefit from it.

---
 .gitlab-ci.yml | 24 ++++--------------------
 1 file changed, 4 insertions(+), 20 deletions(-)
.gitlab-ci.yml +4 -20
index dd5f9a4c505..27a4cbc5171 100644
@@ -79,6 +79,10 @@ include:
       then
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
 
 .native_test_job:
   extends: .native_common_job
@@ -121,10 +125,6 @@ build-system-ubuntu:
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
 
 check-system-ubuntu:
   extends: .native_test_job
@@ -151,10 +151,6 @@ build-system-debian:
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
       riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
     MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
 
 check-system-debian:
   extends: .native_test_job
@@ -182,10 +178,6 @@ build-system-fedora:
     TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu
       xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
     MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
 
 check-system-fedora:
   extends: .native_test_job
@@ -213,10 +205,6 @@ build-system-centos:
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
 
 check-system-centos:
   extends: .native_test_job
@@ -317,10 +305,6 @@ build-deprecated:
     MAKE_CHECK_ARGS: build-tcg
     TARGETS: ppc64abi32-linux-user tilegx-linux-user lm32-softmmu
       unicore32-softmmu
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
 
 # We split the check-tcg step as test failures are expected but we still
 # want to catch the build breaking.
Signature · 1 line
2.26.2
Reviewed-by on v3 1/11
#2 lore raw

Re: [PATCH v3 01/11] gitlab-ci: Drop generic cache rule

In reply to Philippe Mathieu-Daudé
Quoted 14 lines · Philippe Mathieu-Daudé

This cache rule is meant for Avocado artifacts, but affects all jobs. Moreover the 'acceptance_template' template already include a more detailled rule to cache artifacts.

---
 .gitlab-ci.yml | 6 ------
 1 file changed, 6 deletions(-)
.gitlab-ci.yml +0 -0
index 3b15ae5c302..5763318d375 100644

.gitlab-ci.yml stages: · line 14

Quoted 8 lines
.gitlab-ci.yml +0 -5
@@ -7,12 +7,6 @@ stages:
   - build
   - test
 
-# We assume GitLab has it's own caching set up for RPM/APT repositories so we
-# just take care of avocado assets here.
-cache:
-  paths:
-    - $HOME/avocado/data/cache

Right, IIRC the path in $HOME wasn't working as expected anyway, that's why we switched to a more complicated logic in the acceptance_template.

#18 lore raw

Re: [PATCH v3 07/11] gitlab-ci: Extract common job definition as 'cross_common_job'

In reply to Philippe Mathieu-Daudé
Quoted 7 lines · Philippe Mathieu-Daudé

Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.

---
 .gitlab-ci.d/crossbuilds.yml | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

5 insertions, 4 deletions ... is it really that much better?

Thomas

#24 lore raw

Re: [PATCH v3 09/11] gitlab-ci: Add rules to select cross-jobs to build

In reply to Philippe Mathieu-Daudé
Quoted 9 lines · Philippe Mathieu-Daudé

Add rules to select some crossbuild jobs.

The following tags are available to restrict the CI jobs:
- all (select all jobs, this is default)
- cross (select all cross-jobs)
- system (select all cross-system jobs)
- user (select all cross-user jobs)
- $ARCH (select an architecture: arm/mips/ppc/sparc/...)

Quoted 5 lines

Developers can combine tags in the QEMU_BUILD variable when pushing a branch (or tag) to repositories. Examples:

  $ git push -o ci.variable="QEMU_BUILD=user"        myrepo mybranch
  $ git push -o ci.variable="QEMU_BUILD=user,system" myrepo mybranch

That looks interesting, but I think the changes are too big for including them at this point in time, so I'd like to postpone this to the 6.0 cycle (so please respin the series after the 5.2 release).

Also this certainly needs to be documented in a file in the docs/ folder - otherwise you'll be the only one who's using this feature, I guess...

Thanks,
Thomas

#27 lore raw

Re: [PATCH v3 11/11] gitlab-ci: Move artifacts expiry rule to common 'native_build_job'

In reply to Philippe Mathieu-Daudé
Quoted 14 lines · Philippe Mathieu-Daudé

Build jobs include the 'native_build_job' template. Move the 'artifacts expiry' rule there. Now all build jobs benefit from it.

---
 .gitlab-ci.yml | 24 ++++--------------------
 1 file changed, 4 insertions(+), 20 deletions(-)
.gitlab-ci.yml +0 -0
index dd5f9a4c505..27a4cbc5171 100644

.gitlab-ci.yml include: · line 85

Quoted 7 lines
.gitlab-ci.yml +4 -0
@@ -79,6 +79,10 @@ include:
       then
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build

Should we also keep artifacts for pipelines that do not need it to pass them from one job to the next? ... not sure ... considering that gitlab is thinking about cutting down CI minutes etc., we should maybe be more conservative and only keep artifacts where they are really needed?

Thomas

#19 lore raw

Re: [PATCH v3 07/11] gitlab-ci: Extract common job definition as 'cross_common_job'

In reply to Thomas Huth
Quoted 10 lines · Thomas Huth
Quoted 7 lines · Philippe Mathieu-Daudé

Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.

---
 .gitlab-ci.d/crossbuilds.yml | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

5 insertions, 4 deletions ... is it really that much better?

It becomes useful later we we add a lot into cross_common_job, rather than duplicating in cross_[system/user]_build_job.

I understand this commit description lacks that information.

Thanks.

Reviewed-by on v3 2/11
#4 lore raw

Re: [PATCH v3 02/11] gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)

In reply to Philippe Mathieu-Daudé
Quoted 3 lines · Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

Good idea!

Quoted 5 lines
---
  .gitlab-ci.d/crossbuilds.yml | 40 ++++++++++++++++++------------------
  1 file changed, 20 insertions(+), 20 deletions(-)
Quoted 131 lines
.gitlab-ci.d/crossbuilds.yml +20 -20
index 03ebfabb3fa..099949aaef3 100644
@@ -1,5 +1,5 @@
 
-.cross_system_build_job_template: &cross_system_build_job_definition
+.cross_system_build_job:
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    timeout: 80m
@@ -13,7 +13,7 @@
            xtensa-softmmu"
      - make -j$(expr $(nproc) + 1) all check-build
 
-.cross_user_build_job_template: &cross_user_build_job_definition
+.cross_user_build_job:
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
@@ -24,91 +24,91 @@
      - make -j$(expr $(nproc) + 1) all check-build
 
  cross-armel-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-armel-cross
 
  cross-armel-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-armel-cross
 
  cross-armhf-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-armhf-cross
 
  cross-armhf-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-armhf-cross
 
  cross-arm64-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-arm64-cross
 
  cross-arm64-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-arm64-cross
 
  cross-mips-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-mips-cross
 
  cross-mips-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-mips-cross
 
  cross-mipsel-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-mipsel-cross
 
  cross-mipsel-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-mipsel-cross
 
  cross-mips64el-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-mips64el-cross
 
  cross-mips64el-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-mips64el-cross
 
  cross-ppc64el-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-ppc64el-cross
 
  cross-ppc64el-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-ppc64el-cross
 
  cross-s390x-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: debian-s390x-cross
 
  cross-s390x-user:
-  <<: *cross_user_build_job_definition
+  extends: .cross_user_build_job
    variables:
      IMAGE: debian-s390x-cross
 
  cross-win32-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: fedora-win32-cross
 
  cross-win64-system:
-  <<: *cross_system_build_job_definition
+  extends: .cross_system_build_job
    variables:
      IMAGE: fedora-win64-cross
Reviewed-by on v3 3/11
#6 lore raw

Re: [PATCH v3 03/11] gitlab-ci: Replace YAML anchors by extends (native_build_job)

In reply to Philippe Mathieu-Daudé
Quoted 8 lines · Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
  .gitlab-ci.yml | 32 ++++++++++++++++----------------
  1 file changed, 16 insertions(+), 16 deletions(-)

LGTM

Quoted 149 lines
.gitlab-ci.yml +16 -16
index 5763318d375..a96e7dd23e5 100644
@@ -13,7 +13,7 @@ include:
    - local: '/.gitlab-ci.d/containers.yml'
    - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job_template: &native_build_job_definition
+.native_build_job:
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    before_script:
@@ -68,7 +68,7 @@ include:
      - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
  build-system-ubuntu:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: ubuntu2004
      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -99,7 +99,7 @@ acceptance-system-ubuntu:
    <<: *acceptance_definition
 
  build-system-debian:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: debian-amd64
      TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -130,7 +130,7 @@ acceptance-system-debian:
    <<: *acceptance_definition
 
  build-system-fedora:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: fedora
      CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
@@ -162,7 +162,7 @@ acceptance-system-fedora:
    <<: *acceptance_definition
 
  build-system-centos:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: centos8
      CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
@@ -194,7 +194,7 @@ acceptance-system-centos:
    <<: *acceptance_definition
 
  build-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: fedora
      CONFIGURE_ARGS: --disable-attr --disable-avx2 --disable-bochs
@@ -219,7 +219,7 @@ build-disabled:
      MAKE_CHECK_ARGS: check-qtest SPEED=slow
 
  build-tcg-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: centos8
    script:
@@ -239,7 +239,7 @@ build-tcg-disabled:
              260 261 262 263 264 270 272 273 277 279
 
  build-user:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: debian-all-test-cross
      CONFIGURE_ARGS: --disable-tools --disable-system
@@ -249,7 +249,7 @@ build-user:
  # we skip sparc64-linux-user until it has been fixed somewhat
  # we skip cris-linux-user as it doesn't use the common run loop
  build-user-plugins:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: debian-all-test-cross
      CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
@@ -257,7 +257,7 @@ build-user-plugins:
    timeout: 1h 30m
 
  build-clang:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: fedora
      CONFIGURE_ARGS: --cc=clang --cxx=clang++
@@ -267,7 +267,7 @@ build-clang:
 
  # These targets are on the way out
  build-deprecated:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: debian-all-test-cross
      CONFIGURE_ARGS: --disable-docs --disable-tools
@@ -292,7 +292,7 @@ check-deprecated:
    allow_failure: true
 
  build-oss-fuzz:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: fedora
    script:
@@ -310,7 +310,7 @@ build-oss-fuzz:
      - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
  build-tci:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: fedora
    script:
@@ -335,7 +335,7 @@ build-tci:
  # These jobs test old gcrypt and nettle from RHEL7
  # which had some API differences.
  build-crypto-old-nettle:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: centos7
      TARGETS: x86_64-softmmu x86_64-linux-user
@@ -356,7 +356,7 @@ check-crypto-old-nettle:
 
 
  build-crypto-old-gcrypt:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: centos7
      TARGETS: x86_64-softmmu x86_64-linux-user
@@ -377,7 +377,7 @@ check-crypto-old-gcrypt:
 
 
  build-crypto-only-gnutls:
-  <<: *native_build_job_definition
+  extends: .native_build_job
    variables:
      IMAGE: centos7
      TARGETS: x86_64-softmmu x86_64-linux-user
Reviewed-by on v3 4/11
#8 lore raw

Re: [PATCH v3 04/11] gitlab-ci: Replace YAML anchors by extends (native_test_job)

In reply to Philippe Mathieu-Daudé
Quoted 8 lines · Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
  .gitlab-ci.yml | 26 +++++++++++++-------------
  1 file changed, 13 insertions(+), 13 deletions(-)

LGTM

Quoted 122 lines
.gitlab-ci.yml +13 -13
index a96e7dd23e5..e11f80f6d65 100644
@@ -34,7 +34,7 @@ include:
          make -j"$JOBS" $MAKE_CHECK_ARGS ;
        fi
 
-.native_test_job_template: &native_test_job_definition
+.native_test_job:
    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
@@ -80,7 +80,7 @@ build-system-ubuntu:
        - build
 
  check-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-ubuntu
        artifacts: true
@@ -89,7 +89,7 @@ check-system-ubuntu:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-ubuntu
        artifacts: true
@@ -111,7 +111,7 @@ build-system-debian:
        - build
 
  check-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-debian
        artifacts: true
@@ -120,7 +120,7 @@ check-system-debian:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-debian
        artifacts: true
@@ -143,7 +143,7 @@ build-system-fedora:
        - build
 
  check-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-fedora
        artifacts: true
@@ -152,7 +152,7 @@ check-system-fedora:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-fedora
        artifacts: true
@@ -175,7 +175,7 @@ build-system-centos:
        - build
 
  check-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-centos
        artifacts: true
@@ -184,7 +184,7 @@ check-system-centos:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-system-centos
        artifacts: true
@@ -282,7 +282,7 @@ build-deprecated:
  # We split the check-tcg step as test failures are expected but we still
  # want to catch the build breaking.
  check-deprecated:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-deprecated
        artifacts: true
@@ -346,7 +346,7 @@ build-crypto-old-nettle:
        - build
 
  check-crypto-old-nettle:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-crypto-old-nettle
        artifacts: true
@@ -367,7 +367,7 @@ build-crypto-old-gcrypt:
        - build
 
  check-crypto-old-gcrypt:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-crypto-old-gcrypt
        artifacts: true
@@ -388,7 +388,7 @@ build-crypto-only-gnutls:
        - build
 
  check-crypto-only-gnutls:
-  <<: *native_test_job_definition
+  extends: .native_test_job
    needs:
      - job: build-crypto-only-gnutls
        artifacts: true
Reviewed-by on v3 5/11
#10 lore raw

Re: [PATCH v3 05/11] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)

In reply to Philippe Mathieu-Daudé
Quoted 8 lines · Philippe Mathieu-Daudé

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

---
  .gitlab-ci.yml | 15 ++++++---------
  1 file changed, 6 insertions(+), 9 deletions(-)

LGTM

Quoted 79 lines
.gitlab-ci.yml +6 -9
index e11f80f6d65..0ef814764a0 100644
@@ -42,7 +42,8 @@ include:
      - find . -type f -exec touch {} +
      - make $MAKE_CHECK_ARGS
 
-.acceptance_template: &acceptance_definition
+.acceptance_test_job:
+  extends: .native_test_job
    cache:
      key: "${CI_JOB_NAME}-cache"
      paths:
@@ -89,14 +90,13 @@ check-system-ubuntu:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-ubuntu:
-  extends: .native_test_job
+  extends: .acceptance_test_job
    needs:
      - job: build-system-ubuntu
        artifacts: true
    variables:
      IMAGE: ubuntu2004
      MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
  build-system-debian:
    extends: .native_build_job
@@ -120,14 +120,13 @@ check-system-debian:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-debian:
-  extends: .native_test_job
+  extends: .acceptance_test_job
    needs:
      - job: build-system-debian
        artifacts: true
    variables:
      IMAGE: debian-amd64
      MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
  build-system-fedora:
    extends: .native_build_job
@@ -152,14 +151,13 @@ check-system-fedora:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-fedora:
-  extends: .native_test_job
+  extends: .acceptance_test_job
    needs:
      - job: build-system-fedora
        artifacts: true
    variables:
      IMAGE: fedora
      MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
  build-system-centos:
    extends: .native_build_job
@@ -184,14 +182,13 @@ check-system-centos:
      MAKE_CHECK_ARGS: check
 
  acceptance-system-centos:
-  extends: .native_test_job
+  extends: .acceptance_test_job
    needs:
      - job: build-system-centos
        artifacts: true
    variables:
      IMAGE: centos8
      MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
 
  build-disabled:
    extends: .native_build_job
Reviewed-by on v3 6/11
#12 lore raw

Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job

In reply to Philippe Mathieu-Daudé

Once Cleber said "acceptance" wasn't  a good name for those tests. Indeed "integration" is widely used, so okay for this renaming.

Quoted 4 lines · Philippe Mathieu-Daudé
---
  .gitlab-ci.yml | 18 +++++++++---------
  1 file changed, 9 insertions(+), 9 deletions(-)
Quoted 58 lines
.gitlab-ci.yml +9 -9
index 0ef814764a0..d4526323169 100644
@@ -42,7 +42,7 @@ include:
      - find . -type f -exec touch {} +
      - make $MAKE_CHECK_ARGS
 
-.acceptance_test_job:
+.integration_test_job:
    extends: .native_test_job
    cache:
      key: "${CI_JOB_NAME}-cache"
@@ -89,8 +89,8 @@ check-system-ubuntu:
      IMAGE: ubuntu2004
      MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu:
-  extends: .acceptance_test_job
+integration-system-ubuntu:
+  extends: .integration_test_job
    needs:
      - job: build-system-ubuntu
        artifacts: true
@@ -119,8 +119,8 @@ check-system-debian:
      IMAGE: debian-amd64
      MAKE_CHECK_ARGS: check
 
-acceptance-system-debian:
-  extends: .acceptance_test_job
+integration-system-debian:
+  extends: .integration_test_job
    needs:
      - job: build-system-debian
        artifacts: true
@@ -150,8 +150,8 @@ check-system-fedora:
      IMAGE: fedora
      MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora:
-  extends: .acceptance_test_job
+integration-system-fedora:
+  extends: .integration_test_job
    needs:
      - job: build-system-fedora
        artifacts: true
@@ -181,8 +181,8 @@ check-system-centos:
      IMAGE: centos8
      MAKE_CHECK_ARGS: check
 
-acceptance-system-centos:
-  extends: .acceptance_test_job
+integration-system-centos:
+  extends: .integration_test_job
    needs:
      - job: build-system-centos
        artifacts: true
Reviewed-by on v3 7/11
#20 lore raw

Re: [PATCH v3 07/11] gitlab-ci: Extract common job definition as 'cross_common_job'

In reply to Philippe Mathieu-Daudé
Quoted 7 lines · Philippe Mathieu-Daudé

Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.

---
  .gitlab-ci.d/crossbuilds.yml | 9 +++++----
  1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 27 lines
.gitlab-ci.d/crossbuilds.yml +5 -4
index 099949aaef3..701550f028c 100644
@@ -1,7 +1,9 @@
-
-.cross_system_build_job:
+.cross_common_job:
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+
+.cross_system_build_job:
+  extends: .cross_common_job
    timeout: 80m
    script:
      - mkdir build
@@ -14,8 +16,7 @@
      - make -j$(expr $(nproc) + 1) all check-build
 
  .cross_user_build_job:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  extends: .cross_common_job
    script:
      - mkdir build
      - cd build
Reviewed-by on v3 8/11
#22 lore raw

Re: [PATCH v3 08/11] gitlab-ci: Extract common job definition as 'native_common_job'

In reply to Philippe Mathieu-Daudé
Quoted 13 lines · Philippe Mathieu-Daudé

Extract the common definitions shared by '.native_build_job' and '.native_test_job' to '.native_common_job'.

---
  .gitlab-ci.yml | 9 ++++++---
  1 file changed, 6 insertions(+), 3 deletions(-)
.gitlab-ci.yml +0 -0
index d4526323169..f708573884e 100644

.gitlab-ci.yml include: · line 17

Quoted 7 lines
.gitlab-ci.yml +1 -2
@@ -13,9 +13,12 @@ include:
    - local: '/.gitlab-ci.d/containers.yml'
    - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job:
-  stage: build
+.native_common_job:
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest

Do you envision that "native_common_job" with more common properties?

Asking because it creates another indirection to just replace two "image"s.

Anyway,

Quoted 17 lines
.gitlab-ci.yml +5 -1
+
+.native_build_job:
+  extends: .native_common_job
+  stage: build
    before_script:
      - JOBS=$(expr $(nproc) + 1)
      - sed -i s,git.qemu.org/git,gitlab.com/qemu-project, .gitmodules
@@ -35,8 +38,8 @@ include:
        fi
 
  .native_test_job:
+  extends: .native_common_job
    stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
      - cd build
      - find . -type f -exec touch {} +
#13 lore raw

Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job

In reply to Wainer dos Santos Moschetta
Quoted 2 lines · Wainer dos Santos Moschetta

Once Cleber said "acceptance" wasn't  a good name for those tests. Indeed "integration" is widely used, so okay for this renaming.

Should we maybe also rename the folder and "make check-acceptance" into "make check-integration" ?

Thomas

#14 lore raw

Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job

Quoted 5 lines · Thomas Huth
Quoted 2 lines · Wainer dos Santos Moschetta

Once Cleber said "acceptance" wasn't  a good name for those tests. Indeed "integration" is widely used, so okay for this renaming.

Should we maybe also rename the folder and "make check-acceptance" into "make check-integration" ?

I'm okay with that. But I also would like to hear from Cleber.

- Wainer

Quoted 4 lines

Thomas

#15 lore raw

Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job

Quoted 10 lines · Wainer dos Santos Moschetta
Quoted 6 lines · Thomas Huth
Quoted 3 lines · Wainer dos Santos Moschetta

Once Cleber said "acceptance" wasn't  a good name for those tests. Indeed
"integration" is widely used, so okay for this renaming.

Should we maybe also rename the folder and "make check-acceptance" into "make check-integration" ?

I'm okay with that. But I also would like to hear from Cleber.

This is the plan indeed, but out of the scope of this (gitlab-ci specific) series.

'check-acceptance' might then depend on 'check-integration'.

Quoted 8 lines

- Wainer

Quoted 4 lines

  Thomas

#28 lore raw

Re: [PATCH v3 11/11] gitlab-ci: Move artifacts expiry rule to common 'native_build_job'

In reply to Thomas Huth
Quoted 19 lines · Thomas Huth
Quoted 18 lines · Philippe Mathieu-Daudé

Build jobs include the 'native_build_job' template. Move the 'artifacts expiry' rule there. Now all build jobs benefit from it.

---
 .gitlab-ci.yml | 24 ++++--------------------
 1 file changed, 4 insertions(+), 20 deletions(-)
.gitlab-ci.yml +1 -0
index dd5f9a4c505..27a4cbc5171 100644
@@ -79,6 +79,10 @@ include:
       then
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+  artifacts:
Quoted 8 lines

.gitlab-ci.yml include: · line 85

Quoted 3 lines
.gitlab-ci.yml +3 -0
+    expire_in: 2 days
+    paths:
+      - build

Should we also keep artifacts for pipelines that do not need it to pass them from one job to the next? ... not sure ... considering that gitlab is thinking about cutting down CI minutes etc., we should maybe be more conservative and only keep artifacts where they are really needed?

As nothing is released from these jobs (except some roms/ and documentation?) I'd not keep anything at this point. Later we can be stricter and select the minimum we need.

(I don't think nobody ever downloaded a job artifact to test / debug it).

Quoted 3 lines

Thomas

Reviewed-by on v3 6/11
#16 lore raw

Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job

In reply to Philippe Mathieu-Daudé
Quoted 64 lines · Philippe Mathieu-Daudé
---
 .gitlab-ci.yml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
.gitlab-ci.yml +9 -10
index 0ef814764a0..d4526323169 100644
@@ -42,7 +42,7 @@ include:
     - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
-.acceptance_test_job:
+.integration_test_job:
   extends: .native_test_job
   cache:
     key: "${CI_JOB_NAME}-cache"
@@ -89,8 +89,8 @@ check-system-ubuntu:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu:
-  extends: .acceptance_test_job
+integration-system-ubuntu:
+  extends: .integration_test_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -119,8 +119,8 @@ check-system-debian:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
-acceptance-system-debian:
-  extends: .acceptance_test_job
+integration-system-debian:
+  extends: .integration_test_job
   needs:
     - job: build-system-debian
       artifacts: true
@@ -150,8 +150,8 @@ check-system-fedora:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora:
-  extends: .acceptance_test_job
+integration-system-fedora:
+  extends: .integration_test_job
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -181,8 +181,8 @@ check-system-centos:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
-acceptance-system-centos:
-  extends: .acceptance_test_job
+integration-system-centos:
+  extends: .integration_test_job
   needs:
     - job: build-system-centos
       artifacts: true
--
Quoted 3 lines

2.26.2

Makes sense, thanks!