gitlab-ci: Allow forks to select & restrict build jobs
v3 cover gitlab-ci: Allow forks to select & restrict build jobs
Hi,
2 months ago GitLab added time limit to their free CI offer [1]. This series provide developers with the possibility to not run all jobs. By default all jobs are started, but we can restrict by selecting a subset of them.
Since v2:
- no more RFC
- project-wide environment variable works
- do not remove the jobs from the pipeline, mark them 'manual' so user can still run them
Since v1:
- switch from "all but skip some" to "all or select some"
A friendly way to use this feature is with git aliases [3]:
$ git config alias.pushci_system \
'push -o ci.variable="QEMU_BUILD=system"'
$ git config alias.pushci_debian \
'push -o ci.variable="QEMU_BUILD=debian"'Then you can run the jobs based on Debian images (only) using:
$ git pushci_debian gitlab_repo my_branch_for_debian
Or run all system-mode emulation jobs only using:
$ git pushci_system my_gitlab_repo branch_with_system_feature
You can also set the default QEMU_BUILD you want for your fork adding a project-wide environment variable [2]. Your pipelines will be restricted to this set, but you can overwrite it from the git-push command line (using QEMU_BUILD=all).
Comments welcomed!
Regards,
Phil.
[1] https://about.gitlab.com/releases/2020/09/01/ci-minutes-update-free-users/
[2] https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui
[3] https://docs.gitlab.com/ee/user/project/push_options.html#useful-git-aliases
Philippe Mathieu-Daudé (11):
gitlab-ci: Drop generic cache rule
gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
gitlab-ci: Replace YAML anchors by extends (native_build_job)
gitlab-ci: Replace YAML anchors by extends (native_test_job)
gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
gitlab-ci: Rename acceptance_test_job -> integration_test_job
gitlab-ci: Extract common job definition as 'cross_common_job'
gitlab-ci: Extract common job definition as 'native_common_job'
gitlab-ci: Add rules to select cross-jobs to build
gitlab-ci: Add rules to select building/testing native jobs
gitlab-ci: Move artifacts expiry rule to common 'native_build_job'
.gitlab-ci.d/crossbuilds.yml | 82 +++++++++++++------ .gitlab-ci.yml | 153 ++++++++++++++++++++--------------- 2 files changed, 146 insertions(+), 89 deletions(-)
Signature · 1 line
2.26.2
PATCH v3 1/11 gitlab-ci: Drop generic cache rule
This cache rule is meant for Avocado artifacts, but affects all jobs. Moreover the 'acceptance_template' template already include a more detailled rule to cache artifacts.
--- .gitlab-ci.yml | 6 ------ 1 file changed, 6 deletions(-)
.gitlab-ci.yml +0 -6
@@ -7,12 +7,6 @@ stages: - build - test -# We assume GitLab has it's own caching set up for RPM/APT repositories so we -# just take care of avocado assets here. -cache: - paths: - - $HOME/avocado/data/cache - include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml'
Signature · 1 line
2.26.2
PATCH v3 2/11 gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
In reply to Philippe Mathieu-Daudé'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends
--- .gitlab-ci.d/crossbuilds.yml | 40 ++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 20 deletions(-)
.gitlab-ci.d/crossbuilds.yml +20 -20
@@ -1,5 +1,5 @@ -.cross_system_build_job_template: &cross_system_build_job_definition +.cross_system_build_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m
@@ -13,7 +13,7 @@ xtensa-softmmu" - make -j$(expr $(nproc) + 1) all check-build -.cross_user_build_job_template: &cross_user_build_job_definition +.cross_user_build_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
@@ -24,91 +24,91 @@ - make -j$(expr $(nproc) + 1) all check-build cross-armel-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-armel-cross cross-armel-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-armel-cross cross-armhf-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-armhf-cross cross-armhf-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-armhf-cross cross-arm64-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-arm64-cross cross-arm64-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-arm64-cross cross-mips-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-mips-cross cross-mips-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-mips-cross cross-mipsel-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-mipsel-cross cross-mipsel-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-mipsel-cross cross-mips64el-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-mips64el-cross cross-mips64el-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-mips64el-cross cross-ppc64el-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-ppc64el-cross cross-ppc64el-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-ppc64el-cross cross-s390x-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-s390x-cross cross-s390x-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-s390x-cross cross-win32-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: fedora-win32-cross cross-win64-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: fedora-win64-cross
Signature · 1 line
2.26.2
PATCH v3 7/11 gitlab-ci: Extract common job definition as 'cross_common_job'
In reply to Philippe Mathieu-DaudéExtract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.
--- .gitlab-ci.d/crossbuilds.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
.gitlab-ci.d/crossbuilds.yml +5 -4
@@ -1,7 +1,9 @@ - -.cross_system_build_job: +.cross_common_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + +.cross_system_build_job: + extends: .cross_common_job timeout: 80m script: - mkdir build
@@ -14,8 +16,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_user_build_job: - stage: build - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + extends: .cross_common_job script: - mkdir build - cd build
Signature · 1 line
2.26.2
PATCH v3 5/11 gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
In reply to Philippe Mathieu-Daudé'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends
--- .gitlab-ci.yml | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
.gitlab-ci.yml +6 -9
@@ -42,7 +42,8 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.acceptance_template: &acceptance_definition +.acceptance_test_job: + extends: .native_test_job cache: key: "${CI_JOB_NAME}-cache" paths:
@@ -89,14 +90,13 @@ check-system-ubuntu: MAKE_CHECK_ARGS: check acceptance-system-ubuntu: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-ubuntu artifacts: true variables: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-system-debian: extends: .native_build_job
@@ -120,14 +120,13 @@ check-system-debian: MAKE_CHECK_ARGS: check acceptance-system-debian: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-debian artifacts: true variables: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-system-fedora: extends: .native_build_job
@@ -152,14 +151,13 @@ check-system-fedora: MAKE_CHECK_ARGS: check acceptance-system-fedora: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-fedora artifacts: true variables: IMAGE: fedora MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-system-centos: extends: .native_build_job
@@ -184,14 +182,13 @@ check-system-centos: MAKE_CHECK_ARGS: check acceptance-system-centos: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-centos artifacts: true variables: IMAGE: centos8 MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-disabled: extends: .native_build_job
Signature · 1 line
2.26.2
PATCH v3 3/11 gitlab-ci: Replace YAML anchors by extends (native_build_job)
In reply to Philippe Mathieu-Daudé'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends
--- .gitlab-ci.yml | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
.gitlab-ci.yml +16 -16
@@ -13,7 +13,7 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml' -.native_build_job_template: &native_build_job_definition +.native_build_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script:
@@ -68,7 +68,7 @@ include: - du -chs ${CI_PROJECT_DIR}/avocado-cache build-system-ubuntu: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: ubuntu2004 TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -99,7 +99,7 @@ acceptance-system-ubuntu: <<: *acceptance_definition build-system-debian: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-amd64 TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -130,7 +130,7 @@ acceptance-system-debian: <<: *acceptance_definition build-system-fedora: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
@@ -162,7 +162,7 @@ acceptance-system-fedora: <<: *acceptance_definition build-system-centos: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos8 CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
@@ -194,7 +194,7 @@ acceptance-system-centos: <<: *acceptance_definition build-disabled: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora CONFIGURE_ARGS: --disable-attr --disable-avx2 --disable-bochs
@@ -219,7 +219,7 @@ build-disabled: MAKE_CHECK_ARGS: check-qtest SPEED=slow build-tcg-disabled: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos8 script:
@@ -239,7 +239,7 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-tools --disable-system
@@ -249,7 +249,7 @@ build-user: # we skip sparc64-linux-user until it has been fixed somewhat # we skip cris-linux-user as it doesn't use the common run loop build-user-plugins: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
@@ -257,7 +257,7 @@ build-user-plugins: timeout: 1h 30m build-clang: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora CONFIGURE_ARGS: --cc=clang --cxx=clang++
@@ -267,7 +267,7 @@ build-clang: # These targets are on the way out build-deprecated: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-docs --disable-tools
@@ -292,7 +292,7 @@ check-deprecated: allow_failure: true build-oss-fuzz: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora script:
@@ -310,7 +310,7 @@ build-oss-fuzz: - cd build-oss-fuzz && make check-qtest-i386 check-unit build-tci: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora script:
@@ -335,7 +335,7 @@ build-tci: # These jobs test old gcrypt and nettle from RHEL7 # which had some API differences. build-crypto-old-nettle: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos7 TARGETS: x86_64-softmmu x86_64-linux-user
@@ -356,7 +356,7 @@ check-crypto-old-nettle: build-crypto-old-gcrypt: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos7 TARGETS: x86_64-softmmu x86_64-linux-user
@@ -377,7 +377,7 @@ check-crypto-old-gcrypt: build-crypto-only-gnutls: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos7 TARGETS: x86_64-softmmu x86_64-linux-user
Signature · 1 line
2.26.2
PATCH v3 4/11 gitlab-ci: Replace YAML anchors by extends (native_test_job)
In reply to Philippe Mathieu-Daudé'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends
--- .gitlab-ci.yml | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
.gitlab-ci.yml +13 -13
@@ -34,7 +34,7 @@ include: make -j"$JOBS" $MAKE_CHECK_ARGS ; fi -.native_test_job_template: &native_test_job_definition +.native_test_job: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
@@ -80,7 +80,7 @@ build-system-ubuntu: - build check-system-ubuntu: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-ubuntu artifacts: true
@@ -89,7 +89,7 @@ check-system-ubuntu: MAKE_CHECK_ARGS: check acceptance-system-ubuntu: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-ubuntu artifacts: true
@@ -111,7 +111,7 @@ build-system-debian: - build check-system-debian: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-debian artifacts: true
@@ -120,7 +120,7 @@ check-system-debian: MAKE_CHECK_ARGS: check acceptance-system-debian: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-debian artifacts: true
@@ -143,7 +143,7 @@ build-system-fedora: - build check-system-fedora: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-fedora artifacts: true
@@ -152,7 +152,7 @@ check-system-fedora: MAKE_CHECK_ARGS: check acceptance-system-fedora: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-fedora artifacts: true
@@ -175,7 +175,7 @@ build-system-centos: - build check-system-centos: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-centos artifacts: true
@@ -184,7 +184,7 @@ check-system-centos: MAKE_CHECK_ARGS: check acceptance-system-centos: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-centos artifacts: true
@@ -282,7 +282,7 @@ build-deprecated: # We split the check-tcg step as test failures are expected but we still # want to catch the build breaking. check-deprecated: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-deprecated artifacts: true
@@ -346,7 +346,7 @@ build-crypto-old-nettle: - build check-crypto-old-nettle: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-crypto-old-nettle artifacts: true
@@ -367,7 +367,7 @@ build-crypto-old-gcrypt: - build check-crypto-old-gcrypt: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-crypto-old-gcrypt artifacts: true
@@ -388,7 +388,7 @@ build-crypto-only-gnutls: - build check-crypto-only-gnutls: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-crypto-only-gnutls artifacts: true
Signature · 1 line
2.26.2
PATCH v3 6/11 gitlab-ci: Rename acceptance_test_job -> integration_test_job
In reply to Philippe Mathieu-Daudé--- .gitlab-ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
.gitlab-ci.yml +9 -9
@@ -42,7 +42,7 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.acceptance_test_job: +.integration_test_job: extends: .native_test_job cache: key: "${CI_JOB_NAME}-cache"
@@ -89,8 +89,8 @@ check-system-ubuntu: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check -acceptance-system-ubuntu: - extends: .acceptance_test_job +integration-system-ubuntu: + extends: .integration_test_job needs: - job: build-system-ubuntu artifacts: true
@@ -119,8 +119,8 @@ check-system-debian: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check -acceptance-system-debian: - extends: .acceptance_test_job +integration-system-debian: + extends: .integration_test_job needs: - job: build-system-debian artifacts: true
@@ -150,8 +150,8 @@ check-system-fedora: IMAGE: fedora MAKE_CHECK_ARGS: check -acceptance-system-fedora: - extends: .acceptance_test_job +integration-system-fedora: + extends: .integration_test_job needs: - job: build-system-fedora artifacts: true
@@ -181,8 +181,8 @@ check-system-centos: IMAGE: centos8 MAKE_CHECK_ARGS: check -acceptance-system-centos: - extends: .acceptance_test_job +integration-system-centos: + extends: .integration_test_job needs: - job: build-system-centos artifacts: true
Signature · 1 line
2.26.2
PATCH v3 8/11 gitlab-ci: Extract common job definition as 'native_common_job'
In reply to Philippe Mathieu-DaudéExtract the common definitions shared by '.native_build_job' and '.native_test_job' to '.native_common_job'.
--- .gitlab-ci.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
.gitlab-ci.yml +6 -3
@@ -13,9 +13,12 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml' -.native_build_job: - stage: build +.native_common_job: image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + +.native_build_job: + extends: .native_common_job + stage: build before_script: - JOBS=$(expr $(nproc) + 1) - sed -i s,git.qemu.org/git,gitlab.com/qemu-project, .gitmodules
@@ -35,8 +38,8 @@ include: fi .native_test_job: + extends: .native_common_job stage: test - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: - cd build - find . -type f -exec touch {} +
Signature · 1 line
2.26.2
PATCH v3 10/11 gitlab-ci: Add rules to select building/testing native jobs
In reply to Philippe Mathieu-DaudéAdd rules to select various build/test jobs.
The following tags are available to restrict CI jobs:
- user (user-mode jobs) - system (system-mode jobs) - centos (jobs based on CentOS distribution image) - debian (... Debian) - fedora (... Fedora) - ubuntu (... Ubuntu) - crypto (jobs testing the crypto feature) - tci (jobs testing TCI feature) - fuzz (fuzzer job)
- integration (integration tests)
- $ARCH (select an architecture: arm/mips/ppc/sparc/...)
Developers can combine tags in the QEMU_BUILD variable when pushing a branch (or tag) to repositories. Examples:
$ git push -o ci.variable="QEMU_BUILD=user" myrepo mybranch $ git push -o ci.variable="QEMU_BUILD=user,debian,crypto,fuzz" myrepo mybranch
References:
- https://docs.gitlab.com/ee/ci/yaml/#rulesif
- https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
--- .gitlab-ci.yml | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
.gitlab-ci.yml +43 -0
@@ -15,6 +15,49 @@ include: .native_common_job: image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + rules: + # If the if statement is true, the job is added to the pipeline. + # We only filter for push events + - if: '$CI_PIPELINE_SOURCE != "push"' + # Build all when no variable defined, or set to "all" + - if: $QEMU_BUILD == null || $QEMU_BUILD =~ /^all$/ + # Build specific job name + - if: $QEMU_BUILD == $CI_JOB_NAME + # Build jobs using particular distribution image + - if: $QEMU_BUILD =~ /centos/ && $IMAGE =~ /^centos/ + - if: $QEMU_BUILD =~ /debian/ && $IMAGE =~ /^debian/ + - if: $QEMU_BUILD =~ /fedora/ && $IMAGE =~ /^fedora/ + - if: $QEMU_BUILD =~ /ubuntu/ && $IMAGE =~ /^ubuntu/ + # Build set of jobs by feature + - if: $QEMU_BUILD =~ /system/ && ($TARGETS =~ /softmmu/ || $CONFIGURE_ARGS =~ /disable-user/) + - if: $QEMU_BUILD =~ /user/ && ($TARGETS =~ /user/ || $CONFIGURE_ARGS =~ /disable-system/) + - if: $QEMU_BUILD =~ /integration/ && ($CI_JOB_NAME =~ /^integration/ || $CI_JOB_NAME =~ /^build-system/ || $MAKE_CHECK_ARGS =~ /check-acceptance/) # integration depends on build-system + - if: $QEMU_BUILD =~ /crypto/ && $CI_JOB_NAME =~ /crypto/ + - if: $QEMU_BUILD =~ /tci/ && $CI_JOB_NAME =~ /tci$/ + - if: $QEMU_BUILD =~ /fuzz/ && $CI_JOB_NAME =~ /fuzz$/ + # Build set of jobs by arch + - if: $QEMU_BUILD =~ /aarch64/ && ($CI_JOB_NAME =~ /aarch64/ || $TARGETS =~ /aarch64/) + - if: $QEMU_BUILD =~ /alpha/ && ($CI_JOB_NAME =~ /alpha/ || $TARGETS =~ /alpha/) + - if: $QEMU_BUILD =~ /arm/ && ($CI_JOB_NAME =~ /arm/ || $TARGETS =~ /arm/) + - if: $QEMU_BUILD =~ /avr/ && ($CI_JOB_NAME =~ /avr/ || $TARGETS =~ /avr/) + - if: $QEMU_BUILD =~ /hppa/ && ($CI_JOB_NAME =~ /hppa/ || $TARGETS =~ /hppa/) + - if: $QEMU_BUILD =~ /i386/ && ($CI_JOB_NAME =~ /i386/ || $TARGETS =~ /i386/) + - if: $QEMU_BUILD =~ /lm32/ && ($CI_JOB_NAME =~ /lm32/ || $TARGETS =~ /lm32/) + - if: $QEMU_BUILD =~ /m68k/ && ($CI_JOB_NAME =~ /m68k/ || $TARGETS =~ /m68k/) + - if: $QEMU_BUILD =~ /mips/ && ($CI_JOB_NAME =~ /mips/ || $IMAGE =~ /mips/) + - if: $QEMU_BUILD =~ /ppc/ && ($CI_JOB_NAME =~ /ppc/ || $TARGETS =~ /ppc/) + - if: $QEMU_BUILD =~ /riscv/ && ($CI_JOB_NAME =~ /riscv/ || $TARGETS =~ /riscv/) + - if: $QEMU_BUILD =~ /s390x/ && ($CI_JOB_NAME =~ /s390x/ || $TARGETS =~ /s390x/) + - if: $QEMU_BUILD =~ /sparc/ && ($CI_JOB_NAME =~ /sparc/ || $TARGETS =~ /sparc/) + - if: $QEMU_BUILD =~ /tricore/ && ($CI_JOB_NAME =~ /tricore/ || $TARGETS =~ /tricore/) + - if: $QEMU_BUILD =~ /x86/ && ($CI_JOB_NAME =~ /x86/ || $TARGETS =~ /x86/) + - if: $QEMU_BUILD =~ /xtensa/ && ($CI_JOB_NAME =~ /xtensa/ || $TARGETS =~ /xtensa/) + # In all other cases, do not not execute the job automatically. Note the + # job is not excluded from the pipeline, user can still start it manually. + # As the job is optional, we have to mark it 'allow_failure' to not block + # the pipeline. + - when: manual + allow_failure: true .native_build_job: extends: .native_common_job
Signature · 1 line
2.26.2
PATCH v3 9/11 gitlab-ci: Add rules to select cross-jobs to build
In reply to Philippe Mathieu-DaudéAdd rules to select some crossbuild jobs.
The following tags are available to restrict the CI jobs:
- all (select all jobs, this is default)
- cross (select all cross-jobs)
- system (select all cross-system jobs)
- user (select all cross-user jobs)
- $ARCH (select an architecture: arm/mips/ppc/sparc/...)
Developers can combine tags in the QEMU_BUILD variable when pushing a branch (or tag) to repositories. Examples:
$ git push -o ci.variable="QEMU_BUILD=user" myrepo mybranch $ git push -o ci.variable="QEMU_BUILD=user,system" myrepo mybranch
References:
- https://docs.gitlab.com/ee/ci/yaml/#rulesif
- https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
--- .gitlab-ci.d/crossbuilds.yml | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
.gitlab-ci.d/crossbuilds.yml +35 -0
@@ -1,6 +1,41 @@ .cross_common_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + rules: + # If the if statement is true, the job is added to the pipeline. + # We only filter for push events + - if: '$CI_PIPELINE_SOURCE != "push"' + # Build all when no variable defined, or set to "all" + - if: $QEMU_BUILD == null || $QEMU_BUILD =~ /^all$/ + # Build specific job name + - if: $QEMU_BUILD =~ /^$CI_JOB_NAME$/ + # Build set of jobs by feature + - if: $QEMU_BUILD =~ /cross/ + - if: $QEMU_BUILD =~ /system/ && $CI_JOB_NAME =~ /system/ + - if: $QEMU_BUILD =~ /user/ && $CI_JOB_NAME =~ /user/ + # Build set of jobs by arch + - if: $QEMU_BUILD =~ /aarch64/ && ($CI_JOB_NAME =~ /aarch64/ || $IMAGE =~ /aarch64/) + - if: $QEMU_BUILD =~ /alpha/ && ($CI_JOB_NAME =~ /alpha/ || $IMAGE =~ /alpha/) + - if: $QEMU_BUILD =~ /arm/ && ($CI_JOB_NAME =~ /arm/ || $IMAGE =~ /arm/) + - if: $QEMU_BUILD =~ /avr/ && ($CI_JOB_NAME =~ /avr/ || $IMAGE =~ /avr/) + - if: $QEMU_BUILD =~ /hppa/ && ($CI_JOB_NAME =~ /hppa/ || $IMAGE =~ /hppa/) + - if: $QEMU_BUILD =~ /i386/ && ($CI_JOB_NAME =~ /i386/ || $IMAGE =~ /i386/) + - if: $QEMU_BUILD =~ /lm32/ && ($CI_JOB_NAME =~ /lm32/ || $IMAGE =~ /lm32/) + - if: $QEMU_BUILD =~ /m68k/ && ($CI_JOB_NAME =~ /m68k/ || $IMAGE =~ /m68k/) + - if: $QEMU_BUILD =~ /mips/ && ($CI_JOB_NAME =~ /mips/ || $IMAGE =~ /mips/) + - if: $QEMU_BUILD =~ /ppc/ && ($CI_JOB_NAME =~ /ppc/ || $IMAGE =~ /ppc/) + - if: $QEMU_BUILD =~ /riscv/ && ($CI_JOB_NAME =~ /riscv/ || $IMAGE =~ /riscv/) + - if: $QEMU_BUILD =~ /s390x/ && ($CI_JOB_NAME =~ /s390x/ || $IMAGE =~ /s390x/) + - if: $QEMU_BUILD =~ /sparc/ && ($CI_JOB_NAME =~ /sparc/ || $IMAGE =~ /sparc/) + - if: $QEMU_BUILD =~ /tricore/ && ($CI_JOB_NAME =~ /tricore/ || $IMAGE =~ /tricore/) + - if: $QEMU_BUILD =~ /x86/ && ($CI_JOB_NAME =~ /x86/ || $IMAGE =~ /x86/) + - if: $QEMU_BUILD =~ /xtensa/ && ($CI_JOB_NAME =~ /xtensa/ || $IMAGE =~ /xtensa/) + # In all other cases, do not not execute the job automatically. Note the + # job is not excluded from the pipeline, user can still start it manually. + # As the job is optional, we have to mark it 'allow_failure' to not block + # the pipeline. + - when: manual + allow_failure: true .cross_system_build_job: extends: .cross_common_job
Signature · 1 line
2.26.2
PATCH v3 11/11 gitlab-ci: Move artifacts expiry rule to common 'native_build_job'
In reply to Philippe Mathieu-DaudéBuild jobs include the 'native_build_job' template. Move the 'artifacts expiry' rule there. Now all build jobs benefit from it.
--- .gitlab-ci.yml | 24 ++++-------------------- 1 file changed, 4 insertions(+), 20 deletions(-)
.gitlab-ci.yml +4 -20
@@ -79,6 +79,10 @@ include: then make -j"$JOBS" $MAKE_CHECK_ARGS ; fi + artifacts: + expire_in: 2 days + paths: + - build .native_test_job: extends: .native_common_job
@@ -121,10 +125,6 @@ build-system-ubuntu: TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu moxie-softmmu microblazeel-softmmu mips64el-softmmu MAKE_CHECK_ARGS: check-build - artifacts: - expire_in: 2 days - paths: - - build check-system-ubuntu: extends: .native_test_job
@@ -151,10 +151,6 @@ build-system-debian: TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu MAKE_CHECK_ARGS: check-build - artifacts: - expire_in: 2 days - paths: - - build check-system-debian: extends: .native_test_job
@@ -182,10 +178,6 @@ build-system-fedora: TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu MAKE_CHECK_ARGS: check-build - artifacts: - expire_in: 2 days - paths: - - build check-system-fedora: extends: .native_test_job
@@ -213,10 +205,6 @@ build-system-centos: TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu MAKE_CHECK_ARGS: check-build - artifacts: - expire_in: 2 days - paths: - - build check-system-centos: extends: .native_test_job
@@ -317,10 +305,6 @@ build-deprecated: MAKE_CHECK_ARGS: build-tcg TARGETS: ppc64abi32-linux-user tilegx-linux-user lm32-softmmu unicore32-softmmu - artifacts: - expire_in: 2 days - paths: - - build # We split the check-tcg step as test failures are expected but we still # want to catch the build breaking.
Signature · 1 line
2.26.2
Re: [PATCH v3 01/11] gitlab-ci: Drop generic cache rule
In reply to Philippe Mathieu-DaudéQuoted 14 lines · Philippe Mathieu-Daudé
This cache rule is meant for Avocado artifacts, but affects all jobs. Moreover the '
acceptance_template' template already include a more detailled rule to cache artifacts.Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 6 ------ 1 file changed, 6 deletions(-).gitlab-ci.yml +0
-0
.gitlab-ci.yml stages: · line 14
Quoted 8 lines
.gitlab-ci.yml +0
-5@@ -7,12 +7,6 @@ stages: - build - test -# We assume GitLab has it's own caching set up for RPM/APT repositories so we -# just take care of avocado assets here. -cache: - paths: - - $HOME/avocado/data/cache
Right, IIRC the path in $HOME wasn't working as expected anyway, that's why we switched to a more complicated logic in the acceptance_template.
Re: [PATCH v3 07/11] gitlab-ci: Extract common job definition as 'cross_common_job'
In reply to Philippe Mathieu-DaudéQuoted 7 lines · Philippe Mathieu-Daudé
Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
5 insertions, 4 deletions ... is it really that much better?
Thomas
Re: [PATCH v3 09/11] gitlab-ci: Add rules to select cross-jobs to build
In reply to Philippe Mathieu-DaudéQuoted 9 lines · Philippe Mathieu-Daudé
Add rules to select some crossbuild jobs.
The following tags are available to restrict the CI jobs:
- all (select all jobs, this is default)
- cross (select all cross-jobs)
- system (select all cross-system jobs)
- user (select all cross-user jobs)
- $ARCH (select an architecture: arm/mips/ppc/sparc/...)
Quoted 5 lines
Developers can combine tags in the
QEMU_BUILDvariable when pushing a branch (or tag) to repositories. Examples:$ git push -o ci.variable="QEMU_BUILD=user" myrepo mybranch $ git push -o ci.variable="QEMU_BUILD=user,system" myrepo mybranch
That looks interesting, but I think the changes are too big for including them at this point in time, so I'd like to postpone this to the 6.0 cycle (so please respin the series after the 5.2 release).
Also this certainly needs to be documented in a file in the docs/ folder - otherwise you'll be the only one who's using this feature, I guess...
Thanks,
Thomas
Re: [PATCH v3 11/11] gitlab-ci: Move artifacts expiry rule to common 'native_build_job'
In reply to Philippe Mathieu-DaudéQuoted 14 lines · Philippe Mathieu-Daudé
Build jobs include the '
native_build_job' template. Move the 'artifacts expiry' rule there. Now all build jobs benefit from it.Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 24 ++++-------------------- 1 file changed, 4 insertions(+), 20 deletions(-).gitlab-ci.yml +0
-0
.gitlab-ci.yml include: · line 85
Quoted 7 lines
.gitlab-ci.yml +4
-0@@ -79,6 +79,10 @@ include: then make -j"$JOBS" $MAKE_CHECK_ARGS ; fi + artifacts: + expire_in: 2 days + paths: + - build
Should we also keep artifacts for pipelines that do not need it to pass them from one job to the next? ... not sure ... considering that gitlab is thinking about cutting down CI minutes etc., we should maybe be more conservative and only keep artifacts where they are really needed?
Thomas
Re: [PATCH v3 07/11] gitlab-ci: Extract common job definition as 'cross_common_job'
In reply to Thomas HuthQuoted 10 lines · Thomas Huth
Quoted 7 lines · Philippe Mathieu-Daudé
Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)5 insertions, 4 deletions ... is it really that much better?
It becomes useful later we we add a lot into cross_common_job, rather than duplicating in cross_[system/user]_build_job.
I understand this commit description lacks that information.
Thanks.
Re: [PATCH v3 02/11] gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
In reply to Philippe Mathieu-DaudéQuoted 3 lines · Philippe Mathieu-Daudé
'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends
Good idea!
Quoted 5 lines
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 40 ++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 20 deletions(-)
Quoted 131 lines
.gitlab-ci.d/crossbuilds.yml +20
-20@@ -1,5 +1,5 @@ -.cross_system_build_job_template: &cross_system_build_job_definition +.cross_system_build_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m@@ -13,7 +13,7 @@ xtensa-softmmu" - make -j$(expr $(nproc) + 1) all check-build -.cross_user_build_job_template: &cross_user_build_job_definition +.cross_user_build_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:@@ -24,91 +24,91 @@ - make -j$(expr $(nproc) + 1) all check-build cross-armel-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-armel-cross cross-armel-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-armel-cross cross-armhf-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-armhf-cross cross-armhf-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-armhf-cross cross-arm64-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-arm64-cross cross-arm64-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-arm64-cross cross-mips-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-mips-cross cross-mips-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-mips-cross cross-mipsel-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-mipsel-cross cross-mipsel-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-mipsel-cross cross-mips64el-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-mips64el-cross cross-mips64el-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-mips64el-cross cross-ppc64el-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-ppc64el-cross cross-ppc64el-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-ppc64el-cross cross-s390x-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: debian-s390x-cross cross-s390x-user: - <<: *cross_user_build_job_definition + extends: .cross_user_build_job variables: IMAGE: debian-s390x-cross cross-win32-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: fedora-win32-cross cross-win64-system: - <<: *cross_system_build_job_definition + extends: .cross_system_build_job variables: IMAGE: fedora-win64-cross
Re: [PATCH v3 03/11] gitlab-ci: Replace YAML anchors by extends (native_build_job)
In reply to Philippe Mathieu-DaudéQuoted 8 lines · Philippe Mathieu-Daudé
'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extendsSigned-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
LGTM
Quoted 149 lines
.gitlab-ci.yml +16
-16@@ -13,7 +13,7 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml' -.native_build_job_template: &native_build_job_definition +.native_build_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script:@@ -68,7 +68,7 @@ include: - du -chs ${CI_PROJECT_DIR}/avocado-cache build-system-ubuntu: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: ubuntu2004 TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu@@ -99,7 +99,7 @@ acceptance-system-ubuntu: <<: *acceptance_definition build-system-debian: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-amd64 TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu@@ -130,7 +130,7 @@ acceptance-system-debian: <<: *acceptance_definition build-system-fedora: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora CONFIGURE_ARGS: --disable-gcrypt --enable-nettle@@ -162,7 +162,7 @@ acceptance-system-fedora: <<: *acceptance_definition build-system-centos: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos8 CONFIGURE_ARGS: --disable-nettle --enable-gcrypt@@ -194,7 +194,7 @@ acceptance-system-centos: <<: *acceptance_definition build-disabled: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora CONFIGURE_ARGS: --disable-attr --disable-avx2 --disable-bochs@@ -219,7 +219,7 @@ build-disabled: MAKE_CHECK_ARGS: check-qtest SPEED=slow build-tcg-disabled: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos8 script:@@ -239,7 +239,7 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-tools --disable-system@@ -249,7 +249,7 @@ build-user: # we skip sparc64-linux-user until it has been fixed somewhat # we skip cris-linux-user as it doesn't use the common run loop build-user-plugins: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user@@ -257,7 +257,7 @@ build-user-plugins: timeout: 1h 30m build-clang: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora CONFIGURE_ARGS: --cc=clang --cxx=clang++@@ -267,7 +267,7 @@ build-clang: # These targets are on the way out build-deprecated: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-docs --disable-tools@@ -292,7 +292,7 @@ check-deprecated: allow_failure: true build-oss-fuzz: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora script:@@ -310,7 +310,7 @@ build-oss-fuzz: - cd build-oss-fuzz && make check-qtest-i386 check-unit build-tci: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: fedora script:@@ -335,7 +335,7 @@ build-tci: # These jobs test old gcrypt and nettle from RHEL7 # which had some API differences. build-crypto-old-nettle: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos7 TARGETS: x86_64-softmmu x86_64-linux-user@@ -356,7 +356,7 @@ check-crypto-old-nettle: build-crypto-old-gcrypt: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos7 TARGETS: x86_64-softmmu x86_64-linux-user@@ -377,7 +377,7 @@ check-crypto-old-gcrypt: build-crypto-only-gnutls: - <<: *native_build_job_definition + extends: .native_build_job variables: IMAGE: centos7 TARGETS: x86_64-softmmu x86_64-linux-user
Re: [PATCH v3 04/11] gitlab-ci: Replace YAML anchors by extends (native_test_job)
In reply to Philippe Mathieu-DaudéQuoted 8 lines · Philippe Mathieu-Daudé
'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extendsSigned-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
LGTM
Quoted 122 lines
.gitlab-ci.yml +13
-13@@ -34,7 +34,7 @@ include: make -j"$JOBS" $MAKE_CHECK_ARGS ; fi -.native_test_job_template: &native_test_job_definition +.native_test_job: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:@@ -80,7 +80,7 @@ build-system-ubuntu: - build check-system-ubuntu: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-ubuntu artifacts: true@@ -89,7 +89,7 @@ check-system-ubuntu: MAKE_CHECK_ARGS: check acceptance-system-ubuntu: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-ubuntu artifacts: true@@ -111,7 +111,7 @@ build-system-debian: - build check-system-debian: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-debian artifacts: true@@ -120,7 +120,7 @@ check-system-debian: MAKE_CHECK_ARGS: check acceptance-system-debian: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-debian artifacts: true@@ -143,7 +143,7 @@ build-system-fedora: - build check-system-fedora: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-fedora artifacts: true@@ -152,7 +152,7 @@ check-system-fedora: MAKE_CHECK_ARGS: check acceptance-system-fedora: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-fedora artifacts: true@@ -175,7 +175,7 @@ build-system-centos: - build check-system-centos: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-centos artifacts: true@@ -184,7 +184,7 @@ check-system-centos: MAKE_CHECK_ARGS: check acceptance-system-centos: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-system-centos artifacts: true@@ -282,7 +282,7 @@ build-deprecated: # We split the check-tcg step as test failures are expected but we still # want to catch the build breaking. check-deprecated: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-deprecated artifacts: true@@ -346,7 +346,7 @@ build-crypto-old-nettle: - build check-crypto-old-nettle: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-crypto-old-nettle artifacts: true@@ -367,7 +367,7 @@ build-crypto-old-gcrypt: - build check-crypto-old-gcrypt: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-crypto-old-gcrypt artifacts: true@@ -388,7 +388,7 @@ build-crypto-only-gnutls: - build check-crypto-only-gnutls: - <<: *native_test_job_definition + extends: .native_test_job needs: - job: build-crypto-only-gnutls artifacts: true
Re: [PATCH v3 05/11] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
In reply to Philippe Mathieu-DaudéQuoted 8 lines · Philippe Mathieu-Daudé
'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extendsSigned-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
LGTM
Quoted 79 lines
.gitlab-ci.yml +6
-9@@ -42,7 +42,8 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.acceptance_template: &acceptance_definition +.acceptance_test_job: + extends: .native_test_job cache: key: "${CI_JOB_NAME}-cache" paths:@@ -89,14 +90,13 @@ check-system-ubuntu: MAKE_CHECK_ARGS: check acceptance-system-ubuntu: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-ubuntu artifacts: true variables: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-system-debian: extends: .native_build_job@@ -120,14 +120,13 @@ check-system-debian: MAKE_CHECK_ARGS: check acceptance-system-debian: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-debian artifacts: true variables: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-system-fedora: extends: .native_build_job@@ -152,14 +151,13 @@ check-system-fedora: MAKE_CHECK_ARGS: check acceptance-system-fedora: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-fedora artifacts: true variables: IMAGE: fedora MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-system-centos: extends: .native_build_job@@ -184,14 +182,13 @@ check-system-centos: MAKE_CHECK_ARGS: check acceptance-system-centos: - extends: .native_test_job + extends: .acceptance_test_job needs: - job: build-system-centos artifacts: true variables: IMAGE: centos8 MAKE_CHECK_ARGS: check-acceptance - <<: *acceptance_definition build-disabled: extends: .native_build_job
Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job
In reply to Philippe Mathieu-DaudéOnce Cleber said "acceptance" wasn't a good name for those tests. Indeed "integration" is widely used, so okay for this renaming.
Quoted 4 lines · Philippe Mathieu-Daudé
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
Quoted 58 lines
.gitlab-ci.yml +9
-9@@ -42,7 +42,7 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.acceptance_test_job: +.integration_test_job: extends: .native_test_job cache: key: "${CI_JOB_NAME}-cache"@@ -89,8 +89,8 @@ check-system-ubuntu: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check -acceptance-system-ubuntu: - extends: .acceptance_test_job +integration-system-ubuntu: + extends: .integration_test_job needs: - job: build-system-ubuntu artifacts: true@@ -119,8 +119,8 @@ check-system-debian: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check -acceptance-system-debian: - extends: .acceptance_test_job +integration-system-debian: + extends: .integration_test_job needs: - job: build-system-debian artifacts: true@@ -150,8 +150,8 @@ check-system-fedora: IMAGE: fedora MAKE_CHECK_ARGS: check -acceptance-system-fedora: - extends: .acceptance_test_job +integration-system-fedora: + extends: .integration_test_job needs: - job: build-system-fedora artifacts: true@@ -181,8 +181,8 @@ check-system-centos: IMAGE: centos8 MAKE_CHECK_ARGS: check -acceptance-system-centos: - extends: .acceptance_test_job +integration-system-centos: + extends: .integration_test_job needs: - job: build-system-centos artifacts: true
Re: [PATCH v3 07/11] gitlab-ci: Extract common job definition as 'cross_common_job'
In reply to Philippe Mathieu-DaudéQuoted 7 lines · Philippe Mathieu-Daudé
Extract the common definitions shared by '.cross_system_build_job' and '.cross_user_build_job' to '.cross_common_job'.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.d/crossbuilds.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 27 lines
.gitlab-ci.d/crossbuilds.yml +5
-4@@ -1,7 +1,9 @@ - -.cross_system_build_job: +.cross_common_job: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + +.cross_system_build_job: + extends: .cross_common_job timeout: 80m script: - mkdir build@@ -14,8 +16,7 @@ - make -j$(expr $(nproc) + 1) all check-build .cross_user_build_job: - stage: build - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + extends: .cross_common_job script: - mkdir build - cd build
Re: [PATCH v3 08/11] gitlab-ci: Extract common job definition as 'native_common_job'
In reply to Philippe Mathieu-DaudéQuoted 13 lines · Philippe Mathieu-Daudé
Extract the common definitions shared by '.native_build_job' and '.native_test_job' to '.native_common_job'.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-).gitlab-ci.yml +0
-0
.gitlab-ci.yml include: · line 17
Quoted 7 lines
.gitlab-ci.yml +1
-2@@ -13,9 +13,12 @@ include: - local: '/.gitlab-ci.d/containers.yml' - local: '/.gitlab-ci.d/crossbuilds.yml' -.native_build_job: - stage: build +.native_common_job: image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
Do you envision that "native_common_job" with more common properties?
Asking because it creates another indirection to just replace two "image"s.
Anyway,
Quoted 17 lines
.gitlab-ci.yml +5
-1+ +.native_build_job: + extends: .native_common_job + stage: build before_script: - JOBS=$(expr $(nproc) + 1) - sed -i s,git.qemu.org/git,gitlab.com/qemu-project, .gitmodules@@ -35,8 +38,8 @@ include: fi .native_test_job: + extends: .native_common_job stage: test - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: - cd build - find . -type f -exec touch {} +
Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job
In reply to Wainer dos Santos MoschettaQuoted 2 lines · Wainer dos Santos Moschetta
Once Cleber said "acceptance" wasn't a good name for those tests. Indeed "integration" is widely used, so okay for this renaming.
Should we maybe also rename the folder and "make check-acceptance" into "make check-integration" ?
Thomas
Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job
Quoted 5 lines · Thomas Huth
Quoted 2 lines · Wainer dos Santos Moschetta
Once Cleber said "acceptance" wasn't a good name for those tests. Indeed "integration" is widely used, so okay for this renaming.
Should we maybe also rename the folder and "make check-acceptance" into "make check-integration" ?
I'm okay with that. But I also would like to hear from Cleber.
- Wainer
Quoted 4 lines
Thomas
Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job
Quoted 10 lines · Wainer dos Santos Moschetta
Quoted 6 lines · Thomas Huth
Quoted 3 lines · Wainer dos Santos Moschetta
Once Cleber said "acceptance" wasn't a good name for those tests. Indeed
"integration" is widely used, so okay for this renaming.Should we maybe also rename the folder and "make check-acceptance" into "make check-integration" ?
I'm okay with that. But I also would like to hear from Cleber.
This is the plan indeed, but out of the scope of this (gitlab-ci specific) series.
'check-acceptance' might then depend on 'check-integration'.
Quoted 8 lines
- Wainer
Quoted 4 lines
Thomas
Re: [PATCH v3 11/11] gitlab-ci: Move artifacts expiry rule to common 'native_build_job'
In reply to Thomas HuthQuoted 19 lines · Thomas Huth
Quoted 18 lines · Philippe Mathieu-Daudé
Build jobs include the '
native_build_job' template. Move the 'artifacts expiry' rule there. Now all build jobs benefit from it.Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 24 ++++-------------------- 1 file changed, 4 insertions(+), 20 deletions(-).gitlab-ci.yml +1
-0@@ -79,6 +79,10 @@ include: then make -j"$JOBS" $MAKE_CHECK_ARGS ; fi + artifacts:
Quoted 8 lines
.gitlab-ci.yml include: · line 85
Quoted 3 lines
.gitlab-ci.yml +3
-0+ expire_in: 2 days + paths: + - buildShould we also keep artifacts for pipelines that do not need it to pass them from one job to the next? ... not sure ... considering that gitlab is thinking about cutting down CI minutes etc., we should maybe be more conservative and only keep artifacts where they are really needed?
As nothing is released from these jobs (except some roms/ and documentation?) I'd not keep anything at this point. Later we can be stricter and select the minimum we need.
(I don't think nobody ever downloaded a job artifact to test / debug it).
Quoted 3 lines
Thomas
Re: [PATCH v3 06/11] gitlab-ci: Rename acceptance_test_job -> integration_test_job
In reply to Philippe Mathieu-DaudéQuoted 64 lines · Philippe Mathieu-Daudé
Signed-off-by Philippe Mathieu-Daudé <[email protected]>--- .gitlab-ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-).gitlab-ci.yml +9
-10@@ -42,7 +42,7 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.acceptance_test_job: +.integration_test_job: extends: .native_test_job cache: key: "${CI_JOB_NAME}-cache"@@ -89,8 +89,8 @@ check-system-ubuntu: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check -acceptance-system-ubuntu: - extends: .acceptance_test_job +integration-system-ubuntu: + extends: .integration_test_job needs: - job: build-system-ubuntu artifacts: true@@ -119,8 +119,8 @@ check-system-debian: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check -acceptance-system-debian: - extends: .acceptance_test_job +integration-system-debian: + extends: .integration_test_job needs: - job: build-system-debian artifacts: true@@ -150,8 +150,8 @@ check-system-fedora: IMAGE: fedora MAKE_CHECK_ARGS: check -acceptance-system-fedora: - extends: .acceptance_test_job +integration-system-fedora: + extends: .integration_test_job needs: - job: build-system-fedora artifacts: true@@ -181,8 +181,8 @@ check-system-centos: IMAGE: centos8 MAKE_CHECK_ARGS: check -acceptance-system-centos: - extends: .acceptance_test_job +integration-system-centos: + extends: .integration_test_job needs: - job: build-system-centos artifacts: true --
Quoted 3 lines
2.26.2
Makes sense, thanks!