gitlab: limit jobs executed in most pipelines by default
cover gitlab: limit jobs executed in most pipelines by default
The QEMU pipeline has as many as 120+ jobs, almost all of which are always run. This is wasteful in many ways. When merging to the main repo, Peter runs CI on 'staging' branch, which then gets immediately re-run on the 'master' branch. The first patch in this series culls that redundancy by always skipping CI on pushes to 'master'.
I suggest we setup a nightly scheduled job that runs on current contents of 'master' once a day, as a way to catch bitrot from OS distro's changing under us.
I think we want the first patch as a high priority, since I think the duplicated jobs on master are really impacting our integration with Cirrus CI, delaying jobs since Cirrus has very strictly applied concurrency limits.
The second patch is more speculative, and illustrates how to we can reduce the number of CI jobs run for *all* contributors, when they push to gitlab. There is a key decision to make here between:
a. run no jobs at all by default on push
b. run a few jobs by default on push to catch 90% of problems c. run all jobs on by default on push
Currently we're doing (c), and patch 2 implements (b), with an easy opt-in to trigger (c).
There is a credible variant of patch 2 though, which could easily implement (a) with an opt-in to run (b), and a further opt-in to run (c).
Or we can get more advanced and add ability for individual contributors to decide between (a) and (b) for default behaviour, based on a variable configured globally in their gitlab fork.
There is an illustration of results of (b) here:
https://gitlab.com/berrange/qemu/-/pipelines/339281325
Note that for all the non-defualt jobs, I've left them as manually triggerable. This makes a the pipeline UI a bit cluttered by default but lets you trigger jobs from the web UI individually. An alternative would be to never create the non-defualt jobs at all. They would have to be requested explicitly at time of push in some way.
Daniel P. Berrangé (2):
gitlab: prevent CI jobs running pushes to default branch or tags
gitlab: cut down on jobs run by default in user forks
.gitlab-ci.d/buildtest-template.yml | 17 +--- .gitlab-ci.d/buildtest.yml | 14 +++ .gitlab-ci.d/cirrus.yml | 5 +- .gitlab-ci.d/container-core.yml | 3 + .gitlab-ci.d/container-cross.yml | 9 +- .gitlab-ci.d/container-template.yml | 1 + .gitlab-ci.d/containers.yml | 2 + .gitlab-ci.d/crossbuild-template.yml | 3 + .gitlab-ci.d/crossbuilds.yml | 3 + .gitlab-ci.d/qemu-project.yml | 1 + .gitlab-ci.d/workflow.yml | 146 +++++++++++++++++++++++++++ .gitlab-ci.yml | 21 +--- 12 files changed, 188 insertions(+), 37 deletions(-) create mode 100644 .gitlab-ci.d/workflow.yml
Signature · 1 line
2.31.1
PATCH 1/2 gitlab: prevent CI jobs running pushes to default branch or tags
When patch series are accepted they are first pushed to the 'staging' branch, which triggers a CI pipeline. If-and-only-if the pipeline succeeds, the code is then pushed to the default branch. The latter step triggers a second pipeline that duplicates what was just validated in the 'staging' branch. This wastes CI resources and can delay jobs needed by a subsequently pushed 'staging' branch.
This is particularly evident with jobs delegated to Cirrus CI where there is very limited concurrency of jobs. The duplicate jobs in 'master' often delay the next 'staging' jobs enough to cause 1 hour timeout to be hit.
There is similar wasted CI resource when release tags are pushed to the repo, since the branch associated with the release will have already passed CI.
For user forks is is reasonable to skip CI on the default branch since that's either usually unused or follows the main repo default branch. Meanwhile tags in user forks are typically used to track postings of specific patch series versions. It is expected that a contributor will have tested before sending the patch series. Thus pipelines for the default branch and tag pushes in forks are redundant too.
This patch makes use of the 'workflow:' configuration to setup rules that apply to the entire pipeline. The rules allow pipelines to be created ONLY in response to:
- Push events to a non-default branch
- Scheduled pipelines
- REST API pipeline triggers
- Web UI pipeline triggers
--- .gitlab-ci.d/qemu-project.yml | 1 + .gitlab-ci.d/workflow.yml | 56 +++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 .gitlab-ci.d/workflow.yml
.gitlab-ci.d/qemu-project.yml +1 -0
@@ -2,6 +2,7 @@ # https://gitlab.com/qemu-project/qemu/-/pipelines include: + - local: '/.gitlab-ci.d/workflow.yml' - local: '/.gitlab-ci.d/stages.yml' - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml'
.gitlab-ci.d/workflow.yml new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH 2/2 gitlab: cut down on jobs run by default in user forks
In reply to Daniel P. BerrangéCurrently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.
This patch sets up rules to cut down on the jobs run by default to only only cover a fairly typical set of combinations, which ought to catch the majority of common breakage
- Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
- Linux user static build
- Cross build i386 (to identify any 32-bit build issues)
- Cross build s390x (to identify any big endian build issues)
- Containers needed for the above
- Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
That ends up being the following jobs:
Containers stage
- amd64-centos8-container
- amd64-debian10-container
- amd64-fedora-container
- amd64-ubuntu2004-container
- win32-fedora-container
Containers layer2 stage
- amd64-debian-container
- i386-fedora-cross-container
- s390x-debian-cross-container
Build stage
- build-system-centos
- build-system-debian
- build-system-fedora
- build-system-ubuntu
- build-tools-and-docs-debian
- build-user-static
- check-dco
- check-patch
- cross-i386-system
- cross-s390x-system
- cross-win32-system
- x64-freebsd-13-build
- x64-macos-11-build
Test stage
- check-system-centos
- check-system-debian
- check-system-fedora
- check-system-ubuntu
- pages
This gives a reduction from 120+ jobs to 26. It could potentially be cut down more if we re-arrange the split of targets for the four build-system-XXX jobs, if we don't want to test all possible arch targets by default for all contributors (maybe only x86_64 and aarch64 by default) This is left as a future area to explore.
When using inherited templates together with rules, it is ill-defined what the semantics are when both the template and job define rules. Thus logic is controlled by create a template containining job rules, that is then inherited by all jobs / job templates. Individual jobs can tweak the effect of the inherited rules by setting variables.
The set of default jobs are identified by existance of a special variable:
- QEMU_CI_DEFAULT_JOB: 1
To trigger remaining non-default jobs, push to a branch called 'ci-XXXXX', or set a git push option. e.g. one of:
$ git push ci-somefeature $ git push -o "ci.variable=QEMU_CI=1" somefeature
Jobs can be marked to never run automatically using the variable:
- QEMU_CI_MANUAL_JOB: 1
Jobs that provide acceptance (functional/integration) tests are designated using
- QEMU_CI_ACCEPTANCE_JOB: 1
These are not run unless pushing to the 'staging' branch of the main repo. They can be optionally run in other scenarios by pushing to a branch 'ci-acceptance-XXX', or by setting an additional git push option, together with the previously CI configurations. e.g. one of:
$ git push ci-acceptance-somefeature $ git push -o "ci.variable=QEMU_CI_ACEPTANCE=1" ci-somefeature $ git push -o "ci.variable=QEMU_CI_ACEPTANCE=1" -o "ci.variable=QEMU_CI=1" somefeature
--- .gitlab-ci.d/buildtest-template.yml | 17 ++---- .gitlab-ci.d/buildtest.yml | 14 +++++ .gitlab-ci.d/cirrus.yml | 5 +- .gitlab-ci.d/container-core.yml | 3 + .gitlab-ci.d/container-cross.yml | 9 +-- .gitlab-ci.d/container-template.yml | 1 + .gitlab-ci.d/containers.yml | 2 + .gitlab-ci.d/crossbuild-template.yml | 3 + .gitlab-ci.d/crossbuilds.yml | 3 + .gitlab-ci.d/workflow.yml | 90 ++++++++++++++++++++++++++++ .gitlab-ci.yml | 21 +------ 11 files changed, 131 insertions(+), 37 deletions(-)
.gitlab-ci.d/buildtest-template.yml +5 -12
@@ -1,4 +1,6 @@ + .native_build_job_template: + extends: .job_workflow stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script:
@@ -27,6 +29,7 @@ fi .native_test_job_template: + extends: .job_workflow stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
@@ -67,15 +70,5 @@ after_script: - cd build - du -chs ${CI_PROJECT_DIR}/avocado-cache - rules: - # Only run these jobs if running on the mainstream namespace, - # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either - # in its namespace setting or via git-push option, see documentation - # in /.gitlab-ci.yml of this repository). - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: always - - if: '$QEMU_CI_AVOCADO_TESTING' - when: always - # Otherwise, set to manual (the jobs are created but not run). - - when: manual - allow_failure: true + variables: + - QEMU_CI_ACCEPTANCE_JOB: 1
.gitlab-ci.d/buildtest.yml +14 -0
@@ -40,6 +40,7 @@ build-system-ubuntu: needs: job: amd64-ubuntu2004-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: ubuntu2004 CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -56,6 +57,7 @@ check-system-ubuntu: - job: build-system-ubuntu artifacts: true variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check
@@ -73,6 +75,7 @@ build-system-debian: needs: job: amd64-debian-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: debian-amd64 CONFIGURE_ARGS: --enable-fdt=system TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -89,6 +92,7 @@ check-system-debian: - job: build-system-debian artifacts: true variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: debian-amd64 MAKE_CHECK_ARGS: check
@@ -106,6 +110,7 @@ build-system-fedora: needs: job: amd64-fedora-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: fedora CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs --enable-fdt=system --enable-slirp=system --enable-capstone=system
@@ -123,6 +128,7 @@ check-system-fedora: - job: build-system-fedora artifacts: true variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: fedora MAKE_CHECK_ARGS: check
@@ -140,6 +146,7 @@ build-system-centos: needs: job: amd64-centos8-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: centos8 CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system --enable-modules --enable-trace-backends=dtrace
@@ -157,6 +164,7 @@ check-system-centos: - job: build-system-centos artifacts: true variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: centos8 MAKE_CHECK_ARGS: check
@@ -325,6 +333,7 @@ build-user-static: needs: job: amd64-debian-user-cross-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: debian-all-test-cross CONFIGURE_ARGS: --disable-tools --disable-system --static MAKE_CHECK_ARGS: check-tcg
@@ -634,6 +643,7 @@ build-without-default-features: MAKE_CHECK_ARGS: check-unit build-libvhost-user: + extends: .job_workflow stage: build image: $CI_REGISTRY_IMAGE/qemu/fedora:latest needs:
@@ -653,6 +663,7 @@ build-tools-and-docs-debian: needs: job: amd64-debian-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
@@ -664,10 +675,13 @@ build-tools-and-docs-debian: # Prepare for GitLab pages deployment. Anything copied into the # "public" directory will be deployed to $USER.gitlab.io/$PROJECT pages: + extends: .native_build_job_template image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest stage: test needs: - job: build-tools-and-docs-debian + variables: + QEMU_CI_DEFAULT_JOB: 1 script: - mkdir -p public # HTML-ised source tree
.gitlab-ci.d/cirrus.yml +3 -2
@@ -11,6 +11,7 @@ # special care, because we can't just override it at the GitLab CI job # definition level or we risk breaking it completely. .cirrus_build_job: + extends: .job_workflow stage: build image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: []
@@ -39,8 +40,6 @@ <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml - cat .gitlab-ci.d/cirrus/$NAME.yml - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml - rules: - - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" x64-freebsd-12-build: extends: .cirrus_build_job
@@ -61,6 +60,7 @@ x64-freebsd-12-build: x64-freebsd-13-build: extends: .cirrus_build_job variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: freebsd-13 CIRRUS_VM_INSTANCE_TYPE: freebsd_instance CIRRUS_VM_IMAGE_SELECTOR: image_family
@@ -74,6 +74,7 @@ x64-freebsd-13-build: x64-macos-11-base-build: extends: .cirrus_build_job variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: macos-11 CIRRUS_VM_INSTANCE_TYPE: osx_instance CIRRUS_VM_IMAGE_SELECTOR: image
.gitlab-ci.d/container-core.yml +3 -0
@@ -4,14 +4,17 @@ include: amd64-centos8-container: extends: .container_job_template variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: centos8 amd64-fedora-container: extends: .container_job_template variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: fedora amd64-debian10-container: extends: .container_job_template variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: debian10
.gitlab-ci.d/container-cross.yml +5 -4
@@ -50,13 +50,11 @@ armhf-debian-cross-container: # We never want to build hexagon in the CI system and by default we # always want to refer to the master registry where it lives. hexagon-cross-container: + extends: .job_workflow image: docker:stable stage: containers - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - when: always variables: + QEMU_CI_MANUAL_JOB: 1 NAME: debian-hexagon-cross GIT_DEPTH: 1 services:
@@ -143,6 +141,7 @@ s390x-debian-cross-container: stage: containers-layer2 needs: ['amd64-debian10-container'] variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: debian-s390x-cross sh4-debian-cross-container:
@@ -179,11 +178,13 @@ cris-fedora-cross-container: i386-fedora-cross-container: extends: .container_job_template variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: fedora-i386-cross win32-fedora-cross-container: extends: .container_job_template variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: fedora-win32-cross win64-fedora-cross-container:
.gitlab-ci.d/container-template.yml +1 -0
@@ -1,4 +1,5 @@ .container_job_template: + extends: .job_workflow image: docker:stable stage: containers services:
.gitlab-ci.d/containers.yml +2 -0
@@ -17,6 +17,7 @@ amd64-debian-container: stage: containers-layer2 needs: ['amd64-debian10-container'] variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: debian-amd64 amd64-ubuntu1804-container:
@@ -27,6 +28,7 @@ amd64-ubuntu1804-container: amd64-ubuntu2004-container: extends: .container_job_template variables: + QEMU_CI_DEFAULT_JOB: 1 NAME: ubuntu2004 amd64-ubuntu-container:
.gitlab-ci.d/crossbuild-template.yml +3 -0
@@ -1,4 +1,5 @@ .cross_system_build_job: + extends: .job_workflow stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m
@@ -18,6 +19,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $EXTRA_CONFIGURE_OPTS variable. .cross_accel_build_job: + extends: .job_workflow stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 30m
@@ -30,6 +32,7 @@ - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS .cross_user_build_job: + extends: .job_workflow stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script:
.gitlab-ci.d/crossbuilds.yml +3 -0
@@ -48,6 +48,7 @@ cross-i386-system: needs: job: i386-fedora-cross-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: fedora-i386-cross MAKE_CHECK_ARGS: check-qtest
@@ -129,6 +130,7 @@ cross-s390x-system: needs: job: s390x-debian-cross-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: debian-s390x-cross cross-s390x-user:
@@ -159,6 +161,7 @@ cross-win32-system: needs: job: win32-fedora-cross-container variables: + QEMU_CI_DEFAULT_JOB: 1 IMAGE: fedora-win32-cross cross-win64-system:
.gitlab-ci.d/workflow.yml +90 -0
93 lines not on this page · raw message
.gitlab-ci.yml +2 -19
@@ -15,25 +15,8 @@ # # QEMU CI jobs are based on templates. Some templates provide # user-configurable options, modifiable via configuration variables. -# -# These variables can be set globally in the user's CI namespace -# setting: -# https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui -# or set manually each time a branch/tag is pushed, as a git-push -# command line argument: -# https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd -# -# Example setting the QEMU_CI_EXAMPLE_VAR variable: -# -# git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch -# -# ---------------------------------------------------------------------- -# -# List of environment variables that can be use to modify the set -# of jobs selected: -# -# - QEMU_CI_AVOCADO_TESTING -# If set, tests using the Avocado framework will be run +# Refer to 'workflow.yml' for guidance on how to trigger non-default +# jobs using git push options, or special branch names include: - local: '/.gitlab-ci.d/qemu-project.yml'
Signature · 1 line
2.31.1