Skip to contentSkip to conversation

gitlab: prepare for limited CI minutes by not running by default

PATCH In review Daniel P. Berrangé 6 participants 19 messages Aug 12, 2021 – May 10, 2022

lore
Share & tools
lore
#0 lore raw

cover gitlab: prepare for limited CI minutes by not running by default

A while ago GitLab announced it would be ending the free unlimited CI minutes for public projects. It hasn't happened yet, but it is certainly still on the radar. It is possible for OSS projects to get a greater allowance, but it is still unclear how this will work for contributors' forks. Even with the greater allowance, we'll burn through it in just a few pipeline runs. People creating brand new gitlab accounts are already getting hurt by the new limits IIUC.

This series isn't a full solution, but it gets us started in a more sustainable direction. See the second patch commit message and the docs that it adds for full details.

At a high level though, with this series applied, we no longer run any CI jobs by default, except for the 'staging' branch in the 'qemu-project' namespace.

Users can manually start any individual job from the web UI still though. Further, they can opt-in to various sets of jobs

- 100% manual jobs (the new default):

    https://gitlab.com/berrange/qemu/-/pipelines/352393697

- A minimal set of jobs

    https://gitlab.com/berrange/qemu/-/pipelines/352397093

- A full set of jobs (the old default)

    https://gitlab.com/berrange/qemu/-/pipelines/352123582

- A gating set of jobs (identical to what's run on staging)

    https://gitlab.com/berrange/qemu/-/pipelines/352195185

With this starting point, the common "rules" definition across all jobs, makes it easy for us to add more refinements. We can defining further sets of interesting jobs.

eg could define a set of cross-compilation, or a set for non-Linux, or a set of Windows, or sets for particular host architectures, etc.

Daniel P. Berrangé (2):
docs: split the CI docs into two files
gitlab: don't run CI jobs by default on push to user forks

 .gitlab-ci.d/buildtest-template.yml  |  17 +--
 .gitlab-ci.d/buildtest.yml           |  43 +++----
 .gitlab-ci.d/cirrus.yml              |   6 +-
 .gitlab-ci.d/container-core.yml      |   4 +
 .gitlab-ci.d/container-cross.yml     |   9 +-
 .gitlab-ci.d/container-template.yml  |   1 +
 .gitlab-ci.d/containers.yml          |   4 +
 .gitlab-ci.d/crossbuild-template.yml |   3 +
 .gitlab-ci.d/crossbuilds.yml         |   3 +
 .gitlab-ci.d/qemu-project.yml        |   1 +
 .gitlab-ci.d/rules.yml               | 116 +++++++++++++++++++
 .gitlab-ci.d/static_checks.yml       |  18 +--
 .gitlab-ci.yml                       |   4 -
 docs/devel/ci-jobs.rst               | 161 +++++++++++++++++++++++++++
 docs/devel/ci-runners.rst            | 117 +++++++++++++++++++
 docs/devel/ci.rst                    | 159 +-------------------------
 16 files changed, 459 insertions(+), 207 deletions(-)
 create mode 100644 .gitlab-ci.d/rules.yml
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/2 docs: split the CI docs into two files

This splits the CI docs into one file talking about job setup and usage and another file describing provisioning of custom runners.

---
 docs/devel/ci-jobs.rst    |  40 ++++++++++
 docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
 docs/devel/ci.rst         | 159 +-------------------------------------
 3 files changed, 159 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst
docs/devel/ci-jobs.rst new file +40 -0
new file mode 100644
index 0000000000..9cd9819786
@@ -0,0 +1,40 @@
+Custom CI/CD variables
+======================
+
+QEMU CI pipelines can be tuned by setting some CI environment variables.
+
+Set variable globally in the user's CI namespace
+------------------------------------------------
+
+Variables can be set globally in the user's CI namespace setting.
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
+
+Set variable manually when pushing a branch or tag to the user's repository
+---------------------------------------------------------------------------
+
+Variables can be set manually when pushing a branch or tag, using
+git-push command line arguments.
+
+Example setting the QEMU_CI_EXAMPLE_VAR variable:
+
+.. code::
+
+   git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
+
+Here is a list of the most used variables:
+
+QEMU_CI_AVOCADO_TESTING
+~~~~~~~~~~~~~~~~~~~~~~~
+By default, tests using the Avocado framework are not run automatically in
+the pipelines (because multiple artifacts have to be downloaded, and if
+these artifacts are not already cached, downloading them make the jobs
+reach the timeout limit). Set this variable to have the tests using the
+Avocado framework run automatically.
+
docs/devel/ci-runners.rst new file +117 -0

117 lines not on this page · raw message

docs/devel/ci.rst +2 -157

162 lines not on this page · raw message

Signature · 1 line
2.31.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] docs: split the CI docs into two files

Quoted 2 lines · Daniel P. Berrangé

This splits the CI docs into one file talking about job setup and usage and another file describing provisioning of custom runners.

Thanks.

Quoted 8 lines
---
 docs/devel/ci-jobs.rst    |  40 ++++++++++
 docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
 docs/devel/ci.rst         | 159 +-------------------------------------
 3 files changed, 159 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] docs: split the CI docs into two files

In reply to Daniel P. Berrangé
Quoted 4 lines · Daniel P. Berrangé

This splits the CI docs into one file talking about job setup and usage and another file describing provisioning of custom runners.

Quoted 9 lines
---
 docs/devel/ci-jobs.rst    |  40 ++++++++++
 docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
 docs/devel/ci.rst         | 159 +-------------------------------------
 3 files changed, 159 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst
#4 lore raw

PATCH 2/2 gitlab: don't run CI jobs by default on push to user forks

In reply to Daniel P. Berrangé

Currently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.

In addition, with the expected change in GitLab to set a finite CI minute allowance on all users we need to be much more conservative.

This patch thus sets up rules so that no CI jobs will run by default on push, except

- Pushes to branch name prefix 'staging' on 'qemu-project' repo run full set of jobs
- Pushes to 'master' branch, only run jobs needed to publish the website.

In any pushes to users forks, CI is now strictly opt-in. The pipeline will always be created by every singe job will be marked manual. You can then use the web UI to start individual jobs.

For more convenience there are three levels of increasing comprehensive CI job sets that can be chosen between

- Gating - run exactly the same as pushes to 'staging' branch (except for jobs needing custom runners)

Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1

- Full - same as 'Gating', except that acceptance tests don't get run. This is equivalent to historical CI behaviour for pushes to user forks.

Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1

- Minimal - a significantly cut down set of jobs to get a decent sanity check of builds without burning massive amounts of CI time.

Push to 'ci-min-xxx' branch, or set env QEMU_CI=1

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

This is all controlled by defining a set of 'rules' entries that apply to (almost) all jobs in the CI pipeline. The exceptions are those using custom runners, and the firmware build jobs.

Using inherited templates together with rules, doesn't work in the way you might expect. Any rules defined in the child completely replace those from the parent, rather than augmenting them. Thus we need to avoid having rules defined in any jobs at all, by inheriting from a parent job template that contains the rules. The characteristics of individual jobs are then defined by variables set against the job. This is described in the updated content for the docs/devel/ci-jobs.rst file.

---
 .gitlab-ci.d/buildtest-template.yml  |  17 +--
 .gitlab-ci.d/buildtest.yml           |  43 ++++----
 .gitlab-ci.d/cirrus.yml              |   6 +-
 .gitlab-ci.d/container-core.yml      |   4 +
 .gitlab-ci.d/container-cross.yml     |   9 +-
 .gitlab-ci.d/container-template.yml  |   1 +
 .gitlab-ci.d/containers.yml          |   4 +
 .gitlab-ci.d/crossbuild-template.yml |   3 +
 .gitlab-ci.d/crossbuilds.yml         |   3 +
 .gitlab-ci.d/qemu-project.yml        |   1 +
 .gitlab-ci.d/rules.yml               | 116 ++++++++++++++++++++
 .gitlab-ci.d/static_checks.yml       |  18 ++--
 .gitlab-ci.yml                       |   4 -
 docs/devel/ci-jobs.rst               | 155 ++++++++++++++++++++++++---
 14 files changed, 317 insertions(+), 67 deletions(-)
 create mode 100644 .gitlab-ci.d/rules.yml
.gitlab-ci.d/buildtest-template.yml +5 -12
index fcbcc4e627..8578b3ed11 100644
@@ -1,4 +1,6 @@
+
 .native_build_job_template:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -27,6 +29,7 @@
       fi
 
 .native_test_job_template:
+  extends: .job_rules
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -67,15 +70,5 @@
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_GATING: 1
.gitlab-ci.d/buildtest.yml +24 -19

144 lines not on this page · raw message

.gitlab-ci.d/cirrus.yml +4 -2
index 675db69622..4334736796 100644
@@ -11,6 +11,7 @@
 # special care, because we can't just override it at the GitLab CI job
 # definition level or we risk breaking it completely.
 .cirrus_build_job:
+  extends: .job_rules
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
@@ -39,8 +40,6 @@
       <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
 
 x64-freebsd-12-build:
   extends: .cirrus_build_job
@@ -61,6 +60,8 @@ x64-freebsd-12-build:
 x64-freebsd-13-build:
   extends: .cirrus_build_job
   variables:
+    QEMU_JOB_SOFT_FAIL: 1
+    QEMU_JOB_MINIMAL: 1
     NAME: freebsd-13
     CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
     CIRRUS_VM_IMAGE_SELECTOR: image_family
@@ -74,6 +75,7 @@ x64-freebsd-13-build:
 x64-macos-11-base-build:
   extends: .cirrus_build_job
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: macos-11
     CIRRUS_VM_INSTANCE_TYPE: osx_instance
     CIRRUS_VM_IMAGE_SELECTOR: image
.gitlab-ci.d/container-core.yml +4 -0
index e8dd1f476a..8983c32b3b 100644
@@ -4,14 +4,18 @@ include:
 amd64-centos8-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: centos8
 
 amd64-fedora-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: fedora
 
 amd64-debian10-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
     NAME: debian10
.gitlab-ci.d/container-cross.yml +5 -4
index 0fcebe363a..ef316668eb 100644
@@ -50,13 +50,11 @@ armhf-debian-cross-container:
 # We never want to build hexagon in the CI system and by default we
 # always want to refer to the master registry where it lives.
 hexagon-cross-container:
+  extends: .job_rules
   image: docker:stable
   stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
   variables:
+    QEMU_JOB_MANUAL: 1
     NAME: debian-hexagon-cross
     GIT_DEPTH: 1
   services:
@@ -143,6 +141,7 @@ s390x-debian-cross-container:
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: debian-s390x-cross
 
 sh4-debian-cross-container:
@@ -179,11 +178,13 @@ cris-fedora-cross-container:
 i386-fedora-cross-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: fedora-i386-cross
 
 win32-fedora-cross-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: fedora-win32-cross
 
 win64-fedora-cross-container:
.gitlab-ci.d/container-template.yml +1 -0
index 1baecd9460..e2954e4ed3 100644
@@ -1,4 +1,5 @@
 .container_job_template:
+  extends: .job_rules
   image: docker:stable
   stage: containers
   services:
.gitlab-ci.d/containers.yml +4 -0
index cd06d3f5f4..6948923892 100644
@@ -17,6 +17,8 @@ amd64-debian-container:
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
     NAME: debian-amd64
 
 amd64-ubuntu1804-container:
@@ -27,6 +29,7 @@ amd64-ubuntu1804-container:
 amd64-ubuntu2004-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: ubuntu2004
 
 amd64-ubuntu-container:
@@ -42,4 +45,5 @@ amd64-opensuse-leap-container:
 python-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: python
.gitlab-ci.d/crossbuild-template.yml +3 -0
index 7d3ad00a1e..997ec02f79 100644
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -23,6 +24,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $EXTRA_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -35,6 +37,7 @@
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
.gitlab-ci.d/crossbuilds.yml +3 -0
index 4ff3aa3cfc..9fcd2ac567 100644
@@ -48,6 +48,7 @@ cross-i386-system:
   needs:
     job: i386-fedora-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: fedora-i386-cross
     MAKE_CHECK_ARGS: check-qtest
 
@@ -129,6 +130,7 @@ cross-s390x-system:
   needs:
     job: s390x-debian-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: debian-s390x-cross
 
 cross-s390x-user:
@@ -159,6 +161,7 @@ cross-win32-system:
   needs:
     job: win32-fedora-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: fedora-win32-cross
   artifacts:
     paths:
.gitlab-ci.d/qemu-project.yml +1 -0
index b3d79bc429..7007193fc0 100644
@@ -2,6 +2,7 @@
 # https://gitlab.com/qemu-project/qemu/-/pipelines
 
 include:
+  - local: '/.gitlab-ci.d/rules.yml'
   - local: '/.gitlab-ci.d/stages.yml'
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
.gitlab-ci.d/rules.yml new file +116 -0

116 lines not on this page · raw message

.gitlab-ci.d/static_checks.yml +9 -9
index 96dbd9e310..4ba3e038dd 100644
@@ -1,4 +1,5 @@
 check-patch:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
   needs:
@@ -6,42 +7,41 @@ check-patch:
   script:
     - .gitlab-ci.d/check-patch.py
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1000
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
-      allow_failure: true
+  allow_failure: true
 
 check-dco:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
   needs:
     job: amd64-centos8-container
   script: .gitlab-ci.d/check-dco.py
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1000
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
 
 check-python-pipenv:
+  extends: .job_rules
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
     - make -C python check-pipenv
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1
   needs:
     job: python-container
 
 check-python-tox:
+  extends: .job_rules
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
     - make -C python check-tox
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1
     QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
   needs:
.gitlab-ci.yml +0 -4
index 9762dda2ee..d5bd31f294 100644
@@ -15,10 +15,6 @@
 #
 # QEMU CI jobs are based on templates. Some templates provide
 # user-configurable options, modifiable via configuration variables.
-#
-# See https://qemu-project.gitlab.io/qemu/devel/ci.html#custom-ci-cd-variables
-# for more information.
-#
 
 include:
   - local: '/.gitlab-ci.d/qemu-project.yml'
docs/devel/ci-jobs.rst +138 -17

173 lines not on this page · raw message

Signature · 1 line
2.31.1
#5 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 12 lines · Daniel P. Berrangé

Currently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.

In addition, with the expected change in GitLab to set a finite CI minute allowance on all users we need to be much more conservative.

This patch thus sets up rules so that no CI jobs will run by default on push, except

Quoted 8 lines

- Pushes to branch name prefix 'staging' on 'qemu-project' repo run full set of jobs
- Pushes to 'master' branch, only run jobs needed to publish the website.

In any pushes to users forks, CI is now strictly opt-in. The pipeline will always be created by every singe job will be marked manual. You can then use the web UI to start individual jobs.

Is a random user notified in some way that a pipeline with jobs that can be triggered manually has been created?

Quoted 31 lines

For more convenience there are three levels of increasing comprehensive CI job sets that can be chosen between

- Gating - run exactly the same as pushes to 'staging' branch (except for jobs needing custom runners)

Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1

- Full - same as 'Gating', except that acceptance tests don't get run. This is equivalent to historical CI behaviour for pushes to user forks.

Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1

- Minimal - a significantly cut down set of jobs to get a decent sanity check of builds without burning massive amounts of CI time.

Push to 'ci-min-xxx' branch, or set env QEMU_CI=1

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

Quoted 3 lines

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

Where do you think we should start to cut them down? Limit the set of tested arch targets to the most common ones?

Generally speaking, this makes sense; but I think we have different situations which need different kinds of testing, and we should make it as easy as possible to run the right set of tests.

(a) an individual contributor is doing some changes

In that case, I assume (hope?) that the contributor has actually compiled the code for the relevant targets and has done some manual testing. Running acceptance tests locally would also be good, or things like iotests or check-tcg, when applicable.

(b) a subsystem maintainer is queuing some changes

Some more comprehensive automated testing, and likely some subsystem-specific testing on top (like testing that requires specialized hardware, manual interventions, etc.) Since the introduction of gitlab-ci, I had relied on what you call the 'full' set to be run automatically when I push to my staging branch, plus my manual testing. Prior to that, I relied on patchew and ran tests locally. Drawbacks of that: it hogs my development machine, and it is easy to forget something.

If I pull from a sub-submaintainer, I assume that everything is already in good order, and only do some very light sanity/integration checking. Although we're hardly doing subsystem pull request nowadays.

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Couple of ideas from my side:
- include some scripts/make targets that guide an individual contributor to run a good subset of automated tests locally
- define a 'ci-subsystem' set that covers usual pain points for a subsystem
- have subsystem maintainers run a pull req test in the qemu-project context (i.e. using extra CI minutes that the project may have), or put them on a special list on subsystem maintainers so they can use more minutes

#6 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 19 lines · Cornelia Huck
Quoted 17 lines · Daniel P. Berrangé

Currently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.

In addition, with the expected change in GitLab to set a finite CI minute allowance on all users we need to be much more conservative.

This patch thus sets up rules so that no CI jobs will run by default on push, except

- Pushes to branch name prefix 'staging' on 'qemu-project' repo run full set of jobs
- Pushes to 'master' branch, only run jobs needed to publish the website.

Quoted 6 lines
Quoted 3 lines

In any pushes to users forks, CI is now strictly opt-in. The pipeline will always be created by every singe job will be marked manual. You can then use the web UI to start individual jobs.

Is a random user notified in some way that a pipeline with jobs that can be triggered manually has been created?

I don't believe there's any notification.

Quoted 11 lines
Quoted 11 lines

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

Quoted 6 lines
Quoted 3 lines

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

Where do you think we should start to cut them down? Limit the set of tested arch targets to the most common ones?

Some of our targets are obviously much more important and frequently changed than others. For contributors our goal is to mimimize breakage after patches are submitted. Most of our contributors changes will be well covered by x86-64 + aarch64 alone. Other targets give varying degrees of extra benefit.

On the other hand the contributors are likely to have tested x86_64 or aarch64 themselves since that'll be their dev platform. So the benefit of CI is testing bits that they didnt bother to test.

No clear easy answer here, but I feel like we could benefit from classifying our target archs tier 1/2/3 and tailoring our default testing matrix accordingly.

The other way to cut down the "minimal" set is to reduce the OS containers that we build. The jobs above end up requiring something like 8 container builds - we should try to cut this down to perhaps 2-3 container builds

Quoted 10 lines

Generally speaking, this makes sense; but I think we have different situations which need different kinds of testing, and we should make it as easy as possible to run the right set of tests.

(a) an individual contributor is doing some changes

In that case, I assume (hope?) that the contributor has actually compiled the code for the relevant targets and has done some manual testing. Running acceptance tests locally would also be good, or things like iotests or check-tcg, when applicable.

With my contributor hat on, I like GitLab CI to validate the platforms I always forget. Changes I do are 95% tested on Fedora x86_64. I have often broken stuff for non-Linux builds (Windows in particular), or have broken non-x86_64 target arches. CI lets me see this before sending patches. Unfortunately this means I benefit most from the "full" set, but this won't be sustainable with limited CI minutes :-(

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

Quoted 11 lines

(b) a subsystem maintainer is queuing some changes

Some more comprehensive automated testing, and likely some subsystem-specific testing on top (like testing that requires specialized hardware, manual interventions, etc.) Since the introduction of gitlab-ci, I had relied on what you call the 'full' set to be run automatically when I push to my staging branch, plus my manual testing. Prior to that, I relied on patchew and ran tests locally. Drawbacks of that: it hogs my development machine, and it is easy to forget something.

Quoted 3 lines

If I pull from a sub-submaintainer, I assume that everything is already in good order, and only do some very light sanity/integration checking. Although we're hardly doing subsystem pull request nowadays.

Yes, I think that 'minimal' ought to be a good "smoke test" to run when queuing changes, on the assumption you'll run a full/gating set before sending the final pull.

Quoted 5 lines

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

Quoted 3 lines

Couple of ideas from my side:
- include some scripts/make targets that guide an individual contributor to run a good subset of automated tests locally

We've got various make targets right now, but they don't guide users as to what is the most beneficial thing to run.

Quoted 6 lines

- define a 'ci-subsystem' set that covers usual pain points for a subsystem
- have subsystem maintainers run a pull req test in the qemu-project context (i.e. using extra CI minutes that the project may have), or put them on a special list on subsystem maintainers so they can use more minutes

I think ultimately we should be looking to take email out of the loop for merging pull requests from subsys maintainers.

If we expect subsys maintainers to use gitlab CI before sending, then we have a crazy situation where subsys maintainers pushes to gitlab, has CI run, then send email PULL to list, then Peter downloads and applies the mails, and pushes back to gitlab and runs CI again and then pushes to gitlab again for master.

It would be much simpler if the subsys maintainer pushes to gitlab and opens a merge request. This would trigger the CI gating runs, consuming minutes from the QEMU project, not the maintainer. Peter can ignore merge request until CI passes. The subsys maintainer is responsible for getting CI to pass. Once CI passes, the merge request could be apporved and merged. The person approving it for merge should really only need to validate the the non-automatable stuff. eg it meets the current freeze rules; the maintainer is the owner of the subsystem being modified, etc.

Of course this relies on us being able to use GitLab for 100% of merge time gating. Cleber's custom runners recently enabled get us closer, but I think Peter still uses some other hardware outside of GitLab for some testing.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 26 lines · Daniel P. Berrangé
Quoted 12 lines · Cornelia Huck
Quoted 11 lines · Daniel P. Berrangé

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

Quoted 6 lines
Quoted 3 lines

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

Where do you think we should start to cut them down? Limit the set of tested arch targets to the most common ones?

Some of our targets are obviously much more important and frequently changed than others. For contributors our goal is to mimimize breakage after patches are submitted. Most of our contributors changes will be well covered by x86-64 + aarch64 alone. Other targets give varying degrees of extra benefit.

Quoted 8 lines

On the other hand the contributors are likely to have tested x86_64 or aarch64 themselves since that'll be their dev platform. So the benefit of CI is testing bits that they didnt bother to test.

No clear easy answer here, but I feel like we could benefit from classifying our target archs tier 1/2/3 and tailoring our default testing matrix accordingly.

[*]

Quoted 16 lines

The other way to cut down the "minimal" set is to reduce the OS containers that we build. The jobs above end up requiring something like 8 container builds - we should try to cut this down to perhaps 2-3 container builds

Quoted 10 lines

Generally speaking, this makes sense; but I think we have different situations which need different kinds of testing, and we should make it as easy as possible to run the right set of tests.

(a) an individual contributor is doing some changes

In that case, I assume (hope?) that the contributor has actually compiled the code for the relevant targets and has done some manual testing. Running acceptance tests locally would also be good, or things like iotests or check-tcg, when applicable.

Quoted 6 lines

With my contributor hat on, I like GitLab CI to validate the platforms I always forget. Changes I do are 95% tested on Fedora x86_64. I have often broken stuff for non-Linux builds (Windows in particular), or have broken non-x86_64 target arches. CI lets me see this before sending patches. Unfortunately this means I benefit most from the "full" set, but this won't be sustainable with limited CI minutes :-(

Hmmm somehow. What we said 2 years ago was "if a vendor is interested in have QEMU working on a target, it should provide hardware to the project, otherwise it can't be considered a tier 1 platform" [*]. Some did, but we have been wasting their resources doing almost nothing, mostly because we were not ready / organized to use them. Hopefully this might change soon.

Quoted 3 lines

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

Which is why I stopped doing gitlab-related patches and reduced my testing (or even worst for me, before I was simply doing 'git-push' and wait for the pipeline result, now I'm back running 6 different scripts locally).

Quoted 10 lines
Quoted 5 lines

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

"run" or "pass"?

Quoted 4 lines

Of course this relies on us being able to use GitLab for 100% of merge time gating. Cleber's custom runners recently enabled get us closer, but I think Peter still uses some other hardware outside of GitLab for some testing.

There is hope!

#8 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Daniel P. Berrangé
Quoted 5 lines

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Quoted 6 lines
Quoted 4 lines

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

"run" or "pass"?

In 'full' set the acceptance test jobs are marked to allow to be triggered manually and won't affect pipeline status if they fail

In 'gating' set the acceptance test jobs always run and are required to pass.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 13 lines · Daniel P. Berrangé
Quoted 12 lines · Philippe Mathieu-Daudé
Quoted 10 lines · Daniel P. Berrangé
Quoted 5 lines

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

Quoted 4 lines
Quoted 1 line

"run" or "pass"?

In 'full' set the acceptance test jobs are marked to allow to be triggered manually and won't affect pipeline status if they fail

I'd expect 'full' to *run* all tests, having acceptance tests tagged with 'allow_failure', not 'manual', but I realize this is a waste of resources. OK then.

Quoted 7 lines

In 'gating' set the acceptance test jobs always run and are required to pass.

Regards,
Daniel

#10 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

In reply to Daniel P. Berrangé
Quoted 20 lines · Daniel P. Berrangé
Quoted 19 lines · Cornelia Huck
Quoted 17 lines · Daniel P. Berrangé

Currently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.

In addition, with the expected change in GitLab to set a finite CI minute allowance on all users we need to be much more conservative.

This patch thus sets up rules so that no CI jobs will run by default on push, except

- Pushes to branch name prefix 'staging' on 'qemu-project' repo run full set of jobs
- Pushes to 'master' branch, only run jobs needed to publish the website.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

In any pushes to users forks, CI is now strictly opt-in. The pipeline will always be created by every singe job will be marked manual. You can then use the web UI to start individual jobs.

Is a random user notified in some way that a pipeline with jobs that can be triggered manually has been created?

I don't believe there's any notification.

So a random user might be entirely unaware that they could trigger some jobs. OTOH, they already consciously pushed to gitlab...

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

Quoted 8 lines
Quoted 2 lines

Where do you think we should start to cut them down? Limit the set of tested arch targets to the most common ones?

Some of our targets are obviously much more important and frequently changed than others. For contributors our goal is to mimimize breakage after patches are submitted. Most of our contributors changes will be well covered by x86-64 + aarch64 alone. Other targets give varying degrees of extra benefit.

I'd probably add s390x to that list, not just because of personal bias :), but also because it has a unique set of devices, is big endian, and has been broken in the past.

Quoted 5 lines

On the other hand the contributors are likely to have tested x86_64 or aarch64 themselves since that'll be their dev platform. So the benefit of CI is testing bits that they didnt bother to test.

Contributors are unlikely to have tested s390x unless they explicitly develop for that target, so that might be a net benefit.

Quoted 4 lines

No clear easy answer here, but I feel like we could benefit from classifying our target archs tier 1/2/3 and tailoring our default testing matrix accordingly.

Nod, that seems useful in any case.

Quoted 5 lines

The other way to cut down the "minimal" set is to reduce the OS containers that we build. The jobs above end up requiring something like 8 container builds - we should try to cut this down to perhaps 2-3 container builds

Yes, that is also what usually takes ages.

Quoted 12 lines
Quoted 11 lines

Generally speaking, this makes sense; but I think we have different situations which need different kinds of testing, and we should make it as easy as possible to run the right set of tests.

(a) an individual contributor is doing some changes

In that case, I assume (hope?) that the contributor has actually compiled the code for the relevant targets and has done some manual testing. Running acceptance tests locally would also be good, or things like iotests or check-tcg, when applicable.

Quoted 6 lines

With my contributor hat on, I like GitLab CI to validate the platforms I always forget. Changes I do are 95% tested on Fedora x86_64. I have often broken stuff for non-Linux builds (Windows in particular), or have broken non-x86_64 target arches. CI lets me see this before sending patches. Unfortunately this means I benefit most from the "full" set, but this won't be sustainable with limited CI minutes :-(

Agreed :(

Quoted 4 lines

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

I hope that will not make gitlab CI a complete non-starter -- if you cannot easily debug a test case that is failing, it's mostly useless. We've seen too many cases where a failure could not be reproduced when the test case was running locally.

Quoted 22 lines
Quoted 12 lines

(b) a subsystem maintainer is queuing some changes

Some more comprehensive automated testing, and likely some subsystem-specific testing on top (like testing that requires specialized hardware, manual interventions, etc.) Since the introduction of gitlab-ci, I had relied on what you call the 'full' set to be run automatically when I push to my staging branch, plus my manual testing. Prior to that, I relied on patchew and ran tests locally. Drawbacks of that: it hogs my development machine, and it is easy to forget something.

Quoted 3 lines

If I pull from a sub-submaintainer, I assume that everything is already in good order, and only do some very light sanity/integration checking. Although we're hardly doing subsystem pull request nowadays.

Yes, I think that 'minimal' ought to be a good "smoke test" to run when queuing changes, on the assumption you'll run a full/gating set before sending the final pull.

Quoted 2 lines

(c) a subsystem maintainer is preparing a pull request

Quoted 8 lines
Quoted 3 lines

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

I can at least run a subset of the acceptance tests locally, but I think I may be missing some platforms? Still, better than nothing.

Quoted 7 lines
Quoted 4 lines

Couple of ideas from my side:
- include some scripts/make targets that guide an individual contributor to run a good subset of automated tests locally

We've got various make targets right now, but they don't guide users as to what is the most beneficial thing to run.

There's also the question what the most beneficial subset actually is. "Run make check-tcg when you are doing tcg changes" is an unfortunately rare obvious case.

Quoted 8 lines
Quoted 7 lines

- define a 'ci-subsystem' set that covers usual pain points for a subsystem
- have subsystem maintainers run a pull req test in the qemu-project context (i.e. using extra CI minutes that the project may have), or put them on a special list on subsystem maintainers so they can use more minutes

Quoted 8 lines

I think ultimately we should be looking to take email out of the loop for merging pull requests from subsys maintainers.

If we expect subsys maintainers to use gitlab CI before sending, then we have a crazy situation where subsys maintainers pushes to gitlab, has CI run, then send email PULL to list, then Peter downloads and applies the mails, and pushes back to gitlab and runs CI again and then pushes to gitlab again for master.

Hm, I thought Peter pulled the subsys maintainers' signed tags?

[As a side remark, I recommend using b4 for applying patches. The only advantage for me when pulling vs applying with b4 is the usage of signed tags. Not that I needed to pull anything recently :)]

Quoted 10 lines

It would be much simpler if the subsys maintainer pushes to gitlab and opens a merge request. This would trigger the CI gating runs, consuming minutes from the QEMU project, not the maintainer. Peter can ignore merge request until CI passes. The subsys maintainer is responsible for getting CI to pass. Once CI passes, the merge request could be apporved and merged. The person approving it for merge should really only need to validate the the non-automatable stuff. eg it meets the current freeze rules; the maintainer is the owner of the subsystem being modified, etc.

I hate gitlab MR tooling. (Same for any forge. I'm contemplating using magit/forge to at least avoid a medium break; anyone been using it sucessfully?) At the very least, we should have notifications about pull requests and when they are merged on the mailing list, just to keep information about development in one page.

Quoted 5 lines

Of course this relies on us being able to use GitLab for 100% of merge time gating. Cleber's custom runners recently enabled get us closer, but I think Peter still uses some other hardware outside of GitLab for some testing.

#11 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 28 lines · Cornelia Huck
Quoted 17 lines · Daniel P. Berrangé
Quoted 16 lines · Cornelia Huck
Quoted 14 lines · Daniel P. Berrangé

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

Quoted 8 lines
Quoted 2 lines

Where do you think we should start to cut them down? Limit the set of tested arch targets to the most common ones?

Some of our targets are obviously much more important and frequently changed than others. For contributors our goal is to mimimize breakage after patches are submitted. Most of our contributors changes will be well covered by x86-64 + aarch64 alone. Other targets give varying degrees of extra benefit.

Quoted 3 lines

I'd probably add s390x to that list, not just because of personal bias :), but also because it has a unique set of devices, is big endian, and has been broken in the past.

The desire to have something covering big endian is why I included the s390x cross-build in the "minimal" set. Even if we don't run the tests, building it is still worthwhile.

Quoted 8 lines
Quoted 3 lines

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

I hope that will not make gitlab CI a complete non-starter -- if you cannot easily debug a test case that is failing, it's mostly useless. We've seen too many cases where a failure could not be reproduced when the test case was running locally.

One of the best things about GitLab compared to what we had with Travis is that the build environment is 100% container based (until Cleber's custom runners arrived). This meant that when something does fail in GitLab, you can pull the container image down from the gitlab registry and run the build locally. You still have differences due to hardware or CPUs resources, but its a hell of alot easier to reproduce than before. This is good enough for most contributors in general, but not for the CI maintainers, who need to debug especially the CI system as it exists on GitLab.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Quoted 7 lines
Quoted 4 lines

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

I can at least run a subset of the acceptance tests locally, but I think I may be missing some platforms? Still, better than nothing.

As ever the big problem for most people is non-x86_64 platforms. The custom runners solve this for gitlab, and in theory people can deploy a VM using QEMU TCG to do this locally, but massively slower

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

- define a 'ci-subsystem' set that covers usual pain points for a subsystem
- have subsystem maintainers run a pull req test in the qemu-project context (i.e. using extra CI minutes that the project may have), or put them on a special list on subsystem maintainers so they can use more minutes

I think ultimately we should be looking to take email out of the loop for merging pull requests from subsys maintainers.

Quoted 7 lines
Quoted 5 lines

If we expect subsys maintainers to use gitlab CI before sending, then we have a crazy situation where subsys maintainers pushes to gitlab, has CI run, then send email PULL to list, then Peter downloads and applies the mails, and pushes back to gitlab and runs CI again and then pushes to gitlab again for master.

Hm, I thought Peter pulled the subsys maintainers' signed tags?

We could have a CI job that validates the merge request is associated with a signed tag whose key is on a designated keyring containing the known maintainers.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#12 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 12 lines · Daniel P. Berrangé
Quoted 10 lines · Cornelia Huck
Quoted 3 lines · Daniel P. Berrangé

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

I hope that will not make gitlab CI a complete non-starter -- if you cannot easily debug a test case that is failing, it's mostly useless. We've seen too many cases where a failure could not be reproduced when the test case was running locally.

Quoted 9 lines

One of the best things about GitLab compared to what we had with Travis is that the build environment is 100% container based (until Cleber's custom runners arrived). This meant that when something does fail in GitLab, you can pull the container image down from the gitlab registry and run the build locally. You still have differences due to hardware or CPUs resources, but its a hell of alot easier to reproduce than before. This is good enough for most contributors in general, but not for the CI maintainers, who need to debug especially the CI system as it exists on GitLab.

Correct me if I'm wrong, but I remember that some of the more aggravating failures in particular could not be reproduced outside of the gitlab environment, even while using the same image.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces of the pull request; plus some subsystem-specific manual testing on top. In practice, the 'full' set might be good enough.

Yeah, the full/gating set is what I would thing subsys maintainers would want to use, to minimize risk that Peter's tests throw back the merge due to failure. The only difference of gating vs full is whether the acceptance tests run.

Quoted 6 lines
Quoted 2 lines

I can at least run a subset of the acceptance tests locally, but I think I may be missing some platforms? Still, better than nothing.

As ever the big problem for most people is non-x86_64 platforms. The custom runners solve this for gitlab, and in theory people can deploy a VM using QEMU TCG to do this locally, but massively slower

Still, the acceptance tests are usually small enough that running under tcg is bearable.

#13 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 24 lines · Cornelia Huck
Quoted 12 lines · Daniel P. Berrangé
Quoted 10 lines · Cornelia Huck
Quoted 3 lines · Daniel P. Berrangé

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

I hope that will not make gitlab CI a complete non-starter -- if you cannot easily debug a test case that is failing, it's mostly useless. We've seen too many cases where a failure could not be reproduced when the test case was running locally.

Quoted 9 lines

One of the best things about GitLab compared to what we had with Travis is that the build environment is 100% container based (until Cleber's custom runners arrived). This meant that when something does fail in GitLab, you can pull the container image down from the gitlab registry and run the build locally. You still have differences due to hardware or CPUs resources, but its a hell of alot easier to reproduce than before. This is good enough for most contributors in general, but not for the CI maintainers, who need to debug especially the CI system as it exists on GitLab.

Quoted 3 lines

Correct me if I'm wrong, but I remember that some of the more aggravating failures in particular could not be reproduced outside of the gitlab environment, even while using the same image.

There will always be some like that for sure. Some problems are very sensitive to timing issues that are affected by load or CPUs parallism, etc.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

In reply to Daniel P. Berrangé
Quoted 13 lines · Daniel P. Berrangé
Quoted 11 lines · Cornelia Huck
Quoted 5 lines · Daniel P. Berrangé
Quoted 1 line · Daniel P. Berrangé

When I'm working on changing gitlab CI rules, then I burn loads of minutes which is especially troubling - limited CI minutes will make it very hard for me to debug future CI rule changes :-(

I hope that will not make gitlab CI a complete non-starter -- if you cannot easily debug a test case that is failing, it's mostly useless. We've seen too many cases where a failure could not be reproduced when the test case was running locally.

Quoted 9 lines

One of the best things about GitLab compared to what we had with Travis is that the build environment is 100% container based (until Cleber's custom runners arrived). This meant that when something does fail in GitLab, you can pull the container image down from the gitlab registry and run the build locally. You still have differences due to hardware or CPUs resources, but its a hell of alot easier to reproduce than before. This is good enough for most contributors in general, but not for the CI maintainers, who need to debug especially the CI system as it exists on GitLab.

FWIW we could do that with Travis-CI too using:

 $ make docker-test-build@travis
#15 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

In reply to Daniel P. Berrangé

(meta note: patch does not apply anymore and needs to be refreshed)

On 12/08/2021 20.04, Daniel P. Berrangé wrote:
[...]

Quoted 23 lines
.gitlab-ci.d/rules.yml new file +17 -0
new file mode 100644
index 0000000000..3399722ca9
@@ -0,0 +1,116 @@
+
+# This defines rules used to control individual job execution
+# See docs/devel/ci-jobs.rst for an explanation of the various
+# variables and branch naming conventions applied here.
+#
+.job_rules:
+  rules:
+    # ======================================================================
+    # Rules that apply regardless of whether the primary qemu repo or a fork
+    # ======================================================================
+
+    # Skip any cirrus job if either repo or api token are missing
+    # as we won't be able to talk to cirrus
+    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
+      when: never
+
+    # Any jobs marked as manual, never get automatically run in any scenario

.gitlab-ci.d/rules.yml line 27

Quoted 10 lines
.gitlab-ci.d/rules.yml new file +10 -0
+    # and don't contribute to pipeline status
+    - if: '$QEMU_JOB_MANUAL'
+      when: manual
+      allow_failure: true
+
+    # For the main repo, tags represent official releases.
+    # The branch associated with the release will have passed
+    # a CI pipeline already
+    #
+    # For user forks, tags are tyically added by tools like

s/tyically/typically/

.gitlab-ci.d/rules.yml line 33

Quoted 6 lines
.gitlab-ci.d/rules.yml new file +6 -0
+    # git-publish at the same time as pushing the branch prior
+    # to sending out for review
+    #
+    # In neither case do we wish to run CI pipelines for tags
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
+      when: never

Not sure whether I like this rule ... First, tags are very seldom compared to normal pushes to branches, so this should not affect us that much. Second, I think it might be good for "documentation" purposes to be able to say that the CI ran properly for a certain tag. Ok, you could still look it up in the push to a branch that might have happened before, but that's cumbersome. Just my 0.02 €.

Quoted 13 lines
.gitlab-ci.d/rules.yml new file +13 -0
+
+    # ====================================
+    # Rules for running jobs in user forks
+    # ====================================
+
+    # Part 1: gating jobs
+    # -------------------
+
+    # If on a branch with name prefix 'ci-acceptance-', then run
+    # everything, just as a gating job on 'staging' branch would
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
+      when: on_success
+

.gitlab-ci.d/rules.yml line 83

Quoted 10 lines
.gitlab-ci.d/rules.yml new file +10 -0
+    # If user set QEMU_CI_GATING=1, then run everything just as
+    # a gating job on 'staging' branch would
+    - if: '$QEMU_CI_GATING'
+      when: on_success
+
+    # Otherwise never run jobs marked as gating, but allow manual trigger
+    # without affecting pipeline status
+    - if: '$QEMU_JOB_GATING'
+      when: manual
+      allow_failure: true

IMHO if it's "gating", then we should not use "allow_failure: true", no matter whether the job is manual or not. Otherwise this is very confusing.

I guess the problem is rather whether the "acceptance" tests should be called "gating" or not. Currently they are in a rather gray area in the gitlab-CI...

.gitlab-ci.d/rules.yml line 96

Quoted 12 lines
.gitlab-ci.d/rules.yml new file +12 -0
+
+    # =============================================
+    # Part 2: opt-in for all CI, except gating jobs
+    # =============================================
+
+    # If pushing to a branch with name prefix 'ci-all', then run all jobs
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
+      when: on_success
+
+    # If user QEMU_CI_ALL=1, then run all jobs
+    - if: '$QEMU_CI_ALL'
+      when: on_success

Uh, so "all" is not running all jobs? ... that's confusing, too. Could you come up with some better naming? QEMU_CI_CORE maybe?

[...]

.gitlab-ci.d/rules.yml

Quoted 7 lines
.gitlab-ci.d/rules.yml new file +4 -1
+Set variable manually when pushing to git
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
  Variables can be set manually when pushing a branch or tag, using
-git-push command line arguments.
+git-push command line arguments. This is useful for fine tuning the
+jobs on an adhoc basis.

s/adhoc/ad hoc/ (I think)

Thomas

#16 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 48 lines · Thomas Huth

(meta note: patch does not apply anymore and needs to be refreshed)

On 12/08/2021 20.04, Daniel P. Berrangé wrote:
[...]

Quoted 23 lines
.gitlab-ci.d/rules.yml new file +17 -0
new file mode 100644
index 0000000000..3399722ca9
@@ -0,0 +1,116 @@
+
+# This defines rules used to control individual job execution
+# See docs/devel/ci-jobs.rst for an explanation of the various
+# variables and branch naming conventions applied here.
+#
+.job_rules:
+  rules:
+    # ======================================================================
+    # Rules that apply regardless of whether the primary qemu repo or a fork
+    # ======================================================================
+
+    # Skip any cirrus job if either repo or api token are missing
+    # as we won't be able to talk to cirrus
+    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
+      when: never
+
+    # Any jobs marked as manual, never get automatically run in any scenario

.gitlab-ci.d/rules.yml line 27

Quoted 10 lines
.gitlab-ci.d/rules.yml new file +10 -0
+    # and don't contribute to pipeline status
+    - if: '$QEMU_JOB_MANUAL'
+      when: manual
+      allow_failure: true
+
+    # For the main repo, tags represent official releases.
+    # The branch associated with the release will have passed
+    # a CI pipeline already
+    #
+    # For user forks, tags are tyically added by tools like

s/tyically/typically/

.gitlab-ci.d/rules.yml line 33

Quoted 6 lines
.gitlab-ci.d/rules.yml new file +6 -0
+    # git-publish at the same time as pushing the branch prior
+    # to sending out for review
+    #
+    # In neither case do we wish to run CI pipelines for tags
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
+      when: never
Quoted 6 lines

Not sure whether I like this rule ... First, tags are very seldom compared to normal pushes to branches, so this should not affect us that much. Second, I think it might be good for "documentation" purposes to be able to say that the CI ran properly for a certain tag. Ok, you could still look it up in the push to a branch that might have happened before, but that's cumbersome. Just my 0.02 €.

"git-publish" creates tags for every version and pushes them to your repo when you use --pull arg. I don't want jobs triggered when git-publish pushes tags, because I will have already tested the code before I ask git-publish to send the pull. So IMHO skipping CI pipelines on forks is important to avoid git-publish burning everyone's CI minutes.

In terms of the upstream repo, a tag is only pushed when the changes for the release have already been pushed. Those changes would have undergone CI already. There's no point running CI again for the tag, especially if no one is going to do anything with failures it might report.

Quoted 17 lines
Quoted 17 lines
.gitlab-ci.d/rules.yml new file +17 -0
+    # ====================================
+    # Rules for running jobs in user forks
+    # ====================================
+
+    # Part 1: gating jobs
+    # -------------------
+
+    # If on a branch with name prefix 'ci-acceptance-', then run
+    # everything, just as a gating job on 'staging' branch would
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
+      when: on_success
+
+    # If user set QEMU_CI_GATING=1, then run everything just as
+    # a gating job on 'staging' branch would
+    - if: '$QEMU_CI_GATING'
+      when: on_success
+
Quoted 8 lines

.gitlab-ci.d/rules.yml line 83

Quoted 5 lines
.gitlab-ci.d/rules.yml new file +5 -0
+    # Otherwise never run jobs marked as gating, but allow manual trigger
+    # without affecting pipeline status
+    - if: '$QEMU_JOB_GATING'
+      when: manual
+      allow_failure: true

IMHO if it's "gating", then we should not use "allow_failure: true", no matter whether the job is manual or not. Otherwise this is very confusing.

I think you're misinterpreting this.

Gating jobs run on the upstream pushes to 'staging' (not shown in this quoted context - its higher in rules.yml), or if you have pushed to a branch 'ci-gating-' (the first rule here), or if you set QEMU_CI_GATING env var when pushing (the second rule here).

This 3rd rule is about ensuring gating jobs do NOT get launched in any other scenario. We could use 'when: never' but that's a big hammer that prevents users to opt-ing in to running a single job from the web UI, so we use 'when: manual'. If you use 'when: manual' on its own, the pipeline will never complete, as it'll be waiting for someone to trigger the manual job which is not what we need. 'allow_failure: true' means that the pipeline will complete without waiting for the manual job.

Ideally there would be a way to say "Let this job be manually run, and make its failure affect the pipeline status, but ignore the job if not ru nmanually". GitLab doesn't support that AFAIK though, so this is the next best option, that isn't 'when: never'.

Quoted 7 lines
Quoted 7 lines
.gitlab-ci.d/rules.yml new file +7 -0
+
+    # =============================================
+    # Part 2: opt-in for all CI, except gating jobs
+    # =============================================
+
+    # If pushing to a branch with name prefix 'ci-all', then run all jobs
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
Quoted 8 lines

.gitlab-ci.d/rules.yml line 96

Quoted 5 lines
.gitlab-ci.d/rules.yml new file +5 -0
+      when: on_success
+
+    # If user QEMU_CI_ALL=1, then run all jobs
+    - if: '$QEMU_CI_ALL'
+      when: on_success

Uh, so "all" is not running all jobs? ... that's confusing, too. Could you come up with some better naming? QEMU_CI_CORE maybe?

"all" is running everything that currently runs today when you do a git push in a fork. I admit it is odd that 'gating' runs more than 'all'.

We could change it to "default" instead.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#17 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

In reply to Daniel P. Berrangé
Quoted 5 lines · Daniel P. Berrangé

Currently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.

Hi Daniel!

Now that gitlab introduces limits for the CI for all users, I think we should revisit this topic ... could you maybe respin your patch to the current git master branch?

Thomas

Quoted 865 lines

In addition, with the expected change in GitLab to set a finite CI minute allowance on all users we need to be much more conservative.

This patch thus sets up rules so that no CI jobs will run by default on push, except

- Pushes to branch name prefix 'staging' on 'qemu-project' repo run full set of jobs
- Pushes to 'master' branch, only run jobs needed to publish the website.

In any pushes to users forks, CI is now strictly opt-in. The pipeline will always be created by every singe job will be marked manual. You can then use the web UI to start individual jobs.

For more convenience there are three levels of increasing comprehensive CI job sets that can be chosen between

- Gating - run exactly the same as pushes to 'staging' branch (except for jobs needing custom runners)

Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1

- Full - same as 'Gating', except that acceptance tests don't get run. This is equivalent to historical CI behaviour for pushes to user forks.

Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1

- Minimal - a significantly cut down set of jobs to get a decent sanity check of builds without burning massive amounts of CI time.

Push to 'ci-min-xxx' branch, or set env QEMU_CI=1

The minimal job set covers:

* Fedora, CentOS, Ubuntu & Debian system emulator builds to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc

This gives about 30 jobs instead of 120 from the "Full" group. It is likely reasonable to cut the minimal set down even more, as IMHO there are too many system emulator jobs there.

This is all controlled by defining a set of 'rules' entries that apply to (almost) all jobs in the CI pipeline. The exceptions are those using custom runners, and the firmware build jobs.

Using inherited templates together with rules, doesn't work in the way you might expect. Any rules defined in the child completely replace those from the parent, rather than augmenting them. Thus we need to avoid having rules defined in any jobs at all, by inheriting from a parent job template that contains the rules. The characteristics of individual jobs are then defined by variables set against the job. This is described in the updated content for the docs/devel/ci-jobs.rst file.

---
  .gitlab-ci.d/buildtest-template.yml  |  17 +--
  .gitlab-ci.d/buildtest.yml           |  43 ++++----
  .gitlab-ci.d/cirrus.yml              |   6 +-
  .gitlab-ci.d/container-core.yml      |   4 +
  .gitlab-ci.d/container-cross.yml     |   9 +-
  .gitlab-ci.d/container-template.yml  |   1 +
  .gitlab-ci.d/containers.yml          |   4 +
  .gitlab-ci.d/crossbuild-template.yml |   3 +
  .gitlab-ci.d/crossbuilds.yml         |   3 +
  .gitlab-ci.d/qemu-project.yml        |   1 +
  .gitlab-ci.d/rules.yml               | 116 ++++++++++++++++++++
  .gitlab-ci.d/static_checks.yml       |  18 ++--
  .gitlab-ci.yml                       |   4 -
  docs/devel/ci-jobs.rst               | 155 ++++++++++++++++++++++++---
  14 files changed, 317 insertions(+), 67 deletions(-)

create mode 100644 .gitlab-ci.d/rules.yml

.gitlab-ci.d/buildtest-template.yml +5 -12
index fcbcc4e627..8578b3ed11 100644
@@ -1,4 +1,6 @@
+
  .native_build_job_template:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    before_script:
@@ -27,6 +29,7 @@
        fi
 
  .native_test_job_template:
+  extends: .job_rules
    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
@@ -67,15 +70,5 @@
    after_script:
      - cd build
      - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_GATING: 1
.gitlab-ci.d/buildtest.yml +24 -19
index 903ee65f32..c8e03373c5 100644
@@ -40,6 +40,7 @@ build-system-ubuntu:
    needs:
      job: amd64-ubuntu2004-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: ubuntu2004
      CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -56,6 +57,7 @@ check-system-ubuntu:
      - job: build-system-ubuntu
        artifacts: true
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: ubuntu2004
      MAKE_CHECK_ARGS: check
 
@@ -73,6 +75,7 @@ build-system-debian:
    needs:
      job: amd64-debian-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: debian-amd64
      CONFIGURE_ARGS: --enable-fdt=system
      TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -89,6 +92,7 @@ check-system-debian:
      - job: build-system-debian
        artifacts: true
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: debian-amd64
      MAKE_CHECK_ARGS: check
 
@@ -106,6 +110,7 @@ build-system-fedora:
    needs:
      job: amd64-fedora-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: fedora
      CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
               --enable-fdt=system --enable-slirp=system --enable-capstone=system
@@ -123,6 +128,7 @@ check-system-fedora:
      - job: build-system-fedora
        artifacts: true
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: fedora
      MAKE_CHECK_ARGS: check
 
@@ -140,6 +146,7 @@ build-system-centos:
    needs:
      job: amd64-centos8-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: centos8
      CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
                      --enable-modules --enable-trace-backends=dtrace
@@ -157,6 +164,7 @@ check-system-centos:
      - job: build-system-centos
        artifacts: true
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: centos8
      MAKE_CHECK_ARGS: check
 
@@ -325,6 +333,7 @@ build-user-static:
    needs:
      job: amd64-debian-user-cross-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: debian-all-test-cross
      CONFIGURE_ARGS: --disable-tools --disable-system --static
      MAKE_CHECK_ARGS: check-tcg
@@ -411,17 +420,15 @@ build-cfi-aarch64:
        --enable-safe-stack --enable-slirp=git
      TARGETS: aarch64-softmmu
      MAKE_CHECK_ARGS: check-build
+    # FIXME: This job is often failing, likely due to out-of-memory problems in
+    # the constrained containers of the shared runners. Thus this is marked as
+    # manual until the situation has been solved.
+    QEMU_JOB_MANUAL: 1
    timeout: 70m
    artifacts:
      expire_in: 2 days
      paths:
        - build
-  rules:
-    # FIXME: This job is often failing, likely due to out-of-memory problems in
-    # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
 
  check-cfi-aarch64:
    extends: .native_test_job_template
@@ -453,17 +460,15 @@ build-cfi-ppc64-s390x:
        --enable-safe-stack --enable-slirp=git
      TARGETS: ppc64-softmmu s390x-softmmu
      MAKE_CHECK_ARGS: check-build
+    # FIXME: This job is often failing, likely due to out-of-memory problems in
+    # the constrained containers of the shared runners. Thus this is marked as
+    # manual until the situation has been solved.
+    QEMU_JOB_MANUAL: 1
    timeout: 70m
    artifacts:
      expire_in: 2 days
      paths:
        - build
-  rules:
-    # FIXME: This job is often failing, likely due to out-of-memory problems in
-    # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
 
  check-cfi-ppc64-s390x:
    extends: .native_test_job_template
@@ -657,6 +662,7 @@ build-without-default-features:
      MAKE_CHECK_ARGS: check-unit
 
  build-libvhost-user:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
    needs:
@@ -676,6 +682,8 @@ build-tools-and-docs-debian:
    needs:
      job: amd64-debian-container
    variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
      IMAGE: debian-amd64
      MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
      CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
@@ -698,10 +706,14 @@ build-tools-and-docs-debian:
  # that users can see the results of their commits, regardless
  # of what topic branch they're currently using
  pages:
+  extends: .native_build_job_template
    image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
    stage: test
    needs:
      - job: build-tools-and-docs-debian
+  variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
    script:
      - mkdir -p public
      # HTML-ised source tree
@@ -715,10 +727,3 @@ pages:
    artifacts:
      paths:
        - public
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: on_success
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: on_success
.gitlab-ci.d/cirrus.yml +4 -2
index 675db69622..4334736796 100644
@@ -11,6 +11,7 @@
  # special care, because we can't just override it at the GitLab CI job
  # definition level or we risk breaking it completely.
  .cirrus_build_job:
+  extends: .job_rules
    stage: build
    image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
    needs: []
@@ -39,8 +40,6 @@
        <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
      - cat .gitlab-ci.d/cirrus/$NAME.yml
      - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
 
  x64-freebsd-12-build:
    extends: .cirrus_build_job
@@ -61,6 +60,8 @@ x64-freebsd-12-build:
  x64-freebsd-13-build:
    extends: .cirrus_build_job
    variables:
+    QEMU_JOB_SOFT_FAIL: 1
+    QEMU_JOB_MINIMAL: 1
      NAME: freebsd-13
      CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
      CIRRUS_VM_IMAGE_SELECTOR: image_family
@@ -74,6 +75,7 @@ x64-freebsd-13-build:
  x64-macos-11-base-build:
    extends: .cirrus_build_job
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: macos-11
      CIRRUS_VM_INSTANCE_TYPE: osx_instance
      CIRRUS_VM_IMAGE_SELECTOR: image
.gitlab-ci.d/container-core.yml +4 -0
index e8dd1f476a..8983c32b3b 100644
@@ -4,14 +4,18 @@ include:
  amd64-centos8-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: centos8
 
  amd64-fedora-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: fedora
 
  amd64-debian10-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
      NAME: debian10
.gitlab-ci.d/container-cross.yml +5 -4
index 0fcebe363a..ef316668eb 100644
@@ -50,13 +50,11 @@ armhf-debian-cross-container:
  # We never want to build hexagon in the CI system and by default we
  # always want to refer to the master registry where it lives.
  hexagon-cross-container:
+  extends: .job_rules
    image: docker:stable
    stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
    variables:
+    QEMU_JOB_MANUAL: 1
      NAME: debian-hexagon-cross
      GIT_DEPTH: 1
    services:
@@ -143,6 +141,7 @@ s390x-debian-cross-container:
    stage: containers-layer2
    needs: ['amd64-debian10-container']
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: debian-s390x-cross
 
  sh4-debian-cross-container:
@@ -179,11 +178,13 @@ cris-fedora-cross-container:
  i386-fedora-cross-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: fedora-i386-cross
 
  win32-fedora-cross-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: fedora-win32-cross
 
  win64-fedora-cross-container:
.gitlab-ci.d/container-template.yml +1 -0
index 1baecd9460..e2954e4ed3 100644
@@ -1,4 +1,5 @@
  .container_job_template:
+  extends: .job_rules
    image: docker:stable
    stage: containers
    services:
.gitlab-ci.d/containers.yml +4 -0
index cd06d3f5f4..6948923892 100644
@@ -17,6 +17,8 @@ amd64-debian-container:
    stage: containers-layer2
    needs: ['amd64-debian10-container']
    variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
      NAME: debian-amd64
 
  amd64-ubuntu1804-container:
@@ -27,6 +29,7 @@ amd64-ubuntu1804-container:
  amd64-ubuntu2004-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: ubuntu2004
 
  amd64-ubuntu-container:
@@ -42,4 +45,5 @@ amd64-opensuse-leap-container:
  python-container:
    extends: .container_job_template
    variables:
+    QEMU_JOB_MINIMAL: 1
      NAME: python
.gitlab-ci.d/crossbuild-template.yml +3 -0
index 7d3ad00a1e..997ec02f79 100644
@@ -1,4 +1,5 @@
  .cross_system_build_job:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    timeout: 80m
@@ -23,6 +24,7 @@
  # KVM), and set extra options (such disabling other accelerators) via the
  # $EXTRA_CONFIGURE_OPTS variable.
  .cross_accel_build_job:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    timeout: 30m
@@ -35,6 +37,7 @@
      - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
  .cross_user_build_job:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
.gitlab-ci.d/crossbuilds.yml +3 -0
index 4ff3aa3cfc..9fcd2ac567 100644
@@ -48,6 +48,7 @@ cross-i386-system:
    needs:
      job: i386-fedora-cross-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: fedora-i386-cross
      MAKE_CHECK_ARGS: check-qtest
 
@@ -129,6 +130,7 @@ cross-s390x-system:
    needs:
      job: s390x-debian-cross-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: debian-s390x-cross
 
  cross-s390x-user:
@@ -159,6 +161,7 @@ cross-win32-system:
    needs:
      job: win32-fedora-cross-container
    variables:
+    QEMU_JOB_MINIMAL: 1
      IMAGE: fedora-win32-cross
    artifacts:
      paths:
.gitlab-ci.d/qemu-project.yml +1 -0
index b3d79bc429..7007193fc0 100644
@@ -2,6 +2,7 @@
  # https://gitlab.com/qemu-project/qemu/-/pipelines
 
  include:
+  - local: '/.gitlab-ci.d/rules.yml'
    - local: '/.gitlab-ci.d/stages.yml'
    - local: '/.gitlab-ci.d/edk2.yml'
    - local: '/.gitlab-ci.d/opensbi.yml'
.gitlab-ci.d/rules.yml new file +116 -0
new file mode 100644
index 0000000000..3399722ca9
@@ -0,0 +1,116 @@
+
+# This defines rules used to control individual job execution
+# See docs/devel/ci-jobs.rst for an explanation of the various
+# variables and branch naming conventions applied here.
+#
+.job_rules:
+  rules:
+    # ======================================================================
+    # Rules that apply regardless of whether the primary qemu repo or a fork
+    # ======================================================================
+
+    # Skip any cirrus job if either repo or api token are missing
+    # as we won't be able to talk to cirrus
+    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
+      when: never
+
+    # Any jobs marked as manual, never get automatically run in any scenario
+    # and don't contribute to pipeline status
+    - if: '$QEMU_JOB_MANUAL'
+      when: manual
+      allow_failure: true
+
+    # For the main repo, tags represent official releases.
+    # The branch associated with the release will have passed
+    # a CI pipeline already
+    #
+    # For user forks, tags are tyically added by tools like
+    # git-publish at the same time as pushing the branch prior
+    # to sending out for review
+    #
+    # In neither case do we wish to run CI pipelines for tags
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
+      when: never
+
+
+    # ==================================================
+    # Rules to control what happens in primary qemu repo
+    # ==================================================
+
+    # The 'pages' job must never run on a non-default branch as we don't
+    # want to publish the website from pre-merged content
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_JOB_NAME == "pages"'
+      when: never
+
+    # Run all jobs for main repo staging branch
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == "staging"'
+      when: on_success
+
+    # Run jobs needed to support website publishing on the main branch
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $QEMU_JOB_PUBLISH'
+      when: on_success
+
+    # Prevent jobs from running on pushes to any other branch / tag
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != "staging"'
+      when: never
+
+
+    # Remainining pipeline sources follow the same rules as for fork repos
+
+
+
+    # ====================================
+    # Rules for running jobs in user forks
+    # ====================================
+
+    # Part 1: gating jobs
+    # -------------------
+
+    # If on a branch with name prefix 'ci-acceptance-', then run
+    # everything, just as a gating job on 'staging' branch would
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
+      when: on_success
+
+    # If user set QEMU_CI_GATING=1, then run everything just as
+    # a gating job on 'staging' branch would
+    - if: '$QEMU_CI_GATING'
+      when: on_success
+
+    # Otherwise never run jobs marked as gating, but allow manual trigger
+    # without affecting pipeline status
+    - if: '$QEMU_JOB_GATING'
+      when: manual
+      allow_failure: true
+
+
+    # =============================================
+    # Part 2: opt-in for all CI, except gating jobs
+    # =============================================
+
+    # If pushing to a branch with name prefix 'ci-all', then run all jobs
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
+      when: on_success
+
+    # If user QEMU_CI_ALL=1, then run all jobs
+    - if: '$QEMU_CI_ALL'
+      when: on_success
+
+
+    # ===============================
+    # Part 3: the minimal set of jobs
+    # ===============================
+
+    # If pushing to a branch with name prefix 'ci-min', then run
+    # only a minimal set of jobs
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-min/ && $QEMU_JOB_MINIMAL'
+      when: on_success
+
+    # If user set QEMU_CI=1, then run only a minimal set of jobs
+    - if: '$QEMU_CI && $QEMU_JOB_MINIMAL'
+      when: on_success
+
+    # Any other scenario, let the jobs be manually triggered only.
+    # Set to be non-fatal, so that pending manual jobs don't
+    # affect overall pipeline status
+    - when: manual
+      allow_failure: true
.gitlab-ci.d/static_checks.yml +9 -9
index 96dbd9e310..4ba3e038dd 100644
@@ -1,4 +1,5 @@
  check-patch:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
    needs:
@@ -6,42 +7,41 @@ check-patch:
    script:
      - .gitlab-ci.d/check-patch.py
    variables:
+    QEMU_JOB_MINIMAL: 1
      GIT_DEPTH: 1000
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
-      allow_failure: true
+  allow_failure: true
 
  check-dco:
+  extends: .job_rules
    stage: build
    image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
    needs:
      job: amd64-centos8-container
    script: .gitlab-ci.d/check-dco.py
    variables:
+    QEMU_JOB_MINIMAL: 1
      GIT_DEPTH: 1000
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
 
  check-python-pipenv:
+  extends: .job_rules
    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/python:latest
    script:
      - make -C python check-pipenv
    variables:
+    QEMU_JOB_MINIMAL: 1
      GIT_DEPTH: 1
    needs:
      job: python-container
 
  check-python-tox:
+  extends: .job_rules
    stage: test
    image: $CI_REGISTRY_IMAGE/qemu/python:latest
    script:
      - make -C python check-tox
    variables:
+    QEMU_JOB_MINIMAL: 1
      GIT_DEPTH: 1
      QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
    needs:
.gitlab-ci.yml +0 -4
index 9762dda2ee..d5bd31f294 100644
@@ -15,10 +15,6 @@
  #
  # QEMU CI jobs are based on templates. Some templates provide
  # user-configurable options, modifiable via configuration variables.
-#
-# See https://qemu-project.gitlab.io/qemu/devel/ci.html#custom-ci-cd-variables
-# for more information.
-#
 
  include:
    - local: '/.gitlab-ci.d/qemu-project.yml'
docs/devel/ci-jobs.rst +138 -17
index 9cd9819786..6d3f13d495 100644
@@ -1,22 +1,49 @@
-Custom CI/CD variables
-======================
+CI Job Rules
+============
 
-QEMU CI pipelines can be tuned by setting some CI environment variables.
+The following documents how CI rules control execution of jobs in a pipeline
 
-Set variable globally in the user's CI namespace
-------------------------------------------------
+Job variable naming
+-------------------
 
-Variables can be set globally in the user's CI namespace setting.
+The rules for controlling jobs in a pipeline will be driven by a number of
+variables:
+
+ - ``CI_XXX``, ``GITLAB_XXX`` - variables defined by GitLab:
+
+   https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
+
+ - ``QEMU_JOB_XXX`` - variables defined in the YAML files that express
+   characteristics of the job used to control default behaviour
+
+ - ``QEMU_CI_XXX`` - variables defined by the user that are used to fine
+   tune which jobs are run dynamically
+
+and in some cases based on the branch name prefixes.
+
+Job fine tuning strategies
+--------------------------
+
+Using a combination of the ``QEMU_CI_XXX`` variables and ``ci-``
+branch name prefix, users can fine tune what jobs are run.
+
+Set variable globally in the gitlab repository CI config
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Variables can be set globally in the user's gitlab repository CI config.
+These variables will apply to all pipelines associated with the repository
+thereafter. This is useful for fine tuning the jobs indefinitely.
 
  For further information about how to set these variables, please refer to::
 
    https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
 
-Set variable manually when pushing a branch or tag to the user's repository
----------------------------------------------------------------------------
+Set variable manually when pushing to git
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
  Variables can be set manually when pushing a branch or tag, using
-git-push command line arguments.
+git-push command line arguments. This is useful for fine tuning the
+jobs on an adhoc basis.
 
  Example setting the QEMU_CI_EXAMPLE_VAR variable:
 
@@ -28,13 +55,107 @@ For further information about how to set these variables, please refer to::
 
    https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
 
-Here is a list of the most used variables:
+Pushing to specially named branches
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Branch names starting with a 'ci-' prefix can be used as an alternative
+to setting variables. Details of supported branch names are detailed
+later in this document.
+
+Provide a custom gitlab CI configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above strategies all provide a way to fine tune the jobs defined by the
+standard QEMU gitlab CI configuration file. If this is not sufficient it is
+possible to completely replace the CI configuration file with a custom
+version. This allows the contributor to achieve essentially anything they
+desire, within the scope of what GitLab supports.
+
+Replacing the ``.gitlab-ci.yml`` file is done in the repository settings:
+
+  https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file
+
+Note that it is possible to reference an arbitrary URL. This could point
+to a replacement .gitlab-ci.yml on a specific branch in the same repo,
+or can point to any external site.
+
+
+Job grouping sets
+-----------------
+
+There are many different jobs defined in the GitLab CI pipeline used by QEMU.
+It is not practical to run all jobs in every scenario, so there are a set of
+rules defined that control which jobs are executed on each pipeline run. At
+a high level the jobs can be grouped into a number of mutually exclusive
+sets.
+
+ - Manual jobs
+
+   This is a set of jobs that will never be run automatically in any scenario.
+   The reason a job will be marked manual is if it is known to exhibit some
+   non-deterministic failures, or liable to trigger timeouts. Ideally jobs are
+   only present in this set for a short period of time while the problems are
+   investigated and resolved. Users can manually trigger these jobs from the
+   the pipelines web UI if desired, but they will never contribute to the
+   overall pipeline status.
+
+   Identified by the variable ``QEMU_JOB_MANUAL: 1``
+
+
+ - Minimal jobs
+
+   This is a minimal set of jobs that give a reasonable build and test sanity
+   check, which will be run by default in all possible scenarios. This set of
+   jobs is intended to be controlled fairly strictly to avoid wasting CI
+   minutes of maintainers/contributors. The intent is to get 80-90% coverage
+   of build and unit tests across the most important host platforms and
+   architectures.
+
+   Identified by the variable ``QEMU_JOB_MINIMAL: 1``
+
+   Run by setting the variable ``QEMU_CI: 1`` or pushing to a branch
+   named ``ci-min-XXX``.
+
+
+ - Gating jobs
+
+   This is a set of jobs will will always run as a gating test prior to code
+   merging into the default branch of the primary QEMU git repository. In user
+   forks the jobs will not run by default, but the user can opt-in to trigger
+   their execution. These jobs may run a particularly thorough set of scenarios
+   that maintainers are not normally going to exercise before sending series.
+
+   Identified by the variable ``QEMU_JOB_GATING: 1``
+
+   Run by setting the variable ``QEMU_CI_GATING: 1`` or pushing to a branch
+   named ``ci-gating-XXX``
+
+
+ - Optional jobs
+
+   This is all remaining jobs that don't fall into one of the above buckets.
+   These jobs will always be run as a gating test prior to code merging into
+   the default branch of the primary QEMU git repository. In user forks the
+   jobs will not run by default, but the user can opt-in to trigger their
+   execution. These jobs will aim to expand the host/platform build coverage
+   gaps left by the default jobs.
+
+   Identified by not having any of the above variables set.
+
+   Run by setting the variable ``QEMU_CI_ALL: 1`` or pushing to a branch
+   named ``ci-all-XXXX``
+
+Since the sets are mutually exclusive, if more than one of the variables above
+is set, then only one of them will be honoured. The precedence matches the
+ordering of the above docs.
+
+In addition to the above, there are some special extra sets that can augment
+any of the above sets
+
+ - Publishing jobs
 
-QEMU_CI_AVOCADO_TESTING
-~~~~~~~~~~~~~~~~~~~~~~~
-By default, tests using the Avocado framework are not run automatically in
-the pipelines (because multiple artifacts have to be downloaded, and if
-these artifacts are not already cached, downloading them make the jobs
-reach the timeout limit). Set this variable to have the tests using the
-Avocado framework run automatically.
+   These are jobs that are needed to publish the QEMU website. For user fork
+   repos these follow the normal rules set out above. For the primary QEMU
+   repo, however, the jobs will also run on the default branch.
 
+   Identified by the variable ``QEMU_JOB_PUBLISH: 1``
#18 lore raw

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

Quoted 7 lines · Thomas Huth
Quoted 5 lines · Daniel P. Berrangé

Currently pushes to user forks run the same set of build / test jobs as pushes to the main repo. This results in creation of 120+ individual jobs. While it is useful for subsystem maintainers, and even regular contributors to be able to run the full set of jobs, it is wasteful to run all of them all the time.

Quoted 5 lines

Hi Daniel!

Now that gitlab introduces limits for the CI for all users, I think we should revisit this topic ... could you maybe respin your patch to the current git master branch?

Yes, I really do need to revisit this patch.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|