Skip to contentSkip to conversation

gitlab: restrict running jobs in forks and upstream master

PATCH In review Daniel P. Berrangé 4 participants 11 messages May 26, 2022 – Jun 6, 2022

lore
Share & tools
lore
#0 lore raw

cover gitlab: restrict running jobs in forks and upstream master

Currently on upstream most jobs will run in both staging and master. This is quite wasteful of CI credits. The only need to run in master is for the jobs related to publishing the website

In forks we run jobs on every push. With restricted CI allowance this is quickly going to cause problems.

With this series jobs will no longer run on forks at all, without an opt-in with QEMU_CI=1 (pipeline with manual jobs) or QEMU_CI=2 (pipeline with immediate jobs)

This is a rewrite of a previous proposal:

https://lists.nongnu.org/archive/html/qemu-devel/2021-08/msg02104.html

where I've kept it simpler and also split up the patches into more understandable chunks

Daniel P. Berrangé (5):
gitlab: introduce a common base job template
gitlab: convert Cirrus jobs to .base_job_template
gitlab: convert static checks to .base_job_template
gitlab: convert build/container jobs to .base_job_template
gitlab: don't run CI jobs in forks by default

 .gitlab-ci.d/base.yml                | 72 +++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml  | 16 ++---
 .gitlab-ci.d/buildtest.yml           | 28 ++++-----
 .gitlab-ci.d/cirrus.yml              | 16 ++---
 .gitlab-ci.d/container-cross.yml     |  6 +-
 .gitlab-ci.d/container-template.yml  |  1 +
 .gitlab-ci.d/crossbuild-template.yml |  3 +
 .gitlab-ci.d/qemu-project.yml        |  1 +
 .gitlab-ci.d/static_checks.yml       | 19 +++---
 .gitlab-ci.d/windows.yml             |  1 +
 docs/devel/ci-jobs.rst.inc           | 88 +++++++++++++++++++++++++++-
 11 files changed, 199 insertions(+), 52 deletions(-)
 create mode 100644 .gitlab-ci.d/base.yml
Signature · 1 line
2.36.1
#1 lore raw

PATCH 1/5 gitlab: introduce a common base job template

Currently job rules are spread across the various templates and jobs, making it hard to understand exactly what runs in what scenario. This leads to inconsistency in the rules and increased maint burden.

The intent is that we introduce a common '.base_job_template' which will have a general purpose 'rules:' block. No other template or job should define 'rules:', but instead they must rely on the inherited rules. To allow behaviour to be tweaked, rules will be influenced by a number of variables with the naming scheme 'QEMU_JOB_nnnn'.

---
 .gitlab-ci.d/base.yml         | 28 +++++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml |  1 +
 docs/devel/ci-jobs.rst.inc    | 36 ++++++++++++++++++++++++++++++++++-
 3 files changed, 64 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci.d/base.yml
.gitlab-ci.d/base.yml new file +28 -0
new file mode 100644
index 0000000000..10eb6ab8bc
@@ -0,0 +1,28 @@
+
+# The order of rules defined here is critically important.
+# They are evaluated in order and first match wins.
+#
+# Thus we group them into a number of stages, ordered from
+# most restrictive to least restrictive
+#
+.base_job_template:
+  rules:
+    #############################################################
+    # Stage 1: exclude scenarios where we definitely don't
+    # want jobs to run
+    #############################################################
+
+
+    #############################################################
+    # Stage 2: fine tune execution of jobs in specific scenarios
+    # where the catch all logic is inapprorpaite
+    #############################################################
+
+
+    #############################################################
+    # Stage 3: catch all logic applying to any job not matching
+    # an earlier criteria
+    #############################################################
+
+    # Jobs can run if any jobs they depend on were successfull
+    - when: on_success
.gitlab-ci.d/qemu-project.yml +1 -0
index 871262fe0e..691d9bf5dc 100644
@@ -2,6 +2,7 @@
 # https://gitlab.com/qemu-project/qemu/-/pipelines
 
 include:
+  - local: '/.gitlab-ci.d/base.yml'
   - local: '/.gitlab-ci.d/stages.yml'
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
docs/devel/ci-jobs.rst.inc +35 -1
index 92e25872aa..eb6a9e6122 100644
@@ -28,7 +28,35 @@ For further information about how to set these variables, please refer to::
 
   https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
 
-Here is a list of the most used variables:
+Variable naming and grouping
+----------------------------
+
+The variables used by QEMU's CI configuration are grouped together
+in a handful of namespaces
+
+ * QEMU_JOB_nnnn - variables to be defined in individual jobs
+   or templates, to influence the shared rules defined in the
+   .base_job_template.
+
+ * QEMU_CI_nnn - variables to be set by contributors in their
+   repository CI settings, or as git push variables, to influence
+   which jobs get run in a pipeline
+
+ * nnn - other misc variables not falling into the above
+   categories, or using different names for historical reasons
+   and not yet converted.
+
+Maintainer controlled job variables
+-----------------------------------
+
+The following variables may be set when defining a job in the
+CI configuration file.
+
+Contributor controlled runtime variables
+----------------------------------------
+
+The following variables may be set by contributors to control
+job execution
 
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -38,6 +66,12 @@ these artifacts are not already cached, downloading them make the jobs
 reach the timeout limit). Set this variable to have the tests using the
 Avocado framework run automatically.
 
+Other misc variables
+--------------------
+
+These variables are primarily to control execution of jobs on
+private runners
+
 AARCH64_RUNNER_AVAILABLE
 ~~~~~~~~~~~~~~~~~~~~~~~~
 If you've got access to an aarch64 host that can be used as a gitlab-CI
Signature · 1 line
2.36.1
#2 lore raw

PATCH 2/5 gitlab: convert Cirrus jobs to .base_job_template

In reply to Daniel P. Berrangé

This folds the Cirrus job rules into the base job
template, introducing two new variables

- QEMU_JOB_CIRRUS - identifies the job as making use of Cirrus CI via cirrus-run

- QEMU_JOB_OPTIONAL - identifies the job as one that is not run by default, primarily due to resource constraints. It can be manually invoked by users if they wish to validate that scenario.

---
 .gitlab-ci.d/base.yml      |  9 +++++++++
 .gitlab-ci.d/cirrus.yml    | 16 ++++++++--------
 docs/devel/ci-jobs.rst.inc | 14 ++++++++++++++
 3 files changed, 31 insertions(+), 8 deletions(-)
.gitlab-ci.d/base.yml +9 -0
index 10eb6ab8bc..5734caf9fe 100644
@@ -12,12 +12,21 @@
     # want jobs to run
     #############################################################
 
+    # Cirrus jobs can't run unless the creds / target repo are set
+    - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
     # where the catch all logic is inapprorpaite
     #############################################################
 
+    # Optional jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_OPTIONAL'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
.gitlab-ci.d/cirrus.yml +8 -8
index b96b22e269..609c364308 100644
@@ -11,6 +11,7 @@
 # special care, because we can't just override it at the GitLab CI job
 # definition level or we risk breaking it completely.
 .cirrus_build_job:
+  extends: .base_job_template
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
@@ -40,11 +41,8 @@
       <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    # Allow on 'staging' branch and 'stable-X.Y-staging' branches only
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
-      when: never
-    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+  variables:
+    QEMU_JOB_CIRRUS: 1
 
 x64-freebsd-12-build:
   extends: .cirrus_build_job
@@ -90,11 +88,11 @@ x64-macos-11-base-build:
 
 # The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job
 .cirrus_kvm_job:
+  extends: .base_job_template
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
   timeout: 80m
-  allow_failure: true
   script:
     - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
           -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
@@ -105,8 +103,10 @@ x64-macos-11-base-build:
       <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    - when: manual
+  variables:
+    QEMU_JOB_CIRRUS: 1
+    QEMU_JOB_OPTIONAL: 1
+
 
 x86-netbsd:
   extends: .cirrus_kvm_job
docs/devel/ci-jobs.rst.inc +14 -0
index eb6a9e6122..a539f502da 100644
@@ -52,6 +52,20 @@ Maintainer controlled job variables
 The following variables may be set when defining a job in the
 CI configuration file.
 
+QEMU_JOB_CIRRUS
+~~~~~~~~~~~~~~~
+
+The job makes use of Cirrus CI infrastructure, requiring the
+configuration setup for cirrus-run to be present in the repository
+
+QEMU_JOB_OPTIONAL
+~~~~~~~~~~~~~~~~~
+
+The job is expected to be successful in general, but is not run
+by default due to need to conserve limited CI resources. It is
+available to be started manually by the contributor in the CI
+pipelines UI.
+
 Contributor controlled runtime variables
 ----------------------------------------
 
Signature · 1 line
2.36.1
#3 lore raw

PATCH 3/5 gitlab: convert static checks to .base_job_template

In reply to Daniel P. Berrangé

This folds the static checks into using the base job template rules, introducing one new variable

- QEMU_JOB_ONLY_FORKS - a job that should never run on an upstream pipeline. The information it reports is only applicable to contributors in a pre-submission scenario, not time of merge.

---
 .gitlab-ci.d/base.yml          |  4 ++++
 .gitlab-ci.d/static_checks.yml | 19 +++++++------------
 docs/devel/ci-jobs.rst.inc     |  7 +++++++
 3 files changed, 18 insertions(+), 12 deletions(-)
.gitlab-ci.d/base.yml +4 -0
index 5734caf9fe..9a0b8d7f97 100644
@@ -16,6 +16,10 @@
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Jobs only intended for forks should always be skipped on upstram
+    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
.gitlab-ci.d/static_checks.yml +7 -12
index 94858e3272..289ad1359e 100644
@@ -1,4 +1,5 @@
 check-patch:
+  extends: .base_job_template
   stage: build
   image: python:3.10-alpine
   needs: []
@@ -6,15 +7,13 @@ check-patch:
     - .gitlab-ci.d/check-patch.py
   variables:
     GIT_DEPTH: 1000
+    QEMU_JOB_ONLY_FORKS: 1
   before_script:
     - apk -U add git perl
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: on_success
-      allow_failure: true
+  allow_failure: true
 
 check-dco:
+  extends: .base_job_template
   stage: build
   image: python:3.10-alpine
   needs: []
@@ -23,12 +22,9 @@ check-dco:
     GIT_DEPTH: 1000
   before_script:
     - apk -U add git
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
 
 check-python-pipenv:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
@@ -39,6 +35,7 @@ check-python-pipenv:
     job: python-container
 
 check-python-tox:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
@@ -46,8 +43,6 @@ check-python-tox:
   variables:
     GIT_DEPTH: 1
     QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
+    QEMU_JOB_OPTIONAL: 1
   needs:
     job: python-container
-  rules:
-    - when: manual
-  allow_failure: true
docs/devel/ci-jobs.rst.inc +7 -0
index a539f502da..4c7e30ab08 100644
@@ -66,6 +66,13 @@ by default due to need to conserve limited CI resources. It is
 available to be started manually by the contributor in the CI
 pipelines UI.
 
+QEMU_JOB_ONLY_FORKS
+~~~~~~~~~~~~~~~~~~~
+
+The job results are only of interest to contributors prior to
+submitting code. They are not required as part of the gating
+CI pipeline.
+
 Contributor controlled runtime variables
 ----------------------------------------
 
Signature · 1 line
2.36.1
#4 lore raw

PATCH 4/5 gitlab: convert build/container jobs to .base_job_template

In reply to Daniel P. Berrangé

This converts the main build and container jobs to use the base job rules, defining the following new variables

- QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired.

- QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness.

- QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream

---
 .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
 .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
 .gitlab-ci.d/container-cross.yml     |  6 ++----
 .gitlab-ci.d/container-template.yml  |  1 +
 .gitlab-ci.d/crossbuild-template.yml |  3 +++
 .gitlab-ci.d/windows.yml             |  1 +
 docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
 8 files changed, 65 insertions(+), 31 deletions(-)
.gitlab-ci.d/base.yml +22 -0
index 9a0b8d7f97..6a918abbda 100644
@@ -16,10 +16,22 @@
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Publishing jobs should only run on the default branch in upstream
+    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
+      when: never
+
+    # Non-publishing jobs should only run on staging branches in upstream
+    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
+      when: never
+
     # Jobs only intended for forks should always be skipped on upstram
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
+    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
@@ -31,6 +43,16 @@
       when: manual
       allow_failure: true
 
+    # Skipped jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_SKIPPED'
+      when: manual
+      allow_failure: true
+
+    # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
+    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
.gitlab-ci.d/buildtest-template.yml +4 -12
index dc6d67aacf..1874c07a4b 100644
@@ -1,4 +1,5 @@
 .native_build_job_template:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -27,6 +28,7 @@
       fi
 
 .common_test_job_template:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -75,15 +77,5 @@
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_AVOCADO: 1
.gitlab-ci.d/buildtest.yml +13 -15
index e9620c3074..ecac3ec50c 100644
@@ -360,12 +360,11 @@ build-cfi-aarch64:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-aarch64:
   extends: .native_test_job_template
@@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-ppc64-s390x:
   extends: .native_test_job_template
@@ -579,6 +577,7 @@ build-without-default-features:
     MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow
 
 build-libvhost-user:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
@@ -595,10 +594,13 @@ build-tools-and-docs-debian:
   extends: .native_build_job_template
   needs:
     job: amd64-debian-container
+    # when running on 'master' we use pre-existing container
+    optional: true
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
     CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
+    QEMU_JOB_PUBLISH: 1
   artifacts:
     expire_in: 2 days
     paths:
@@ -618,6 +620,7 @@ build-tools-and-docs-debian:
 # that users can see the results of their commits, regardless
 # of what topic branch they're currently using
 pages:
+  extends: .base_job_template
   image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
   needs:
@@ -635,10 +638,5 @@ pages:
   artifacts:
     paths:
       - public
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: on_success
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: on_success
+  variables:
+    QEMU_JOB_PUBLISH: 1
.gitlab-ci.d/container-cross.yml +2 -4
index e622ac2d21..90ecfd8da0 100644
@@ -42,15 +42,13 @@ armhf-debian-cross-container:
 # We never want to build hexagon in the CI system and by default we
 # always want to refer to the master registry where it lives.
 hexagon-cross-container:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
   variables:
     NAME: debian-hexagon-cross
     GIT_DEPTH: 1
+    QEMU_JOB_ONLY_FORKS: 1
   services:
     - docker:dind
   before_script:
.gitlab-ci.d/container-template.yml +1 -0
index 1baecd9460..c434b9c8f3 100644
@@ -1,4 +1,5 @@
 .container_job_template:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
   services:
.gitlab-ci.d/crossbuild-template.yml +3 -0
index 29c3c2b826..28b2142ec2 100644
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -24,6 +25,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $EXTRA_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -36,6 +38,7 @@
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
.gitlab-ci.d/windows.yml +1 -0
index cf7724b8e5..1b2ede49e1 100644
@@ -1,4 +1,5 @@
 .shared_msys2_builder:
+  extends: .base_job_template
   tags:
   - shared-windows
   - windows
docs/devel/ci-jobs.rst.inc +19 -0
index 4c7e30ab08..0b4926e537 100644
@@ -73,6 +73,25 @@ The job results are only of interest to contributors prior to
 submitting code. They are not required as part of the gating
 CI pipeline.
 
+QEMU_JOB_SKIPPED
+~~~~~~~~~~~~~~~~
+
+The job is not reliably successsful in general, so is not
+currently suitable to be run by default. Ideally this should
+be a temporary marker until the problems can be addressed, or
+the job permanently removed.
+
+QEMU_JOB_PUBLISH
+~~~~~~~~~~~~~~~~
+
+The job is for publishing content after a branch has been
+merged into the upstream default branch.
+
+QEMU_JOB_AVOCADO
+~~~~~~~~~~~~~~~~
+
+The job runs the Avocado integration test suite
+
 Contributor controlled runtime variables
 ----------------------------------------
 
Signature · 1 line
2.36.1
#5 lore raw

Re: [PATCH 4/5] gitlab: convert build/container jobs to .base_job_template

Quoted 24 lines · Daniel P. Berrangé

This converts the main build and container jobs to use the base job rules, defining the following new variables

- QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired.

- QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness.

- QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream

---
  .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
  .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
  .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
  .gitlab-ci.d/container-cross.yml     |  6 ++----
  .gitlab-ci.d/container-template.yml  |  1 +
  .gitlab-ci.d/crossbuild-template.yml |  3 +++
  .gitlab-ci.d/windows.yml             |  1 +
  docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
  8 files changed, 65 insertions(+), 31 deletions(-)

...

Quoted 23 lines
.gitlab-ci.d/buildtest.yml +3 -4
index e9620c3074..ecac3ec50c 100644
@@ -360,12 +360,11 @@ build-cfi-aarch64:
      expire_in: 2 days
      paths:
        - build
-  rules:
+  variables:
      # FIXME: This job is often failing, likely due to out-of-memory problems in
      # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
  check-cfi-aarch64:
    extends: .native_test_job_template
@@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
      expire_in: 2 days
      paths:

.gitlab-ci.d/buildtest.yml build-cfi-ppc64-s390x: · line 408

Quoted 10 lines
.gitlab-ci.d/buildtest.yml +3 -4
        - build
-  rules:
+  variables:
      # FIXME: This job is often failing, likely due to out-of-memory problems in
      # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1

FYI, this patch broke the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs since they've now got two "variables:" sections and apparently only the second one is taken into account...

Thomas

#6 lore raw

Re: [PATCH 4/5] gitlab: convert build/container jobs to .base_job_template

Quoted 54 lines · Thomas Huth
Quoted 24 lines · Daniel P. Berrangé

This converts the main build and container jobs to use the base job rules, defining the following new variables

- QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired.

- QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness.

- QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream

---
  .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
  .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
  .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
  .gitlab-ci.d/container-cross.yml     |  6 ++----
  .gitlab-ci.d/container-template.yml  |  1 +
  .gitlab-ci.d/crossbuild-template.yml |  3 +++
  .gitlab-ci.d/windows.yml             |  1 +
  docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
  8 files changed, 65 insertions(+), 31 deletions(-)

...

Quoted 28 lines
.gitlab-ci.d/buildtest.yml +4 -6
index e9620c3074..ecac3ec50c 100644
@@ -360,12 +360,11 @@ build-cfi-aarch64:
      expire_in: 2 days
      paths:
        - build
-  rules:
+  variables:
      # FIXME: This job is often failing, likely due to out-of-memory problems in
      # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
  check-cfi-aarch64:
    extends: .native_test_job_template
@@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
      expire_in: 2 days
      paths:
        - build
-  rules:
+  variables:
      # FIXME: This job is often failing, likely due to out-of-memory problems in
      # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
Quoted 8 lines

.gitlab-ci.d/buildtest.yml build-cfi-aarch64: · line 367

Quoted 4 lines
.gitlab-ci.d/buildtest.yml +2 -2
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1

FYI, this patch broke the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs since they've now got two "variables:" sections and apparently only the second one is taken into account...

Opps, sorry about that, and of course I didn't notice as it is a skipped job and I forgot to try running it :-(

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw

PATCH 5/5 gitlab: don't run CI jobs in forks by default

In reply to Daniel P. Berrangé

To preserve CI shared runner credits we don't want to run pipelines on every push.

This sets up the config to that pipelines are never created for contributors by default. To override this the QEMU_CI variable can be set to a non-zero value. If set to 1, the pipeline will be created but all jobs will remain manually started. The contributor can selectively run jobs that they care about. If set to 2, the pipeline will be created and all jobs will immediately start.

This behavior can be controlled using push variables

git push -o ci.variable=QEMU_CI=1

To make this more convenient define an alias

   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"

Which lets you run

git push-ci

to create the pipeline, or

git push-ci-now

to create and run the pipeline

---
 .gitlab-ci.d/base.yml      |  9 +++++++++
 docs/devel/ci-jobs.rst.inc | 12 ++++++++++++
 2 files changed, 21 insertions(+)
.gitlab-ci.d/base.yml +9 -0
index 6a918abbda..62f2a850c3 100644
@@ -28,6 +28,10 @@
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
     - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
       when: never
@@ -59,5 +63,10 @@
     # an earlier criteria
     #############################################################
 
+    # Forks pipeline jobs don't start automatically unless
+    # QEMU_CI=2 is set
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successfull
     - when: on_success
docs/devel/ci-jobs.rst.inc +12 -0
index 0b4926e537..e36c8cc68d 100644
@@ -98,6 +98,18 @@ Contributor controlled runtime variables
 The following variables may be set by contributors to control
 job execution
 
+QEMU_CI
+~~~~~~~
+
+By default, no pipelines will be created on contributor forks
+in order to preserve CI credits
+
+Set this variable to 1 to create the pipelines, but leave all
+the jobs to be manually started from the UI
+
+Set this variable to 2 to create the pipelines and run all
+the jobs immediately, as was historicaly behaviour
+
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
 By default, tests using the Avocado framework are not run automatically in
Signature · 1 line
2.36.1
#8 lore raw

Re: [PATCH 5/5] gitlab: don't run CI jobs in forks by default

Hi

Quoted 5 lines · Daniel P. Berrangé

To preserve CI shared runner credits we don't want to run pipelines on every push.

This sets up the config to that pipelines are never created

to/so ?

Quoted 18 lines

for contributors by default. To override this the QEMU_CI variable can be set to a non-zero value. If set to 1, the pipeline will be created but all jobs will remain manually started. The contributor can selectively run jobs that they care about. If set to 2, the pipeline will be created and all jobs will immediately start.

This behavior can be controlled using push variables

git push -o ci.variable=QEMU_CI=1

To make this more convenient define an alias

   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"

Which lets you run

Quoted 8 lines

git push-ci

to create the pipeline, or

git push-ci-now

to create and run the pipeline

Nice
Is there a place to put that in the devel documentation? Either in submiting-a-patch.rst "Test your patches" (there is a broken link Testing#Continuous_Integration here, wonder why sphinx doesn't catch it) or somewhere in ci.rst documents perhaps.

Quoted 64 lines
---
 .gitlab-ci.d/base.yml      |  9 +++++++++
 docs/devel/ci-jobs.rst.inc | 12 ++++++++++++
 2 files changed, 21 insertions(+)
.gitlab-ci.d/base.yml +0 -0
index 6a918abbda..62f2a850c3 100644
@@ -28,6 +28,10 @@
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE ==

"qemu-project"'
when: never

.gitlab-ci.d/base.yml +2 -0
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE !=

"qemu-project"'

.gitlab-ci.d/base.yml +2 -0
+      when: never
+
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is

set
- if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" &&
$CI_PROJECT_NAMESPACE != "qemu-project"'
when: never

.gitlab-ci.d/base.yml +5 -0
@@ -59,5 +63,10 @@
     # an earlier criteria
     #############################################################
 
+    # Forks pipeline jobs don't start automatically unless
+    # QEMU_CI=2 is set
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successfull
     - when: on_success
docs/devel/ci-jobs.rst.inc +12 -1
index 0b4926e537..e36c8cc68d 100644
@@ -98,6 +98,18 @@ Contributor controlled runtime variables
 The following variables may be set by contributors to control
 job execution
 
+QEMU_CI
+~~~~~~~
+
+By default, no pipelines will be created on contributor forks
+in order to preserve CI credits
+
+Set this variable to 1 to create the pipelines, but leave all
+the jobs to be manually started from the UI
+
+Set this variable to 2 to create the pipelines and run all
+the jobs immediately, as was historicaly behaviour
+
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
 By default, tests using the Avocado framework are not run automatically in
--

2.36.1

Signature · 1 line
Marc-André Lureau
#9 lore raw

Re: [PATCH 5/5] gitlab: don't run CI jobs in forks by default

In reply to Daniel P. Berrangé
Quoted 22 lines · Daniel P. Berrangé

To preserve CI shared runner credits we don't want to run pipelines on every push.

This sets up the config to that pipelines are never created for contributors by default. To override this the QEMU_CI variable can be set to a non-zero value. If set to 1, the pipeline will be created but all jobs will remain manually started. The contributor can selectively run jobs that they care about. If set to 2, the pipeline will be created and all jobs will immediately start.

This behavior can be controlled using push variables

git push -o ci.variable=QEMU_CI=1

To make this more convenient define an alias

   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"

Which lets you run

Quoted 7 lines

git push-ci

to create the pipeline, or

git push-ci-now

to create and run the pipeline

I think these useful tips on running the pipelines should go in the rst so you don't have to grep the commit log to find them.

Quoted 56 lines
---
 .gitlab-ci.d/base.yml      |  9 +++++++++
 docs/devel/ci-jobs.rst.inc | 12 ++++++++++++
 2 files changed, 21 insertions(+)
.gitlab-ci.d/base.yml +9 -0
index 6a918abbda..62f2a850c3 100644
@@ -28,6 +28,10 @@
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
     - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
       when: never
@@ -59,5 +63,10 @@
     # an earlier criteria
     #############################################################
 
+    # Forks pipeline jobs don't start automatically unless
+    # QEMU_CI=2 is set
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successfull
     - when: on_success
docs/devel/ci-jobs.rst.inc +12 -0
index 0b4926e537..e36c8cc68d 100644
@@ -98,6 +98,18 @@ Contributor controlled runtime variables
 The following variables may be set by contributors to control
 job execution
 
+QEMU_CI
+~~~~~~~
+
+By default, no pipelines will be created on contributor forks
+in order to preserve CI credits
+
+Set this variable to 1 to create the pipelines, but leave all
+the jobs to be manually started from the UI
+
+Set this variable to 2 to create the pipelines and run all
+the jobs immediately, as was historicaly behaviour
+
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
 By default, tests using the Avocado framework are not run automatically in
Signature · 1 line
Alex Bennée
#10 lore raw
In reply to Daniel P. Berrangé
Quoted 12 lines · Daniel P. Berrangé

Currently on upstream most jobs will run in both staging and master. This is quite wasteful of CI credits. The only need to run in master is for the jobs related to publishing the website

In forks we run jobs on every push. With restricted CI allowance this is quickly going to cause problems.

With this series jobs will no longer run on forks at all, without an opt-in with QEMU_CI=1 (pipeline with manual jobs) or QEMU_CI=2 (pipeline with immediate jobs)

Quoted 6 lines

This is a rewrite of a previous proposal:

https://lists.nongnu.org/archive/html/qemu-devel/2021-08/msg02104.html

where I've kept it simpler and also split up the patches into more understandable chunks

Queued to testing/next, thanks.

I'll fix up the comment and move some stuff into the rst.

Quoted 21 lines

Daniel P. Berrangé (5):
gitlab: introduce a common base job template
gitlab: convert Cirrus jobs to .base_job_template
gitlab: convert static checks to .base_job_template
gitlab: convert build/container jobs to .base_job_template
gitlab: don't run CI jobs in forks by default

 .gitlab-ci.d/base.yml                | 72 +++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml  | 16 ++---
 .gitlab-ci.d/buildtest.yml           | 28 ++++-----
 .gitlab-ci.d/cirrus.yml              | 16 ++---
 .gitlab-ci.d/container-cross.yml     |  6 +-
 .gitlab-ci.d/container-template.yml  |  1 +
 .gitlab-ci.d/crossbuild-template.yml |  3 +
 .gitlab-ci.d/qemu-project.yml        |  1 +
 .gitlab-ci.d/static_checks.yml       | 19 +++---
 .gitlab-ci.d/windows.yml             |  1 +
 docs/devel/ci-jobs.rst.inc           | 88 +++++++++++++++++++++++++++-
 11 files changed, 199 insertions(+), 52 deletions(-)
 create mode 100644 .gitlab-ci.d/base.yml
Signature · 1 line
Alex Bennée