Skip to contentSkip to conversation

gitlab: improvements to handling of stable staging branches

PATCH Superseded Daniel P. Berrangé 3 participants 13 messages May 26, 2023 – Jun 8, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover gitlab: improvements to handling of stable staging branches

We just (re)discovered that our gitlab rules don't work nicely with pipelines running from stable staging branches. Every pipeline gets published with the 'latest' tag, whether its the main staging branch or one of the stable staging branches. If pipelines for multiple staging branches run concurrently they'll get very confused and end up using the wrong container content. eg a 8.0 stable job will get run with a container from the development branch, or vica-verca.

With this series we dynamically change the tag so that the 'staging' branch still uses 'latest', but the stable 'staging-X.Y' branaches use a 'staging-X-Y' container tag.

We also let the container tag be set explicitly via the new variable

QEMU_CI_CONTAINER_TAG

to facilitate CI testing, the new variable

QEMU_CI_UPSTREAM

can be set to the fork namespace, to allow contributors to run a pipeline as if their fork were upstream.

Finally 'QEMU_CI=1' is also made to work in upstream, so that it is possible to re-run selective jobs on staging branches.

Changed in v2:

- Allow QEMU_CI=1 to work on all staging branches, not just stable
- Rebased

Daniel P. Berrangé (5):
gitlab: centralize the container tag name
gitlab: allow overriding name of the upstream repository
gitlab: stable staging branches publish containers in a separate tag
gitlab: avoid extra pipelines for tags and stable branches
gitlab: support disabling job auto-run in upstream

 .gitlab-ci.d/base.yml                | 63 ++++++++++++++++++++++++----
 .gitlab-ci.d/buildtest-template.yml  |  4 +-
 .gitlab-ci.d/buildtest.yml           |  4 +-
 .gitlab-ci.d/container-template.yml  |  3 +-
 .gitlab-ci.d/crossbuild-template.yml |  6 +--
 .gitlab-ci.d/static_checks.yml       |  4 +-
 docs/devel/ci-jobs.rst.inc           | 11 +++++
 7 files changed, 78 insertions(+), 17 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH v2 1/5 gitlab: centralize the container tag name

We use a fixed container tag of 'latest' so that contributors' forks don't end up with an ever growing number of containers as they work on throwaway feature branches.

This fixed tag causes problems running CI upstream in stable staging branches, however, because the stable staging branch will publish old container content that clashes with that needed by primary staging branch. This makes it impossible to reliably run CI pipelines in parallel in upstream for different staging branches.

This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to change which tag container publishing uses. Initially it can be set by contributors as a git push option if they want to override the default use of 'latest' eg

git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish

this is useful if contributors need to run pipelines for different branches concurrently in their forks.

---
 .gitlab-ci.d/base.yml                | 6 ++++++
 .gitlab-ci.d/buildtest-template.yml  | 4 ++--
 .gitlab-ci.d/buildtest.yml           | 4 ++--
 .gitlab-ci.d/container-template.yml  | 3 ++-
 .gitlab-ci.d/crossbuild-template.yml | 6 +++---
 .gitlab-ci.d/static_checks.yml       | 4 ++--
 docs/devel/ci-jobs.rst.inc           | 5 +++++
 7 files changed, 22 insertions(+), 10 deletions(-)
.gitlab-ci.d/base.yml +6 -0
index 2fbb58d2a3..fba9d31cc6 100644
@@ -1,4 +1,10 @@
 
+variables:
+  # On stable branches this needs changing. Should also be
+  # overridden per pipeline if running pipelines concurrently
+  # for different branches in contributor forks.
+  QEMU_CI_CONTAINER_TAG: latest
+
 # The order of rules defined here is critically important.
 # They are evaluated in order and first match wins.
 #
.gitlab-ci.d/buildtest-template.yml +2 -2
index c9f2e737c0..4cb80785ca 100644
@@ -1,7 +1,7 @@
 .native_build_job_template:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   before_script:
     - JOBS=$(expr $(nproc) + 1)
   script:
@@ -40,7 +40,7 @@
 .common_test_job_template:
   extends: .base_job_template
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   script:
     - scripts/git-submodule.sh update
         $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
.gitlab-ci.d/buildtest.yml +2 -2
index 0f1be14cb6..0fde900e92 100644
@@ -532,7 +532,7 @@ build-without-defaults:
 build-libvhost-user:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
   needs:
     job: amd64-fedora-container
   script:
@@ -572,7 +572,7 @@ build-tools-and-docs-debian:
 # of what topic branch they're currently using
 pages:
   extends: .base_job_template
-  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
+  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG
   stage: test
   needs:
     - job: build-tools-and-docs-debian
.gitlab-ci.d/container-template.yml +2 -1
index 9ac4a0ee25..971065918a 100644
@@ -5,7 +5,8 @@
   services:
     - docker:stable-dind
   before_script:
-    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
+    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:$QEMU_CI_CONTAINER_TAG"
+    # Always ':latest' because we always use upstream as a common cache source
     - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
     - apk add python3
     - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
.gitlab-ci.d/crossbuild-template.yml +3 -3
index 4f93b9e4e5..6efb0d2a54 100644
@@ -1,7 +1,7 @@
 .cross_system_build_job:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   timeout: 80m
   script:
     - mkdir build
@@ -27,7 +27,7 @@
 .cross_accel_build_job:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   timeout: 30m
   script:
     - mkdir build
@@ -39,7 +39,7 @@
 .cross_user_build_job:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   script:
     - mkdir build
     - cd build
.gitlab-ci.d/static_checks.yml +2 -2
index b4cbdbce2a..ad9f426a52 100644
@@ -26,7 +26,7 @@ check-dco:
 check-python-minreqs:
   extends: .base_job_template
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/python:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:$QEMU_CI_CONTAINER_TAG
   script:
     - make -C python check-minreqs
   variables:
@@ -37,7 +37,7 @@ check-python-minreqs:
 check-python-tox:
   extends: .base_job_template
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/python:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:$QEMU_CI_CONTAINER_TAG
   script:
     - make -C python check-tox
   variables:
docs/devel/ci-jobs.rst.inc +5 -0
index 1f28fec0d0..f72537853b 100644
@@ -70,6 +70,11 @@ in a handful of namespaces
    repository CI settings, or as git push variables, to influence
    which jobs get run in a pipeline
 
+ * QEMU_CI_CONTAINER_TAG - the tag used to publish containers
+   in stage 1, for use by build jobs in stage 2. Defaults to
+   'latest', but if running pipelines for different branches
+   concurrently, it should be overridden per pipeline.
+
  * nnn - other misc variables not falling into the above
    categories, or using different names for historical reasons
    and not yet converted.
Signature · 1 line
2.40.1
#2 lore raw

PATCH v2 2/5 gitlab: allow overriding name of the upstream repository

In reply to Daniel P. Berrangé

The CI rules have special logic for what happens in upstream. To enable contributors who modify CI rules to test this logic, however, they need to be able to override which repo is considered upstream. This introduces the 'QEMU_CI_UPSTREAM' variable

git push gitlab <branch> -o ci.variable=QEMU_CI_UPSTREAM=berrange

to make it look as if my namespace is the actual upstream. Namespace in this context refers to the path fragement in gitlab URLs that is above the repository. Typically this will be the contributor's gitlab login name.

---
 .gitlab-ci.d/base.yml      | 19 ++++++++++++-------
 docs/devel/ci-jobs.rst.inc |  6 ++++++
 2 files changed, 18 insertions(+), 7 deletions(-)
.gitlab-ci.d/base.yml +12 -7
index fba9d31cc6..a1d734267a 100644
@@ -5,6 +5,11 @@ variables:
   # for different branches in contributor forks.
   QEMU_CI_CONTAINER_TAG: latest
 
+  # For purposes of CI rules, upstream is the gitlab.com/qemu-project
+  # namespace. When testing CI, it might be usefult to override this
+  # to point to a fork repo
+  QEMU_CI_UPSTREAM: qemu-project
+
 # The order of rules defined here is critically important.
 # They are evaluated in order and first match wins.
 #
@@ -30,23 +35,23 @@ variables:
       when: never
 
     # Publishing jobs should only run on the default branch in upstream
-    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
+    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
       when: never
 
     # Non-publishing jobs should only run on staging branches in upstream
-    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
+    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH !~ /staging/'
       when: never
 
     # Jobs only intended for forks should always be skipped on upstream
-    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
+    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
       when: never
 
     # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
-    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: never
 
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
-    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: never
 
 
@@ -66,7 +71,7 @@ variables:
       allow_failure: true
 
     # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
-    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
       allow_failure: true
 
@@ -78,7 +83,7 @@ variables:
 
     # Forks pipeline jobs don't start automatically unless
     # QEMU_CI=2 is set
-    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
 
     # Jobs can run if any jobs they depend on were successful
docs/devel/ci-jobs.rst.inc +6 -0
index f72537853b..f9ef14f2eb 100644
@@ -75,6 +75,12 @@ in a handful of namespaces
    'latest', but if running pipelines for different branches
    concurrently, it should be overridden per pipeline.
 
+ * QEMU_CI_UPSTREAM - gitlab namespace that is considerd to be
+   the 'upstream'. This defaults to 'qemu-project'. Contributors
+   may choose to override this if they are modifying rules in
+   base.yml and need to validate how they will operate when in
+   an upstream context, as opposed to their fork context.
+
  * nnn - other misc variables not falling into the above
    categories, or using different names for historical reasons
    and not yet converted.
Signature · 1 line
2.40.1
Reviewed-by on v2 2/5
#3 lore raw

Re: [PATCH v2 2/5] gitlab: allow overriding name of the upstream repository

Quoted 12 lines · Daniel P. Berrangé

The CI rules have special logic for what happens in upstream. To enable contributors who modify CI rules to test this logic, however, they need to be able to override which repo is considered upstream. This introduces the 'QEMU_CI_UPSTREAM' variable

git push gitlab <branch> -o ci.variable=QEMU_CI_UPSTREAM=berrange

to make it look as if my namespace is the actual upstream. Namespace in this context refers to the path fragement in gitlab URLs that is above the repository. Typically this will be the contributor's gitlab login name.

Quoted 6 lines
---
  .gitlab-ci.d/base.yml      | 19 ++++++++++++-------
  docs/devel/ci-jobs.rst.inc |  6 ++++++
  2 files changed, 18 insertions(+), 7 deletions(-)

r~

#4 lore raw

PATCH v2 3/5 gitlab: stable staging branches publish containers in a separate tag

In reply to Daniel P. Berrangé

If the stable staging branches publish containers under the 'latest' tag they will clash with containers published on the primary staging branch, as well as with each other. This introduces logic that overrides the container tag when jobs run against the stable staging branches.

The CI_COMMIT_REF_SLUG variable we use expands to the git branch name, but with most special characters removed, such that it is valid as a docker tag name. eg 'staging-8.0' will get a slug of 'staging-8-0'

---
 .gitlab-ci.d/base.yml | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
.gitlab-ci.d/base.yml +22 -2
index a1d734267a..f379c182a7 100644
@@ -1,7 +1,7 @@
 
 variables:
-  # On stable branches this needs changing. Should also be
-  # overridden per pipeline if running pipelines concurrently
+  # On stable branches this is changed by later rules. Should also
+  # be overridden per pipeline if running pipelines concurrently
   # for different branches in contributor forks.
   QEMU_CI_CONTAINER_TAG: latest
 
@@ -16,6 +16,9 @@ variables:
 # Thus we group them into a number of stages, ordered from
 # most restrictive to least restrictive
 #
+# For pipelines running for stable "staging-X.Y" branches
+# we must override QEMU_CI_CONTAINER_TAG
+#
 .base_job_template:
   variables:
     # Each script line from will be in a collapsible section in the job output
@@ -61,11 +64,23 @@ variables:
     #############################################################
 
     # Optional jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_OPTIONAL && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: manual
+      allow_failure: true
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
     - if: '$QEMU_JOB_OPTIONAL'
       when: manual
       allow_failure: true
 
     # Skipped jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_SKIPPED && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: manual
+      allow_failure: true
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
     - if: '$QEMU_JOB_SKIPPED'
       when: manual
       allow_failure: true
@@ -87,4 +102,9 @@ variables:
       when: manual
 
     # Jobs can run if any jobs they depend on were successful
+    - if: '$QEMU_JOB_SKIPPED && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: on_success
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
     - when: on_success
Signature · 1 line
2.40.1
Reviewed-by on v2 3/5
#5 lore raw

Re: [PATCH v2 3/5] gitlab: stable staging branches publish containers in a separate tag

Quoted 9 lines · Daniel P. Berrangé

If the stable staging branches publish containers under the 'latest' tag they will clash with containers published on the primary staging branch, as well as with each other. This introduces logic that overrides the container tag when jobs run against the stable staging branches.

The CI_COMMIT_REF_SLUG variable we use expands to the git branch name, but with most special characters removed, such that it is valid as a docker tag name. eg 'staging-8.0' will get a slug of 'staging-8-0'

Quoted 5 lines
---
  .gitlab-ci.d/base.yml | 24 ++++++++++++++++++++++--
  1 file changed, 22 insertions(+), 2 deletions(-)

r~

#6 lore raw

PATCH v2 4/5 gitlab: avoid extra pipelines for tags and stable branches

In reply to Daniel P. Berrangé

In upstream context we only run pipelines on staging branches, and limited publishing jobs on the default branch.

We don't want to run pipelines on stable branches, or tags, because the content will have already been tested on a staging branch before getting pushed.

---
 .gitlab-ci.d/base.yml | 8 ++++++++
 1 file changed, 8 insertions(+)
.gitlab-ci.d/base.yml +8 -0
index f379c182a7..999149852e 100644
@@ -33,6 +33,14 @@ variables:
     # want jobs to run
     #############################################################
 
+    # Never run jobs upstream on stable branch, staging branch jobs already ran
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /^stable-/'
+      when: never
+
+    # Never run jobs upstream on tags, staging branch jobs already ran
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
+      when: never
+
     # Cirrus jobs can't run unless the creds / target repo are set
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
       when: never
Signature · 1 line
2.40.1
Reviewed-by on v2 4/5
#7 lore raw

Re: [PATCH v2 4/5] gitlab: avoid extra pipelines for tags and stable branches

Quoted 7 lines · Daniel P. Berrangé

In upstream context we only run pipelines on staging branches, and limited publishing jobs on the default branch.

We don't want to run pipelines on stable branches, or tags, because the content will have already been tested on a staging branch before getting pushed.

Quoted 5 lines
---
  .gitlab-ci.d/base.yml | 8 ++++++++
  1 file changed, 8 insertions(+)

r~

#8 lore raw

PATCH v2 5/5 gitlab: support disabling job auto-run in upstream

In reply to Daniel P. Berrangé

In forks QEMU_CI=1 can be used to create a pipeline but not auto-run any jobs. In upstream jobs always auto-run, which is equiv of QEMU_CI=2.

This supports setting QEMU_CI=1 in upstream, to disable job auto-run. This can be used to preserve CI minutes if repushing a branch to staging with a specific fix that only needs testing in limited scenarios.

---
 .gitlab-ci.d/base.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)
.gitlab-ci.d/base.yml +10 -0
index 999149852e..739ec07d7b 100644
@@ -109,6 +109,16 @@ variables:
     - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
 
+    # Upstream pipeline jobs start automatically unless told not to
+    # by setting QEMU_CI=1
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging/'
+      when: manual
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successful
     - if: '$QEMU_JOB_SKIPPED && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
       when: on_success
Signature · 1 line
2.40.1
#9 lore raw

Re: [PATCH v2 5/5] gitlab: support disabling job auto-run in upstream

.gitlab-ci.d/base.yml variables: · line 117

Quoted 6 lines · Daniel P. Berrangé
.gitlab-ci.d/base.yml +6 -0
+    # Upstream pipeline jobs start automatically unless told not to
+    # by setting QEMU_CI=1
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~/staging/'
+      when: manual
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG

So.. in this case we don't want 'latest' for 'staging' because we may not run all jobs? Or should this have checked staging-X.Y, since the following rule will handle 'staging'?

.gitlab-ci.d/base.yml variables: · line 121

Quoted 3 lines
.gitlab-ci.d/base.yml +3 -0
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
+      when: manual
+

r~

#10 lore raw

Re: [PATCH v2 5/5] gitlab: support disabling job auto-run in upstream

Quoted 11 lines · Richard Henderson

.gitlab-ci.d/base.yml variables: · line 117

Quoted 6 lines · Daniel P. Berrangé
.gitlab-ci.d/base.yml +6 -0
+    # Upstream pipeline jobs start automatically unless told not to
+    # by setting QEMU_CI=1
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~/staging/'
+      when: manual
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG

So.. in this case we don't want 'latest' for 'staging' because we may not run all jobs? Or should this have checked staging-X.Y, since the following rule will handle 'staging'?

Yes, this was a mistake. My original v1 was in fact correct.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw
In reply to Daniel P. Berrangé
Quoted 14 lines · Daniel P. Berrangé

We just (re)discovered that our gitlab rules don't work nicely with pipelines running from stable staging branches. Every pipeline gets published with the 'latest' tag, whether its the main staging branch or one of the stable staging branches. If pipelines for multiple staging branches run concurrently they'll get very confused and end up using the wrong container content. eg a 8.0 stable job will get run with a container from the development branch, or vica-verca.

With this series we dynamically change the tag so that the 'staging' branch still uses 'latest', but the stable 'staging-X.Y' branaches use a 'staging-X-Y' container tag.

We also let the container tag be set explicitly via the new variable

Quoted 8 lines

QEMU_CI_CONTAINER_TAG

to facilitate CI testing, the new variable

QEMU_CI_UPSTREAM

can be set to the fork namespace, to allow contributors to run a pipeline as if their fork were upstream.

Daniel, can you describe in a bit more detail (or refer to some text to read) about how this whole thing works, aka the "big picture"?

It smells like we're doing huge wasteful job here, but it might be just because I don't understand how it works.

Can't we prepare all containers separately and independently of regular qemu commits, and just use the prepared container images every time we run tests?

Also, why can't several tests (from several different pipelines, maybe run for different branches) use the same images (if we don't re-prepare them on each and every test run)?

I understand different branches might have different requirements for containers, like using older version of some OS, etc, - this is done by naming the container images appropriately, like debian-latest (for master) vs debian-bullseye (for stable-7.2) etc.

Still, preparing container images might be done separately from regular commits, like when ci config changes or when new version of something (fedora/redhat/etc) is released.

How does it all works, why we need to couple container creation to regular commits and re-create containers on every test run?

Thank you!

/mjt

#12 lore raw
Quoted 19 lines · Michael Tokarev
Quoted 18 lines · Daniel P. Berrangé

We just (re)discovered that our gitlab rules don't work nicely with pipelines running from stable staging branches. Every pipeline gets published with the 'latest' tag, whether its the main staging branch or one of the stable staging branches. If pipelines for multiple staging branches run concurrently they'll get very confused and end up using the wrong container content. eg a 8.0 stable job will get run with a container from the development branch, or vica-verca.

With this series we dynamically change the tag so that the 'staging' branch still uses 'latest', but the stable 'staging-X.Y' branaches use a 'staging-X-Y' container tag.

We also let the container tag be set explicitly via the new variable

QEMU_CI_CONTAINER_TAG

to facilitate CI testing, the new variable

Quoted 7 lines
Quoted 4 lines

QEMU_CI_UPSTREAM

can be set to the fork namespace, to allow contributors to run a pipeline as if their fork were upstream.

Daniel, can you describe in a bit more detail (or refer to some text to read) about how this whole thing works, aka the "big picture"?

What docs we have are at docs/devel/ci*rst but they're by no means complete.

Quoted 6 lines

It smells like we're doing huge wasteful job here, but it might be just because I don't understand how it works.

Can't we prepare all containers separately and independently of regular qemu commits, and just use the prepared container images every time we run tests?

Contributor branches, contributor patch series submissions, and pull requests periodically contain updates to the dockerfiles. When we build such code branches, we need to ensure that containers we using building inside match the contents of those dockerfiles otherwise the build will fail or perhaps worse, silently not test the changes in the correct way.

Also we're creating contaniers in a staging branch and there's no guarantee that the staging branch will actually get merged to master, it might get rejected if CI fails, so we're left with containers that might reflect a discard pull request.

A final point is that the distro base images change periodically and we want to pick up this content.

We don't want people triggering the pipelines to have to think about any of this to figure out whether a container rebuild is needed or not, as that is inherantly error prone. We need CI to "do the right thing" at all times.

Thus we will always build the containers in stage 1 of the pipeline. The stage 2 will then do the QEMU builds inside the just refreshed continers.

This is indeed wasteful if the patch series being tested did NOT have any container changes.

To mitigate this wastage, however, we tell docker to use the previously published containers as a cache. So docker build will compare each command in the dockerfile, against the cache and if they match, it will just copy across the contanier layer. This is a major performance win, but even this act of checking the cache does have some wastage.

Essentially with CI reliability is king and generally overules other considerations. A reliable, but computationally wasteful CI, is more usable than an unreliable, but computationally efficient CI.

Obviously the ideal is computationally efficient *and* reliable, and that's what we constantly want to strive towards.

Quoted 3 lines

Also, why can't several tests (from several different pipelines, maybe run for different branches) use the same images (if we don't re-prepare them on each and every test run)?

The cache should mostly address this, within the scope of our release stream.

Quoted 4 lines

I understand different branches might have different requirements for containers, like using older version of some OS, etc, - this is done by naming the container images appropriately, like debian-latest (for master) vs debian-bullseye (for stable-7.2) etc.

The package dependancies have changed frequently enough that each release of QEMU needs distinct containers. So with this patch series, we set container tag names based on the staging branch, so we'll get

   debian11:latest       (for master / staging)
   debian11:staging-8-0  (for 8.0.x release staging)
   debian11:staging-8-2  (for 7.2.x release staging)

Finally, I have at last figured out a way we can improve this that will probably let us remove the redundant container rebuilds for patch series that /don't/ include dockerfile changes. IOW, we may finally be able to achieve a computationally efficient and reliable CI, that doesn't require maintainers to figure out when to rebuild containers. It is on my to do list to try it out....

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|