Skip to contentSkip to conversation

gitlab: stopp edk2/opensbi jobs running in forks by default

PATCH In review Daniel P. Berrangé 3 participants 9 messages Jun 29, 2022 – Jul 4, 2022

lore
Share & tools
lore
#0 lore raw

cover gitlab: stopp edk2/opensbi jobs running in forks by default

When we introducd the QEMU_CI variable to control running of pipelines in gitlab forks, we didn't include the ekd2/opensbi jobs in the rules. This caused pipelines to be unexpectedly created in some cases. This addresses that oversight.

Daniel P. Berrangé (3):
gitlab: normalize indentation in edk2/opensbi rules
gitlab: tweak comments in edk2/opensbi jobs
gitlab: honour QEMU_CI variable in edk2/opensbi jobs

 .gitlab-ci.d/edk2.yml    | 133 ++++++++++++++++++++++----------------
 .gitlab-ci.d/opensbi.yml | 134 +++++++++++++++++++++++----------------
 2 files changed, 158 insertions(+), 109 deletions(-)
Signature · 1 line
2.36.1
#1 lore raw

PATCH 1/3 gitlab: normalize indentation in edk2/opensbi rules

The edk2/opensbi gitlab CI config was using single space indents which is not consistent with the rest of the gitlab CI config files.

---
 .gitlab-ci.d/edk2.yml    | 108 +++++++++++++++++++-------------------
 .gitlab-ci.d/opensbi.yml | 110 +++++++++++++++++++--------------------
 2 files changed, 109 insertions(+), 109 deletions(-)
.gitlab-ci.d/edk2.yml +54 -54
index 13d0f8b019..fbe763a282 100644
@@ -1,60 +1,60 @@
 # All jobs needing docker-edk2 must use the same rules it uses.
 .edk2_job_rules:
- rules: # Only run this job when ...
- - changes:
-   # this file is modified
-   - .gitlab-ci.d/edk2.yml
-   # or the Dockerfile is modified
-   - .gitlab-ci.d/edk2/Dockerfile
-   # or roms/edk2/ is modified (submodule updated)
-   - roms/edk2/*
-   when: on_success
- - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
-   when: on_success
- - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
-   when: on_success
+  rules: # Only run this job when ...
+    - changes:
+        # this file is modified
+        - .gitlab-ci.d/edk2.yml
+        # or the Dockerfile is modified
+        - .gitlab-ci.d/edk2/Dockerfile
+        # or roms/edk2/ is modified (submodule updated)
+        - roms/edk2/*
+      when: on_success
+    - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
+      when: on_success
+    - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
+      when: on_success
 
 docker-edk2:
- extends: .edk2_job_rules
- stage: containers
- image: docker:19.03.1
- services:
- - docker:19.03.1-dind
- variables:
-  GIT_DEPTH: 3
-  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
-  # We don't use TLS
-  DOCKER_HOST: tcp://docker:2375
-  DOCKER_TLS_CERTDIR: ""
- before_script:
- - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- script:
- - docker pull $IMAGE_TAG || true
- - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-                                        --tag $IMAGE_TAG .gitlab-ci.d/edk2
- - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- - docker push $IMAGE_TAG
+  extends: .edk2_job_rules
+  stage: containers
+  image: docker:19.03.1
+  services:
+    - docker:19.03.1-dind
+  variables:
+    GIT_DEPTH: 3
+    IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
+    # We don't use TLS
+    DOCKER_HOST: tcp://docker:2375
+    DOCKER_TLS_CERTDIR: ""
+  before_script:
+    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
+  script:
+    - docker pull $IMAGE_TAG || true
+    - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+                                           --tag $IMAGE_TAG .gitlab-ci.d/edk2
+    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+    - docker push $IMAGE_TAG
 
 build-edk2:
- extends: .edk2_job_rules
- stage: build
- needs: ['docker-edk2']
- artifacts:
-   paths: # 'artifacts.zip' will contains the following files:
-   - pc-bios/edk2*bz2
-   - pc-bios/edk2-licenses.txt
-   - edk2-stdout.log
-   - edk2-stderr.log
- image: $CI_REGISTRY_IMAGE:edk2-cross-build
- variables:
-   GIT_DEPTH: 3
- script: # Clone the required submodules and build EDK2
- - git submodule update --init roms/edk2
- - git -C roms/edk2 submodule update --init --
-     ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
-     BaseTools/Source/C/BrotliCompress/brotli
-     CryptoPkg/Library/OpensslLib/openssl
-     MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
- - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
- - echo "=== Using ${JOBS} simultaneous jobs ==="
- - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
+  extends: .edk2_job_rules
+  stage: build
+  needs: ['docker-edk2']
+  artifacts:
+    paths: # 'artifacts.zip' will contains the following files:
+      - pc-bios/edk2*bz2
+      - pc-bios/edk2-licenses.txt
+      - edk2-stdout.log
+      - edk2-stderr.log
+  image: $CI_REGISTRY_IMAGE:edk2-cross-build
+  variables:
+    GIT_DEPTH: 3
+  script: # Clone the required submodules and build EDK2
+    - git submodule update --init roms/edk2
+    - git -C roms/edk2 submodule update --init --
+       ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
+       BaseTools/Source/C/BrotliCompress/brotli
+       CryptoPkg/Library/OpensslLib/openssl
+       MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
+    - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+    - echo "=== Using ${JOBS} simultaneous jobs ==="
+    - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
.gitlab-ci.d/opensbi.yml +55 -55

116 lines not on this page · raw message

Signature · 1 line
2.36.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] gitlab: normalize indentation in edk2/opensbi rules

Quoted 9 lines · Daniel P. Berrangé

The edk2/opensbi gitlab CI config was using single space indents which is not consistent with the rest of the gitlab CI config files.

---
  .gitlab-ci.d/edk2.yml    | 108 +++++++++++++++++++-------------------
  .gitlab-ci.d/opensbi.yml | 110 +++++++++++++++++++--------------------
  2 files changed, 109 insertions(+), 109 deletions(-)

r~

#3 lore raw

PATCH 2/3 gitlab: tweak comments in edk2/opensbi jobs

In reply to Daniel P. Berrangé

Get rid of comments stating the obvious and re-arrange remaining comments. The opensbi split of rules for file matches is also merged into one rule.

---
 .gitlab-ci.d/edk2.yml    | 14 ++++++++------
 .gitlab-ci.d/opensbi.yml | 15 ++++++++-------
 2 files changed, 16 insertions(+), 13 deletions(-)
.gitlab-ci.d/edk2.yml +8 -6
index fbe763a282..905e02440f 100644
@@ -1,17 +1,19 @@
 # All jobs needing docker-edk2 must use the same rules it uses.
 .edk2_job_rules:
-  rules: # Only run this job when ...
+  rules:
+    # Run if any files affecting the build output are touched
     - changes:
-        # this file is modified
         - .gitlab-ci.d/edk2.yml
-        # or the Dockerfile is modified
         - .gitlab-ci.d/edk2/Dockerfile
-        # or roms/edk2/ is modified (submodule updated)
         - roms/edk2/*
       when: on_success
-    - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
+
+    # Run if the branch/tag starts with 'edk2'
+    - if: '$CI_COMMIT_REF_NAME =~ /^edk2/'
       when: on_success
-    - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
+
+    # Run if last commit msg contains 'EDK2' (case insensitive)
+    - if: '$CI_COMMIT_MESSAGE =~ /edk2/i'
       when: on_success
 
 docker-edk2:
.gitlab-ci.d/opensbi.yml +8 -7
index 0745ccdf10..753a003f93 100644
@@ -1,18 +1,19 @@
 # All jobs needing docker-opensbi must use the same rules it uses.
 .opensbi_job_rules:
-  rules: # Only run this job when ...
+  rules:
+    # Run if any files affecting the build output are touched
     - changes:
-        # this file is modified
         - .gitlab-ci.d/opensbi.yml
-        # or the Dockerfile is modified
         - .gitlab-ci.d/opensbi/Dockerfile
-      when: on_success
-    - changes: # or roms/opensbi/ is modified (submodule updated)
         - roms/opensbi/*
       when: on_success
-    - if: '$CI_COMMIT_REF_NAME =~ /^opensbi/' # or the branch/tag starts with 'opensbi'
+
+    # Run if the branch/tag starts with 'opensbi'
+    - if: '$CI_COMMIT_REF_NAME =~ /^opensbi/'
       when: on_success
-    - if: '$CI_COMMIT_MESSAGE =~ /opensbi/i' # or last commit description contains 'OpenSBI'
+
+    # Run if the last commit msg contains 'OpenSBI' (case insensitive)
+    - if: '$CI_COMMIT_MESSAGE =~ /opensbi/i'
       when: on_success
 
 docker-opensbi:
Signature · 1 line
2.36.1
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] gitlab: tweak comments in edk2/opensbi jobs

Quoted 9 lines · Daniel P. Berrangé

Get rid of comments stating the obvious and re-arrange remaining comments. The opensbi split of rules for file matches is also merged into one rule.

---
  .gitlab-ci.d/edk2.yml    | 14 ++++++++------
  .gitlab-ci.d/opensbi.yml | 15 ++++++++-------
  2 files changed, 16 insertions(+), 13 deletions(-)

r~

#5 lore raw

PATCH 3/3 gitlab: honour QEMU_CI variable in edk2/opensbi jobs

In reply to Daniel P. Berrangé

To preserve contributor CI credits we don't want jobs to run by default unless the QEMU_CI variable is set. For most jobs we can achieve this using the base template, but the edk2/opensbi jobs are a little special as they have some complex conditions we can't easily model in the base template.

We duplicate existing rules and put them under control of QEMU_CI variable, such that QEMU_CI=1 creates manual jobs and QEMU_CI=2 immediately runs jobs.

---
 .gitlab-ci.d/edk2.yml    | 23 +++++++++++++++++++++++
 .gitlab-ci.d/opensbi.yml | 23 +++++++++++++++++++++++
 2 files changed, 46 insertions(+)
.gitlab-ci.d/edk2.yml +23 -0
index 905e02440f..970bdbd315 100644
@@ -1,6 +1,29 @@
 # All jobs needing docker-edk2 must use the same rules it uses.
 .edk2_job_rules:
   rules:
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if any of the files affecting the build are touched
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      changes:
+        - .gitlab-ci.d/edk2.yml
+        - .gitlab-ci.d/edk2/Dockerfile
+        - roms/edk2/*
+      when: manual
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if the branch/tag starts with 'edk2'
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_REF_NAME =~ /^edk2/'
+      when: manual
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if last commit msg contains 'EDK2' (case insensitive)
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /edk2/i'
+      when: on_success
+
     # Run if any files affecting the build output are touched
     - changes:
         - .gitlab-ci.d/edk2.yml
.gitlab-ci.d/opensbi.yml +23 -0
index 753a003f93..04ed5a3ea1 100644
@@ -1,6 +1,29 @@
 # All jobs needing docker-opensbi must use the same rules it uses.
 .opensbi_job_rules:
   rules:
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if any files affecting the build output are touched
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      changes:
+        - .gitlab-ci.d/opensbi.yml
+        - .gitlab-ci.d/opensbi/Dockerfile
+        - roms/opensbi/*
+      when: manual
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if the branch/tag starts with 'opensbi'
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_REF_NAME =~ /^opensbi/'
+      when: manual
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if the last commit msg contains 'OpenSBI' (case insensitive)
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /opensbi/i'
+      when: manual
+
     # Run if any files affecting the build output are touched
     - changes:
         - .gitlab-ci.d/opensbi.yml
Signature · 1 line
2.36.1
#6 lore raw

Re: [PATCH 3/3] gitlab: honour QEMU_CI variable in edk2/opensbi jobs

.gitlab-ci.d/edk2.yml line 25

Quoted 9 lines · Daniel P. Berrangé
.gitlab-ci.d/edk2.yml +9 -0
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if the branch/tag starts with 'edk2'
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_REF_NAME =~ /^edk2/'
+      when: manual
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if last commit msg contains 'EDK2' (case insensitive)
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /edk2/i'
+      when: on_success

manual on last line?

r~

#7 lore raw

Re: [PATCH 3/3] gitlab: honour QEMU_CI variable in edk2/opensbi jobs

Quoted 11 lines · Richard Henderson

.gitlab-ci.d/edk2.yml line 25

Quoted 9 lines · Daniel P. Berrangé
.gitlab-ci.d/edk2.yml +9 -0
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if the branch/tag starts with 'edk2'
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_REF_NAME =~ /^edk2/'
+      when: manual
+
+    # In forks, if QEMU_CI=1 is set, then create manual job
+    # if last commit msg contains 'EDK2' (case insensitive)
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /edk2/i'
+      when: on_success
Quoted 1 line

manual on last line?

/facepalm Yes, of course.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw
In reply to Daniel P. Berrangé
Quoted 5 lines · Daniel P. Berrangé

When we introducd the QEMU_CI variable to control running of pipelines in gitlab forks, we didn't include the ekd2/opensbi jobs in the rules. This caused pipelines to be unexpectedly created in some cases. This addresses that oversight.

Quoted 8 lines

Daniel P. Berrangé (3):
gitlab: normalize indentation in edk2/opensbi rules
gitlab: tweak comments in edk2/opensbi jobs
gitlab: honour QEMU_CI variable in edk2/opensbi jobs

  .gitlab-ci.d/edk2.yml    | 133 ++++++++++++++++++++++----------------
  .gitlab-ci.d/opensbi.yml | 134 +++++++++++++++++++++++----------------
  2 files changed, 158 insertions(+), 109 deletions(-)

Thanks, queued to my testing-next branch now:

https://gitlab.com/thuth/qemu/-/commits/testing-next

(with the "on_success" changed to "manual" in the last patch)

Thomas