Skip to contentSkip to conversation

Some improvement for gitlab CI jobs

PATCH In review Anthony PERARD 2 participants 9 messages Jul 21, 2022

lore
Share & tools
lore
#0 lore raw

v2 cover Some improvement for gitlab CI jobs

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2

v2:
- rebased

Allow build and test jobs to run concurently.

Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.

Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs

 automation/gitlab-ci/build.yaml | 20 ++++---
 automation/gitlab-ci/test.yaml  | 99 +++++++++------------------------
 2 files changed, 40 insertions(+), 79 deletions(-)
Signature · 1 line
Anthony PERARD
#1 lore raw

PATCH v2 1/4 automation: fix typo in .gcc-tmpl

The name of the field doesn't matter because it's use as a YAML achor, but it's nicer to have the proper spelling.

---
 automation/gitlab-ci/build.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
automation/gitlab-ci/build.yaml +1 -1
index 72f2a317ac..5eb7312754 100644
@@ -17,7 +17,7 @@
     - /^stable-.*/
 
 .gcc-tmpl:
-  variabes: &gcc
+  variables: &gcc
     CC: gcc
     CXX: g++
 
Signature · 1 line
Anthony PERARD
#2 lore raw

PATCH v2 2/4 automation: add a templates for test jobs

In reply to Anthony PERARD

Allow to set common configuration from a single place for all tests jobs.

---

Notes:
v2:
- rebased, to include "qemu-smoke-arm32-gcc".

 automation/gitlab-ci/test.yaml | 81 ++++++++--------------------------
 1 file changed, 18 insertions(+), 63 deletions(-)
automation/gitlab-ci/test.yaml +18 -63
index 42cd725a12..53b43801f4 100644
@@ -1,7 +1,15 @@
-# Test jobs
-build-each-commit-gcc:
+.test-jobs-common:
   stage: test
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  except:
+    - master
+    - smoke
+    - /^coverity-tested\/.*/
+    - /^stable-.*/
+
+# Test jobs
+build-each-commit-gcc:
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:stretch
     XEN_TARGET_ARCH: x86_64
@@ -16,15 +24,9 @@ build-each-commit-gcc:
   dependencies: []
   tags:
     - x86_64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-alpine-arm64-gcc:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
@@ -41,15 +43,9 @@ qemu-alpine-arm64-gcc:
     when: always
   tags:
     - arm64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-alpine-x86_64-gcc:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:stretch
   script:
@@ -65,15 +61,9 @@ qemu-alpine-x86_64-gcc:
     when: always
   tags:
     - x86_64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-smoke-arm64-gcc:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
@@ -89,15 +79,9 @@ qemu-smoke-arm64-gcc:
     when: always
   tags:
     - arm64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-smoke-arm32-gcc:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
@@ -112,15 +96,9 @@ qemu-smoke-arm32-gcc:
     when: always
   tags:
     - arm64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-smoke-x86-64-gcc:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:stretch
   script:
@@ -134,15 +112,9 @@ qemu-smoke-x86-64-gcc:
     - debian-stretch-gcc-debug
   tags:
     - x86_64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-smoke-x86-64-clang:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:stretch
   script:
@@ -156,15 +128,9 @@ qemu-smoke-x86-64-clang:
     - debian-unstable-clang-debug
   tags:
     - x86_64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-smoke-x86-64-gcc-pvh:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:stretch
   script:
@@ -178,15 +144,9 @@ qemu-smoke-x86-64-gcc-pvh:
     - debian-stretch-gcc-debug
   tags:
     - x86_64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
 
 qemu-smoke-x86-64-clang-pvh:
-  stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  extends: .test-jobs-common
   variables:
     CONTAINER: debian:stretch
   script:
@@ -200,8 +160,3 @@ qemu-smoke-x86-64-clang-pvh:
     - debian-unstable-clang-debug
   tags:
     - x86_64
-  except:
-    - master
-    - smoke
-    - /^coverity-tested\/.*/
-    - /^stable-.*/
Signature · 1 line
Anthony PERARD
#3 lore raw

PATCH v2 3/4 automation: only run test artifact jobs when needed

In reply to Anthony PERARD

Share the same "except" as the one used for tests.

---

Notes:
v2:
- rebased, to include "qemu-system-aarch64-6.0.0-arm32-export".

 automation/gitlab-ci/build.yaml | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
automation/gitlab-ci/build.yaml +12 -6
index 5eb7312754..23b306e7d0 100644
@@ -567,10 +567,16 @@ alpine-3.12-gcc-debug-arm64:
     CONTAINER: alpine:3.12-arm64v8
 
 
+## Test artifacts common
+
+.test-jobs-artifact-common:
+  stage: build
+  except: !reference [.test-jobs-common, except]
+
 # Arm test artifacts
 
 alpine-3.12-arm64-rootfs-export:
-  stage: build
+  extends: .test-jobs-artifact-common
   image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.12-arm64v8
   script:
     - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
@@ -581,7 +587,7 @@ alpine-3.12-arm64-rootfs-export:
     - arm64
 
 kernel-5.9.9-arm64-export:
-  stage: build
+  extends: .test-jobs-artifact-common
   image: registry.gitlab.com/xen-project/xen/tests-artifacts/kernel:5.9.9-arm64v8
   script:
     - mkdir binaries && cp /Image binaries/Image
@@ -592,7 +598,7 @@ kernel-5.9.9-arm64-export:
     - arm64
 
 qemu-system-aarch64-6.0.0-arm64-export:
-  stage: build
+  extends: .test-jobs-artifact-common
   image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
   script:
     - mkdir binaries && cp /qemu-system-aarch64 binaries/qemu-system-aarch64
@@ -603,7 +609,7 @@ qemu-system-aarch64-6.0.0-arm64-export:
     - arm64
 
 qemu-system-aarch64-6.0.0-arm32-export:
-  stage: build
+  extends: .test-jobs-artifact-common
   image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
   script:
     - mkdir binaries && cp /qemu-system-arm binaries/qemu-system-arm
@@ -616,7 +622,7 @@ qemu-system-aarch64-6.0.0-arm32-export:
 # x86_64 test artifacts
 
 alpine-3.12-rootfs-export:
-  stage: build
+  extends: .test-jobs-artifact-common
   image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.12
   script:
     - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
@@ -627,7 +633,7 @@ alpine-3.12-rootfs-export:
     - x86_64
 
 kernel-5.10.74-export:
-  stage: build
+  extends: .test-jobs-artifact-common
   image: registry.gitlab.com/xen-project/xen/tests-artifacts/kernel:5.10.74
   script:
     - mkdir binaries && cp /bzImage binaries/bzImage
Signature · 1 line
Anthony PERARD
#4 lore raw

PATCH v2 4/4 automation: use "needs" instead of "dependencies" for test jobs

In reply to Anthony PERARD

Like with "dependencies", the jobs will get artifacts from the jobs listed in "needs". But the test jobs can run as soon as the build jobs listed have finished.

---

Notes:
v2:
- rebased, to include "qemu-smoke-arm32-gcc".

 automation/gitlab-ci/test.yaml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
automation/gitlab-ci/test.yaml +9 -9
index 53b43801f4..26bdbcc3ea 100644
@@ -21,7 +21,7 @@ build-each-commit-gcc:
     paths:
       - '*.log'
     when: always
-  dependencies: []
+  needs: []
   tags:
     - x86_64
 
@@ -31,7 +31,7 @@ qemu-alpine-arm64-gcc:
     CONTAINER: debian:unstable-arm64v8
   script:
     - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
-  dependencies:
+  needs:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.9.9-arm64-export
@@ -50,7 +50,7 @@ qemu-alpine-x86_64-gcc:
     CONTAINER: debian:stretch
   script:
     - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee qemu-smoke-x86_64.log
-  dependencies:
+  needs:
     - alpine-3.12-gcc
     - alpine-3.12-rootfs-export
     - kernel-5.10.74-export
@@ -68,7 +68,7 @@ qemu-smoke-arm64-gcc:
     CONTAINER: debian:unstable-arm64v8
   script:
     - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
-  dependencies:
+  needs:
     - debian-unstable-gcc-arm64
     - kernel-5.9.9-arm64-export
     - qemu-system-aarch64-6.0.0-arm64-export
@@ -86,7 +86,7 @@ qemu-smoke-arm32-gcc:
     CONTAINER: debian:unstable-arm64v8
   script:
     - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
-  dependencies:
+  needs:
     - debian-unstable-gcc-arm32
     - qemu-system-aarch64-6.0.0-arm32-export
   artifacts:
@@ -108,7 +108,7 @@ qemu-smoke-x86-64-gcc:
       - smoke.serial
       - '*.log'
     when: always
-  dependencies:
+  needs:
     - debian-stretch-gcc-debug
   tags:
     - x86_64
@@ -124,7 +124,7 @@ qemu-smoke-x86-64-clang:
       - smoke.serial
       - '*.log'
     when: always
-  dependencies:
+  needs:
     - debian-unstable-clang-debug
   tags:
     - x86_64
@@ -140,7 +140,7 @@ qemu-smoke-x86-64-gcc-pvh:
       - smoke.serial
       - '*.log'
     when: always
-  dependencies:
+  needs:
     - debian-stretch-gcc-debug
   tags:
     - x86_64
@@ -156,7 +156,7 @@ qemu-smoke-x86-64-clang-pvh:
       - smoke.serial
       - '*.log'
     when: always
-  dependencies:
+  needs:
     - debian-unstable-clang-debug
   tags:
     - x86_64
Signature · 1 line
Anthony PERARD
#5 lore raw
In reply to Anthony PERARD
Quoted 8 lines · Anthony Perard

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2

v2:
- rebased

Allow build and test jobs to run concurently.

Quoted 8 lines

Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.

Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs

LGTM.  Got a CI run of these?

~Andrew

#6 lore raw
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Anthony Perard

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2

v2:
- rebased

Allow build and test jobs to run concurently.

Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.

Quoted 7 lines
Quoted 5 lines

Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs

LGTM.  Got a CI run of these?

Yes,

patchew's commit is in progress:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/593479157

A run in my private repo:
https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/593340820
(no failure due to this change)

Running on one of my branch called "stable-wip" doesn't create any pipeline, as expected.

Signature · 1 line
Anthony PERARD
#7 lore raw
Quoted 24 lines · Anthony PERARD
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Anthony Perard

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2

v2:
- rebased

Allow build and test jobs to run concurently.

Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.

Quoted 6 lines
Quoted 5 lines

Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs

LGTM.  Got a CI run of these?

Yes,

patchew's commit is in progress:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/593479157

Quoted 7 lines

A run in my private repo:
https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/593340820
(no failure due to this change)

Running on one of my branch called "stable-wip" doesn't create any pipeline, as expected.

Cool.  I'll get these 4 committed in due course.

One thing, looking at this right now.  Is there anything we can we prioritize the builds used to run the qemu smoke tests?  We've currently got x86 smoke tests waiting while the workers are building hypervisors unrelated to the tests.

~Andrew

#8 lore raw
Quoted 29 lines · Andrew Cooper
Quoted 28 lines · Anthony PERARD
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Anthony Perard

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2

v2:
- rebased

Allow build and test jobs to run concurently.

Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.

Quoted 6 lines
Quoted 5 lines

Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs

LGTM.  Got a CI run of these?

Yes,

patchew's commit is in progress:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/593479157

A run in my private repo:
https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/593340820
(no failure due to this change)

Quoted 6 lines
Quoted 3 lines

Running on one of my branch called "stable-wip" doesn't create any pipeline, as expected.

Cool.  I'll get these 4 committed in due course.

Thanks.

Quoted 4 lines

One thing, looking at this right now.  Is there anything we can we prioritize the builds used to run the qemu smoke tests?  We've currently got x86 smoke tests waiting while the workers are building hypervisors unrelated to the tests.

I think the only way would be to create a new stage before the "build" stage which would have all the build jobs needed by the tests jobs. But that kind of became more complicated.

Signature · 1 line
Anthony PERARD