Some improvement for gitlab CI jobs
v2 cover Some improvement for gitlab CI jobs
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2
v2:
- rebased
Allow build and test jobs to run concurently.
Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.
Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs
automation/gitlab-ci/build.yaml | 20 ++++--- automation/gitlab-ci/test.yaml | 99 +++++++++------------------------ 2 files changed, 40 insertions(+), 79 deletions(-)
Signature · 1 line
Anthony PERARD
PATCH v2 1/4 automation: fix typo in .gcc-tmpl
The name of the field doesn't matter because it's use as a YAML achor, but it's nicer to have the proper spelling.
--- automation/gitlab-ci/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
automation/gitlab-ci/build.yaml +1 -1
@@ -17,7 +17,7 @@ - /^stable-.*/ .gcc-tmpl: - variabes: &gcc + variables: &gcc CC: gcc CXX: g++
Signature · 1 line
Anthony PERARD
PATCH v2 2/4 automation: add a templates for test jobs
In reply to Anthony PERARDAllow to set common configuration from a single place for all tests jobs.
---
Notes:
v2:
- rebased, to include "qemu-smoke-arm32-gcc".
automation/gitlab-ci/test.yaml | 81 ++++++++-------------------------- 1 file changed, 18 insertions(+), 63 deletions(-)
automation/gitlab-ci/test.yaml +18 -63
@@ -1,7 +1,15 @@ -# Test jobs -build-each-commit-gcc: +.test-jobs-common: stage: test image: registry.gitlab.com/xen-project/xen/${CONTAINER} + except: + - master + - smoke + - /^coverity-tested\/.*/ + - /^stable-.*/ + +# Test jobs +build-each-commit-gcc: + extends: .test-jobs-common variables: CONTAINER: debian:stretch XEN_TARGET_ARCH: x86_64
@@ -16,15 +24,9 @@ build-each-commit-gcc: dependencies: [] tags: - x86_64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-alpine-arm64-gcc: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:unstable-arm64v8 script:
@@ -41,15 +43,9 @@ qemu-alpine-arm64-gcc: when: always tags: - arm64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-alpine-x86_64-gcc: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:stretch script:
@@ -65,15 +61,9 @@ qemu-alpine-x86_64-gcc: when: always tags: - x86_64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-smoke-arm64-gcc: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:unstable-arm64v8 script:
@@ -89,15 +79,9 @@ qemu-smoke-arm64-gcc: when: always tags: - arm64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-smoke-arm32-gcc: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:unstable-arm64v8 script:
@@ -112,15 +96,9 @@ qemu-smoke-arm32-gcc: when: always tags: - arm64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-smoke-x86-64-gcc: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:stretch script:
@@ -134,15 +112,9 @@ qemu-smoke-x86-64-gcc: - debian-stretch-gcc-debug tags: - x86_64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-smoke-x86-64-clang: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:stretch script:
@@ -156,15 +128,9 @@ qemu-smoke-x86-64-clang: - debian-unstable-clang-debug tags: - x86_64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-smoke-x86-64-gcc-pvh: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:stretch script:
@@ -178,15 +144,9 @@ qemu-smoke-x86-64-gcc-pvh: - debian-stretch-gcc-debug tags: - x86_64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/ qemu-smoke-x86-64-clang-pvh: - stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + extends: .test-jobs-common variables: CONTAINER: debian:stretch script:
@@ -200,8 +160,3 @@ qemu-smoke-x86-64-clang-pvh: - debian-unstable-clang-debug tags: - x86_64 - except: - - master - - smoke - - /^coverity-tested\/.*/ - - /^stable-.*/
Signature · 1 line
Anthony PERARD
PATCH v2 3/4 automation: only run test artifact jobs when needed
In reply to Anthony PERARDShare the same "except" as the one used for tests.
---
Notes:
v2:
- rebased, to include "qemu-system-aarch64-6.0.0-arm32-export".
automation/gitlab-ci/build.yaml | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
automation/gitlab-ci/build.yaml +12 -6
@@ -567,10 +567,16 @@ alpine-3.12-gcc-debug-arm64: CONTAINER: alpine:3.12-arm64v8 +## Test artifacts common + +.test-jobs-artifact-common: + stage: build + except: !reference [.test-jobs-common, except] + # Arm test artifacts alpine-3.12-arm64-rootfs-export: - stage: build + extends: .test-jobs-artifact-common image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.12-arm64v8 script: - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
@@ -581,7 +587,7 @@ alpine-3.12-arm64-rootfs-export: - arm64 kernel-5.9.9-arm64-export: - stage: build + extends: .test-jobs-artifact-common image: registry.gitlab.com/xen-project/xen/tests-artifacts/kernel:5.9.9-arm64v8 script: - mkdir binaries && cp /Image binaries/Image
@@ -592,7 +598,7 @@ kernel-5.9.9-arm64-export: - arm64 qemu-system-aarch64-6.0.0-arm64-export: - stage: build + extends: .test-jobs-artifact-common image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8 script: - mkdir binaries && cp /qemu-system-aarch64 binaries/qemu-system-aarch64
@@ -603,7 +609,7 @@ qemu-system-aarch64-6.0.0-arm64-export: - arm64 qemu-system-aarch64-6.0.0-arm32-export: - stage: build + extends: .test-jobs-artifact-common image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8 script: - mkdir binaries && cp /qemu-system-arm binaries/qemu-system-arm
@@ -616,7 +622,7 @@ qemu-system-aarch64-6.0.0-arm32-export: # x86_64 test artifacts alpine-3.12-rootfs-export: - stage: build + extends: .test-jobs-artifact-common image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.12 script: - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
@@ -627,7 +633,7 @@ alpine-3.12-rootfs-export: - x86_64 kernel-5.10.74-export: - stage: build + extends: .test-jobs-artifact-common image: registry.gitlab.com/xen-project/xen/tests-artifacts/kernel:5.10.74 script: - mkdir binaries && cp /bzImage binaries/bzImage
Signature · 1 line
Anthony PERARD
PATCH v2 4/4 automation: use "needs" instead of "dependencies" for test jobs
In reply to Anthony PERARDLike with "dependencies", the jobs will get artifacts from the jobs listed in "needs". But the test jobs can run as soon as the build jobs listed have finished.
---
Notes:
v2:
- rebased, to include "qemu-smoke-arm32-gcc".
automation/gitlab-ci/test.yaml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
automation/gitlab-ci/test.yaml +9 -9
@@ -21,7 +21,7 @@ build-each-commit-gcc: paths: - '*.log' when: always - dependencies: [] + needs: [] tags: - x86_64
@@ -31,7 +31,7 @@ qemu-alpine-arm64-gcc: CONTAINER: debian:unstable-arm64v8 script: - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log - dependencies: + needs: - alpine-3.12-gcc-arm64 - alpine-3.12-arm64-rootfs-export - kernel-5.9.9-arm64-export
@@ -50,7 +50,7 @@ qemu-alpine-x86_64-gcc: CONTAINER: debian:stretch script: - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee qemu-smoke-x86_64.log - dependencies: + needs: - alpine-3.12-gcc - alpine-3.12-rootfs-export - kernel-5.10.74-export
@@ -68,7 +68,7 @@ qemu-smoke-arm64-gcc: CONTAINER: debian:unstable-arm64v8 script: - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log - dependencies: + needs: - debian-unstable-gcc-arm64 - kernel-5.9.9-arm64-export - qemu-system-aarch64-6.0.0-arm64-export
@@ -86,7 +86,7 @@ qemu-smoke-arm32-gcc: CONTAINER: debian:unstable-arm64v8 script: - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log - dependencies: + needs: - debian-unstable-gcc-arm32 - qemu-system-aarch64-6.0.0-arm32-export artifacts:
@@ -108,7 +108,7 @@ qemu-smoke-x86-64-gcc: - smoke.serial - '*.log' when: always - dependencies: + needs: - debian-stretch-gcc-debug tags: - x86_64
@@ -124,7 +124,7 @@ qemu-smoke-x86-64-clang: - smoke.serial - '*.log' when: always - dependencies: + needs: - debian-unstable-clang-debug tags: - x86_64
@@ -140,7 +140,7 @@ qemu-smoke-x86-64-gcc-pvh: - smoke.serial - '*.log' when: always - dependencies: + needs: - debian-stretch-gcc-debug tags: - x86_64
@@ -156,7 +156,7 @@ qemu-smoke-x86-64-clang-pvh: - smoke.serial - '*.log' when: always - dependencies: + needs: - debian-unstable-clang-debug tags: - x86_64
Signature · 1 line
Anthony PERARD
Quoted 8 lines · Anthony Perard
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2v2:
- rebasedAllow build and test jobs to run concurently.
Quoted 8 lines
Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.
Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobs
LGTM. Got a CI run of these?
~Andrew
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Anthony Perard
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2v2:
- rebasedAllow build and test jobs to run concurently.
Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.
Quoted 7 lines
Quoted 5 lines
Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobsLGTM. Got a CI run of these?
Yes,
patchew's commit is in progress:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/593479157
A run in my private repo:
https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/593340820
(no failure due to this change)
Running on one of my branch called "stable-wip" doesn't create any pipeline, as expected.
Signature · 1 line
Anthony PERARD
Quoted 24 lines · Anthony PERARD
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Anthony Perard
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2v2:
- rebasedAllow build and test jobs to run concurently.
Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.
Quoted 6 lines
Quoted 5 lines
Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobsLGTM. Got a CI run of these?
Yes,
patchew's commit is in progress:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/593479157
Quoted 7 lines
A run in my private repo:
https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/593340820
(no failure due to this change)Running on one of my branch called "stable-wip" doesn't create any pipeline, as expected.
Cool. I'll get these 4 committed in due course.
One thing, looking at this right now. Is there anything we can we prioritize the builds used to run the qemu smoke tests? We've currently got x86 smoke tests waiting while the workers are building hypervisors unrelated to the tests.
~Andrew
Quoted 29 lines · Andrew Cooper
Quoted 28 lines · Anthony PERARD
Quoted 12 lines · Andrew Cooper
Quoted 11 lines · Anthony Perard
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-improvement-v2v2:
- rebasedAllow build and test jobs to run concurently.
Avoid running "test artifact" jobs on branch "master" and other, when test jobs aren't runned.
Quoted 6 lines
Quoted 5 lines
Anthony PERARD (4):
automation: fix typo in .gcc-tmpl
automation: add a templates for test jobs
automation: only run test artifact jobs when needed
automation: use "needs" instead of "dependencies" for test jobsLGTM. Got a CI run of these?
Yes,
patchew's commit is in progress:
https://gitlab.com/xen-project/patchew/xen/-/pipelines/593479157A run in my private repo:
https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/593340820
(no failure due to this change)
Quoted 6 lines
Quoted 3 lines
Running on one of my branch called "stable-wip" doesn't create any pipeline, as expected.
Cool. I'll get these 4 committed in due course.
Thanks.
Quoted 4 lines
One thing, looking at this right now. Is there anything we can we prioritize the builds used to run the qemu smoke tests? We've currently got x86 smoke tests waiting while the workers are building hypervisors unrelated to the tests.
I think the only way would be to create a new stage before the "build" stage which would have all the build jobs needed by the tests jobs. But that kind of became more complicated.
Signature · 1 line
Anthony PERARD