gitlab: build containers to use in build jobs
cover gitlab: build containers to use in build jobs
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.
We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....
...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.
These later jobs are now faster because they're not having to install any software.
There is a performance hit to build the images, however, they are cached in the GitLab docker registry. A user forking the repo will use the cache from qemu-project/qemu and their own local fork. So overall the time penalty to build images will only be seen if the user modifiers tests/docker/dockerfiles in their patch series, or if the base image changes.
The GitLab container registry is public, so we're not limited to using the built images in GitLab CI. Any other CI system that uses docker can consume these images. Similarly we could change the tests/docker/Makefile rules so that developers pull from https://gitlab.com/qemu-project/qemu, instead of building images locally. This is why we're building all the images, instead of just the handful the current jobs want.
The interesting stuff is mostly in patch 2.
Patch 3 is a quick hack I did to convert existing jobs to use the newly built images. I know there's other work being done on the GitLab jobs right now that probably conflicts with this though, so I've not invested much time in patch 3. Consider it more an illustration of what's possible, than a finished proposal for merge. The patch 3 diff is fairly unintelligble, so it is easier to look at the final result:
https://gitlab.com/berrange/qemu/-/blob/ci-cont/.gitlab-ci.yml
An example pipeline can be viewed here:
https://gitlab.com/berrange/qemu/-/pipelines/158824359
The cached images are here:
https://gitlab.com/berrange/qemu/container_registry
Daniel P. Berrangé (3):
gitlab: introduce explicit "container" and "build" stages
gitlab: build all container images during CI
gitlab: convert jobs to use custom built containers
.gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.d/edk2.yml | 3 +- .gitlab-ci.d/opensbi.yml | 3 +- .gitlab-ci.yml | 187 +++++++++++++-------------- 4 files changed, 340 insertions(+), 101 deletions(-) create mode 100644 .gitlab-ci.d/containers.yml
Signature · 1 line
2.24.1
RFC PATCH 1/3 gitlab: introduce explicit "container" and "build" stages
If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.
--- .gitlab-ci.d/edk2.yml | 3 ++- .gitlab-ci.d/opensbi.yml | 3 ++- .gitlab-ci.yml | 11 +++++++++++ 3 files changed, 15 insertions(+), 2 deletions(-)
.gitlab-ci.d/edk2.yml +2 -1
@@ -1,5 +1,5 @@ docker-edk2: - stage: build + stage: containers rules: # Only run this job when the Dockerfile is modified - changes: - .gitlab-ci-edk2.yml
@@ -24,6 +24,7 @@ docker-edk2: - docker push $IMAGE_TAG build-edk2: + stage: build rules: # Only run this job when ... - changes: # ... roms/edk2/ is modified (submodule updated) - roms/edk2/*
.gitlab-ci.d/opensbi.yml +2 -1
@@ -1,5 +1,5 @@ docker-opensbi: - stage: build + stage: containers rules: # Only run this job when the Dockerfile is modified - changes: - .gitlab-ci-opensbi.yml
@@ -24,6 +24,7 @@ docker-opensbi: - docker push $IMAGE_TAG build-opensbi: + stage: build rules: # Only run this job when ... - changes: # ... roms/opensbi/ is modified (submodule updated) - roms/opensbi/*
.gitlab-ci.yml +11 -0
@@ -1,3 +1,7 @@ +stages: + - containers + - build + include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml'
@@ -17,6 +21,7 @@ include: - JOBS=$(expr $(nproc) + 1) build-system1: + stage: build image: ubuntu:19.10 <<: *before_script_apt script:
@@ -31,6 +36,7 @@ build-system1: - make -j"$JOBS" check build-system2: + stage: build image: fedora:latest <<: *before_script_dnf script:
@@ -46,6 +52,7 @@ build-system2: - make -j"$JOBS" check build-disabled: + stage: build image: fedora:latest <<: *before_script_dnf script:
@@ -62,6 +69,7 @@ build-disabled: - make -j"$JOBS" check-qtest SPEED=slow build-tcg-disabled: + stage: build image: centos:8 <<: *before_script_dnf script:
@@ -82,6 +90,7 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: + stage: build <<: *before_script_apt script: - mkdir build
@@ -92,6 +101,7 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: + stage: build image: fedora:latest <<: *before_script_dnf script:
@@ -106,6 +116,7 @@ build-clang: - make -j"$JOBS" check build-tci: + stage: build image: centos:8 <<: *before_script_dnf script:
Signature · 1 line
2.24.1
Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages
Quoted 111 lines · Daniel P. Berrangé
If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/edk2.yml | 3 ++- .gitlab-ci.d/opensbi.yml | 3 ++- .gitlab-ci.yml | 11 +++++++++++ 3 files changed, 15 insertions(+), 2 deletions(-).gitlab-ci.d/edk2.yml +2
-1@@ -1,5 +1,5 @@ docker-edk2: - stage: build + stage: containers rules: # Only run this job when the Dockerfile is modified - changes: - .gitlab-ci-edk2.yml@@ -24,6 +24,7 @@ docker-edk2: - docker push $IMAGE_TAG build-edk2: + stage: build rules: # Only run this job when ... - changes: # ... roms/edk2/ is modified (submodule updated) - roms/edk2/*.gitlab-ci.d/opensbi.yml +2
-1@@ -1,5 +1,5 @@ docker-opensbi: - stage: build + stage: containers rules: # Only run this job when the Dockerfile is modified - changes: - .gitlab-ci-opensbi.yml@@ -24,6 +24,7 @@ docker-opensbi: - docker push $IMAGE_TAG build-opensbi: + stage: build rules: # Only run this job when ... - changes: # ... roms/opensbi/ is modified (submodule updated) - roms/opensbi/*.gitlab-ci.yml +10
-0@@ -1,3 +1,7 @@ +stages: + - containers + - build + include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml'@@ -17,6 +21,7 @@ include: - JOBS=$(expr $(nproc) + 1) build-system1: + stage: build image: ubuntu:19.10 <<: *before_script_apt script:@@ -31,6 +36,7 @@ build-system1: - make -j"$JOBS" check build-system2: + stage: build image: fedora:latest <<: *before_script_dnf script:@@ -46,6 +52,7 @@ build-system2: - make -j"$JOBS" check build-disabled: + stage: build image: fedora:latest <<: *before_script_dnf script:@@ -62,6 +69,7 @@ build-disabled: - make -j"$JOBS" check-qtest SPEED=slow build-tcg-disabled: + stage: build image: centos:8 <<: *before_script_dnf script:@@ -82,6 +90,7 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: + stage: build <<: *before_script_apt script: - mkdir build@@ -92,6 +101,7 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: + stage: build image: fedora:latest <<: *before_script_dnf script:
.gitlab-ci.yml build-clang: · line 122
Quoted 7 lines
.gitlab-ci.yml +1
-0@@ -106,6 +116,7 @@ build-clang: - make -j"$JOBS" check build-tci: + stage: build image: centos:8 <<: *before_script_dnf script:
Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages
In reply to Daniel P. BerrangéQuoted 16 lines · Daniel P. Berrangé
If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/edk2.yml | 3 ++- .gitlab-ci.d/opensbi.yml | 3 ++- .gitlab-ci.yml | 11 +++++++++++ 3 files changed, 15 insertions(+), 2 deletions(-).gitlab-ci.d/edk2.yml +0
-0
.gitlab-ci.d/edk2.yml line 5
Quoted 6 lines
.gitlab-ci.d/edk2.yml +1
-1@@ -1,5 +1,5 @@ docker-edk2: - stage: build + stage: containers rules: # Only run this job when the Dockerfile is modified - changes: - .gitlab-ci-edk2.yml
Uh, oh, I guess I should have changed that line to .gitlab-ci.d/edk2.yml when I renamed that file .... will send a patch...
Quoted 13 lines
.gitlab-ci.d/edk2.yml +1
-0@@ -24,6 +24,7 @@ docker-edk2: - docker push $IMAGE_TAG build-edk2: + stage: build rules: # Only run this job when ... - changes: # ... roms/edk2/ is modified (submodule updated) - roms/edk2/*.gitlab-ci.d/opensbi.yml +0
-0
.gitlab-ci.d/opensbi.yml line 5
Quoted 6 lines
.gitlab-ci.d/opensbi.yml +1
-1@@ -1,5 +1,5 @@ docker-opensbi: - stage: build + stage: containers rules: # Only run this job when the Dockerfile is modified - changes: - .gitlab-ci-opensbi.yml
dito
Quoted 70 lines
.gitlab-ci.d/opensbi.yml +1
-0@@ -24,6 +24,7 @@ docker-opensbi: - docker push $IMAGE_TAG build-opensbi: + stage: build rules: # Only run this job when ... - changes: # ... roms/opensbi/ is modified (submodule updated) - roms/opensbi/*.gitlab-ci.yml +10
-0@@ -1,3 +1,7 @@ +stages: + - containers + - build + include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml'@@ -17,6 +21,7 @@ include: - JOBS=$(expr $(nproc) + 1) build-system1: + stage: build image: ubuntu:19.10 <<: *before_script_apt script:@@ -31,6 +36,7 @@ build-system1: - make -j"$JOBS" check build-system2: + stage: build image: fedora:latest <<: *before_script_dnf script:@@ -46,6 +52,7 @@ build-system2: - make -j"$JOBS" check build-disabled: + stage: build image: fedora:latest <<: *before_script_dnf script:@@ -62,6 +69,7 @@ build-disabled: - make -j"$JOBS" check-qtest SPEED=slow build-tcg-disabled: + stage: build image: centos:8 <<: *before_script_dnf script:@@ -82,6 +90,7 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: + stage: build <<: *before_script_apt script: - mkdir build@@ -92,6 +101,7 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: + stage: build image: fedora:latest <<: *before_script_dnf script:
.gitlab-ci.yml build-clang: · line 122
Quoted 7 lines
.gitlab-ci.yml +1
-0@@ -106,6 +116,7 @@ build-clang: - make -j"$JOBS" check build-tci: + stage: build image: centos:8 <<: *before_script_dnf script:
Philippe Mathieu-Daudé
Reviewed-byRe: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages
Quoted 12 lines · Thomas Huth
Quoted 11 lines · Daniel P. Berrangé
If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.
Signed-off-by Daniel P. Berrangé <[email protected]>---
.gitlab-ci.d/edk2.yml | 3 ++-
.gitlab-ci.d/opensbi.yml| 3 ++-
.gitlab-ci.yml | 11 +++++++++++3 files changed, 15 insertions(+), 2 deletions(-)
Quoted 14 lines
Quoted 11 lines
.gitlab-ci.d/edk2.yml +0
-0@@ -1,5 +1,5 @@docker-edk2:
- stage: build
+ stage: containers
rules: # Only run this job when the Dockerfile is modified
- changes:
- .gitlab-ci-edk2.ymlUh, oh, I guess I should have changed that line to
.gitlab-ci.d/edk2.ymlwhen I renamed that file .... will send a patch...
Well, your patch has been reviewed...
Quoted 13 lines
Quoted 13 lines
.gitlab-ci.d/edk2.yml +0
-0@@ -24,6 +24,7 @@ docker-edk2:- docker push $IMAGE_TAG
build-edk2:
+ stage: build
rules: # Only run this job when ...
- changes: # ... roms/edk2/ is modified (submodule updated)
- roms/edk2/*.gitlab-ci.d/opensbi.yml +0
-0@@ -1,5 +1,5 @@
Quoted 8 lines
Quoted 6 lines
docker-opensbi:
- stage: build
+ stage: containers
rules: # Only run this job when the Dockerfile is modified
- changes:
- .gitlab-ci-opensbi.ymldito
Oops...
Quoted 64 lines
Quoted 64 lines
.gitlab-ci.d/opensbi.yml +0
-0@@ -24,6 +24,7 @@ docker-opensbi:- docker push $IMAGE_TAG
build-opensbi:
+ stage: build
rules: # Only run this job when ...
- changes: # ... roms/opensbi/ is modified (submodule updated)
- roms/opensbi/*.gitlab-ci.yml +4
-0@@ -1,3 +1,7 @@ +stages: + - containers + - build +include:
- local: '/.gitlab-ci.d/edk2.yml'
- local: '/.gitlab-ci.d/opensbi.yml'.gitlab-ci.yml +0
-0@@ -17,6 +21,7 @@ include: - JOBS=$(expr $(nproc) + 1)build-system1:
+ stage: build
image: ubuntu:19.10
<<:*before_script_apt
script:.gitlab-ci.yml +0
-0@@ -31,6 +36,7 @@ build-system1:- make -j"$JOBS" check
build-system2:
+ stage: build
image: fedora:latest
<<:*before_script_dnf
script:.gitlab-ci.yml +0
-0@@ -46,6 +52,7 @@ build-system2:- make -j"$JOBS" check
build-disabled:
+ stage: build
image: fedora:latest
<<:*before_script_dnf
script:.gitlab-ci.yml +0
-0@@ -62,6 +69,7 @@ build-disabled:- make -j"$JOBS" check-qtest SPEED=slow
build-tcg-disabled:
+ stage: build
image: centos:8
<<:*before_script_dnf
script:.gitlab-ci.yml +0
-0@@ -82,6 +90,7 @@ build-tcg-disabled:260 261 262 263 264 270 272 273 277 279 build-user:
+ stage: build
<<:*before_script_apt
script:
- mkdir build.gitlab-ci.yml +0
-0@@ -92,6 +101,7 @@ build-user:- make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang:
+ stage: build
image: fedora:latest
<<:*before_script_dnf
script:.gitlab-ci.yml +0
-0@@ -106,6 +116,7 @@ build-clang:- make -j"$JOBS" check
Quoted 9 lines
Quoted 6 lines
build-tci:
+ stage: build
image: centos:8
<<:*before_script_dnf
script:Reviewed-by Thomas Huth <[email protected]>
RFC PATCH 2/3 gitlab: build all container images during CI
In reply to Daniel P. BerrangéWe have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.yml
.gitlab-ci.d/containers.yml new file +248 -0
@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout + +amd64-centos7-container: + <<: *container_job_definition + variables: + NAME: centos7 + +amd64-centos8-container: + <<: *container_job_definition + variables: + NAME: centos8 + +amd64-debian10-container: + <<: *container_job_definition + variables: + NAME: debian10 + +amd64-debian11-container: + <<: *container_job_definition + variables: + NAME: debian11 + +amd64-debian9-container: + <<: *container_job_definition + variables: + NAME: debian9 + +amd64-debian9-mxe-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian9-mxe + +alpha-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-alpha-cross + +amd64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64-cross + +amd64-debian-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64 + +arm64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-arm64-cross + +arm64-test-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian11-container'] + variables: + NAME: debian-arm64-test-cross + +armel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armel-cross + +armhf-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armhf-cross + +hppa-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-hppa-cross + +m68k-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-m68k-cross + +mips64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64-cross + +mips64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64el-cross + +mips-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips-cross + +mipsel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mipsel-cross + +powerpc-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-powerpc-cross + +ppc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64-cross + +ppc64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64el-cross + +riscv64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-riscv64-cross + +s390x-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-s390x-cross + +sh4-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sh4-cross + +sparc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sparc64-cross + +tricore-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian-tricore-cross + +win32-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win32-cross + +win64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win64-cross + +xtensa-debian-cross-container: + <<: *container_job_definition + variables: + NAME: debian-xtensa-cross + +cris-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-cris-cross + +amd64-fedora-container: + <<: *container_job_definition + variables: + NAME: fedora + +i386-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-i386-cross + +amd64-ubuntu1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu1804 + +amd64-ubuntu2004-container: + <<: *container_job_definition + variables: + NAME: ubuntu2004 + +amd64-ubuntu-container: + <<: *container_job_definition + variables: + NAME: ubuntu +
.gitlab-ci.yml +3 -0
@@ -1,10 +1,13 @@ stages: - containers + - containers-layer2 + - containers-layer3 - build include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' + - local: '/.gitlab-ci.d/containers.yml' .update_apt_template: &before_script_apt before_script:
Signature · 1 line
2.24.1
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 11 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
OMG you did it! Lovely... 😍
Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't populated on all the runners yet? Or we have to use the same runner again to hit its cache?
Quoted 38 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +24
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout + +amd64-centos7-container: + <<: *container_job_definition + variables: + NAME: centos7
[...]
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Quoted 6 lines
OMG you did it! Lovely... 😍
Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't populated on all the runners yet? Or we have to use the same runner again to hit its cache?
It is caching it but it isn't obvious what to look for.
The "docker build" command is always still run, but you'll notice each of the actual commands in the dockerfile are followed by a message " ---> Using cache", instead of the normal command output.
Compare the "amd64-debian9-container" job as an example.
Here is the output seen in the original job which actually did a real build:
https://gitlab.com/berrange/qemu/-/jobs/605783351
And here is the output from the pipeline you point to above:
https://gitlab.com/berrange/qemu/-/jobs/606175855
The cached case is still taking 2 mins 30 seconds, but the original full build took 4 mins 50 seconds.
The difference will be more noticable if we expand the list of packages install in the images to cover more of QEMU's possible dependancies.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 21 lines · Daniel P. Berrangé
Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Quoted 6 lines
OMG you did it! Lovely... 😍
Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't populated on all the runners yet? Or we have to use the same runner again to hit its cache?
Quoted 6 lines
It is caching it but it isn't obvious what to look for.
The "docker build" command is always still run, but you'll notice each of the actual commands in the dockerfile are followed by a message " ---> Using cache", instead of the normal command output.
Indeed!
Quoted 10 lines
Compare the "amd64-debian9-container" job as an example.
Here is the output seen in the original job which actually did a real build:
https://gitlab.com/berrange/qemu/-/jobs/605783351And here is the output from the pipeline you point to above:
Quoted 7 lines
https://gitlab.com/berrange/qemu/-/jobs/606175855The cached case is still taking 2 mins 30 seconds, but the original full build took 4 mins 50 seconds.
The difference will be more noticable if we expand the list of packages install in the images to cover more of QEMU's possible dependancies.
The difference is already very noticeable on my host:
$ time docker pull registry.gitlab.com/berrange/qemu/ci-centos8:latest latest: Pulling from berrange/qemu/ci-centos8 8a29a15cefae: Pull complete 4a748adf2909: Pull complete 3c034506a458: Pull complete cd48de325fa1: Pull complete Digest: sha256:a6580433f5456d7c49e2b9b796c717c79a5fa15f4eecf71bdbcffd9df6b8217a Status: Downloaded newer image for registry.gitlab.com/berrange/qemu/ci-centos8:latest registry.gitlab.com/berrange/qemu/ci-centos8:latest real 0m53.566s user 0m0.115s sys 0m0.081s
Quoted 4 lines
Regards,
Daniel
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
In reply to Daniel P. BerrangéQuoted 35 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +10
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export
Quoted 1 line
COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
It would be nice if we could keep the same form as they have in the local registry which would make it easier to integrate with the rest of the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"
Otherwise it looks good:
Signature · 1 line
Alex Bennée
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 35 lines · Alex Bennée
Quoted 32 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +7
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind
Quoted 8 lines
Quoted 4 lines
.gitlab-ci.d/containers.yml new file +3
-0+ before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - exportCOMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
It would be nice if we could keep the same form as they have in the local registry which would make it easier to integrate with the rest of the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"
Every time I re-discover it, I find the QEMU container naming really surprising. It is not following the normal best practice for naming containers. Expected container naming convention is that the image name reflects the general content set, and the tag reflects a version number. QEMU has shifted it along, so container name is just a fixed string, and the tag reflects the contenet set, and there is no version.
QEMU's naming will cause problems with caching in GitLab. As GitLab expects the normal container naming scheme, it has a job which expires old versions of an image once there are more than 10 tags. So we have to use the normal naming scheme. Ideally we would change rest of QEMU to use the normal scheme too.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 52 lines · Daniel P. Berrangé
Quoted 35 lines · Alex Bennée
Quoted 32 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +7
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dindQuoted 8 lines
Quoted 4 lines
.gitlab-ci.d/containers.yml new file +3
-0+ before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - exportCOMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
It would be nice if we could keep the same form as they have in the local registry which would make it easier to integrate with the rest of the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"
Every time I re-discover it, I find the QEMU container naming really surprising. It is not following the normal best practice for naming containers. Expected container naming convention is that the image name reflects the general content set, and the tag reflects a version number. QEMU has shifted it along, so container name is just a fixed string, and the tag reflects the contenet set, and there is no version.
Quoted 5 lines
QEMU's naming will cause problems with caching in GitLab. As GitLab expects the normal container naming scheme, it has a job which expires old versions of an image once there are more than 10 tags. So we have to use the normal naming scheme. Ideally we would change rest of QEMU to use the normal scheme too.
Fair enough.. I'll look into it.
Quoted 3 lines
Regards,
Daniel
Signature · 1 line
Alex Bennée
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
In reply to Daniel P. BerrangéQuoted 27 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +2
-0@@ -0,0 +1,248 @@ + +
.gitlab-ci.d/containers.yml line 12
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
I can see this in the output:
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
I have to admit that I have only little knowledge about docker ... but could there be an issue here? Should this be done in a different way?
Quoted 245 lines
.gitlab-ci.d/containers.yml new file +236
-0+ script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout + +amd64-centos7-container: + <<: *container_job_definition + variables: + NAME: centos7 + +amd64-centos8-container: + <<: *container_job_definition + variables: + NAME: centos8 + +amd64-debian10-container: + <<: *container_job_definition + variables: + NAME: debian10 + +amd64-debian11-container: + <<: *container_job_definition + variables: + NAME: debian11 + +amd64-debian9-container: + <<: *container_job_definition + variables: + NAME: debian9 + +amd64-debian9-mxe-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian9-mxe + +alpha-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-alpha-cross + +amd64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64-cross + +amd64-debian-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64 + +arm64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-arm64-cross + +arm64-test-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian11-container'] + variables: + NAME: debian-arm64-test-cross + +armel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armel-cross + +armhf-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armhf-cross + +hppa-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-hppa-cross + +m68k-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-m68k-cross + +mips64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64-cross + +mips64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64el-cross + +mips-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips-cross + +mipsel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mipsel-cross + +powerpc-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-powerpc-cross + +ppc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64-cross + +ppc64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64el-cross + +riscv64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-riscv64-cross + +s390x-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-s390x-cross + +sh4-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sh4-cross + +sparc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sparc64-cross + +tricore-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian-tricore-cross + +win32-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win32-cross + +win64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win64-cross + +xtensa-debian-cross-container: + <<: *container_job_definition + variables: + NAME: debian-xtensa-cross + +cris-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-cris-cross + +amd64-fedora-container: + <<: *container_job_definition + variables: + NAME: fedora + +i386-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-i386-cross + +amd64-ubuntu1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu1804 + +amd64-ubuntu2004-container: + <<: *container_job_definition + variables: + NAME: ubuntu2004 + +amd64-ubuntu-container: + <<: *container_job_definition + variables: + NAME: ubuntu +.gitlab-ci.yml +1
-0@@ -1,10 +1,13 @@ stages: - containers + - containers-layer2
.gitlab-ci.yml line 13
Quoted 10 lines
.gitlab-ci.yml +2
-0+ - containers-layer3 - build include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' + - local: '/.gitlab-ci.d/containers.yml' .update_apt_template: &before_script_apt before_script:
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 41 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +2
-0@@ -0,0 +1,248 @@ + +.gitlab-ci.d/containers.yml line 12
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"I can see this in the output:
Quoted 4 lines
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
I have to admit that I have only little knowledge about docker ... but could there be an issue here? Should this be done in a different way?
In general the warning is correct, because other users on the same host can see the process CLI args, and thus the password is susceptible to snooping.
In this case, however, it is a non-issue. This is running inside a docker container already which has a PID namespace. Thus the only things that can see our password on the CLI are things inside our own container which already all have the env variable, and processes running in host OS context which are only things GitLab admins control. So there's no data leakage to anyone who doesn't already have access to the password
This particular docker login command is the documented solution:
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 58 lines · Daniel P. Berrangé
Quoted 41 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +2
-0@@ -0,0 +1,248 @@ + +.gitlab-ci.d/containers.yml line 12
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"I can see this in the output:
Quoted 4 lines
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
I have to admit that I have only little knowledge about docker ... but could there be an issue here? Should this be done in a different way?
In general the warning is correct, because other users on the same host can see the process CLI args, and thus the password is susceptible to snooping.
In this case, however, it is a non-issue. This is running inside a docker container already which has a PID namespace. Thus the only things that can see our password on the CLI are things inside our own container which already all have the env variable, and processes running in host OS context which are only things GitLab admins control. So there's no data leakage to anyone who doesn't already have access to the password
Quoted 3 lines
This particular docker login command is the documented solution:
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
(
Purely theoretically, we could use a "here string":
docker [...] --password-stdin <<< "$CI_REGISTRY_PASSWORD"
The password is then not exposed on any process's command line; it's a (bash) shell redirection. (It's not in POSIX.)
)
Thanks
Laszlo
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
In reply to Daniel P. BerrangéQuoted 12 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Quoted 6 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.yml
Thinking about this at 3am with insomnia, I started considering possible ways this could result in CI failures. I came up with
- Distro repos are unavialable due to transient network/mirror problems
- Distro pushes bad packages or changes something that invalidates QEMU assumptions
The first one can hit at any time, but I think that we're reasonably well insulated from it due to our usage of caching. Only a small number of CI jobs are going to actually trigger a full rebuild of an image, most will just hit the cache.
The second one is probably not likely, *provided* we only use stable branches of distros. It looks like we're ok in that regard as we're not using Debian Sid, or Fedora rawhide for any images currently.
If we did want to reduce the risk though, we could mark some of the container jobs as non-fatal. We could mark the subset of images that are not actually required by later build jobs that we currently have.
Quoted 275 lines
.gitlab-ci.d/containers.yml new file +248
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout + +amd64-centos7-container: + <<: *container_job_definition + variables: + NAME: centos7 + +amd64-centos8-container: + <<: *container_job_definition + variables: + NAME: centos8 + +amd64-debian10-container: + <<: *container_job_definition + variables: + NAME: debian10 + +amd64-debian11-container: + <<: *container_job_definition + variables: + NAME: debian11 + +amd64-debian9-container: + <<: *container_job_definition + variables: + NAME: debian9 + +amd64-debian9-mxe-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian9-mxe + +alpha-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-alpha-cross + +amd64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64-cross + +amd64-debian-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64 + +arm64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-arm64-cross + +arm64-test-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian11-container'] + variables: + NAME: debian-arm64-test-cross + +armel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armel-cross + +armhf-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armhf-cross + +hppa-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-hppa-cross + +m68k-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-m68k-cross + +mips64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64-cross + +mips64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64el-cross + +mips-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips-cross + +mipsel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mipsel-cross + +powerpc-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-powerpc-cross + +ppc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64-cross + +ppc64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64el-cross + +riscv64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-riscv64-cross + +s390x-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-s390x-cross + +sh4-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sh4-cross + +sparc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sparc64-cross + +tricore-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian-tricore-cross + +win32-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win32-cross + +win64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win64-cross + +xtensa-debian-cross-container: + <<: *container_job_definition + variables: + NAME: debian-xtensa-cross + +cris-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-cris-cross + +amd64-fedora-container: + <<: *container_job_definition + variables: + NAME: fedora + +i386-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-i386-cross + +amd64-ubuntu1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu1804 + +amd64-ubuntu2004-container: + <<: *container_job_definition + variables: + NAME: ubuntu2004 + +amd64-ubuntu-container: + <<: *container_job_definition + variables: + NAME: ubuntu +.gitlab-ci.yml +3
-0@@ -1,10 +1,13 @@ stages: - containers + - containers-layer2 + - containers-layer3 - build include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' + - local: '/.gitlab-ci.d/containers.yml' .update_apt_template: &before_script_apt before_script:Signature · 1 line
2.24.1
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 28 lines · Daniel P. Berrangé
Quoted 12 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Quoted 6 lines
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+) create mode 100644 .gitlab-ci.d/containers.ymlThinking about this at 3am with insomnia, I started considering possible ways this could result in CI failures. I came up with
- Distro repos are unavialable due to transient network/mirror problems
- Distro pushes bad packages or changes something that invalidates QEMU assumptions
Quoted 8 lines
The first one can hit at any time, but I think that we're reasonably well insulated from it due to our usage of caching. Only a small number of CI jobs are going to actually trigger a full rebuild of an image, most will just hit the cache.
The second one is probably not likely, *provided* we only use stable branches of distros. It looks like we're ok in that regard as we're not using Debian Sid, or Fedora rawhide for any images currently.
We use debian11 (which I think is debian-testing) for one of the aarch64 compilers because we need fairlyu bleeding edge gcc's but we won't go back to sid anytime soon.
Quoted 283 lines
If we did want to reduce the risk though, we could mark some of the container jobs as non-fatal. We could mark the subset of images that are not actually required by later build jobs that we currently have.
Quoted 275 lines
.gitlab-ci.d/containers.yml new file +248
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout + +amd64-centos7-container: + <<: *container_job_definition + variables: + NAME: centos7 + +amd64-centos8-container: + <<: *container_job_definition + variables: + NAME: centos8 + +amd64-debian10-container: + <<: *container_job_definition + variables: + NAME: debian10 + +amd64-debian11-container: + <<: *container_job_definition + variables: + NAME: debian11 + +amd64-debian9-container: + <<: *container_job_definition + variables: + NAME: debian9 + +amd64-debian9-mxe-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian9-mxe + +alpha-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-alpha-cross + +amd64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64-cross + +amd64-debian-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-amd64 + +arm64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-arm64-cross + +arm64-test-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian11-container'] + variables: + NAME: debian-arm64-test-cross + +armel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armel-cross + +armhf-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-armhf-cross + +hppa-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-hppa-cross + +m68k-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-m68k-cross + +mips64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64-cross + +mips64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips64el-cross + +mips-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mips-cross + +mipsel-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-mipsel-cross + +powerpc-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-powerpc-cross + +ppc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64-cross + +ppc64el-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-ppc64el-cross + +riscv64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-riscv64-cross + +s390x-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-s390x-cross + +sh4-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sh4-cross + +sparc64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian10-container'] + variables: + NAME: debian-sparc64-cross + +tricore-debian-cross-container: + <<: *container_job_definition + stage: containers-layer2 + needs: ['amd64-debian9-container'] + variables: + NAME: debian-tricore-cross + +win32-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win32-cross + +win64-debian-cross-container: + <<: *container_job_definition + stage: containers-layer3 + needs: ['amd64-debian9-mxe-container'] + variables: + NAME: debian-win64-cross + +xtensa-debian-cross-container: + <<: *container_job_definition + variables: + NAME: debian-xtensa-cross + +cris-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-cris-cross + +amd64-fedora-container: + <<: *container_job_definition + variables: + NAME: fedora + +i386-fedora-cross-container: + <<: *container_job_definition + variables: + NAME: fedora-i386-cross + +amd64-ubuntu1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu1804 + +amd64-ubuntu2004-container: + <<: *container_job_definition + variables: + NAME: ubuntu2004 + +amd64-ubuntu-container: + <<: *container_job_definition + variables: + NAME: ubuntu +.gitlab-ci.yml +3
-0@@ -1,10 +1,13 @@ stages: - containers + - containers-layer2 + - containers-layer3 - build include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' + - local: '/.gitlab-ci.d/containers.yml' .update_apt_template: &before_script_apt before_script:Signature · 1 line
2.24.1Regards,
Daniel
Signature · 1 line
Alex Bennée
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
In reply to Daniel P. BerrangéQuoted 34 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +9
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
.gitlab-ci.d/containers.yml line 19
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout
.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:
rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*
?
Thomas
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 50 lines · Thomas Huth
Quoted 34 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +9
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest".gitlab-ci.d/containers.yml line 19
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout
.gitlab-ci.d/edk2.ymluses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:
Quoted 7 lines
rules:
- changes:
-.gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*?
If the OS distro base image changes, we'll never pick it up with that kind of filtering. For the main gitlab.com/qemu-project/qemu you could configure a nightly/weekly/whatever job to force rebuild on a periodic basis to pick up base image changes. The downside of this is that any users who fork qemu won't have that periodic job and so will be testing their work against potentially outdated content.
Having said all that, I'm not 100% convinced I'm actually picking up changed base images right now anyway, given our use of caching.
It is possible that I would need todo an explict "docker pull" of the base image to force it to trigger a refresh othrewise I have a feeling we're always cached.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 66 lines · Daniel P. Berrangé
Quoted 50 lines · Thomas Huth
Quoted 34 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +9
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest".gitlab-ci.d/containers.yml line 19
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout
.gitlab-ci.d/edk2.ymluses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:Quoted 7 lines
rules:
- changes:
-.gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*?
If the OS distro base image changes, we'll never pick it up with that kind of filtering. For the main gitlab.com/qemu-project/qemu you could configure a nightly/weekly/whatever job to force rebuild on a periodic basis to pick up base image changes. The downside of this is that any users who fork qemu won't have that periodic job and so will be testing their work against potentially outdated content.
Quoted 6 lines
Having said all that, I'm not 100% convinced I'm actually picking up changed base images right now anyway, given our use of caching.
It is possible that I would need todo an explict "docker pull" of the base image to force it to trigger a refresh othrewise I have a feeling we're always cached.
But currently, each of the container stages currently takes > 2 minutes, even with the cached containers. I had a quick look, and it takes 7 minutes 'till the "build" stage begins. So all the advantages of not having to do "yum/apt-get install" in the build containers anymore seem to be crushed by the time that the three container stages take now?
Thomas
Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Quoted 74 lines · Thomas Huth
Quoted 66 lines · Daniel P. Berrangé
Quoted 50 lines · Thomas Huth
Quoted 34 lines · Daniel P. Berrangé
We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 3 + 2 files changed, 251 insertions(+)create mode 100644
.gitlab-ci.d/containers.yml.gitlab-ci.d/containers.yml new file +9
-0@@ -0,0 +1,248 @@ + + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest".gitlab-ci.d/containers.yml line 19
Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10
-0+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles + - docker push "$TAG" + after_script: + - docker logout
.gitlab-ci.d/edk2.ymluses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:Quoted 7 lines
rules:
- changes:
-.gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*?
If the OS distro base image changes, we'll never pick it up with that kind of filtering. For the main gitlab.com/qemu-project/qemu you could configure a nightly/weekly/whatever job to force rebuild on a periodic basis to pick up base image changes. The downside of this is that any users who fork qemu won't have that periodic job and so will be testing their work against potentially outdated content.
Quoted 6 lines
Having said all that, I'm not 100% convinced I'm actually picking up changed base images right now anyway, given our use of caching.
It is possible that I would need todo an explict "docker pull" of the base image to force it to trigger a refresh othrewise I have a feeling we're always cached.
Quoted 5 lines
But currently, each of the container stages currently takes > 2 minutes, even with the cached containers. I had a quick look, and it takes 7 minutes 'till the "build" stage begins. So all the advantages of not having to do "yum/apt-get install" in the build containers anymore seem to be crushed by the time that the three container stages take now?
That's still not an apples/apples comparison though. The containers have many mre packages installed and so test more features. Add all those extra packages into the original apt-get/dnf commands and then compare times.
I do agree though that I don't much like the 3 containers stages. I set them up so they would parallelize. ie stage 2 can start without waiting for entire of stage 1 to finish - only the parent container needs to finish. The builds stage still waits for all the containers to complete.
We could do similar optimize for the actual build jobs, so they don't have to wait for all containers to finish - they only need wait for their own required container to finish.
Ultimately it might be a better tradeoff to not have inheritance between the containers, and instead just duplicate the common packages in each. This leads to bigger container images, but simpler builds. Libvirt went this way for its cross compiler images, just duplicating the shared parts in each.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH 3/3 gitlab: convert jobs to use custom built containers
In reply to Daniel P. BerrangéNow that we're building standard container images from dockerfiles in tests/docker/dockerfiles, we can convert the build jobs to use them. The key benefit of this is that a contributor can now more easily replicate the CI environment on their local machine. The container images are cached too, so we are not spending time waiting for the apt-get/dnf package installs to complete.
--- .gitlab-ci.yml | 187 +++++++++++++++++++++---------------------------- 1 file changed, 81 insertions(+), 106 deletions(-)
.gitlab-ci.yml +81 -106
@@ -9,133 +9,108 @@ include: - local: '/.gitlab-ci.d/opensbi.yml' - local: '/.gitlab-ci.d/containers.yml' -.update_apt_template: &before_script_apt - before_script: - - apt-get update -qq - - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make - genisoimage - - JOBS=$(expr $(nproc) + 1) - -.update_dnf_template: &before_script_dnf - before_script: - - dnf update -y - - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel - make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel - - JOBS=$(expr $(nproc) + 1) +.native_build_job_template: &native_build_job_definition + stage: build + image: $CI_REGISTRY_IMAGE/ci-$IMAGE:latest + before_script: + - JOBS=$(expr $(nproc) + 1) + script: + - mkdir build + - cd build + - if test -n "$TARGETS"; + then + ../configure --enable-werror $CONFIGURE_ARGS --target-list="$TARGETS" ; + else + ../configure --enable-werror $CONFIGURE_ARGS ; + fi + - make -j"$JOBS" + - make -j"$JOBS" $MAKE_CHECK_ARGS build-system1: - stage: build - image: ubuntu:19.10 - <<: *before_script_apt - script: - - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev - libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev - - mkdir build - - cd build - - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu - cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu - mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu" - - make -j"$JOBS" - - make -j"$JOBS" check + <<: *native_build_job_definition + variables: + IMAGE: ubuntu2004 + TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu + moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu + riscv64-softmmu sparc-softmmu + MAKE_CHECK_ARGS: check build-system2: - stage: build - image: fedora:latest - <<: *before_script_dnf - script: - - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel - libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel - libzstd-devel - - mkdir build - - cd build - - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu - microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu - sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu" - - make -j"$JOBS" - - make -j"$JOBS" check + <<: *native_build_job_definition + variables: + IMAGE: fedora + TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu + riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu + xtensa-softmmu nios2-softmmu or1k-softmmu + MAKE_CHECK_ARGS: check build-disabled: - stage: build - image: fedora:latest - <<: *before_script_dnf - script: - - mkdir build - - cd build - - ../configure --enable-werror --disable-rdma --disable-slirp --disable-curl + <<: *native_build_job_definition + variables: + IMAGE: fedora + CONFIGURE_ARGS: --disable-rdma --disable-slirp --disable-curl --disable-capstone --disable-live-block-migration --disable-glusterfs --disable-replication --disable-coroutine-pool --disable-smartcard --disable-guest-agent --disable-curses --disable-libxml2 --disable-tpm --disable-qom-cast-debug --disable-spice --disable-vhost-vsock --disable-vhost-net --disable-vhost-crypto --disable-vhost-user - --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user" - - make -j"$JOBS" - - make -j"$JOBS" check-qtest SPEED=slow + TARGETS: i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user + MAKE_CHECK_ARGS: check-qtest SPEED=slow build-tcg-disabled: - stage: build - image: centos:8 - <<: *before_script_dnf - script: - - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel - - mkdir build - - cd build - - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list="" - - make -j"$JOBS" - - make check-unit - - make check-qapi-schema - - cd tests/qemu-iotests/ - - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048 + <<: *native_build_job_definition + variables: + IMAGE: centos8 + script: + - mkdir build + - cd build + - ../configure --disable-tcg --audio-drv-list="" + - make -j"$JOBS" + - make check-unit + - make check-qapi-schema + - cd tests/qemu-iotests/ + - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048 052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163 170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277 - - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122 + - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122 124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202 208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258 260 261 262 263 264 270 272 273 277 279 build-user: - stage: build - <<: *before_script_apt - script: - - mkdir build - - cd build - - ../configure --enable-werror --disable-system --disable-guest-agent - --disable-capstone --disable-slirp --disable-fdt - - make -j"$JOBS" - - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user + <<: *native_build_job_definition + variables: + IMAGE: ubuntu2004 + CONFIGURE_ARGS: --disable-system --disable-guest-agent + --disable-capstone --disable-slirp --disable-fdt + MAKE_CHECK_ARGS: run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: - stage: build - image: fedora:latest - <<: *before_script_dnf - script: - - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel - libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel - - mkdir build - - cd build - - ../configure --cc=clang --cxx=clang++ --enable-werror - --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu - ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user" - - make -j"$JOBS" - - make -j"$JOBS" check + <<: *native_build_job_definition + variables: + IMAGE: fedora + CONFIGURE_ARGS: --cc=clang --cxx=clang++ + TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu + ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user + MAKE_CHECK_ARGS: check build-tci: - stage: build - image: centos:8 - <<: *before_script_dnf - script: - - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" - - mkdir build - - cd build - - ../configure --enable-tcg-interpreter - --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" - - make -j"$JOBS" - - make run-tcg-tests-x86_64-softmmu - - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test - - for tg in $TARGETS ; do - export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ; - ./tests/qtest/boot-serial-test || exit 1 ; - ./tests/qtest/cdrom-test || exit 1 ; - done - - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test - - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" - ./tests/qtest/pxe-test -m slow + <<: *native_build_job_definition + variables: + IMAGE: fedora + script: + - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" + - mkdir build + - cd build + - ../configure --enable-tcg-interpreter + --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" + - make -j"$JOBS" + - make run-tcg-tests-x86_64-softmmu + - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test + - for tg in $TARGETS ; do + export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ; + ./tests/qtest/boot-serial-test || exit 1 ; + ./tests/qtest/cdrom-test || exit 1 ; + done + - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test + - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" ./tests/qtest/pxe-test -m slow
Signature · 1 line
2.24.1
Re: [PATCH RFC 3/3] gitlab: convert jobs to use custom built containers
Quoted 223 lines · Daniel P. Berrangé
Now that we're building standard container images from dockerfiles in tests/docker/dockerfiles, we can convert the build jobs to use them. The key benefit of this is that a contributor can now more easily replicate the CI environment on their local machine. The container images are cached too, so we are not spending time waiting for the apt-get/dnf package installs to complete.
Signed-off-by Daniel P. Berrangé <[email protected]>--- .gitlab-ci.yml | 187 +++++++++++++++++++++---------------------------- 1 file changed, 81 insertions(+), 106 deletions(-).gitlab-ci.yml +71
-106@@ -9,133 +9,108 @@ include: - local: '/.gitlab-ci.d/opensbi.yml' - local: '/.gitlab-ci.d/containers.yml' -.update_apt_template: &before_script_apt - before_script: - - apt-get update -qq - - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make - genisoimage - - JOBS=$(expr $(nproc) + 1) - -.update_dnf_template: &before_script_dnf - before_script: - - dnf update -y - - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel - make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel - - JOBS=$(expr $(nproc) + 1) +.native_build_job_template: &native_build_job_definition + stage: build + image: $CI_REGISTRY_IMAGE/ci-$IMAGE:latest + before_script: + - JOBS=$(expr $(nproc) + 1) + script: + - mkdir build + - cd build + - if test -n "$TARGETS"; + then + ../configure --enable-werror $CONFIGURE_ARGS --target-list="$TARGETS" ; + else + ../configure --enable-werror $CONFIGURE_ARGS ; + fi + - make -j"$JOBS" + - make -j"$JOBS" $MAKE_CHECK_ARGS build-system1: - stage: build - image: ubuntu:19.10 - <<: *before_script_apt - script: - - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev - libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev - - mkdir build - - cd build - - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu - cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu - mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu" - - make -j"$JOBS" - - make -j"$JOBS" check + <<: *native_build_job_definition + variables: + IMAGE: ubuntu2004 + TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu + moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu + riscv64-softmmu sparc-softmmu + MAKE_CHECK_ARGS: check build-system2: - stage: build - image: fedora:latest - <<: *before_script_dnf - script: - - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel - libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel - libzstd-devel - - mkdir build - - cd build - - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu - microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu - sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu" - - make -j"$JOBS" - - make -j"$JOBS" check + <<: *native_build_job_definition + variables: + IMAGE: fedora + TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu + riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu + xtensa-softmmu nios2-softmmu or1k-softmmu + MAKE_CHECK_ARGS: check build-disabled: - stage: build - image: fedora:latest - <<: *before_script_dnf - script: - - mkdir build - - cd build - - ../configure --enable-werror --disable-rdma --disable-slirp --disable-curl + <<: *native_build_job_definition + variables: + IMAGE: fedora + CONFIGURE_ARGS: --disable-rdma --disable-slirp --disable-curl --disable-capstone --disable-live-block-migration --disable-glusterfs --disable-replication --disable-coroutine-pool --disable-smartcard --disable-guest-agent --disable-curses --disable-libxml2 --disable-tpm --disable-qom-cast-debug --disable-spice --disable-vhost-vsock --disable-vhost-net --disable-vhost-crypto --disable-vhost-user - --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user" - - make -j"$JOBS" - - make -j"$JOBS" check-qtest SPEED=slow + TARGETS: i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user + MAKE_CHECK_ARGS: check-qtest SPEED=slow build-tcg-disabled: - stage: build - image: centos:8 - <<: *before_script_dnf - script: - - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel - - mkdir build - - cd build - - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list="" - - make -j"$JOBS" - - make check-unit - - make check-qapi-schema - - cd tests/qemu-iotests/ - - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048 + <<: *native_build_job_definition + variables: + IMAGE: centos8 + script: + - mkdir build + - cd build + - ../configure --disable-tcg --audio-drv-list="" + - make -j"$JOBS" + - make check-unit + - make check-qapi-schema + - cd tests/qemu-iotests/ + - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048 052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163 170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277 - - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122 + - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122 124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202 208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258 260 261 262 263 264 270 272 273 277 279 build-user: - stage: build - <<: *before_script_apt - script: - - mkdir build - - cd build - - ../configure --enable-werror --disable-system --disable-guest-agent - --disable-capstone --disable-slirp --disable-fdt - - make -j"$JOBS" - - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user + <<: *native_build_job_definition + variables: + IMAGE: ubuntu2004 + CONFIGURE_ARGS: --disable-system --disable-guest-agent + --disable-capstone --disable-slirp --disable-fdt + MAKE_CHECK_ARGS: run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: - stage: build - image: fedora:latest - <<: *before_script_dnf - script: - - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel - libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel - - mkdir build - - cd build - - ../configure --cc=clang --cxx=clang++ --enable-werror - --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu - ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user" - - make -j"$JOBS" - - make -j"$JOBS" check + <<: *native_build_job_definition + variables: + IMAGE: fedora + CONFIGURE_ARGS: --cc=clang --cxx=clang++ + TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu + ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user + MAKE_CHECK_ARGS: check build-tci: - stage: build - image: centos:8 - <<: *before_script_dnf - script: - - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" - - mkdir build - - cd build - - ../configure --enable-tcg-interpreter - --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" - - make -j"$JOBS" - - make run-tcg-tests-x86_64-softmmu - - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test - - for tg in $TARGETS ; do - export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ; - ./tests/qtest/boot-serial-test || exit 1 ; - ./tests/qtest/cdrom-test || exit 1 ; - done - - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test - - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" - ./tests/qtest/pxe-test -m slow + <<: *native_build_job_definition + variables: + IMAGE: fedora + script: + - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" + - mkdir build + - cd build + - ../configure --enable-tcg-interpreter + --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
.gitlab-ci.yml include: · line 116
Quoted 10 lines
.gitlab-ci.yml +10
-0+ - make -j"$JOBS" + - make run-tcg-tests-x86_64-softmmu + - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test + - for tg in $TARGETS ; do + export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ; + ./tests/qtest/boot-serial-test || exit 1 ; + ./tests/qtest/cdrom-test || exit 1 ; + done + - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test + - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" ./tests/qtest/pxe-test -m slow
Looks pretty cool, thanks for doing this!
Quoted 3 lines · Daniel P. Berrangé
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
<snip>
I should say I've queued this into testing/next and tweaked it slightly.
Signature · 1 line
Alex Bennée
Quoted 12 lines · Daniel P. Berrangé
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.
We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....
Quoted 6 lines
...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.
These later jobs are now faster because they're not having to install any software.
Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:
https://gitlab.com/huth/qemu/-/jobs/610924897
and after your change:
https://gitlab.com/huth/qemu/-/jobs/611069374
Duration went up from 35 minutes to 42 minutes.
Seems also to happen in your builds, before the change:
https://gitlab.com/berrange/qemu/-/jobs/582995084
and after the change:
https://gitlab.com/berrange/qemu/-/jobs/606175927
... went from 36 minutes up to 42 minutes.
Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...
Thomas
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.
We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....
Quoted 6 lines
...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.
These later jobs are now faster because they're not having to install any software.
Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:
https://gitlab.com/huth/qemu/-/jobs/610924897
and after your change:
https://gitlab.com/huth/qemu/-/jobs/611069374
Duration went up from 35 minutes to 42 minutes.
Seems also to happen in your builds, before the change:
Quoted 8 lines
and after the change:
https://gitlab.com/berrange/qemu/-/jobs/606175927
... went from 36 minutes up to 42 minutes.
Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...
I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.
If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 50 lines · Daniel P. Berrangé
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.
We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....
Quoted 6 lines
...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.
These later jobs are now faster because they're not having to install any software.
Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:
https://gitlab.com/huth/qemu/-/jobs/610924897
and after your change:
https://gitlab.com/huth/qemu/-/jobs/611069374
Duration went up from 35 minutes to 42 minutes.
Seems also to happen in your builds, before the change:
Quoted 8 lines
and after the change:
https://gitlab.com/berrange/qemu/-/jobs/606175927
... went from 36 minutes up to 42 minutes.
Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...
I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.
Quoted 5 lines
If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.
Oh a missed a lot more actually - there's also spice, opengl, libiscsi, libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev too.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 57 lines · Daniel P. Berrangé
Quoted 50 lines · Daniel P. Berrangé
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.
We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....
Quoted 6 lines
...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.
These later jobs are now faster because they're not having to install any software.
Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:
https://gitlab.com/huth/qemu/-/jobs/610924897
and after your change:
https://gitlab.com/huth/qemu/-/jobs/611069374
Duration went up from 35 minutes to 42 minutes.
Seems also to happen in your builds, before the change:
Quoted 8 lines
and after the change:
https://gitlab.com/berrange/qemu/-/jobs/606175927
... went from 36 minutes up to 42 minutes.
Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...
I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.
Quoted 5 lines
If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.
Quoted 3 lines
Oh a missed a lot more actually - there's also spice, opengl, libiscsi, libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev too.
Ok, that's fair, I think it's ok to spend some additional minutes for the extended test coverage here.
Thomas
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Daniel P. Berrangé
Quoted 55 lines · Daniel P. Berrangé
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé
The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.
The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.
We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....
Quoted 6 lines
...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.
These later jobs are now faster because they're not having to install any software.
Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:
https://gitlab.com/huth/qemu/-/jobs/610924897
and after your change:
https://gitlab.com/huth/qemu/-/jobs/611069374
Duration went up from 35 minutes to 42 minutes.
Seems also to happen in your builds, before the change:
Quoted 8 lines
and after the change:
https://gitlab.com/berrange/qemu/-/jobs/606175927
... went from 36 minutes up to 42 minutes.
Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...
I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.
If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.
Quoted 6 lines
Quoted 3 lines
Oh a missed a lot more actually - there's also spice, opengl, libiscsi, libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev too.
Ok, that's fair, I think it's ok to spend some additional minutes for the extended test coverage here.
Unlike Travis which limits to 5 concurrent jobs, GitLab doesn't seem to have a fixed concurrency limit (at least I've not managed to hit one yet). So if we want faster overall build time, we have alot more scope for splitting jobs in half and making use of much better parallelism in CI to get this to complete sooner.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|