Skip to contentSkip to conversation

gitlab: build containers to use in build jobs

RFC PATCH In review Daniel P. Berrangé 5 participants 29 messages Jun 22, 2020 – Jun 25, 2020

lore
Share & tools
lore
#0 lore raw

cover gitlab: build containers to use in build jobs

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.

We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....

...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.

These later jobs are now faster because they're not having to install any software.

There is a performance hit to build the images, however, they are cached in the GitLab docker registry. A user forking the repo will use the cache from qemu-project/qemu and their own local fork. So overall the time penalty to build images will only be seen if the user modifiers tests/docker/dockerfiles in their patch series, or if the base image changes.

The GitLab container registry is public, so we're not limited to using the built images in GitLab CI. Any other CI system that uses docker can consume these images. Similarly we could change the tests/docker/Makefile rules so that developers pull from https://gitlab.com/qemu-project/qemu, instead of building images locally. This is why we're building all the images, instead of just the handful the current jobs want.

The interesting stuff is mostly in patch 2.

Patch 3 is a quick hack I did to convert existing jobs to use the newly built images. I know there's other work being done on the GitLab jobs right now that probably conflicts with this though, so I've not invested much time in patch 3. Consider it more an illustration of what's possible, than a finished proposal for merge. The patch 3 diff is fairly unintelligble, so it is easier to look at the final result:

https://gitlab.com/berrange/qemu/-/blob/ci-cont/.gitlab-ci.yml

An example pipeline can be viewed here:

https://gitlab.com/berrange/qemu/-/pipelines/158824359

The cached images are here:

https://gitlab.com/berrange/qemu/container_registry

Daniel P. Berrangé (3):
gitlab: introduce explicit "container" and "build" stages
gitlab: build all container images during CI
gitlab: convert jobs to use custom built containers

 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/edk2.yml       |   3 +-
 .gitlab-ci.d/opensbi.yml    |   3 +-
 .gitlab-ci.yml              | 187 +++++++++++++--------------
 4 files changed, 340 insertions(+), 101 deletions(-)
 create mode 100644 .gitlab-ci.d/containers.yml
Signature · 1 line
2.24.1
#1 lore raw

RFC PATCH 1/3 gitlab: introduce explicit "container" and "build" stages

If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.

---
 .gitlab-ci.d/edk2.yml    |  3 ++-
 .gitlab-ci.d/opensbi.yml |  3 ++-
 .gitlab-ci.yml           | 11 +++++++++++
 3 files changed, 15 insertions(+), 2 deletions(-)
.gitlab-ci.d/edk2.yml +2 -1
index 088ba4b43a..d4e7dfcba6 100644
@@ -1,5 +1,5 @@
 docker-edk2:
- stage: build
+ stage: containers
  rules: # Only run this job when the Dockerfile is modified
  - changes:
    - .gitlab-ci-edk2.yml
@@ -24,6 +24,7 @@ docker-edk2:
  - docker push $IMAGE_TAG
 
 build-edk2:
+ stage: build
  rules: # Only run this job when ...
  - changes: # ... roms/edk2/ is modified (submodule updated)
    - roms/edk2/*
.gitlab-ci.d/opensbi.yml +2 -1
index dd051c0124..ec1c1f4cab 100644
@@ -1,5 +1,5 @@
 docker-opensbi:
- stage: build
+ stage: containers
  rules: # Only run this job when the Dockerfile is modified
  - changes:
    - .gitlab-ci-opensbi.yml
@@ -24,6 +24,7 @@ docker-opensbi:
  - docker push $IMAGE_TAG
 
 build-opensbi:
+ stage: build
  rules: # Only run this job when ...
  - changes: # ... roms/opensbi/ is modified (submodule updated)
    - roms/opensbi/*
.gitlab-ci.yml +11 -0
index 349c77aa58..9fdc752ea6 100644
@@ -1,3 +1,7 @@
+stages:
+  - containers
+  - build
+
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
@@ -17,6 +21,7 @@ include:
   - JOBS=$(expr $(nproc) + 1)
 
 build-system1:
+ stage: build
  image: ubuntu:19.10
  <<: *before_script_apt
  script:
@@ -31,6 +36,7 @@ build-system1:
  - make -j"$JOBS" check
 
 build-system2:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -46,6 +52,7 @@ build-system2:
  - make -j"$JOBS" check
 
 build-disabled:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -62,6 +69,7 @@ build-disabled:
  - make -j"$JOBS" check-qtest SPEED=slow
 
 build-tcg-disabled:
+ stage: build
  image: centos:8
  <<: *before_script_dnf
  script:
@@ -82,6 +90,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ stage: build
  <<: *before_script_apt
  script:
  - mkdir build
@@ -92,6 +101,7 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -106,6 +116,7 @@ build-clang:
  - make -j"$JOBS" check
 
 build-tci:
+ stage: build
  image: centos:8
  <<: *before_script_dnf
  script:
Signature · 1 line
2.24.1
Acked-by on v1 1/3
#2 lore raw

Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

Quoted 111 lines · Daniel P. Berrangé

If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.

---
 .gitlab-ci.d/edk2.yml    |  3 ++-
 .gitlab-ci.d/opensbi.yml |  3 ++-
 .gitlab-ci.yml           | 11 +++++++++++
 3 files changed, 15 insertions(+), 2 deletions(-)
.gitlab-ci.d/edk2.yml +2 -1
index 088ba4b43a..d4e7dfcba6 100644
@@ -1,5 +1,5 @@
 docker-edk2:
- stage: build
+ stage: containers
  rules: # Only run this job when the Dockerfile is modified
  - changes:
    - .gitlab-ci-edk2.yml
@@ -24,6 +24,7 @@ docker-edk2:
  - docker push $IMAGE_TAG
 
 build-edk2:
+ stage: build
  rules: # Only run this job when ...
  - changes: # ... roms/edk2/ is modified (submodule updated)
    - roms/edk2/*
.gitlab-ci.d/opensbi.yml +2 -1
index dd051c0124..ec1c1f4cab 100644
@@ -1,5 +1,5 @@
 docker-opensbi:
- stage: build
+ stage: containers
  rules: # Only run this job when the Dockerfile is modified
  - changes:
    - .gitlab-ci-opensbi.yml
@@ -24,6 +24,7 @@ docker-opensbi:
  - docker push $IMAGE_TAG
 
 build-opensbi:
+ stage: build
  rules: # Only run this job when ...
  - changes: # ... roms/opensbi/ is modified (submodule updated)
    - roms/opensbi/*
.gitlab-ci.yml +10 -0
index 349c77aa58..9fdc752ea6 100644
@@ -1,3 +1,7 @@
+stages:
+  - containers
+  - build
+
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
@@ -17,6 +21,7 @@ include:
   - JOBS=$(expr $(nproc) + 1)
 
 build-system1:
+ stage: build
  image: ubuntu:19.10
  <<: *before_script_apt
  script:
@@ -31,6 +36,7 @@ build-system1:
  - make -j"$JOBS" check
 
 build-system2:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -46,6 +52,7 @@ build-system2:
  - make -j"$JOBS" check
 
 build-disabled:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -62,6 +69,7 @@ build-disabled:
  - make -j"$JOBS" check-qtest SPEED=slow
 
 build-tcg-disabled:
+ stage: build
  image: centos:8
  <<: *before_script_dnf
  script:
@@ -82,6 +90,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ stage: build
  <<: *before_script_apt
  script:
  - mkdir build
@@ -92,6 +101,7 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:

.gitlab-ci.yml build-clang: · line 122

Quoted 7 lines
.gitlab-ci.yml +1 -0
@@ -106,6 +116,7 @@ build-clang:
  - make -j"$JOBS" check
 
 build-tci:
+ stage: build
  image: centos:8
  <<: *before_script_dnf
  script:
Reviewed-by on v1 1/3
#3 lore raw

Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

In reply to Daniel P. Berrangé
Quoted 16 lines · Daniel P. Berrangé

If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.

---
  .gitlab-ci.d/edk2.yml    |  3 ++-
  .gitlab-ci.d/opensbi.yml |  3 ++-
  .gitlab-ci.yml           | 11 +++++++++++
  3 files changed, 15 insertions(+), 2 deletions(-)
.gitlab-ci.d/edk2.yml +0 -0
index 088ba4b43a..d4e7dfcba6 100644

.gitlab-ci.d/edk2.yml line 5

Quoted 6 lines
.gitlab-ci.d/edk2.yml +1 -1
@@ -1,5 +1,5 @@
  docker-edk2:
- stage: build
+ stage: containers
   rules: # Only run this job when the Dockerfile is modified
   - changes:
     - .gitlab-ci-edk2.yml

Uh, oh, I guess I should have changed that line to .gitlab-ci.d/edk2.yml when I renamed that file .... will send a patch...

Quoted 13 lines
.gitlab-ci.d/edk2.yml +1 -0
@@ -24,6 +24,7 @@ docker-edk2:
   - docker push $IMAGE_TAG
 
  build-edk2:
+ stage: build
   rules: # Only run this job when ...
   - changes: # ... roms/edk2/ is modified (submodule updated)
     - roms/edk2/*
.gitlab-ci.d/opensbi.yml +0 -0
index dd051c0124..ec1c1f4cab 100644

.gitlab-ci.d/opensbi.yml line 5

Quoted 6 lines
.gitlab-ci.d/opensbi.yml +1 -1
@@ -1,5 +1,5 @@
  docker-opensbi:
- stage: build
+ stage: containers
   rules: # Only run this job when the Dockerfile is modified
   - changes:
     - .gitlab-ci-opensbi.yml

dito

Quoted 70 lines
.gitlab-ci.d/opensbi.yml +1 -0
@@ -24,6 +24,7 @@ docker-opensbi:
   - docker push $IMAGE_TAG
 
  build-opensbi:
+ stage: build
   rules: # Only run this job when ...
   - changes: # ... roms/opensbi/ is modified (submodule updated)
     - roms/opensbi/*
.gitlab-ci.yml +10 -0
index 349c77aa58..9fdc752ea6 100644
@@ -1,3 +1,7 @@
+stages:
+  - containers
+  - build
+
  include:
    - local: '/.gitlab-ci.d/edk2.yml'
    - local: '/.gitlab-ci.d/opensbi.yml'
@@ -17,6 +21,7 @@ include:
    - JOBS=$(expr $(nproc) + 1)
 
  build-system1:
+ stage: build
   image: ubuntu:19.10
   <<: *before_script_apt
   script:
@@ -31,6 +36,7 @@ build-system1:
   - make -j"$JOBS" check
 
  build-system2:
+ stage: build
   image: fedora:latest
   <<: *before_script_dnf
   script:
@@ -46,6 +52,7 @@ build-system2:
   - make -j"$JOBS" check
 
  build-disabled:
+ stage: build
   image: fedora:latest
   <<: *before_script_dnf
   script:
@@ -62,6 +69,7 @@ build-disabled:
   - make -j"$JOBS" check-qtest SPEED=slow
 
  build-tcg-disabled:
+ stage: build
   image: centos:8
   <<: *before_script_dnf
   script:
@@ -82,6 +90,7 @@ build-tcg-disabled:
              260 261 262 263 264 270 272 273 277 279
 
  build-user:
+ stage: build
   <<: *before_script_apt
   script:
   - mkdir build
@@ -92,6 +101,7 @@ build-user:
   - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
  build-clang:
+ stage: build
   image: fedora:latest
   <<: *before_script_dnf
   script:

.gitlab-ci.yml build-clang: · line 122

Quoted 7 lines
.gitlab-ci.yml +1 -0
@@ -106,6 +116,7 @@ build-clang:
   - make -j"$JOBS" check
 
  build-tci:
+ stage: build
   image: centos:8
   <<: *before_script_dnf
   script:
#4 lore raw

Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

Quoted 12 lines · Thomas Huth
Quoted 11 lines · Daniel P. Berrangé

If no stage is listed, jobs get put in an implicit "test" stage. Some jobs which create container images to be used by later stages are currently listed as in a "build" stages.

---
  .gitlab-ci.d/edk2.yml    |  3 ++-
  .gitlab-ci.d/opensbi.yml |  3 ++-
  .gitlab-ci.yml           | 11 +++++++++++

  3 files changed, 15 insertions(+), 2 deletions(-)
Quoted 14 lines
Quoted 11 lines
.gitlab-ci.d/edk2.yml +0 -0
index 088ba4b43a..d4e7dfcba6 100644
@@ -1,5 +1,5 @@

  docker-edk2:
- stage: build
+ stage: containers
   rules: # Only run this job when the Dockerfile is modified
   - changes:
     - .gitlab-ci-edk2.yml

Uh, oh, I guess I should have changed that line to .gitlab-ci.d/edk2.yml when I renamed that file .... will send a patch...

Well, your patch has been reviewed...

Quoted 13 lines
Quoted 13 lines
.gitlab-ci.d/edk2.yml +0 -0
@@ -24,6 +24,7 @@ docker-edk2:

   - docker push $IMAGE_TAG
    build-edk2:
+ stage: build
   rules: # Only run this job when ...
   - changes: # ... roms/edk2/ is modified (submodule updated)
     - roms/edk2/*

.gitlab-ci.d/opensbi.yml +0 -0
index dd051c0124..ec1c1f4cab 100644
@@ -1,5 +1,5 @@
Quoted 8 lines
Quoted 6 lines

  docker-opensbi:
- stage: build
+ stage: containers
   rules: # Only run this job when the Dockerfile is modified
   - changes:
     - .gitlab-ci-opensbi.yml

dito

Oops...

Quoted 64 lines
Quoted 64 lines
.gitlab-ci.d/opensbi.yml +0 -0
@@ -24,6 +24,7 @@ docker-opensbi:

   - docker push $IMAGE_TAG
    build-opensbi:
+ stage: build
   rules: # Only run this job when ...
   - changes: # ... roms/opensbi/ is modified (submodule updated)
     - roms/opensbi/*

.gitlab-ci.yml +4 -0
index 349c77aa58..9fdc752ea6 100644
@@ -1,3 +1,7 @@
+stages:
+  - containers
+  - build
+

  include:
    - local: '/.gitlab-ci.d/edk2.yml'
    - local: '/.gitlab-ci.d/opensbi.yml'

.gitlab-ci.yml +0 -0
@@ -17,6 +21,7 @@ include:
     - JOBS=$(expr $(nproc) + 1)

    build-system1:
+ stage: build
   image: ubuntu:19.10
   <<: *before_script_apt
   script:

.gitlab-ci.yml +0 -0
@@ -31,6 +36,7 @@ build-system1:

   - make -j"$JOBS" check
    build-system2:
+ stage: build
   image: fedora:latest
   <<: *before_script_dnf
   script:

.gitlab-ci.yml +0 -0
@@ -46,6 +52,7 @@ build-system2:

   - make -j"$JOBS" check
    build-disabled:
+ stage: build
   image: fedora:latest
   <<: *before_script_dnf
   script:

.gitlab-ci.yml +0 -0
@@ -62,6 +69,7 @@ build-disabled:

   - make -j"$JOBS" check-qtest SPEED=slow
    build-tcg-disabled:
+ stage: build
   image: centos:8
   <<: *before_script_dnf
   script:

.gitlab-ci.yml +0 -0
@@ -82,6 +90,7 @@ build-tcg-disabled:

              260 261 262 263 264 270 272 273 277 279 build-user:
+ stage: build
   <<: *before_script_apt
   script:
   - mkdir build

.gitlab-ci.yml +0 -0
@@ -92,6 +101,7 @@ build-user:

   - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang:
+ stage: build
   image: fedora:latest
   <<: *before_script_dnf
   script:

.gitlab-ci.yml +0 -0
@@ -106,6 +116,7 @@ build-clang:

   - make -j"$JOBS" check

Quoted 9 lines
Quoted 6 lines

    build-tci:
+ stage: build
   image: centos:8
   <<: *before_script_dnf
   script:

#5 lore raw

RFC PATCH 2/3 gitlab: build all container images during CI

In reply to Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml
.gitlab-ci.d/containers.yml new file +248 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7
+
+amd64-centos8-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos8
+
+amd64-debian10-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian10
+
+amd64-debian11-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian11
+
+amd64-debian9-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian9
+
+amd64-debian9-mxe-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian9-mxe
+
+alpha-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64-cross
+
+amd64-debian-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64
+
+arm64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian11-container']
+  variables:
+    NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win32-cross
+
+win64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-cris-cross
+
+amd64-fedora-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora
+
+i386-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu2004
+
+amd64-ubuntu-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu
+
.gitlab-ci.yml +3 -0
index 9fdc752ea6..72d688875f 100644
@@ -1,10 +1,13 @@
 stages:
   - containers
+  - containers-layer2
+  - containers-layer3
   - build
 
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
Signature · 1 line
2.24.1
#6 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 11 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

OMG you did it! Lovely... 😍

Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't populated on all the runners yet? Or we have to use the same runner again to hit its cache?

Quoted 38 lines
---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml
.gitlab-ci.d/containers.yml new file +24 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7

[...]

#7 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

Quoted 6 lines

OMG you did it! Lovely... 😍

Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't populated on all the runners yet? Or we have to use the same runner again to hit its cache?

It is caching it but it isn't obvious what to look for.

The "docker build" command is always still run, but you'll notice each of the actual commands in the dockerfile are followed by a message " ---> Using cache", instead of the normal command output.

Compare the "amd64-debian9-container" job as an example.

Here is the output seen in the original job which actually did a real build:

https://gitlab.com/berrange/qemu/-/jobs/605783351

And here is the output from the pipeline you point to above:

https://gitlab.com/berrange/qemu/-/jobs/606175855

The cached case is still taking 2 mins 30 seconds, but the original full build took 4 mins 50 seconds.

The difference will be more noticable if we expand the list of packages install in the images to cover more of QEMU's possible dependancies.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 21 lines · Daniel P. Berrangé
Quoted 13 lines · Philippe Mathieu-Daudé
Quoted 11 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

Quoted 6 lines

OMG you did it! Lovely... 😍

Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't populated on all the runners yet? Or we have to use the same runner again to hit its cache?

Quoted 6 lines

It is caching it but it isn't obvious what to look for.

The "docker build" command is always still run, but you'll notice each of the actual commands in the dockerfile are followed by a message " ---> Using cache", instead of the normal command output.

Indeed!

Quoted 10 lines

Compare the "amd64-debian9-container" job as an example.

Here is the output seen in the original job which actually did a real build:

   https://gitlab.com/berrange/qemu/-/jobs/605783351

And here is the output from the pipeline you point to above:

Quoted 7 lines
   https://gitlab.com/berrange/qemu/-/jobs/606175855

The cached case is still taking 2 mins 30 seconds, but the original full build took 4 mins 50 seconds.

The difference will be more noticable if we expand the list of packages install in the images to cover more of QEMU's possible dependancies.

The difference is already very noticeable on my host:

$ time docker pull registry.gitlab.com/berrange/qemu/ci-centos8:latest
latest: Pulling from berrange/qemu/ci-centos8
8a29a15cefae: Pull complete
4a748adf2909: Pull complete
3c034506a458: Pull complete
cd48de325fa1: Pull complete
Digest:
sha256:a6580433f5456d7c49e2b9b796c717c79a5fa15f4eecf71bdbcffd9df6b8217a
Status: Downloaded newer image for
registry.gitlab.com/berrange/qemu/ci-centos8:latest
registry.gitlab.com/berrange/qemu/ci-centos8:latest
real    0m53.566s
user    0m0.115s
sys     0m0.081s
Quoted 4 lines

Regards,
Daniel

Reviewed-by on v1 2/3
#9 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

In reply to Daniel P. Berrangé
Quoted 35 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml
.gitlab-ci.d/containers.yml new file +10 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export
Quoted 1 line

COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"

It would be nice if we could keep the same form as they have in the local registry which would make it easier to integrate with the rest of the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Otherwise it looks good:

Signature · 1 line
Alex Bennée
#10 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 35 lines · Alex Bennée
Quoted 32 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml
.gitlab-ci.d/containers.yml new file +7 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
Quoted 8 lines
Quoted 4 lines
.gitlab-ci.d/containers.yml new file +3 -0
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export

COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"

It would be nice if we could keep the same form as they have in the local registry which would make it easier to integrate with the rest of the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Every time I re-discover it, I find the QEMU container naming really surprising. It is not following the normal best practice for naming containers. Expected container naming convention is that the image name reflects the general content set, and the tag reflects a version number. QEMU has shifted it along, so container name is just a fixed string, and the tag reflects the contenet set, and there is no version.

QEMU's naming will cause problems with caching in GitLab. As GitLab expects the normal container naming scheme, it has a job which expires old versions of an image once there are more than 10 tags. So we have to use the normal naming scheme. Ideally we would change rest of QEMU to use the normal scheme too.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 52 lines · Daniel P. Berrangé
Quoted 35 lines · Alex Bennée
Quoted 32 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml
.gitlab-ci.d/containers.yml new file +7 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
Quoted 8 lines
Quoted 4 lines
.gitlab-ci.d/containers.yml new file +3 -0
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export

COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"

It would be nice if we could keep the same form as they have in the local registry which would make it easier to integrate with the rest of the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Every time I re-discover it, I find the QEMU container naming really surprising. It is not following the normal best practice for naming containers. Expected container naming convention is that the image name reflects the general content set, and the tag reflects a version number. QEMU has shifted it along, so container name is just a fixed string, and the tag reflects the contenet set, and there is no version.

Quoted 5 lines

QEMU's naming will cause problems with caching in GitLab. As GitLab expects the normal container naming scheme, it has a job which expires old versions of an image once there are more than 10 tags. So we have to use the normal naming scheme. Ideally we would change rest of QEMU to use the normal scheme too.

Fair enough.. I'll look into it.

Quoted 3 lines

Regards,
Daniel

Signature · 1 line
Alex Bennée
Acked-by on v1 2/3
#12 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

In reply to Daniel P. Berrangé
Quoted 27 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml              |   3 +
  2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +2 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+

.gitlab-ci.d/containers.yml line 12

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"

I can see this in the output:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

I have to admit that I have only little knowledge about docker ... but could there be an issue here? Should this be done in a different way?

Quoted 245 lines
.gitlab-ci.d/containers.yml new file +236 -0
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7
+
+amd64-centos8-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos8
+
+amd64-debian10-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian10
+
+amd64-debian11-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian11
+
+amd64-debian9-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian9
+
+amd64-debian9-mxe-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian9-mxe
+
+alpha-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64-cross
+
+amd64-debian-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64
+
+arm64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian11-container']
+  variables:
+    NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win32-cross
+
+win64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-cris-cross
+
+amd64-fedora-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora
+
+i386-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu2004
+
+amd64-ubuntu-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu
+
.gitlab-ci.yml +1 -0
index 9fdc752ea6..72d688875f 100644
@@ -1,10 +1,13 @@
  stages:
    - containers
+  - containers-layer2

.gitlab-ci.yml line 13

Quoted 10 lines
.gitlab-ci.yml +2 -0
+  - containers-layer3
    - build
 
  include:
    - local: '/.gitlab-ci.d/edk2.yml'
    - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
 
  .update_apt_template: &before_script_apt
   before_script:
#13 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 41 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml              |   3 +
  2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +2 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+

.gitlab-ci.d/containers.yml line 12

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"

I can see this in the output:

Quoted 4 lines

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

I have to admit that I have only little knowledge about docker ... but could there be an issue here? Should this be done in a different way?

In general the warning is correct, because other users on the same host can see the process CLI args, and thus the password is susceptible to snooping.

In this case, however, it is a non-issue. This is running inside a docker container already which has a PID namespace. Thus the only things that can see our password on the CLI are things inside our own container which already all have the env variable, and processes running in host OS context which are only things GitLab admins control. So there's no data leakage to anyone who doesn't already have access to the password

This particular docker login command is the documented solution:

https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 58 lines · Daniel P. Berrangé
Quoted 41 lines · Thomas Huth
Quoted 27 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml              |   3 +
  2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +2 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+

.gitlab-ci.d/containers.yml line 12

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"

I can see this in the output:

Quoted 4 lines

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

I have to admit that I have only little knowledge about docker ... but could there be an issue here? Should this be done in a different way?

In general the warning is correct, because other users on the same host can see the process CLI args, and thus the password is susceptible to snooping.

In this case, however, it is a non-issue. This is running inside a docker container already which has a PID namespace. Thus the only things that can see our password on the CLI are things inside our own container which already all have the env variable, and processes running in host OS context which are only things GitLab admins control. So there's no data leakage to anyone who doesn't already have access to the password

Quoted 3 lines

This particular docker login command is the documented solution:

https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

(

Purely theoretically, we could use a "here string":

docker [...] --password-stdin <<< "$CI_REGISTRY_PASSWORD"

The password is then not exposed on any process's command line; it's a (bash) shell redirection. (It's not in POSIX.)

)

Thanks
Laszlo

#15 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

In reply to Daniel P. Berrangé
Quoted 12 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

Quoted 6 lines
---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml

Thinking about this at 3am with insomnia, I started considering possible ways this could result in CI failures. I came up with

- Distro repos are unavialable due to transient network/mirror problems

- Distro pushes bad packages or changes something that invalidates QEMU assumptions

The first one can hit at any time, but I think that we're reasonably well insulated from it due to our usage of caching. Only a small number of CI jobs are going to actually trigger a full rebuild of an image, most will just hit the cache.

The second one is probably not likely, *provided* we only use stable branches of distros. It looks like we're ok in that regard as we're not using Debian Sid, or Fedora rawhide for any images currently.

If we did want to reduce the risk though, we could mark some of the container jobs as non-fatal. We could mark the subset of images that are not actually required by later build jobs that we currently have.

Quoted 275 lines
.gitlab-ci.d/containers.yml new file +248 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7
+
+amd64-centos8-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos8
+
+amd64-debian10-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian10
+
+amd64-debian11-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian11
+
+amd64-debian9-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian9
+
+amd64-debian9-mxe-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian9-mxe
+
+alpha-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64-cross
+
+amd64-debian-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64
+
+arm64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian11-container']
+  variables:
+    NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win32-cross
+
+win64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-cris-cross
+
+amd64-fedora-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora
+
+i386-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu2004
+
+amd64-ubuntu-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu
+
.gitlab-ci.yml +3 -0
index 9fdc752ea6..72d688875f 100644
@@ -1,10 +1,13 @@
 stages:
   - containers
+  - containers-layer2
+  - containers-layer3
   - build
 
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
Signature · 1 line
2.24.1

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#16 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 28 lines · Daniel P. Berrangé
Quoted 12 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

Quoted 6 lines
---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml

Thinking about this at 3am with insomnia, I started considering possible ways this could result in CI failures. I came up with

- Distro repos are unavialable due to transient network/mirror problems

- Distro pushes bad packages or changes something that invalidates QEMU assumptions

Quoted 8 lines

The first one can hit at any time, but I think that we're reasonably well insulated from it due to our usage of caching. Only a small number of CI jobs are going to actually trigger a full rebuild of an image, most will just hit the cache.

The second one is probably not likely, *provided* we only use stable branches of distros. It looks like we're ok in that regard as we're not using Debian Sid, or Fedora rawhide for any images currently.

We use debian11 (which I think is debian-testing) for one of the aarch64 compilers because we need fairlyu bleeding edge gcc's but we won't go back to sid anytime soon.

Quoted 283 lines

If we did want to reduce the risk though, we could mark some of the container jobs as non-fatal. We could mark the subset of images that are not actually required by later build jobs that we currently have.

Quoted 275 lines
.gitlab-ci.d/containers.yml new file +248 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7
+
+amd64-centos8-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos8
+
+amd64-debian10-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian10
+
+amd64-debian11-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian11
+
+amd64-debian9-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian9
+
+amd64-debian9-mxe-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian9-mxe
+
+alpha-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64-cross
+
+amd64-debian-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64
+
+arm64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian11-container']
+  variables:
+    NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win32-cross
+
+win64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-cris-cross
+
+amd64-fedora-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora
+
+i386-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu2004
+
+amd64-ubuntu-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu
+
.gitlab-ci.yml +3 -0
index 9fdc752ea6..72d688875f 100644
@@ -1,10 +1,13 @@
 stages:
   - containers
+  - containers-layer2
+  - containers-layer3
   - build
 
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
Signature · 1 line
2.24.1

Regards,
Daniel

Signature · 1 line
Alex Bennée
#17 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

In reply to Daniel P. Berrangé
Quoted 34 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml              |   3 +
  2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +9 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"

.gitlab-ci.d/containers.yml line 19

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout

.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:

rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*

?

Thomas

#18 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 50 lines · Thomas Huth
Quoted 34 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
  .gitlab-ci.yml              |   3 +
  2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +9 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"

.gitlab-ci.d/containers.yml line 19

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout

.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:

Quoted 7 lines

rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*

?

If the OS distro base image changes, we'll never pick it up with that kind of filtering. For the main gitlab.com/qemu-project/qemu you could configure a nightly/weekly/whatever job to force rebuild on a periodic basis to pick up base image changes. The downside of this is that any users who fork qemu won't have that periodic job and so will be testing their work against potentially outdated content.

Having said all that, I'm not 100% convinced I'm actually picking up changed base images right now anyway, given our use of caching.

It is possible that I would need todo an explict "docker pull" of the base image to force it to trigger a refresh othrewise I have a feeling we're always cached.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#19 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 66 lines · Daniel P. Berrangé
Quoted 50 lines · Thomas Huth
Quoted 34 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
   .gitlab-ci.yml              |   3 +
   2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +9 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"

.gitlab-ci.d/containers.yml line 19

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout

.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:

Quoted 7 lines

rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*

?

If the OS distro base image changes, we'll never pick it up with that kind of filtering. For the main gitlab.com/qemu-project/qemu you could configure a nightly/weekly/whatever job to force rebuild on a periodic basis to pick up base image changes. The downside of this is that any users who fork qemu won't have that periodic job and so will be testing their work against potentially outdated content.

Quoted 6 lines

Having said all that, I'm not 100% convinced I'm actually picking up changed base images right now anyway, given our use of caching.

It is possible that I would need todo an explict "docker pull" of the base image to force it to trigger a refresh othrewise I have a feeling we're always cached.

But currently, each of the container stages currently takes > 2 minutes, even with the cached containers. I had a quick look, and it takes 7 minutes 'till the "build" stage begins. So all the advantages of not having to do "yum/apt-get install" in the build containers anymore seem to be crushed by the time that the three container stages take now?

Thomas

#20 lore raw

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

Quoted 74 lines · Thomas Huth
Quoted 66 lines · Daniel P. Berrangé
Quoted 50 lines · Thomas Huth
Quoted 34 lines · Daniel P. Berrangé

We have a number of container images in tests/docker/dockerfiles that are intended to provide well defined environments for doing test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the CI, so that the built containers are available for later build jobs. The containers are setup to use the GitLab container registry as the cache, so we only pay the penalty of the full build when the dockerfiles change. The main qemu-project/qemu repo is used as a second cache, so that users forking QEMU will see a fast turnaround time on their CI jobs.

---
   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
   .gitlab-ci.yml              |   3 +
   2 files changed, 251 insertions(+)

create mode 100644 .gitlab-ci.d/containers.yml

.gitlab-ci.d/containers.yml new file +9 -0
new file mode 100644
index 0000000000..ea1edbb196
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"

.gitlab-ci.d/containers.yml line 19

Quoted 10 lines
.gitlab-ci.d/containers.yml new file +10 -0
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout

.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if something really has been changed. Could you use something similar here? E.g.:

Quoted 7 lines

rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*

?

If the OS distro base image changes, we'll never pick it up with that kind of filtering. For the main gitlab.com/qemu-project/qemu you could configure a nightly/weekly/whatever job to force rebuild on a periodic basis to pick up base image changes. The downside of this is that any users who fork qemu won't have that periodic job and so will be testing their work against potentially outdated content.

Quoted 6 lines

Having said all that, I'm not 100% convinced I'm actually picking up changed base images right now anyway, given our use of caching.

It is possible that I would need todo an explict "docker pull" of the base image to force it to trigger a refresh othrewise I have a feeling we're always cached.

Quoted 5 lines

But currently, each of the container stages currently takes > 2 minutes, even with the cached containers. I had a quick look, and it takes 7 minutes 'till the "build" stage begins. So all the advantages of not having to do "yum/apt-get install" in the build containers anymore seem to be crushed by the time that the three container stages take now?

That's still not an apples/apples comparison though. The containers have many mre packages installed and so test more features. Add all those extra packages into the original apt-get/dnf commands and then compare times.

I do agree though that I don't much like the 3 containers stages. I set them up so they would parallelize. ie stage 2 can start without waiting for entire of stage 1 to finish - only the parent container needs to finish. The builds stage still waits for all the containers to complete.

We could do similar optimize for the actual build jobs, so they don't have to wait for all containers to finish - they only need wait for their own required container to finish.

Ultimately it might be a better tradeoff to not have inheritance between the containers, and instead just duplicate the common packages in each. This leads to bigger container images, but simpler builds. Libvirt went this way for its cross compiler images, just duplicating the shared parts in each.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#21 lore raw

RFC PATCH 3/3 gitlab: convert jobs to use custom built containers

In reply to Daniel P. Berrangé

Now that we're building standard container images from dockerfiles in tests/docker/dockerfiles, we can convert the build jobs to use them. The key benefit of this is that a contributor can now more easily replicate the CI environment on their local machine. The container images are cached too, so we are not spending time waiting for the apt-get/dnf package installs to complete.

---
 .gitlab-ci.yml | 187 +++++++++++++++++++++----------------------------
 1 file changed, 81 insertions(+), 106 deletions(-)
.gitlab-ci.yml +81 -106
index 72d688875f..3297a402f7 100644
@@ -9,133 +9,108 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
 
-.update_apt_template: &before_script_apt
- before_script:
-  - apt-get update -qq
-  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
-        genisoimage
-  - JOBS=$(expr $(nproc) + 1)
-
-.update_dnf_template: &before_script_dnf
- before_script:
-  - dnf update -y
-  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
-        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
-  - JOBS=$(expr $(nproc) + 1)
+.native_build_job_template: &native_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/ci-$IMAGE:latest
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
+  script:
+    - mkdir build
+    - cd build
+    - if test -n "$TARGETS";
+      then
+        ../configure --enable-werror $CONFIGURE_ARGS --target-list="$TARGETS" ;
+      else
+        ../configure --enable-werror $CONFIGURE_ARGS ;
+      fi
+    - make -j"$JOBS"
+    - make -j"$JOBS" $MAKE_CHECK_ARGS
 
 build-system1:
- stage: build
- image: ubuntu:19.10
- <<: *before_script_apt
- script:
- - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
-      libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
- - mkdir build
- - cd build
- - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
-      cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
-      mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
+      riscv64-softmmu sparc-softmmu
+    MAKE_CHECK_ARGS: check
 
 build-system2:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
-       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
-       libzstd-devel
- - mkdir build
- - cd build
- - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
-      microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
-      sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
+      riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
+      xtensa-softmmu nios2-softmmu or1k-softmmu
+    MAKE_CHECK_ARGS: check
 
 build-disabled:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - mkdir build
- - cd build
- - ../configure --enable-werror --disable-rdma --disable-slirp --disable-curl
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --disable-rdma --disable-slirp --disable-curl
       --disable-capstone --disable-live-block-migration --disable-glusterfs
       --disable-replication --disable-coroutine-pool --disable-smartcard
       --disable-guest-agent --disable-curses --disable-libxml2 --disable-tpm
       --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
       --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
-      --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j"$JOBS"
- - make -j"$JOBS" check-qtest SPEED=slow
+    TARGETS: i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user
+    MAKE_CHECK_ARGS: check-qtest SPEED=slow
 
 build-tcg-disabled:
- stage: build
- image: centos:8
- <<: *before_script_dnf
- script:
- - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
- - mkdir build
- - cd build
- - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
- - make -j"$JOBS"
- - make check-unit
- - make check-qapi-schema
- - cd tests/qemu-iotests/
- - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: centos8
+  script:
+    - mkdir build
+    - cd build
+    - ../configure --disable-tcg --audio-drv-list=""
+    - make -j"$JOBS"
+    - make check-unit
+    - make check-qapi-schema
+    - cd tests/qemu-iotests/
+    - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
             052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
             170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
- - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
+    - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
             124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
             208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
- stage: build
- <<: *before_script_apt
- script:
- - mkdir build
- - cd build
- - ../configure --enable-werror --disable-system --disable-guest-agent
-               --disable-capstone --disable-slirp --disable-fdt
- - make -j"$JOBS"
- - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --disable-system --disable-guest-agent
+      --disable-capstone --disable-slirp --disable-fdt
+    MAKE_CHECK_ARGS:  run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
-       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
- - mkdir build
- - cd build
- - ../configure --cc=clang --cxx=clang++ --enable-werror
-      --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
-                     ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++
+    TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
+      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user
+    MAKE_CHECK_ARGS: check
 
 build-tci:
- stage: build
- image: centos:8
- <<: *before_script_dnf
- script:
- - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
- - mkdir build
- - cd build
- - ../configure --enable-tcg-interpreter
-      --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
- - make -j"$JOBS"
- - make run-tcg-tests-x86_64-softmmu
- - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
- - for tg in $TARGETS ; do
-     export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
-     ./tests/qtest/boot-serial-test || exit 1 ;
-     ./tests/qtest/cdrom-test || exit 1 ;
-   done
- - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
- - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x"
-   ./tests/qtest/pxe-test -m slow
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+  script:
+    - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
+    - mkdir build
+    - cd build
+    - ../configure --enable-tcg-interpreter
+        --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
+    - make -j"$JOBS"
+    - make run-tcg-tests-x86_64-softmmu
+    - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
+    - for tg in $TARGETS ; do
+        export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
+        ./tests/qtest/boot-serial-test || exit 1 ;
+        ./tests/qtest/cdrom-test || exit 1 ;
+      done
+    - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
+    - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" ./tests/qtest/pxe-test -m slow
Signature · 1 line
2.24.1
Acked-by on v1 3/3
#22 lore raw

Re: [PATCH RFC 3/3] gitlab: convert jobs to use custom built containers

Quoted 223 lines · Daniel P. Berrangé

Now that we're building standard container images from dockerfiles in tests/docker/dockerfiles, we can convert the build jobs to use them. The key benefit of this is that a contributor can now more easily replicate the CI environment on their local machine. The container images are cached too, so we are not spending time waiting for the apt-get/dnf package installs to complete.

---
  .gitlab-ci.yml | 187 +++++++++++++++++++++----------------------------
  1 file changed, 81 insertions(+), 106 deletions(-)
.gitlab-ci.yml +71 -106
index 72d688875f..3297a402f7 100644
@@ -9,133 +9,108 @@ include:
    - local: '/.gitlab-ci.d/opensbi.yml'
    - local: '/.gitlab-ci.d/containers.yml'
 
-.update_apt_template: &before_script_apt
- before_script:
-  - apt-get update -qq
-  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
-        genisoimage
-  - JOBS=$(expr $(nproc) + 1)
-
-.update_dnf_template: &before_script_dnf
- before_script:
-  - dnf update -y
-  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
-        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
-  - JOBS=$(expr $(nproc) + 1)
+.native_build_job_template: &native_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/ci-$IMAGE:latest
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
+  script:
+    - mkdir build
+    - cd build
+    - if test -n "$TARGETS";
+      then
+        ../configure --enable-werror $CONFIGURE_ARGS --target-list="$TARGETS" ;
+      else
+        ../configure --enable-werror $CONFIGURE_ARGS ;
+      fi
+    - make -j"$JOBS"
+    - make -j"$JOBS" $MAKE_CHECK_ARGS
 
  build-system1:
- stage: build
- image: ubuntu:19.10
- <<: *before_script_apt
- script:
- - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
-      libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
- - mkdir build
- - cd build
- - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
-      cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
-      mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
+      riscv64-softmmu sparc-softmmu
+    MAKE_CHECK_ARGS: check
 
  build-system2:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
-       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
-       libzstd-devel
- - mkdir build
- - cd build
- - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
-      microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
-      sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
+      riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
+      xtensa-softmmu nios2-softmmu or1k-softmmu
+    MAKE_CHECK_ARGS: check
 
  build-disabled:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - mkdir build
- - cd build
- - ../configure --enable-werror --disable-rdma --disable-slirp --disable-curl
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --disable-rdma --disable-slirp --disable-curl
        --disable-capstone --disable-live-block-migration --disable-glusterfs
        --disable-replication --disable-coroutine-pool --disable-smartcard
        --disable-guest-agent --disable-curses --disable-libxml2 --disable-tpm
        --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
        --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
-      --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j"$JOBS"
- - make -j"$JOBS" check-qtest SPEED=slow
+    TARGETS: i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user
+    MAKE_CHECK_ARGS: check-qtest SPEED=slow
 
  build-tcg-disabled:
- stage: build
- image: centos:8
- <<: *before_script_dnf
- script:
- - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
- - mkdir build
- - cd build
- - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
- - make -j"$JOBS"
- - make check-unit
- - make check-qapi-schema
- - cd tests/qemu-iotests/
- - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: centos8
+  script:
+    - mkdir build
+    - cd build
+    - ../configure --disable-tcg --audio-drv-list=""
+    - make -j"$JOBS"
+    - make check-unit
+    - make check-qapi-schema
+    - cd tests/qemu-iotests/
+    - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
              052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
              170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
- - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
+    - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
              124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
              208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
              260 261 262 263 264 270 272 273 277 279
 
  build-user:
- stage: build
- <<: *before_script_apt
- script:
- - mkdir build
- - cd build
- - ../configure --enable-werror --disable-system --disable-guest-agent
-               --disable-capstone --disable-slirp --disable-fdt
- - make -j"$JOBS"
- - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --disable-system --disable-guest-agent
+      --disable-capstone --disable-slirp --disable-fdt
+    MAKE_CHECK_ARGS:  run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
  build-clang:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
-       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
- - mkdir build
- - cd build
- - ../configure --cc=clang --cxx=clang++ --enable-werror
-      --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
-                     ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++
+    TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
+      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user
+    MAKE_CHECK_ARGS: check
 
  build-tci:
- stage: build
- image: centos:8
- <<: *before_script_dnf
- script:
- - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
- - mkdir build
- - cd build
- - ../configure --enable-tcg-interpreter
-      --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
- - make -j"$JOBS"
- - make run-tcg-tests-x86_64-softmmu
- - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
- - for tg in $TARGETS ; do
-     export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
-     ./tests/qtest/boot-serial-test || exit 1 ;
-     ./tests/qtest/cdrom-test || exit 1 ;
-   done
- - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
- - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x"
-   ./tests/qtest/pxe-test -m slow
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+  script:
+    - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
+    - mkdir build
+    - cd build
+    - ../configure --enable-tcg-interpreter
+        --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"

.gitlab-ci.yml include: · line 116

Quoted 10 lines
.gitlab-ci.yml +10 -0
+    - make -j"$JOBS"
+    - make run-tcg-tests-x86_64-softmmu
+    - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
+    - for tg in $TARGETS ; do
+        export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
+        ./tests/qtest/boot-serial-test || exit 1 ;
+        ./tests/qtest/cdrom-test || exit 1 ;
+      done
+    - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
+    - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" ./tests/qtest/pxe-test -m slow

Looks pretty cool, thanks for doing this!

#23 lore raw
In reply to Daniel P. Berrangé
Quoted 3 lines · Daniel P. Berrangé

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

<snip>

I should say I've queued this into testing/next and tweaked it slightly.

Signature · 1 line
Alex Bennée
#24 lore raw
In reply to Daniel P. Berrangé
Quoted 12 lines · Daniel P. Berrangé

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.

We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....

Quoted 6 lines

...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.

These later jobs are now faster because they're not having to install any software.

Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:

https://gitlab.com/huth/qemu/-/jobs/610924897

and after your change:

https://gitlab.com/huth/qemu/-/jobs/611069374

Duration went up from 35 minutes to 42 minutes.

Seems also to happen in your builds, before the change:

https://gitlab.com/berrange/qemu/-/jobs/582995084

and after the change:

https://gitlab.com/berrange/qemu/-/jobs/606175927

... went from 36 minutes up to 42 minutes.

Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...

Thomas

#25 lore raw
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.

We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....

Quoted 6 lines

...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.

These later jobs are now faster because they're not having to install any software.

Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:

https://gitlab.com/huth/qemu/-/jobs/610924897

and after your change:

https://gitlab.com/huth/qemu/-/jobs/611069374

Duration went up from 35 minutes to 42 minutes.

Seems also to happen in your builds, before the change:

https://gitlab.com/berrange/qemu/-/jobs/582995084

Quoted 8 lines

and after the change:

https://gitlab.com/berrange/qemu/-/jobs/606175927

... went from 36 minutes up to 42 minutes.

Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...

I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.

If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#26 lore raw
Quoted 50 lines · Daniel P. Berrangé
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.

We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....

Quoted 6 lines

...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.

These later jobs are now faster because they're not having to install any software.

Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:

https://gitlab.com/huth/qemu/-/jobs/610924897

and after your change:

https://gitlab.com/huth/qemu/-/jobs/611069374

Duration went up from 35 minutes to 42 minutes.

Seems also to happen in your builds, before the change:

https://gitlab.com/berrange/qemu/-/jobs/582995084

Quoted 8 lines

and after the change:

https://gitlab.com/berrange/qemu/-/jobs/606175927

... went from 36 minutes up to 42 minutes.

Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...

I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.

Quoted 5 lines

If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.

Oh a missed a lot more actually - there's also spice, opengl, libiscsi, libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev too.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#27 lore raw
Quoted 57 lines · Daniel P. Berrangé
Quoted 50 lines · Daniel P. Berrangé
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.

We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....

Quoted 6 lines

...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.

These later jobs are now faster because they're not having to install any software.

Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:

https://gitlab.com/huth/qemu/-/jobs/610924897

and after your change:

https://gitlab.com/huth/qemu/-/jobs/611069374

Duration went up from 35 minutes to 42 minutes.

Seems also to happen in your builds, before the change:

https://gitlab.com/berrange/qemu/-/jobs/582995084

Quoted 8 lines

and after the change:

https://gitlab.com/berrange/qemu/-/jobs/606175927

... went from 36 minutes up to 42 minutes.

Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...

I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.

Quoted 5 lines

If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.

Quoted 3 lines

Oh a missed a lot more actually - there's also spice, opengl, libiscsi, libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev too.

Ok, that's fair, I think it's ok to spend some additional minutes for the extended test coverage here.

Thomas

#28 lore raw
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Daniel P. Berrangé
Quoted 55 lines · Daniel P. Berrangé
Quoted 36 lines · Thomas Huth
Quoted 12 lines · Daniel P. Berrangé

The current gitlab CI jobs are quite inefficient because they use the generic distro images and then apt-get/dnf install extra packages every time.

The other downside is that the container environment used is only defined in thte .gitlab-ci.yml file, so it tedious to reproduce locally.

We already have containers defined in tests/docker for use by developers building locally. We can use these for CI systems too if we just had a way to build them....

Quoted 6 lines

...GitLab CI offers such a way. We can use docker-in-docker to build the images at the start of the CI cycle, and use the built images in later jobs.

These later jobs are now faster because they're not having to install any software.

Did you see any speed-up? I had a look at some pipelines, and it seems to me that they rather got slower now? For example, this is the system1 pipeline before your change:

https://gitlab.com/huth/qemu/-/jobs/610924897

and after your change:

https://gitlab.com/huth/qemu/-/jobs/611069374

Duration went up from 35 minutes to 42 minutes.

Seems also to happen in your builds, before the change:

https://gitlab.com/berrange/qemu/-/jobs/582995084

Quoted 8 lines

and after the change:

https://gitlab.com/berrange/qemu/-/jobs/606175927

... went from 36 minutes up to 42 minutes.

Could be a coincidence due to the load on the shared runners, but it looks at least a little bit suspicious...

I think the difference is because we're building more features now. The dockerfiles have provided more build pre-requisites that the old gitlab recipe did.

If you compare the configure summary, I see the new build now covers SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs and a few more. So we've saved time by not intsallling many packages each time, but consumed a greater amount of time by compiling more features.

Quoted 6 lines
Quoted 3 lines

Oh a missed a lot more actually - there's also spice, opengl, libiscsi, libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev too.

Ok, that's fair, I think it's ok to spend some additional minutes for the extended test coverage here.

Unlike Travis which limits to 5 concurrent jobs, GitLab doesn't seem to have a fixed concurrency limit (at least I've not managed to hit one yet). So if we want faster overall build time, we have alot more scope for splitting jobs in half and making use of much better parallelism in CI to get this to complete sooner.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|