Skip to contentSkip to conversation

Gitlab-CI improvements

PATCH Applied Thomas Huth 4 participants 9 messages Jul 30, 2021 – Aug 3, 2021

lore
Share & tools
lore
#0 lore raw

cover Gitlab-CI improvements

Here are three patches for some small issues that I noticed in our gitlab-CI files recently...

Thomas Huth (3):
gitlab-ci: Merge "build-disabled" with "build-without-default-features"
gitlab-ci: Remove superfluous "dnf install" statement
gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

 .gitlab-ci.d/buildtest.yml      | 99 +++++----------------------------
 .gitlab-ci.d/custom-runners.yml | 12 ++--
 docs/devel/ci.rst               | 13 +++++
 3 files changed, 32 insertions(+), 92 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH 1/3 gitlab-ci: Merge "build-disabled" with "build-without-default-features"

Both jobs are testing more or less the same thing (building QEMU with features disabled), so we are wasting precious CI cycles here by doing this twice. Merge the jobs by using --without-default-features by default and just adding some additional --disable-... switches which are not covered by the generic switch (yet). And while we're at it, also test compilation with "--disable-fdt" (which forces us to change the list of targets in this job, though, since some targets do not work without fdt).

---
 .gitlab-ci.d/buildtest.yml | 97 +++++---------------------------------
 1 file changed, 13 insertions(+), 84 deletions(-)
.gitlab-ci.d/buildtest.yml +13 -84
index 903ee65f32..f390f98044 100644
@@ -202,85 +202,6 @@ acceptance-system-opensuse:
     MAKE_CHECK_ARGS: check-acceptance
 
 
-build-disabled:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-fedora-container
-  variables:
-    IMAGE: fedora
-    CONFIGURE_ARGS:
-      --disable-attr
-      --disable-auth-pam
-      --disable-avx2
-      --disable-bochs
-      --disable-brlapi
-      --disable-bzip2
-      --disable-cap-ng
-      --disable-capstone
-      --disable-cloop
-      --disable-coroutine-pool
-      --disable-curl
-      --disable-curses
-      --disable-dmg
-      --disable-docs
-      --disable-gcrypt
-      --disable-glusterfs
-      --disable-gnutls
-      --disable-gtk
-      --disable-guest-agent
-      --disable-iconv
-      --disable-keyring
-      --disable-kvm
-      --disable-libiscsi
-      --disable-libpmem
-      --disable-libssh
-      --disable-libudev
-      --disable-libusb
-      --disable-libxml2
-      --disable-linux-aio
-      --disable-live-block-migration
-      --disable-lzo
-      --disable-malloc-trim
-      --disable-mpath
-      --disable-nettle
-      --disable-numa
-      --disable-opengl
-      --disable-parallels
-      --disable-pie
-      --disable-qcow1
-      --disable-qed
-      --disable-qom-cast-debug
-      --disable-rbd
-      --disable-rdma
-      --disable-replication
-      --disable-sdl
-      --disable-seccomp
-      --disable-slirp
-      --disable-smartcard
-      --disable-snappy
-      --disable-sparse
-      --disable-spice
-      --disable-strip
-      --disable-tpm
-      --disable-usb-redir
-      --disable-vdi
-      --disable-vhost-crypto
-      --disable-vhost-net
-      --disable-vhost-scsi
-      --disable-vhost-kernel
-      --disable-vhost-user
-      --disable-vhost-vdpa
-      --disable-vhost-vsock
-      --disable-virglrenderer
-      --disable-vnc
-      --disable-vte
-      --disable-vvfat
-      --disable-xen
-      --disable-zstd
-    TARGETS: arm-softmmu i386-softmmu ppc64-softmmu mips64-softmmu
-      s390x-softmmu i386-linux-user
-    MAKE_CHECK_ARGS: check-qtest SPEED=slow
-
 # This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
 # the configure script. The container doesn't contain Xen headers so
 # Xen accelerator is not detected / selected. As result it build the
@@ -649,12 +570,20 @@ build-without-default-devices:
 build-without-default-features:
   extends: .native_build_job_template
   needs:
-    job: amd64-debian-container
+    job: amd64-fedora-container
   variables:
-    IMAGE: debian-amd64
-    CONFIGURE_ARGS: --without-default-features --disable-user
-        --target-list-exclude=arm-softmmu,i386-softmmu,mipsel-softmmu,mips64-softmmu,ppc-softmmu
-    MAKE_CHECK_ARGS: check-unit
+    IMAGE: fedora
+    CONFIGURE_ARGS:
+      --without-default-features
+      --disable-capstone
+      --disable-fdt
+      --disable-pie
+      --disable-qom-cast-debug
+      --disable-slirp
+      --disable-strip
+    TARGETS: avr-softmmu i386-softmmu mips64-softmmu s390x-softmmu sh4-softmmu
+      sparc64-softmmu hexagon-linux-user i386-linux-user s390x-linux-user
+    MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow
 
 build-libvhost-user:
   stage: build
Signature · 1 line
2.27.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] gitlab-ci: Merge "build-disabled" with "build-without-default-features"

Quoted 10 lines · Thomas Huth

Both jobs are testing more or less the same thing (building QEMU with features disabled), so we are wasting precious CI cycles here by doing this twice. Merge the jobs by using --without-default-features by default and just adding some additional --disable-... switches which are not covered by the generic switch (yet). And while we're at it, also test compilation with "--disable-fdt" (which forces us to change the list of targets in this job, though, since some targets do not work without fdt).

Quoted 5 lines
---
 .gitlab-ci.d/buildtest.yml | 97 +++++---------------------------------
 1 file changed, 13 insertions(+), 84 deletions(-)
#3 lore raw

PATCH 2/3 gitlab-ci: Remove superfluous "dnf install" statement

In reply to Thomas Huth

The container already features meson and ninja, so there is no need to try to install it with dnf again.

---
 .gitlab-ci.d/buildtest.yml | 2 --
 1 file changed, 2 deletions(-)
.gitlab-ci.d/buildtest.yml +0 -2
index f390f98044..38f08452f1 100644
@@ -590,8 +590,6 @@ build-libvhost-user:
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
     job: amd64-fedora-container
-  before_script:
-    - dnf install -y meson ninja-build
   script:
     - mkdir subprojects/libvhost-user/build
     - cd subprojects/libvhost-user/build
Signature · 1 line
2.27.0
#5 lore raw

PATCH 3/3 gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

In reply to Thomas Huth

The patch that recently introduced the S390X_RUNNER_AVAILABLE variable in custom-runners.yml missed that the bottom half of the file is rather about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to AARCH64_RUNNER_AVAILABLE in those jobs.

Finally mention both variables in our CI documentation, too.

Fixes c5dd0f0342 ("Improve rules for the staging branch")
---
 .gitlab-ci.d/custom-runners.yml | 12 ++++++------
 docs/devel/ci.rst               | 13 +++++++++++++
 2 files changed, 19 insertions(+), 6 deletions(-)
.gitlab-ci.d/custom-runners.yml +6 -6
index 564b94565d..0d3e4a7b4b 100644
@@ -137,7 +137,7 @@ ubuntu-20.04-aarch64-all-linux-static:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
  # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
@@ -157,7 +157,7 @@ ubuntu-20.04-aarch64-all:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -174,7 +174,7 @@ ubuntu-20.04-aarch64-alldbg:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -193,7 +193,7 @@ ubuntu-20.04-aarch64-clang:
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
    when: manual
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
    when: manual
  script:
  - mkdir build
@@ -211,7 +211,7 @@ ubuntu-20.04-aarch64-tci:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -228,7 +228,7 @@ ubuntu-20.04-aarch64-notcg:
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
    when: manual
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
    when: manual
  script:
  - mkdir build
docs/devel/ci.rst +13 -0
index 205572510c..558327457c 100644
@@ -48,6 +48,19 @@ these artifacts are not already cached, downloading them make the jobs
 reach the timeout limit). Set this variable to have the tests using the
 Avocado framework run automatically.
 
+AARCH64_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an aarch64 host that can be used as a gitlab-CI
+runner, you can set this variable to enable the tests that require this
+kind of host. The runner should be tagged with "aarch64".
+
+S390X_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an IBM Z host that can be used as a gitlab-CI
+runner, you can set this variable to enable the tests that require this
+kind of host. The runner should be tagged with "s390x".
+
+
 Jobs on Custom Runners
 ======================
 
Signature · 1 line
2.27.0
Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

Quoted 4 lines · Thomas Huth

The patch that recently introduced the S390X_RUNNER_AVAILABLE variable in custom-runners.yml missed that the bottom half of the file is rather about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to AARCH64_RUNNER_AVAILABLE in those jobs.

Oops.

Quoted 1 line

Finally mention both variables in our CI documentation, too.

Thanks.

Maybe "Fix ${arch}_RUNNER_AVAILABLE" as subject.

Quoted 90 lines
Fixes c5dd0f0342 ("Improve rules for the staging branch")
---
 .gitlab-ci.d/custom-runners.yml | 12 ++++++------
 docs/devel/ci.rst               | 13 +++++++++++++
 2 files changed, 19 insertions(+), 6 deletions(-)
.gitlab-ci.d/custom-runners.yml +6 -6
index 564b94565d..0d3e4a7b4b 100644
@@ -137,7 +137,7 @@ ubuntu-20.04-aarch64-all-linux-static:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
  # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
@@ -157,7 +157,7 @@ ubuntu-20.04-aarch64-all:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -174,7 +174,7 @@ ubuntu-20.04-aarch64-alldbg:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -193,7 +193,7 @@ ubuntu-20.04-aarch64-clang:
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
    when: manual
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
    when: manual
  script:
  - mkdir build
@@ -211,7 +211,7 @@ ubuntu-20.04-aarch64-tci:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -228,7 +228,7 @@ ubuntu-20.04-aarch64-notcg:
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
    when: manual
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
    when: manual
  script:
  - mkdir build
docs/devel/ci.rst +13 -0
index 205572510c..558327457c 100644
@@ -48,6 +48,19 @@ these artifacts are not already cached, downloading them make the jobs
 reach the timeout limit). Set this variable to have the tests using the
 Avocado framework run automatically.
 
+AARCH64_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an aarch64 host that can be used as a gitlab-CI
+runner, you can set this variable to enable the tests that require this
+kind of host. The runner should be tagged with "aarch64".
+
+S390X_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an IBM Z host that can be used as a gitlab-CI
+runner, you can set this variable to enable the tests that require this
+kind of host. The runner should be tagged with "s390x".
+
+
 Jobs on Custom Runners
 ======================
 
Reviewed-by on v1 3/3
#7 lore raw

Re: [PATCH 3/3] gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

In reply to Thomas Huth
Quoted 6 lines · Thomas Huth

The patch that recently introduced the S390X_RUNNER_AVAILABLE variable in custom-runners.yml missed that the bottom half of the file is rather about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to AARCH64_RUNNER_AVAILABLE in those jobs.

Quoted 9 lines

Finally mention both variables in our CI documentation, too.

Fixes c5dd0f0342 ("Improve rules for the staging branch")
---
 .gitlab-ci.d/custom-runners.yml | 12 ++++++------
 docs/devel/ci.rst               | 13 +++++++++++++
 2 files changed, 19 insertions(+), 6 deletions(-)
Applied
#8 lore raw
In reply to Thomas Huth
Quoted 2 lines · Thomas Huth

Here are three patches for some small issues that I noticed in our gitlab-CI files recently...

Queued to for-6.1/misc-fixes-for-rc2, thanks.

Quoted 10 lines

Thomas Huth (3):
gitlab-ci: Merge "build-disabled" with "build-without-default-features"
gitlab-ci: Remove superfluous "dnf install" statement
gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

 .gitlab-ci.d/buildtest.yml      | 99 +++++----------------------------
 .gitlab-ci.d/custom-runners.yml | 12 ++--
 docs/devel/ci.rst               | 13 +++++
 3 files changed, 32 insertions(+), 92 deletions(-)
Signature · 1 line
Alex Bennée