Skip to contentSkip to conversation

gitlab-ci.yml: Avoid some submodules to speed up the CI a little bit

PATCH In review Thomas Huth 2 participants 3 messages Jan 21, 2021 – Jan 26, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gitlab-ci.yml: Avoid some submodules to speed up the CI a little bit

Since the meson build system rework, the configure script prefers the git submodules over the system libraries. So we are testing compilation with capstone, fdt and libslirp as a submodule all over the place, burning CPU cycles by recompiling these third party modules and wasting some network bandwidth in the CI by cloning the submodules each time. Let's stop doing that in at least a couple of jobs and use the system libraries instead.

While we're at it, also install meson in the Fedora container, since it is new enough already, so we do not need to check out the meson submodule here.

---
 .gitlab-ci.yml                               | 6 +++++-
 tests/docker/dockerfiles/centos8.docker      | 1 +
 tests/docker/dockerfiles/debian-amd64.docker | 1 +
 tests/docker/dockerfiles/fedora.docker       | 3 +++
 tests/docker/dockerfiles/ubuntu2004.docker   | 1 +
 5 files changed, 11 insertions(+), 1 deletion(-)
.gitlab-ci.yml +5 -1
index de3a3d25b5..e2f9c99e27 100644
@@ -109,6 +109,7 @@ build-system-ubuntu:
   <<: *native_build_job_definition
   variables:
     IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --enable-fdt=system --enable-slirp=system
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -140,6 +141,7 @@ build-system-debian:
   <<: *native_build_job_definition
   variables:
     IMAGE: debian-amd64
+    CONFIGURE_ARGS: --enable-fdt=system
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
       riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -185,6 +187,7 @@ build-system-fedora:
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
+             --enable-fdt=system --enable-slirp=system --enable-capstone=system
     TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu
       xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -216,7 +219,7 @@ build-system-centos:
   <<: *native_build_job_definition
   variables:
     IMAGE: centos8
-    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
+    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -248,6 +251,7 @@ build-system-opensuse:
   <<: *native_build_job_definition
   variables:
     IMAGE: opensuse-leap
+    CONFIGURE_ARGS: --enable-fdt=system
     TARGETS: s390x-softmmu x86_64-softmmu aarch64-softmmu
     MAKE_CHECK_ARGS: check-build
   artifacts:
tests/docker/dockerfiles/centos8.docker +1 -0
index 64cb7a6eda..a763d55730 100644
@@ -15,6 +15,7 @@ ENV PACKAGES \
     glib2-devel \
     libaio-devel \
     libepoxy-devel \
+    libfdt-devel \
     libgcrypt-devel \
     lzo-devel \
     make \
tests/docker/dockerfiles/debian-amd64.docker +1 -0
index a98314757d..ed546edcd6 100644
@@ -21,6 +21,7 @@ RUN apt update && \
         libbz2-dev \
         liblzo2-dev \
         libgcrypt20-dev \
+        libfdt-dev \
         librdmacm-dev \
         libsasl2-dev \
         libsnappy-dev \
tests/docker/dockerfiles/fedora.docker +3 -0
index 0b5053f2d0..0d7602abbe 100644
@@ -6,6 +6,7 @@ ENV PACKAGES \
     brlapi-devel \
     bzip2 \
     bzip2-devel \
+    capstone-devel \
     ccache \
     clang \
     cyrus-sasl-devel \
@@ -37,6 +38,7 @@ ENV PACKAGES \
     libpng-devel \
     librbd-devel \
     libseccomp-devel \
+    libslirp-devel \
     libssh-devel \
     libubsan \
     libudev-devel \
@@ -46,6 +48,7 @@ ENV PACKAGES \
     llvm \
     lzo-devel \
     make \
+    meson \
     mingw32-bzip2 \
     mingw32-curl \
     mingw32-glib2 \
tests/docker/dockerfiles/ubuntu2004.docker +1 -0
index ae889d8482..8519584d2b 100644
@@ -37,6 +37,7 @@ ENV PACKAGES flex bison \
     libsasl2-dev \
     libsdl2-dev \
     libseccomp-dev \
+    libslirp-dev \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
Signature · 1 line
2.27.0
#1 lore raw

Hi Thomas,

Quoted 12 lines · Thomas Huth

Since the meson build system rework, the configure script prefers the git submodules over the system libraries. So we are testing compilation with capstone, fdt and libslirp as a submodule all over the place, burning CPU cycles by recompiling these third party modules and wasting some network bandwidth in the CI by cloning the submodules each time. Let's stop doing that in at least a couple of jobs and use the system libraries instead.

While we're at it, also install meson in the Fedora container, since it is new enough already, so we do not need to check out the meson submodule here.

Quoted 8 lines
---
 .gitlab-ci.yml                               | 6 +++++-
 tests/docker/dockerfiles/centos8.docker      | 1 +
 tests/docker/dockerfiles/debian-amd64.docker | 1 +
 tests/docker/dockerfiles/fedora.docker       | 3 +++
 tests/docker/dockerfiles/ubuntu2004.docker   | 1 +
 5 files changed, 11 insertions(+), 1 deletion(-)

So now which are the jobs left building from git submodule clone?

#2 lore raw
Quoted 24 lines · Philippe Mathieu-Daudé

Hi Thomas,

Quoted 12 lines · Thomas Huth

Since the meson build system rework, the configure script prefers the git submodules over the system libraries. So we are testing compilation with capstone, fdt and libslirp as a submodule all over the place, burning CPU cycles by recompiling these third party modules and wasting some network bandwidth in the CI by cloning the submodules each time. Let's stop doing that in at least a couple of jobs and use the system libraries instead.

While we're at it, also install meson in the Fedora container, since it is new enough already, so we do not need to check out the meson submodule here.

Quoted 8 lines
---
  .gitlab-ci.yml                               | 6 +++++-
  tests/docker/dockerfiles/centos8.docker      | 1 +
  tests/docker/dockerfiles/debian-amd64.docker | 1 +
  tests/docker/dockerfiles/fedora.docker       | 3 +++
  tests/docker/dockerfiles/ubuntu2004.docker   | 1 +
  5 files changed, 11 insertions(+), 1 deletion(-)
Quoted 1 line

So now which are the jobs left building from git submodule clone?

All the jobs which don't have a depending test stage jobs should still use the submodules, e.g. build-oss-fuzz, build-tci, etc. Also the centos7-based jobs still use all submodules, e.g. build-crypto-old-nettle etc.

Thomas