Skip to contentSkip to conversation

Run cross-compilation build tests in the gitlab-CI

PATCH In review Thomas Huth 3 participants 20 messages Aug 23, 2020 – Sep 1, 2020

lore
Share & tools
lore
#0 lore raw

v2 cover Run cross-compilation build tests in the gitlab-CI

Now that we can use all our QEMU build containers in the gitlab-CI, we can also run the cross-compilation jobs there. Of course, some problems have to be fixed first, so this is taken care of in the first four patches.

The following two patches make sure that we can also enable WHPX builds with our debian-win64-cross container, so that we can compile-test this accelerator code now, too.

The last patch then finally enables the cross-compilation jobs in the CI.

v2:
- Dropped patches that are not necessary anymore
- Added the first two patches to fix problems with the new meson build system

Thomas Huth (7):
configure: Add system = 'linux' for meson when cross-compiling tests/docker: Install python3-setuptools in the debian9-mxe containers tests/Makefile: test-image-locking needs CONFIG_POSIX tests/Makefile: test-replication needs CONFIG_POSIX dockerfiles/debian-win64-cross: Download WHPX MinGW headers
configure: Allow automatic WHPX detection
gitlab-ci: Add cross-compiling build tests

 .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
 .gitlab-ci.yml                                |   1 +
 MAINTAINERS                                   |   1 +
 configure                                     |   4 +
 tests/Makefile.include                        |   6 +-
 .../dockerfiles/debian-win64-cross.docker     |   9 +-
 tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
 7 files changed, 133 insertions(+), 3 deletions(-)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml
Signature · 1 line
2.18.2
#3 lore raw

PATCH v2 2/7 tests/docker: Install python3-setuptools in the debian9-mxe containers

The python setuptools are a requirement for meson, so we need to install this additional package now.

---
 tests/docker/dockerfiles/debian9-mxe.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/docker/dockerfiles/debian9-mxe.docker +1 -1
index ae2c222a6f..7865c821f4 100644
@@ -8,7 +8,7 @@ FROM qemu/debian9
 MAINTAINER Philippe Mathieu-Daudé <[email protected]>
 
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends gnupg dirmngr
+    apt install -y --no-install-recommends gnupg dirmngr python3-setuptools
 
 # Add the foreign architecture we want and install dependencies
 RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C6BF758A33A3A276 && \
Signature · 1 line
2.18.2
#7 lore raw

PATCH v2 4/7 tests/Makefile: test-replication needs CONFIG_POSIX

In reply to Thomas Huth

test-replication uses sigaction() and friends which are only available on POSIX-like systems.

Message-Id: <[email protected]>

---
 tests/Makefile.include | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tests/Makefile.include +3 -1
index 497f1f21ff..a9746c2ecb 100644
@@ -156,11 +156,13 @@ check-unit-$(CONFIG_BLOCK)  += tests/test-crypto-afsplit$(EXESUF)
 check-unit-$(call land,$(CONFIG_BLOCK),$(CONFIG_QEMU_PRIVATE_XTS)) += tests/test-crypto-xts$(EXESUF)
 check-unit-$(CONFIG_BLOCK)  += tests/test-crypto-block$(EXESUF)
 check-unit-y += tests/test-logging$(EXESUF)
-check-unit-$(call land,$(CONFIG_BLOCK),$(CONFIG_REPLICATION)) += tests/test-replication$(EXESUF)
 check-unit-$(CONFIG_SOFTMMU) += tests/test-bufferiszero$(EXESUF)
 check-unit-y += tests/test-uuid$(EXESUF)
 check-unit-y += tests/ptimer-test$(EXESUF)
 check-unit-y += tests/test-qapi-util$(EXESUF)
+ifeq ($(CONFIG_BLOCK)$(CONFIG_REPLICATION)$(CONFIG_POSIX),yyy)
+check-unit-y += tests/test-replication$(EXESUF)
+endif
 
 generated-files-y += tests/test-qapi-types.h
 generated-files-y += tests/include/test-qapi-types-sub-module.h
Signature · 1 line
2.18.2
#9 lore raw

PATCH v2 5/7 dockerfiles/debian-win64-cross: Download WHPX MinGW headers

In reply to Thomas Huth

To compile-test the WHPX accelerator, we need to download these system headers first (they are unfortunately not part of any released and packaged MinGW toolchain yet).

Idea taken from another patch by Stefan Weil.

Message-Id: <[email protected]>

---
 tests/docker/dockerfiles/debian-win64-cross.docker | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
tests/docker/dockerfiles/debian-win64-cross.docker +8 -1
index 2fc9cfcbc6..4cc4a3f365 100644
@@ -32,7 +32,14 @@ RUN apt-get update && \
         mxe-$TARGET-w64-mingw32.shared-sdl2 \
         mxe-$TARGET-w64-mingw32.shared-sdl2-mixer \
         mxe-$TARGET-w64-mingw32.shared-sdl2-gfx \
-        mxe-$TARGET-w64-mingw32.shared-zlib
+        mxe-$TARGET-w64-mingw32.shared-zlib \
+        curl && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvEmulation.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvemulation.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvPlatform.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatform.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/winhvplatformdefs.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatformdefs.h?format=raw"
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=x86_64-w64-mingw32.shared-
Signature · 1 line
2.18.2
#12 lore raw

PATCH v2 6/7 configure: Allow automatic WHPX detection

In reply to Thomas Huth

The whpx variable is currently initialized to "no" which causes the WHPX check to skip the detection unless the user specified --enable-whpx. Since the detection code should be able to figure it out correctly, let's initialized the variable to "" on MinGW-builds for proper auto-detection instead.

Message-Id: <[email protected]>

---
 configure | 1 +
 1 file changed, 1 insertion(+)
configure +1 -0
index 01204ba0b5..2ca3c1d8e9 100755
@@ -849,6 +849,7 @@ case $targetos in
 MINGW32*)
   mingw32="yes"
   hax="yes"
+  whpx=""
   vhost_user="no"
   audio_possible_drivers="dsound sdl"
   if check_include dsound.h; then
Signature · 1 line
2.18.2
#5 lore raw

PATCH v2 3/7 tests/Makefile: test-image-locking needs CONFIG_POSIX

In reply to Thomas Huth

test-image-locking.c uses the qemu_lock_fd_test() function which is only available on POSIX-like systems.

Message-Id: <[email protected]>

---
 tests/Makefile.include | 2 ++
 1 file changed, 2 insertions(+)
tests/Makefile.include +2 -0
index 9ac8f5b86a..497f1f21ff 100644
@@ -87,7 +87,9 @@ check-unit-$(CONFIG_BLOCK) += tests/test-blockjob$(EXESUF)
 check-unit-$(CONFIG_BLOCK) += tests/test-blockjob-txn$(EXESUF)
 check-unit-$(CONFIG_BLOCK) += tests/test-block-backend$(EXESUF)
 check-unit-$(CONFIG_BLOCK) += tests/test-block-iothread$(EXESUF)
+ifeq ($(CONFIG_POSIX),y)
 check-unit-$(CONFIG_BLOCK) += tests/test-image-locking$(EXESUF)
+endif
 check-unit-y += tests/test-x86-cpuid$(EXESUF)
 # all code tested by test-x86-cpuid is inside topology.h
 ifeq ($(CONFIG_SOFTMMU),y)
Signature · 1 line
2.18.2
#1 lore raw

PATCH v2 1/7 configure: Add system = 'linux' for meson when cross-compiling

In reply to Thomas Huth

Meson needs the "system = xyz" line when cross-compiling. We are already adding a "system = 'windows'" for the MinGW cross-compilation case here, so let's add a "system = 'linux'" now for Linux hosts, too.

---
 configure | 3 +++
 1 file changed, 3 insertions(+)
configure +3 -0
index d9ca87fbbb..01204ba0b5 100755
@@ -8208,6 +8208,9 @@ if test -n "$cross_prefix"; then
             ?:*) pre_prefix=/ ;;
         esac
     fi
+    if test "$linux" = "yes" ; then
+        echo "system = 'linux'" >> $cross
+    fi
     case "$ARCH" in
         i386|x86_64)
             echo "cpu_family = 'x86'" >> $cross
Signature · 1 line
2.18.2
#14 lore raw

PATCH v2 7/7 gitlab-ci: Add cross-compiling build tests

In reply to Thomas Huth

Now that we can use all our QEMU test containers in the gitlab-CI, we can easily add some jobs that test cross-compilation for various architectures. There is just only small ugliness: Since the shared runners on gitlab.com are single-threaded, we have to split each compilation job into two parts (--disable-user and --disable-system), and exclude some additional targets, to avoid that the jobs are running too long and hitting the timeout of 1 h.

---
 .gitlab-ci.d/crossbuilds.yml | 113 +++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml               |   1 +
 MAINTAINERS                  |   1 +
 3 files changed, 115 insertions(+)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml
.gitlab-ci.d/crossbuilds.yml new file +113 -0
new file mode 100644
index 0000000000..4ec7226b5c
@@ -0,0 +1,113 @@
+
+.cross_system_build_job_template: &cross_system_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-user
+        --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
+          mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
+          xtensa-softmmu"
+    - make -j$(expr $(nproc) + 1) all check-build
+
+.cross_user_build_job_template: &cross_user_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
+    - make -j$(expr $(nproc) + 1) all check-build
+
+cross-armel-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-armel-cross
+
+cross-armel-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-armel-cross
+
+cross-armhf-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-armhf-cross
+
+cross-armhf-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-armhf-cross
+
+cross-arm64-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-arm64-cross
+
+cross-arm64-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-arm64-cross
+
+cross-mips-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-mips-cross
+
+cross-mips-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-mips-cross
+
+cross-mipsel-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-mipsel-cross
+
+cross-mipsel-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-mipsel-cross
+
+cross-mips64el-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-mips64el-cross
+
+cross-mips64el-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-mips64el-cross
+
+cross-ppc64el-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-ppc64el-cross
+
+cross-ppc64el-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-ppc64el-cross
+
+cross-s390x-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-s390x-cross
+
+cross-s390x-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-s390x-cross
+
+cross-win32-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-win32-cross
+
+cross-win64-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-win64-cross
.gitlab-ci.yml +1 -0
index b7967b9a13..8a4b67ecca 100644
@@ -18,6 +18,7 @@ include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
+  - local: '/.gitlab-ci.d/crossbuilds.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
MAINTAINERS +1 -0
index 0886eb3d2b..2731f7a594 100644
@@ -3067,6 +3067,7 @@ M: Alex Bennée <[email protected]>
 R: Wainer dos Santos Moschetta <[email protected]>
 S: Maintained
 F: .gitlab-ci.yml
+F: .gitlab-ci.d/crossbuilds.yml
 
 Guest Test Compilation Support
 M: Alex Bennée <[email protected]>
Signature · 1 line
2.18.2
Acked-by on the v2 series
#17 lore raw
In reply to Thomas Huth

Build system parts

Il dom 23 ago 2020, 13:18 Thomas Huth <[email protected]> ha scritto:

Quoted 41 lines

Now that we can use all our QEMU build containers in the gitlab-CI, we can also run the cross-compilation jobs there. Of course, some problems have to be fixed first, so this is taken care of in the first four patches.

The following two patches make sure that we can also enable WHPX builds with
our debian-win64-cross container, so that we can compile-test this accelerator
code now, too.

The last patch then finally enables the cross-compilation jobs in the CI.

v2:
- Dropped patches that are not necessary anymore
- Added the first two patches to fix problems with the new meson build system

Thomas Huth (7):
configure: Add system = 'linux' for meson when cross-compiling tests/docker: Install python3-setuptools in the debian9-mxe containers tests/Makefile: test-image-locking needs CONFIG_POSIX tests/Makefile: test-replication needs CONFIG_POSIX dockerfiles/debian-win64-cross: Download WHPX MinGW headers
configure: Allow automatic WHPX detection
gitlab-ci: Add cross-compiling build tests

 .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
 .gitlab-ci.yml                                |   1 +
 MAINTAINERS                                   |   1 +
 configure                                     |   4 +
 tests/Makefile.include                        |   6 +-
 .../dockerfiles/debian-win64-cross.docker     |   9 +-
 tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
 7 files changed, 133 insertions(+), 3 deletions(-)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml

--
2.18.2

Reviewed-by on v2 1/7
#2 lore raw

Re: [PATCH v2 1/7] configure: Add system = 'linux' for meson when cross-compiling

In reply to Thomas Huth
Quoted 8 lines · Thomas Huth

Meson needs the "system = xyz" line when cross-compiling. We are already adding a "system = 'windows'" for the MinGW cross-compilation case here, so let's add a "system = 'linux'" now for Linux hosts, too.

---
 configure | 3 +++
 1 file changed, 3 insertions(+)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v2 2/7
#4 lore raw

Re: [PATCH v2 2/7] tests/docker: Install python3-setuptools in the debian9-mxe containers

In reply to Thomas Huth
Quoted 7 lines · Thomas Huth

The python setuptools are a requirement for meson, so we need to install this additional package now.

---
 tests/docker/dockerfiles/debian9-mxe.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v2 4/7
#8 lore raw

Re: [PATCH v2 4/7] tests/Makefile: test-replication needs CONFIG_POSIX

In reply to Thomas Huth
Quoted 9 lines · Thomas Huth

test-replication uses sigaction() and friends which are only available on POSIX-like systems.

Message-Id: <[email protected]>

---
 tests/Makefile.include | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v2 3/7
#6 lore raw

Re: [PATCH v2 3/7] tests/Makefile: test-image-locking needs CONFIG_POSIX

In reply to Thomas Huth
Quoted 9 lines · Thomas Huth

test-image-locking.c uses the qemu_lock_fd_test() function which is only available on POSIX-like systems.

Message-Id: <[email protected]>

---
 tests/Makefile.include | 2 ++
 1 file changed, 2 insertions(+)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v2 5/7
#10 lore raw

Re: [PATCH v2 5/7] dockerfiles/debian-win64-cross: Download WHPX MinGW headers

In reply to Thomas Huth
Quoted 21 lines · Thomas Huth

To compile-test the WHPX accelerator, we need to download these system headers first (they are unfortunately not part of any released and packaged MinGW toolchain yet).

Idea taken from another patch by Stefan Weil.

Message-Id: <[email protected]>

---
 tests/docker/dockerfiles/debian-win64-cross.docker | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
tests/docker/dockerfiles/debian-win64-cross.docker +0 -0
index 2fc9cfcbc6..4cc4a3f365 100644
@@ -32,7 +32,14 @@ RUN apt-get update && \
         mxe-$TARGET-w64-mingw32.shared-sdl2 \
         mxe-$TARGET-w64-mingw32.shared-sdl2-mixer \

tests/docker/dockerfiles/debian-win64-cross.docker RUN apt-get update && \ · line 42

Quoted 10 lines
tests/docker/dockerfiles/debian-win64-cross.docker +8 -1
         mxe-$TARGET-w64-mingw32.shared-sdl2-gfx \
-        mxe-$TARGET-w64-mingw32.shared-zlib
+        mxe-$TARGET-w64-mingw32.shared-zlib \
+        curl && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvEmulation.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvemulation.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvPlatform.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatform.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/winhvplatformdefs.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatformdefs.h?format=raw"

My only concern here is whether sourceforge.net URLs are reliable enough for CI, or whether we should instead be mirroring the w64 repo on git.qemu.org ?

I guess this will be a short term issue until a new mingw release is made and arrives i nthe distros, so we can try with the curl approach for now and change if it causes problems

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by on v2 6/7
#13 lore raw

Re: [PATCH v2 6/7] configure: Allow automatic WHPX detection

In reply to Thomas Huth
Quoted 6 lines · Thomas Huth

The whpx variable is currently initialized to "no" which causes the WHPX check to skip the detection unless the user specified --enable-whpx. Since the detection code should be able to figure it out correctly, let's initialized the variable to "" on MinGW-builds for proper auto-detection instead.

Quoted 7 lines

Message-Id: <[email protected]>

---
 configure | 1 +
 1 file changed, 1 insertion(+)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by Reviewed-by on v2 7/7
#15 lore raw

Re: [PATCH v2 7/7] gitlab-ci: Add cross-compiling build tests

In reply to Thomas Huth
Quoted 7 lines · Thomas Huth

Now that we can use all our QEMU test containers in the gitlab-CI, we can easily add some jobs that test cross-compilation for various architectures. There is just only small ugliness: Since the shared runners on gitlab.com are single-threaded, we have to split each compilation job into two parts (--disable-user and --disable-system), and exclude some additional targets, to avoid that the jobs are running too long and hitting the timeout of 1 h.

Quoted 7 lines
---
 .gitlab-ci.d/crossbuilds.yml | 113 +++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml               |   1 +
 MAINTAINERS                  |   1 +
 3 files changed, 115 insertions(+)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml
Quoted 9 lines
.gitlab-ci.d/crossbuilds.yml new file +2 -0
new file mode 100644
index 0000000000..4ec7226b5c
@@ -0,0 +1,113 @@
+
+.cross_system_build_job_template: &cross_system_build_job_definition

.gitlab-ci.d/crossbuilds.yml line 12

Quoted 10 lines
.gitlab-ci.d/crossbuilds.yml new file +10 -0
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-user
+        --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
+          mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
+          xtensa-softmmu"

What does this leave enabled ? Would it be shorter to just say --target-list="...explicit targets we want..." ? It would be clearer to review at least.

.gitlab-ci.d/crossbuilds.yml line 13

Quoted 1 line
.gitlab-ci.d/crossbuilds.yml new file +1 -0
+    - make -j$(expr $(nproc) + 1) all check-build

None the less, it is functionally fine so

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw
In reply to Thomas Huth
Quoted 25 lines · Thomas Huth

Now that we can use all our QEMU build containers in the gitlab-CI, we can also run the cross-compilation jobs there. Of course, some problems have to be fixed first, so this is taken care of in the first four patches.

The following two patches make sure that we can also enable WHPX builds with our debian-win64-cross container, so that we can compile-test this accelerator code now, too.

The last patch then finally enables the cross-compilation jobs in the CI.

v2:
- Dropped patches that are not necessary anymore
- Added the first two patches to fix problems with the new meson build system

Thomas Huth (7):
configure: Add system = 'linux' for meson when cross-compiling tests/docker: Install python3-setuptools in the debian9-mxe containers tests/Makefile: test-image-locking needs CONFIG_POSIX tests/Makefile: test-replication needs CONFIG_POSIX dockerfiles/debian-win64-cross: Download WHPX MinGW headers
configure: Allow automatic WHPX detection
gitlab-ci: Add cross-compiling build tests

Quoted 9 lines
 .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
 .gitlab-ci.yml                                |   1 +
 MAINTAINERS                                   |   1 +
 configure                                     |   4 +
 tests/Makefile.include                        |   6 +-
 .../dockerfiles/debian-win64-cross.docker     |   9 +-
 tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
 7 files changed, 133 insertions(+), 3 deletions(-)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml

You seem to have not sent the mail for

[PATCH 8/7] delete obsolete shippable config

;-P

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw

Re: [PATCH v2 5/7] dockerfiles/debian-win64-cross: Download WHPX MinGW headers

In reply to Daniel P. Berrangé
Quoted 37 lines · Daniel P. Berrangé
Quoted 21 lines · Thomas Huth

To compile-test the WHPX accelerator, we need to download these system headers first (they are unfortunately not part of any released and packaged MinGW toolchain yet).

Idea taken from another patch by Stefan Weil.

Message-Id: <[email protected]>

---
 tests/docker/dockerfiles/debian-win64-cross.docker | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
tests/docker/dockerfiles/debian-win64-cross.docker +0 -0
index 2fc9cfcbc6..4cc4a3f365 100644
@@ -32,7 +32,14 @@ RUN apt-get update && \
         mxe-$TARGET-w64-mingw32.shared-sdl2 \
         mxe-$TARGET-w64-mingw32.shared-sdl2-mixer \

tests/docker/dockerfiles/debian-win64-cross.docker RUN apt-get update && \ · line 42

Quoted 10 lines
tests/docker/dockerfiles/debian-win64-cross.docker +8 -1
         mxe-$TARGET-w64-mingw32.shared-sdl2-gfx \
-        mxe-$TARGET-w64-mingw32.shared-zlib
+        mxe-$TARGET-w64-mingw32.shared-zlib \
+        curl && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvEmulation.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvemulation.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvPlatform.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatform.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/winhvplatformdefs.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatformdefs.h?format=raw"

My only concern here is whether sourceforge.net URLs are reliable enough for CI, or whether we should instead be mirroring the w64 repo on git.qemu.org ?

Quoted 3 lines

I guess this will be a short term issue until a new mingw release is made and arrives i nthe distros, so we can try with the curl approach for now and change if it causes problems

That's my assumption, too. And since the containers are only rebuild when something changed (as far as I understood), the download should only occur quite rarely, so there is hopefully really not a big dependency here.

Quoted 1 line

Thanks!

Thomas

#16 lore raw

Re: [PATCH v2 7/7] gitlab-ci: Add cross-compiling build tests

In reply to Daniel P. Berrangé
Quoted 33 lines · Daniel P. Berrangé
Quoted 7 lines · Thomas Huth

Now that we can use all our QEMU test containers in the gitlab-CI, we can easily add some jobs that test cross-compilation for various architectures. There is just only small ugliness: Since the shared runners on gitlab.com are single-threaded, we have to split each compilation job into two parts (--disable-user and --disable-system), and exclude some additional targets, to avoid that the jobs are running too long and hitting the timeout of 1 h.

Quoted 7 lines
---
 .gitlab-ci.d/crossbuilds.yml | 113 +++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml               |   1 +
 MAINTAINERS                  |   1 +
 3 files changed, 115 insertions(+)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml
Quoted 15 lines
.gitlab-ci.d/crossbuilds.yml new file +8 -0
new file mode 100644
index 0000000000..4ec7226b5c
@@ -0,0 +1,113 @@
+
+.cross_system_build_job_template: &cross_system_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
Quoted 8 lines

.gitlab-ci.d/crossbuilds.yml line 12

Quoted 4 lines
.gitlab-ci.d/crossbuilds.yml new file +4 -0
+      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-user
+        --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
+          mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
+          xtensa-softmmu"

What does this leave enabled ? Would it be shorter to just say --target-list="...explicit targets we want..." ? It would be clearer to review at least.

I basically excluded all targets that have a second compile test coverage, e.g. i386-softmmu code is mostly also covered by x86_64-softmmu, sh4-softmmu is covered by sh4eb-softmmu etc.

The --target-list-exclude also comes in handy when new targets are added to QEMU - you then don't have to remember to add the new targets here, they are picked up automatically.

Quoted 5 lines

.gitlab-ci.d/crossbuilds.yml line 13

Quoted 1 line
.gitlab-ci.d/crossbuilds.yml new file +1 -0
+    - make -j$(expr $(nproc) + 1) all check-build

None the less, it is functionally fine so

Thanks!

Thomas

#19 lore raw
In reply to Daniel P. Berrangé
Quoted 36 lines · Daniel P. Berrangé
Quoted 25 lines · Thomas Huth

Now that we can use all our QEMU build containers in the gitlab-CI, we can also run the cross-compilation jobs there. Of course, some problems have to be fixed first, so this is taken care of in the first four patches.

The following two patches make sure that we can also enable WHPX builds with our debian-win64-cross container, so that we can compile-test this accelerator code now, too.

The last patch then finally enables the cross-compilation jobs in the CI.

v2:
- Dropped patches that are not necessary anymore
- Added the first two patches to fix problems with the new meson build system

Thomas Huth (7):
configure: Add system = 'linux' for meson when cross-compiling tests/docker: Install python3-setuptools in the debian9-mxe containers tests/Makefile: test-image-locking needs CONFIG_POSIX tests/Makefile: test-replication needs CONFIG_POSIX dockerfiles/debian-win64-cross: Download WHPX MinGW headers
configure: Allow automatic WHPX detection
gitlab-ci: Add cross-compiling build tests

Quoted 9 lines
 .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
 .gitlab-ci.yml                                |   1 +
 MAINTAINERS                                   |   1 +
 configure                                     |   4 +
 tests/Makefile.include                        |   6 +-
 .../dockerfiles/debian-win64-cross.docker     |   9 +-
 tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
 7 files changed, 133 insertions(+), 3 deletions(-)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml
Quoted 5 lines

You seem to have not sent the mail for

[PATCH 8/7] delete obsolete shippable config

;-P

Yeah, I wanted to chat with Alex about that first once the gitlab stuff has been mreged...

Thomas