Gitlab CI improvements and fixes
cover Gitlab CI improvements and fixes
Currently, almost all piplines of our Gitlab CI are broken, since there has been a change in the default Debian-based container that causes the QEMU build process to fail (aka. the "BITS_PER_LONG" macro problem).
We could likely either wait for the containers to get fixed, or we could try to work-around that issue in the QEMU sources, but considering that we test with Ubuntu (and thus Debian-based) containers in our Travis-CI already, we should maybe do something different here and rather switch our Gitlab-CI to some RPM-based distros instead, to get even more test coverage. This is now done in the 5th patch. Along the way, there are clean-ups and some minor improvements, and a fix to be able to build QEMU with Clang again (last patch).
After applying this patch series, the Gitlab-CI shows "green" again:
https://gitlab.com/huth/qemu/pipelines/149326595
Alex Bennée (1):
linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
Cleber Rosa (1):
GitLab CI: avoid calling before_scripts on unintended jobs
Philippe Mathieu-Daudé (1):
gitlab-ci: Remove flex/bison packages
Thomas Huth (4):
MAINTAINERS: Add Philippe, Alex and Wainer to the Gitlab-CI section
gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
gitlab-ci: Do not use the standard container images from gitlab
gitlab-ci: Determine the number of jobs dynamically
.gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml | 0 .../opensbi.yml | 0 .gitlab-ci.yml | 67 +++++++++++++------ MAINTAINERS | 5 +- linux-user/elfload.c | 2 + 5 files changed, 51 insertions(+), 23 deletions(-) rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%) rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
Signature · 1 line
2.18.1
PATCH 1/7 MAINTAINERS: Add Philippe, Alex and Wainer to the Gitlab-CI section
Initially, I was the only one who was using Gitlab while most developers had their git trees still on other systems, but that has changed nowadays. There is now much more interest in the Gitlab-CI today, so it would be good to have more than only one maintainer / reviewer for the gitlab-ci.yml file. Alex, Wainer and Philippe kindly offered their help here, so let's add them to the corresponding section in the MAINTAINERS file now.
--- MAINTAINERS | 3 +++ 1 file changed, 3 insertions(+)
MAINTAINERS +3 -0
@@ -2893,6 +2893,9 @@ W: https://cirrus-ci.com/github/qemu/qemu GitLab Continuous Integration M: Thomas Huth <[email protected]> +M: Philippe Mathieu-Daudé <[email protected]> +M: Alex Bennée <[email protected]> +R: Wainer dos Santos Moschetta <[email protected]> S: Maintained F: .gitlab-ci.yml
Signature · 1 line
2.18.1
PATCH 2/7 gitlab-ci: Remove flex/bison packages
In reply to Thomas HuthFrom: Philippe Mathieu-Daudé <[email protected]>
QEMU does not use flex/bison packages.
Message-Id: <[email protected]>
--- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.yml +1 -1
@@ -4,7 +4,7 @@ include: before_script: - apt-get update -qq - - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage + - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage build-system1: script:
Signature · 1 line
2.18.1
Re: [PATCH 2/7] gitlab-ci: Remove flex/bison packages
Quoted 7 lines · Thomas Huth
From: Philippe Mathieu-Daudé <[email protected]>
QEMU does not use flex/bison packages.
Signed-off-by Philippe Mathieu-Daudé <[email protected]>Message-Id: <[email protected]>
Signed-off-by Thomas Huth <[email protected]>
Quoted 17 lines
--- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-).gitlab-ci.yml +1
-1@@ -4,7 +4,7 @@ include: before_script: - apt-get update -qq - - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage + - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage build-system1: script:
Signature · 1 line
Alex Bennée
PATCH 3/7 GitLab CI: avoid calling before_scripts on unintended jobs
In reply to Thomas HuthFrom: Cleber Rosa <[email protected]>
At this point it seems that all jobs depend on those steps, with maybe the EDK2 jobs as exceptions.
The jobs that will be added later will not want those scripts to be run, so let's move these steps to the appropriate jobs, while still trying to avoid repetition.
--- .gitlab-ci.yml | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
.gitlab-ci.yml +11 -3
@@ -2,11 +2,13 @@ include: - local: '/.gitlab-ci-edk2.yml' - local: '/.gitlab-ci-opensbi.yml' -before_script: - - apt-get update -qq - - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage +.update_apt_template: &before_script_apt + before_script: + - apt-get update -qq + - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage build-system1: + <<: *before_script_apt script: - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
@@ -19,6 +21,7 @@ build-system1: - make -j2 check build-system2: + <<: *before_script_apt script: - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
@@ -32,6 +35,7 @@ build-system2: - make -j2 check build-disabled: + <<: *before_script_apt script: - mkdir build - cd build
@@ -46,6 +50,7 @@ build-disabled: - make -j2 check-qtest SPEED=slow build-tcg-disabled: + <<: *before_script_apt script: - apt-get install -y -qq clang libgtk-3-dev libusb-dev - mkdir build
@@ -64,6 +69,7 @@ build-tcg-disabled: 260 261 262 263 264 270 272 273 277 279 build-user: + <<: *before_script_apt script: - mkdir build - cd build
@@ -73,6 +79,7 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: + <<: *before_script_apt script: - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
@@ -85,6 +92,7 @@ build-clang: - make -j2 check build-tci: + <<: *before_script_apt script: - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" - mkdir build
Signature · 1 line
2.18.1
Re: [PATCH 3/7] GitLab CI: avoid calling before_scripts on unintended jobs
Quoted 5 lines · Thomas Huth
From: Cleber Rosa <[email protected]>
At this point it seems that all jobs depend on those steps, with maybe the EDK2 jobs as exceptions.
Quoted 7 lines
The jobs that will be added later will not want those scripts to be run, so let's move these steps to the appropriate jobs, while still trying to avoid repetition.
Signed-off-by Cleber Rosa <[email protected]>[thuth: Rebased to current master branch, use separate template]Signed-off-by Thomas Huth <[email protected]>
Signature · 1 line
Alex Bennée
PATCH 4/7 gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
In reply to Thomas HuthWe have a dedicated folder for the gitlab-ci - so there is no need to clutter the top directory with these .yml files.
--- .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml | 0 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0 .gitlab-ci.yml | 4 ++-- MAINTAINERS | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-) rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%) rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
.gitlab-ci.d/edk2.yml renamed from .gitlab-ci-edk2.yml +0 -0
.gitlab-ci.d/opensbi.yml renamed from .gitlab-ci-opensbi.yml +0 -0
.gitlab-ci.yml +2 -2
@@ -1,6 +1,6 @@ include: - - local: '/.gitlab-ci-edk2.yml' - - local: '/.gitlab-ci-opensbi.yml' + - local: '/.gitlab-ci.d/edk2.yml' + - local: '/.gitlab-ci.d/opensbi.yml' .update_apt_template: &before_script_apt before_script:
MAINTAINERS +1 -1
@@ -2534,7 +2534,7 @@ F: roms/edk2 F: roms/edk2-* F: tests/data/uefi-boot-images/ F: tests/uefi-test-tools/ -F: .gitlab-ci-edk2.yml +F: .gitlab-ci.d/edk2.yml F: .gitlab-ci.d/edk2/ Usermode Emulation
Signature · 1 line
2.18.1
Re: [PATCH 4/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
Quoted 40 lines · Thomas Huth
We have a dedicated folder for the gitlab-ci - so there is no need to clutter the top directory with these .yml files.
Signed-off-by Thomas Huth <[email protected]>--- .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml | 0 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0 .gitlab-ci.yml | 4 ++-- MAINTAINERS | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-) rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%) rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%).gitlab-ci.d/edk2.yml renamed from .gitlab-ci-edk2.yml +0
-0.gitlab-ci.d/opensbi.yml renamed from .gitlab-ci-opensbi.yml +0
-0.gitlab-ci.yml +2
-2@@ -1,6 +1,6 @@ include: - - local: '/.gitlab-ci-edk2.yml' - - local: '/.gitlab-ci-opensbi.yml' + - local: '/.gitlab-ci.d/edk2.yml' + - local: '/.gitlab-ci.d/opensbi.yml' .update_apt_template: &before_script_apt before_script:MAINTAINERS +0
-0
MAINTAINERS F: roms/edk2 · line 2540
Quoted 8 lines
MAINTAINERS +1
-1@@ -2534,7 +2534,7 @@ F: roms/edk2 F: roms/edk2-* F: tests/data/uefi-boot-images/ F: tests/uefi-test-tools/ -F: .gitlab-ci-edk2.yml +F: .gitlab-ci.d/edk2.yml F: .gitlab-ci.d/edk2/ Usermode Emulation
Re: [PATCH 4/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
In reply to Thomas HuthQuoted 4 lines · Thomas Huth
We have a dedicated folder for the gitlab-ci - so there is no need to clutter the top directory with these .yml files.
Signed-off-by Thomas Huth <[email protected]>
Quoted 43 lines
--- .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml | 0 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0 .gitlab-ci.yml | 4 ++-- MAINTAINERS | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-) rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%) rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%).gitlab-ci.d/edk2.yml renamed from .gitlab-ci-edk2.yml +0
-0.gitlab-ci.d/opensbi.yml renamed from .gitlab-ci-opensbi.yml +0
-0.gitlab-ci.yml +2
-2@@ -1,6 +1,6 @@ include: - - local: '/.gitlab-ci-edk2.yml' - - local: '/.gitlab-ci-opensbi.yml' + - local: '/.gitlab-ci.d/edk2.yml' + - local: '/.gitlab-ci.d/opensbi.yml' .update_apt_template: &before_script_apt before_script:MAINTAINERS +1
-1@@ -2534,7 +2534,7 @@ F: roms/edk2 F: roms/edk2-* F: tests/data/uefi-boot-images/ F: tests/uefi-test-tools/ -F: .gitlab-ci-edk2.yml +F: .gitlab-ci.d/edk2.yml F: .gitlab-ci.d/edk2/ Usermode Emulation
Signature · 1 line
Alex Bennée
PATCH 5/7 gitlab-ci: Do not use the standard container images from gitlab
In reply to Thomas HuthCurrently all pipelines of the gitlab CI are failing, except for the "build-user" pipeline. There is an issue with the default container image (likely Debian stable) where they imported something bad in one of the system headers:
/usr/include/linux/swab.h: In function '__swab':
/builds/huth/qemu/include/qemu/bitops.h:20:34: error: "sizeof" is not
defined, evaluates to 0 [-Werror=undef]
#define BITS_PER_LONG (sizeof (unsigned long) * BITS_PER_BYTE)
We could maybe work-around this issue or wait for the default containers to get fixed, but considering that we use Ubuntu (and thus Debian-style) CI in Travis already to a very large extent, we should consider to use some RPM-based distros in our gitlab CI instead. Thus let's change the failing pipelines to use Fedora and CentOS (and also one Ubuntu 19.10, since 20.04 is broken, too) now.
--- .gitlab-ci.yml | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-)
.gitlab-ci.yml +25 -12
@@ -5,9 +5,17 @@ include: .update_apt_template: &before_script_apt before_script: - apt-get update -qq - - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage + - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make + genisoimage + +.update_dnf_template: &before_script_dnf + before_script: + - dnf update -y + - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel + make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel build-system1: + image: ubuntu:19.10 <<: *before_script_apt script: - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
@@ -21,11 +29,12 @@ build-system1: - make -j2 check build-system2: - <<: *before_script_apt + image: fedora:latest + <<: *before_script_dnf script: - - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev - libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev - libzstd-dev + - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel + libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel + libzstd-devel - mkdir build - cd build - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
@@ -35,7 +44,8 @@ build-system2: - make -j2 check build-disabled: - <<: *before_script_apt + image: fedora:latest + <<: *before_script_dnf script: - mkdir build - cd build
@@ -50,9 +60,10 @@ build-disabled: - make -j2 check-qtest SPEED=slow build-tcg-disabled: - <<: *before_script_apt + image: centos:8 + <<: *before_script_dnf script: - - apt-get install -y -qq clang libgtk-3-dev libusb-dev + - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel - mkdir build - cd build - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
@@ -79,10 +90,11 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: - <<: *before_script_apt + image: fedora:latest + <<: *before_script_dnf script: - - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev - xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev + - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel + libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel - mkdir build - cd build - ../configure --cc=clang --cxx=clang++ --enable-werror
@@ -92,7 +104,8 @@ build-clang: - make -j2 check build-tci: - <<: *before_script_apt + image: centos:8 + <<: *before_script_dnf script: - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" - mkdir build
Signature · 1 line
2.18.1
Re: [PATCH 5/7] gitlab-ci: Do not use the standard container images from gitlab
Quoted 10 lines · Thomas Huth
Currently all pipelines of the gitlab CI are failing, except for the "build-user" pipeline. There is an issue with the default container image (likely Debian stable) where they imported something bad in one of the system headers:
/usr/include/linux/swab.h: In function '__swab':
/builds/huth/qemu/include/qemu/bitops.h:20:34: error: "sizeof" is not
defined, evaluates to 0 [-Werror=undef]
#defineBITS_PER_LONG(sizeof (unsigned long) *BITS_PER_BYTE)
Quoted 8 lines
We could maybe work-around this issue or wait for the default containers to get fixed, but considering that we use Ubuntu (and thus Debian-style) CI in Travis already to a very large extent, we should consider to use some RPM-based distros in our gitlab CI instead. Thus let's change the failing pipelines to use Fedora and CentOS (and also one Ubuntu 19.10, since 20.04 is broken, too) now.
Signed-off-by Thomas Huth <[email protected]>
I will say Fedora/CentOS won't be immune to this failure if they update the kernel headers to somewhere between the breakage and the fix.
Quoted 92 lines
--- .gitlab-ci.yml | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-).gitlab-ci.yml +25
-12@@ -5,9 +5,17 @@ include: .update_apt_template: &before_script_apt before_script: - apt-get update -qq - - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage + - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make + genisoimage + +.update_dnf_template: &before_script_dnf + before_script: + - dnf update -y + - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel + make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel build-system1: + image: ubuntu:19.10 <<: *before_script_apt script: - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev@@ -21,11 +29,12 @@ build-system1: - make -j2 check build-system2: - <<: *before_script_apt + image: fedora:latest + <<: *before_script_dnf script: - - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev - libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev - libzstd-dev + - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel + libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel + libzstd-devel - mkdir build - cd build - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu@@ -35,7 +44,8 @@ build-system2: - make -j2 check build-disabled: - <<: *before_script_apt + image: fedora:latest + <<: *before_script_dnf script: - mkdir build - cd build@@ -50,9 +60,10 @@ build-disabled: - make -j2 check-qtest SPEED=slow build-tcg-disabled: - <<: *before_script_apt + image: centos:8 + <<: *before_script_dnf script: - - apt-get install -y -qq clang libgtk-3-dev libusb-dev + - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel - mkdir build - cd build - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""@@ -79,10 +90,11 @@ build-user: - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang: - <<: *before_script_apt + image: fedora:latest + <<: *before_script_dnf script: - - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev - xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev + - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel + libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel - mkdir build - cd build - ../configure --cc=clang --cxx=clang++ --enable-werror@@ -92,7 +104,8 @@ build-clang: - make -j2 check build-tci: - <<: *before_script_apt + image: centos:8 + <<: *before_script_dnf script: - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64" - mkdir build
Signature · 1 line
Alex Bennée
PATCH 6/7 gitlab-ci: Determine the number of jobs dynamically
In reply to Thomas HuthSome people might want to run the gitlab CI pipelines in an environment where multiple CPUs are available to the runners, so let's rather get the number for "-j" from the "nproc" program (increased by 1 to compensate for jobs that wait for I/O) instead of hard-coding it.
--- .gitlab-ci.yml | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
.gitlab-ci.yml +13 -11
@@ -7,12 +7,14 @@ include: - apt-get update -qq - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make genisoimage + - JOBS=$(expr $(nproc) + 1) .update_dnf_template: &before_script_dnf before_script: - dnf update -y - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel + - JOBS=$(expr $(nproc) + 1) build-system1: image: ubuntu:19.10
@@ -25,8 +27,8 @@ build-system1: - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check build-system2: image: fedora:latest
@@ -40,8 +42,8 @@ build-system2: - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check build-disabled: image: fedora:latest
@@ -56,8 +58,8 @@ build-disabled: --disable-qom-cast-debug --disable-spice --disable-vhost-vsock --disable-vhost-net --disable-vhost-crypto --disable-vhost-user --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user" - - make -j2 - - make -j2 check-qtest SPEED=slow + - make -j"$JOBS" + - make -j"$JOBS" check-qtest SPEED=slow build-tcg-disabled: image: centos:8
@@ -67,7 +69,7 @@ build-tcg-disabled: - mkdir build - cd build - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list="" - - make -j2 + - make -j"$JOBS" - make check-unit - make check-qapi-schema - cd tests/qemu-iotests/
@@ -86,7 +88,7 @@ build-user: - cd build - ../configure --enable-werror --disable-system --disable-guest-agent --disable-capstone --disable-slirp --disable-fdt - - make -j2 + - make -j"$JOBS" - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user build-clang:
@@ -100,8 +102,8 @@ build-clang: - ../configure --cc=clang --cxx=clang++ --enable-werror --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check build-tci: image: centos:8
@@ -112,7 +114,7 @@ build-tci: - cd build - ../configure --enable-tcg-interpreter --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" - - make -j2 + - make -j"$JOBS" - make run-tcg-tests-x86_64-softmmu - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test - for tg in $TARGETS ; do
Signature · 1 line
2.18.1
Re: [PATCH 6/7] gitlab-ci: Determine the number of jobs dynamically
Quoted 6 lines · Thomas Huth
Some people might want to run the gitlab CI pipelines in an environment where multiple CPUs are available to the runners, so let's rather get the number for "-j" from the "nproc" program (increased by 1 to compensate for jobs that wait for I/O) instead of hard-coding it.
Signed-off-by Thomas Huth <[email protected]>
<snip>
Quoted 23 lines
.gitlab-ci.yml +4
-4@@ -25,8 +27,8 @@ build-system1: - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check build-system2: image: fedora:latest@@ -40,8 +42,8 @@ build-system2: - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check build-disabled: image: fedora:latest
.gitlab-ci.yml build-disabled: · line 62
Quoted 7 lines
.gitlab-ci.yml +2
-2@@ -56,8 +58,8 @@ build-disabled: --disable-qom-cast-debug --disable-spice --disable-vhost-vsock --disable-vhost-net --disable-vhost-crypto --disable-vhost-user --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user" - - make -j2 - - make -j2 check-qtest SPEED=slow + - make -j"$JOBS" + - make -j"$JOBS" check-qtest SPEED=slow
I would make all the check jobs use a single core as it otherwise gets hard to figure out exactly where something broke/hung.
<snip>
.gitlab-ci.yml build-clang: · line 106
Quoted 8 lines
.gitlab-ci.yml +2
-2@@ -100,8 +102,8 @@ build-clang: - ../configure --cc=clang --cxx=clang++ --enable-werror --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check
Ditto for this check
.gitlab-ci.yml build-tci: · line 120
Quoted 12 lines
.gitlab-ci.yml +1
-1build-tci: image: centos:8@@ -112,7 +114,7 @@ build-tci: - cd build - ../configure --enable-tcg-interpreter --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" - - make -j2 + - make -j"$JOBS" - make run-tcg-tests-x86_64-softmmu - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test - for tg in $TARGETS ; do
Otherwise:
Signature · 1 line
Alex Bennée
Re: [PATCH 6/7] gitlab-ci: Determine the number of jobs dynamically
Quoted 30 lines · Alex Bennée
Quoted 6 lines · Thomas Huth
Some people might want to run the gitlab CI pipelines in an environment where multiple CPUs are available to the runners, so let's rather get the number for "-j" from the "nproc" program (increased by 1 to compensate for jobs that wait for I/O) instead of hard-coding it.
Signed-off-by Thomas Huth <[email protected]><snip>
Quoted 20 lines
.gitlab-ci.yml +4
-4@@ -25,8 +27,8 @@ build-system1: - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check build-system2: image: fedora:latest@@ -40,8 +42,8 @@ build-system2: - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu" - - make -j2 - - make -j2 check + - make -j"$JOBS" + - make -j"$JOBS" check
Quoted 13 lines
.gitlab-ci.yml build-disabled: · line 62
Quoted 10 lines
.gitlab-ci.yml +2
-2build-disabled: image: fedora:latest@@ -56,8 +58,8 @@ build-disabled: --disable-qom-cast-debug --disable-spice --disable-vhost-vsock --disable-vhost-net --disable-vhost-crypto --disable-vhost-user --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user" - - make -j2 - - make -j2 check-qtest SPEED=slow + - make -j"$JOBS" + - make -j"$JOBS" check-qtest SPEED=slowI would make all the check jobs use a single core as it otherwise gets hard to figure out exactly where something broke/hung.
It's a somewhat double-edged sword ... either faster CI test times, or more deterministic output ... so far I didn't suffer the problem with the deterministic output in the gitlab-CI yet (unlike with Travis), so I'd rather keep the -j here for now. We can still remove it later if we hit a bug that is hard to debug otherwise.
Thomas
PATCH 7/7 linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
In reply to Thomas HuthFrom: Alex Bennée <[email protected]>
Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:
linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.
So lets limit the check to 32 bit hosts only.
--- linux-user/elfload.c | 2 ++ 1 file changed, 2 insertions(+)
linux-user/elfload.c +2 -0
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,exit (EXIT_FAILURE ); } }else { +#if HOST_LONG_BITS <TARGET_ABI_BITS if ((guest_hiaddr - guest_base) > ~(uintptr_t )0 ) {error_report ("%s: requires more virtual address space " "than the host can provide (0x%" PRIx64")" , image_name, (uint64_t )guest_hiaddr - guest_base);exit (EXIT_FAILURE ); } +#endif }/*
Signature · 1 line
2.18.1
Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
Le 25/05/2020 à 15:18, Thomas Huth a écrit :
Quoted 33 lines
From: Alex Bennée <[email protected]>
Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:
linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr-guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.So lets limit the check to 32 bit hosts only.
Fixes ee94743034bfReported-by Thomas Huth <[email protected]>Signed-off-by Alex Bennée <[email protected]>[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]Signed-off-by Thomas Huth <[email protected]>--- linux-user/elfload.c | 2 ++ 1 file changed, 2 insertions(+)linux-user/elfload.c +1
-0@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,exit (EXIT_FAILURE ); } }else { +#if HOST_LONG_BITS <TARGET_ABI_BITS
linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086
Quoted 10 lines
linux-user/elfload.c +1
-0if ((guest_hiaddr - guest_base) > ~(uintptr_t )0 ) {error_report ("%s: requires more virtual address space " "than the host can provide (0x%" PRIx64")" , image_name, (uint64_t )guest_hiaddr - guest_base);exit (EXIT_FAILURE ); } +#endif }/*
Philippe sent the same patch:
https://www.mail-archive.com/[email protected]/msg699796.html
Thanks,
Laurent
Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
Quoted 42 lines · Laurent Vivier
Le 25/05/2020 à 15:18, Thomas Huth a écrit :
Quoted 41 lines
From: Alex Bennée <[email protected]>
Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:
linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr-guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.So lets limit the check to 32 bit hosts only.
Fixes ee94743034bfReported-by Thomas Huth <[email protected]>Signed-off-by Alex Bennée <[email protected]>[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]Signed-off-by Thomas Huth <[email protected]>--- linux-user/elfload.c | 2 ++ 1 file changed, 2 insertions(+)linux-user/elfload.c +2
-0@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,exit (EXIT_FAILURE ); } }else { +#if HOST_LONG_BITS <TARGET_ABI_BITS if ((guest_hiaddr - guest_base) > ~(uintptr_t )0 ) {error_report ("%s: requires more virtual address space " "than the host can provide (0x%" PRIx64")" , image_name, (uint64_t )guest_hiaddr - guest_base);exit (EXIT_FAILURE ); } +#endif }
Quoted 6 lines
linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086
Quoted 2 lines
linux-user/elfload.c +0
-0/* Philippe sent the same patch:
https://www.mail-archive.com/[email protected]/msg699796.html
Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?
Thomas
Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
Quoted 48 lines · Thomas Huth
Quoted 47 lines · Laurent Vivier
Quoted 33 lines · Thomas Huth
From: Alex Bennée <[email protected]>
Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:
linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr-guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.So lets limit the check to 32 bit hosts only.
Fixes ee94743034bfReported-by Thomas Huth <[email protected]>Signed-off-by Alex Bennée <[email protected]>[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]Signed-off-by Thomas Huth <[email protected]>--- linux-user/elfload.c | 2 ++ 1 file changed, 2 insertions(+)linux-user/elfload.c +1
-0@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,exit (EXIT_FAILURE ); } }else { +#if HOST_LONG_BITS <TARGET_ABI_BITS linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086
Quoted 10 lines
linux-user/elfload.c +1
-0if ((guest_hiaddr - guest_base) > ~(uintptr_t )0 ) {error_report ("%s: requires more virtual address space " "than the host can provide (0x%" PRIx64")" , image_name, (uint64_t )guest_hiaddr - guest_base);exit (EXIT_FAILURE ); } +#endif }/* Philippe sent the same patch:
Quoted 4 lines
Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?
Richard was more inclined to suppress the warning:
Subject: Re: [PATCH v2] linux-user: limit check to HOST_LONG_BITS == 32
From: Richard Henderson <[email protected]>
Message-ID: <[email protected]>
Date: Thu, 21 May 2020 20:15:51 -0700
One reason I dropped the f32 patch from my last PR was because this wasn't the only warning the latest clang picks up.
Signature · 1 line
Alex Bennée
Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
Quoted 58 lines · Alex Bennée
Quoted 48 lines · Thomas Huth
Quoted 47 lines · Laurent Vivier
Quoted 33 lines · Thomas Huth
From: Alex Bennée <[email protected]>
Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:
linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr-guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.So lets limit the check to 32 bit hosts only.
Fixes ee94743034bfReported-by Thomas Huth <[email protected]>Signed-off-by Alex Bennée <[email protected]>[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]Signed-off-by Thomas Huth <[email protected]>--- linux-user/elfload.c | 2 ++ 1 file changed, 2 insertions(+)linux-user/elfload.c +1
-0@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,exit (EXIT_FAILURE ); } }else { +#if HOST_LONG_BITS <TARGET_ABI_BITS linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086
Quoted 10 lines
linux-user/elfload.c +1
-0if ((guest_hiaddr - guest_base) > ~(uintptr_t )0 ) {error_report ("%s: requires more virtual address space " "than the host can provide (0x%" PRIx64")" , image_name, (uint64_t )guest_hiaddr - guest_base);exit (EXIT_FAILURE ); } +#endif }/* Philippe sent the same patch:
Quoted 4 lines
Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?
Richard was more inclined to suppress the warning:
Quoted 7 lines
Subject: Re: [PATCH v2] linux-user: limit check to
HOST_LONG_BITS== 32
From: Richard Henderson <[email protected]>
Message-ID: <[email protected]>
Date: Thu, 21 May 2020 20:15:51 -0700One reason I dropped the f32 patch from my last PR was because this wasn't the only warning the latest clang picks up.
... but this is currently the only spot that is required to get the gitlab CI going again, so I think we should include this patch until we have a final decision whether to disable the warning or not (and we can still revert this patch after we disabled the warning). Ok?
Thomas
Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS
Quoted 64 lines · Thomas Huth
Quoted 63 lines · Alex Bennée
Quoted 48 lines · Thomas Huth
Quoted 47 lines · Laurent Vivier
Quoted 33 lines · Thomas Huth
From: Alex Bennée <[email protected]>
Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:
linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr-guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.So lets limit the check to 32 bit hosts only.
Fixes ee94743034bfReported-by Thomas Huth <[email protected]>Signed-off-by Alex Bennée <[email protected]>[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]Signed-off-by Thomas Huth <[email protected]>--- linux-user/elfload.c | 2 ++ 1 file changed, 2 insertions(+)linux-user/elfload.c +1
-0@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,exit (EXIT_FAILURE ); } }else { +#if HOST_LONG_BITS <TARGET_ABI_BITS linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086
Quoted 10 lines
linux-user/elfload.c +1
-0if ((guest_hiaddr - guest_base) > ~(uintptr_t )0 ) {error_report ("%s: requires more virtual address space " "than the host can provide (0x%" PRIx64")" , image_name, (uint64_t )guest_hiaddr - guest_base);exit (EXIT_FAILURE ); } +#endif }/* Philippe sent the same patch:
Quoted 4 lines
Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?
Richard was more inclined to suppress the warning:
Subject: Re: [PATCH v2] linux-user: limit check to
HOST_LONG_BITS== 32
From: Richard Henderson <[email protected]>
Message-ID: <[email protected]>
Date: Thu, 21 May 2020 20:15:51 -0700
Quoted 7 lines
Quoted 2 lines
One reason I dropped the f32 patch from my last PR was because this wasn't the only warning the latest clang picks up.
... but this is currently the only spot that is required to get the gitlab CI going again, so I think we should include this patch until we have a final decision whether to disable the warning or not (and we can still revert this patch after we disabled the warning). Ok?
I'm certainly happy with that if it gets gitlab working.
My experience with make docker-test-vlang@fedora (with 32) was there where more things to fix. I guess gitlab didn't trigger them.
Signature · 1 line
Alex Bennée