Skip to contentSkip to conversation

Gitlab CI improvements and fixes

PATCH In review Thomas Huth 4 participants 20 messages May 25, 2020 – May 28, 2020

lore
Share & tools
lore
#0 lore raw

cover Gitlab CI improvements and fixes

Currently, almost all piplines of our Gitlab CI are broken, since there has been a change in the default Debian-based container that causes the QEMU build process to fail (aka. the "BITS_PER_LONG" macro problem).

We could likely either wait for the containers to get fixed, or we could try to work-around that issue in the QEMU sources, but considering that we test with Ubuntu (and thus Debian-based) containers in our Travis-CI already, we should maybe do something different here and rather switch our Gitlab-CI to some RPM-based distros instead, to get even more test coverage. This is now done in the 5th patch. Along the way, there are clean-ups and some minor improvements, and a fix to be able to build QEMU with Clang again (last patch).

After applying this patch series, the Gitlab-CI shows "green" again:

https://gitlab.com/huth/qemu/pipelines/149326595

Alex Bennée (1):
linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

Cleber Rosa (1):
GitLab CI: avoid calling before_scripts on unintended jobs

Philippe Mathieu-Daudé (1):
gitlab-ci: Remove flex/bison packages

Thomas Huth (4):
MAINTAINERS: Add Philippe, Alex and Wainer to the Gitlab-CI section
gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
gitlab-ci: Do not use the standard container images from gitlab
gitlab-ci: Determine the number of jobs dynamically

 .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml  |  0
 .../opensbi.yml                               |  0
 .gitlab-ci.yml                                | 67 +++++++++++++------
 MAINTAINERS                                   |  5 +-
 linux-user/elfload.c                          |  2 +
 5 files changed, 51 insertions(+), 23 deletions(-)
 rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
 rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
Signature · 1 line
2.18.1
#1 lore raw

PATCH 1/7 MAINTAINERS: Add Philippe, Alex and Wainer to the Gitlab-CI section

Initially, I was the only one who was using Gitlab while most developers had their git trees still on other systems, but that has changed nowadays. There is now much more interest in the Gitlab-CI today, so it would be good to have more than only one maintainer / reviewer for the gitlab-ci.yml file. Alex, Wainer and Philippe kindly offered their help here, so let's add them to the corresponding section in the MAINTAINERS file now.

---
 MAINTAINERS | 3 +++
 1 file changed, 3 insertions(+)
MAINTAINERS +3 -0
index 87a412c229..bde5fd480f 100644
@@ -2893,6 +2893,9 @@ W: https://cirrus-ci.com/github/qemu/qemu
 
 GitLab Continuous Integration
 M: Thomas Huth <[email protected]>
+M: Philippe Mathieu-Daudé <[email protected]>
+M: Alex Bennée <[email protected]>
+R: Wainer dos Santos Moschetta <[email protected]>
 S: Maintained
 F: .gitlab-ci.yml
 
Signature · 1 line
2.18.1
#4 lore raw

PATCH 3/7 GitLab CI: avoid calling before_scripts on unintended jobs

In reply to Thomas Huth

From: Cleber Rosa <[email protected]>

At this point it seems that all jobs depend on those steps, with maybe the EDK2 jobs as exceptions.

The jobs that will be added later will not want those scripts to be run, so let's move these steps to the appropriate jobs, while still trying to avoid repetition.

[thuth: Rebased to current master branch, use separate template]
---
 .gitlab-ci.yml | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
.gitlab-ci.yml +11 -3
index 994774250f..bc6aee6aba 100644
@@ -2,11 +2,13 @@ include:
   - local: '/.gitlab-ci-edk2.yml'
   - local: '/.gitlab-ci-opensbi.yml'
 
-before_script:
- - apt-get update -qq
- - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
+.update_apt_template: &before_script_apt
+ before_script:
+  - apt-get update -qq
+  - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
 
 build-system1:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
       libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
@@ -19,6 +21,7 @@ build-system1:
  - make -j2 check
 
 build-system2:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
       libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
@@ -32,6 +35,7 @@ build-system2:
  - make -j2 check
 
 build-disabled:
+ <<: *before_script_apt
  script:
  - mkdir build
  - cd build
@@ -46,6 +50,7 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq clang libgtk-3-dev libusb-dev
  - mkdir build
@@ -64,6 +69,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ <<: *before_script_apt
  script:
  - mkdir build
  - cd build
@@ -73,6 +79,7 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
       xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
@@ -85,6 +92,7 @@ build-clang:
  - make -j2 check
 
 build-tci:
+ <<: *before_script_apt
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
Signature · 1 line
2.18.1
#9 lore raw

PATCH 5/7 gitlab-ci: Do not use the standard container images from gitlab

In reply to Thomas Huth

Currently all pipelines of the gitlab CI are failing, except for the "build-user" pipeline. There is an issue with the default container image (likely Debian stable) where they imported something bad in one of the system headers:

/usr/include/linux/swab.h: In function '__swab':
/builds/huth/qemu/include/qemu/bitops.h:20:34: error: "sizeof" is not
defined, evaluates to 0 [-Werror=undef]
#define BITS_PER_LONG (sizeof (unsigned long) * BITS_PER_BYTE)

We could maybe work-around this issue or wait for the default containers to get fixed, but considering that we use Ubuntu (and thus Debian-style) CI in Travis already to a very large extent, we should consider to use some RPM-based distros in our gitlab CI instead. Thus let's change the failing pipelines to use Fedora and CentOS (and also one Ubuntu 19.10, since 20.04 is broken, too) now.

---
 .gitlab-ci.yml | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)
.gitlab-ci.yml +25 -12
index 5208d93ff8..559ec2ab4d 100644
@@ -5,9 +5,17 @@ include:
 .update_apt_template: &before_script_apt
  before_script:
   - apt-get update -qq
-  - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
+  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
+        genisoimage
+
+.update_dnf_template: &before_script_dnf
+ before_script:
+  - dnf update -y
+  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
+        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
 
 build-system1:
+ image: ubuntu:19.10
  <<: *before_script_apt
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
@@ -21,11 +29,12 @@ build-system1:
  - make -j2 check
 
 build-system2:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
-      libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
-      libzstd-dev
+ - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
+       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
+       libzstd-devel
  - mkdir build
  - cd build
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
@@ -35,7 +44,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
  - mkdir build
  - cd build
@@ -50,9 +60,10 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
- <<: *before_script_apt
+ image: centos:8
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq clang libgtk-3-dev libusb-dev
+ - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
  - mkdir build
  - cd build
  - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
@@ -79,10 +90,11 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
-      xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
+ - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
+       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
  - mkdir build
  - cd build
  - ../configure --cc=clang --cxx=clang++ --enable-werror
@@ -92,7 +104,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
- <<: *before_script_apt
+ image: centos:8
+ <<: *before_script_dnf
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
Signature · 1 line
2.18.1
#2 lore raw

PATCH 2/7 gitlab-ci: Remove flex/bison packages

In reply to Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

QEMU does not use flex/bison packages.

Message-Id: <[email protected]>

---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.yml +1 -1
index b889fb96b6..994774250f 100644
@@ -4,7 +4,7 @@ include:
 
 before_script:
  - apt-get update -qq
- - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
+ - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
 
 build-system1:
  script:
Signature · 1 line
2.18.1
#6 lore raw

PATCH 4/7 gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

In reply to Thomas Huth

We have a dedicated folder for the gitlab-ci - so there is no need to clutter the top directory with these .yml files.

---
 .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
 .gitlab-ci.yml                                     | 4 ++--
 MAINTAINERS                                        | 2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
 rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
.gitlab-ci.d/edk2.yml renamed from .gitlab-ci-edk2.yml +0 -0
similarity index 100%
rename from .gitlab-ci-edk2.yml
rename to .gitlab-ci.d/edk2.yml
.gitlab-ci.d/opensbi.yml renamed from .gitlab-ci-opensbi.yml +0 -0
similarity index 100%
rename from .gitlab-ci-opensbi.yml
rename to .gitlab-ci.d/opensbi.yml
.gitlab-ci.yml +2 -2
index bc6aee6aba..5208d93ff8 100644
@@ -1,6 +1,6 @@
 include:
-  - local: '/.gitlab-ci-edk2.yml'
-  - local: '/.gitlab-ci-opensbi.yml'
+  - local: '/.gitlab-ci.d/edk2.yml'
+  - local: '/.gitlab-ci.d/opensbi.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
MAINTAINERS +1 -1
index bde5fd480f..d43c98115c 100644
@@ -2534,7 +2534,7 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
-F: .gitlab-ci-edk2.yml
+F: .gitlab-ci.d/edk2.yml
 F: .gitlab-ci.d/edk2/
 
 Usermode Emulation
Signature · 1 line
2.18.1
#11 lore raw

PATCH 6/7 gitlab-ci: Determine the number of jobs dynamically

In reply to Thomas Huth

Some people might want to run the gitlab CI pipelines in an environment where multiple CPUs are available to the runners, so let's rather get the number for "-j" from the "nproc" program (increased by 1 to compensate for jobs that wait for I/O) instead of hard-coding it.

---
 .gitlab-ci.yml | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
.gitlab-ci.yml +13 -11
index 559ec2ab4d..349c77aa58 100644
@@ -7,12 +7,14 @@ include:
   - apt-get update -qq
   - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
         genisoimage
+  - JOBS=$(expr $(nproc) + 1)
 
 .update_dnf_template: &before_script_dnf
  before_script:
   - dnf update -y
   - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
         make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
+  - JOBS=$(expr $(nproc) + 1)
 
 build-system1:
  image: ubuntu:19.10
@@ -25,8 +27,8 @@ build-system1:
  - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
       cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
       mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-system2:
  image: fedora:latest
@@ -40,8 +42,8 @@ build-system2:
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
       microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
       sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-disabled:
  image: fedora:latest
@@ -56,8 +58,8 @@ build-disabled:
       --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
       --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
       --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j2
- - make -j2 check-qtest SPEED=slow
+ - make -j"$JOBS"
+ - make -j"$JOBS" check-qtest SPEED=slow
 
 build-tcg-disabled:
  image: centos:8
@@ -67,7 +69,7 @@ build-tcg-disabled:
  - mkdir build
  - cd build
  - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
- - make -j2
+ - make -j"$JOBS"
  - make check-unit
  - make check-qapi-schema
  - cd tests/qemu-iotests/
@@ -86,7 +88,7 @@ build-user:
  - cd build
  - ../configure --enable-werror --disable-system --disable-guest-agent
                --disable-capstone --disable-slirp --disable-fdt
- - make -j2
+ - make -j"$JOBS"
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
@@ -100,8 +102,8 @@ build-clang:
  - ../configure --cc=clang --cxx=clang++ --enable-werror
       --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
                      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-tci:
  image: centos:8
@@ -112,7 +114,7 @@ build-tci:
  - cd build
  - ../configure --enable-tcg-interpreter
       --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
- - make -j2
+ - make -j"$JOBS"
  - make run-tcg-tests-x86_64-softmmu
  - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
  - for tg in $TARGETS ; do
Signature · 1 line
2.18.1
#14 lore raw

PATCH 7/7 linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

In reply to Thomas Huth

From: Alex Bennée <[email protected]>

Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:

linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes ee94743034bf
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)
linux-user/elfload.c +2 -0
index 01a9323a63..ebc663ea0b 100644
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
     /*
Signature · 1 line
2.18.1
Reviewed-by on v1 4/7
#7 lore raw

Re: [PATCH 4/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

In reply to Thomas Huth
Quoted 40 lines · Thomas Huth

We have a dedicated folder for the gitlab-ci - so there is no need to clutter the top directory with these .yml files.

---
 .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
 .gitlab-ci.yml                                     | 4 ++--
 MAINTAINERS                                        | 2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
 rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
.gitlab-ci.d/edk2.yml renamed from .gitlab-ci-edk2.yml +0 -0
similarity index 100%
rename from .gitlab-ci-edk2.yml
rename to .gitlab-ci.d/edk2.yml
.gitlab-ci.d/opensbi.yml renamed from .gitlab-ci-opensbi.yml +0 -0
similarity index 100%
rename from .gitlab-ci-opensbi.yml
rename to .gitlab-ci.d/opensbi.yml
.gitlab-ci.yml +2 -2
index bc6aee6aba..5208d93ff8 100644
@@ -1,6 +1,6 @@
 include:
-  - local: '/.gitlab-ci-edk2.yml'
-  - local: '/.gitlab-ci-opensbi.yml'
+  - local: '/.gitlab-ci.d/edk2.yml'
+  - local: '/.gitlab-ci.d/opensbi.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
MAINTAINERS +0 -0
index bde5fd480f..d43c98115c 100644

MAINTAINERS F: roms/edk2 · line 2540

Quoted 8 lines
MAINTAINERS +1 -1
@@ -2534,7 +2534,7 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
-F: .gitlab-ci-edk2.yml
+F: .gitlab-ci.d/edk2.yml
 F: .gitlab-ci.d/edk2/
 
 Usermode Emulation
#15 lore raw

Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

In reply to Thomas Huth

Le 25/05/2020 à 15:18, Thomas Huth a écrit :

Quoted 33 lines

From: Alex Bennée <[email protected]>

Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:

linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes ee94743034bf
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)
linux-user/elfload.c +1 -0
index 01a9323a63..ebc663ea0b 100644
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS

linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086

Quoted 10 lines
linux-user/elfload.c +1 -0
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
     /*

Philippe sent the same patch:

https://www.mail-archive.com/[email protected]/msg699796.html

Thanks,
Laurent

#16 lore raw

Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

Quoted 42 lines · Laurent Vivier

Le 25/05/2020 à 15:18, Thomas Huth a écrit :

Quoted 41 lines

From: Alex Bennée <[email protected]>

Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:

linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes ee94743034bf
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)
linux-user/elfload.c +2 -0
index 01a9323a63..ebc663ea0b 100644
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
Quoted 6 lines

linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086

Quoted 2 lines
linux-user/elfload.c +0 -0
     /*

Philippe sent the same patch:

https://www.mail-archive.com/[email protected]/msg699796.html

Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?

Thomas

#17 lore raw

Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

Quoted 48 lines · Thomas Huth
Quoted 47 lines · Laurent Vivier
Quoted 33 lines · Thomas Huth

From: Alex Bennée <[email protected]>

Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:

linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes ee94743034bf
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)
linux-user/elfload.c +1 -0
index 01a9323a63..ebc663ea0b 100644
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS

linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086

Quoted 10 lines
linux-user/elfload.c +1 -0
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
     /*

Philippe sent the same patch:

Quoted 4 lines
Quoted 1 line

https://www.mail-archive.com/[email protected]/msg699796.html

Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?

Richard was more inclined to suppress the warning:

Subject: Re: [PATCH v2] linux-user: limit check to HOST_LONG_BITS == 32
From: Richard Henderson <[email protected]>
Message-ID: <[email protected]>
Date: Thu, 21 May 2020 20:15:51 -0700

One reason I dropped the f32 patch from my last PR was because this wasn't the only warning the latest clang picks up.

Signature · 1 line
Alex Bennée
Reviewed-by on v1 2/7
#3 lore raw

Re: [PATCH 2/7] gitlab-ci: Remove flex/bison packages

In reply to Thomas Huth
Quoted 7 lines · Thomas Huth

From: Philippe Mathieu-Daudé <[email protected]>

QEMU does not use flex/bison packages.

Message-Id: <[email protected]>

Quoted 17 lines
---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.yml +1 -1
index b889fb96b6..994774250f 100644
@@ -4,7 +4,7 @@ include:
 
 before_script:
  - apt-get update -qq
- - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
+ - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
 
 build-system1:
  script:
Signature · 1 line
Alex Bennée
Reviewed-by on v1 4/7
#8 lore raw

Re: [PATCH 4/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

In reply to Thomas Huth
Quoted 4 lines · Thomas Huth

We have a dedicated folder for the gitlab-ci - so there is no need to clutter the top directory with these .yml files.

Quoted 43 lines
---
 .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
 .gitlab-ci.yml                                     | 4 ++--
 MAINTAINERS                                        | 2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
 rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
.gitlab-ci.d/edk2.yml renamed from .gitlab-ci-edk2.yml +0 -0
similarity index 100%
rename from .gitlab-ci-edk2.yml
rename to .gitlab-ci.d/edk2.yml
.gitlab-ci.d/opensbi.yml renamed from .gitlab-ci-opensbi.yml +0 -0
similarity index 100%
rename from .gitlab-ci-opensbi.yml
rename to .gitlab-ci.d/opensbi.yml
.gitlab-ci.yml +2 -2
index bc6aee6aba..5208d93ff8 100644
@@ -1,6 +1,6 @@
 include:
-  - local: '/.gitlab-ci-edk2.yml'
-  - local: '/.gitlab-ci-opensbi.yml'
+  - local: '/.gitlab-ci.d/edk2.yml'
+  - local: '/.gitlab-ci.d/opensbi.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
MAINTAINERS +1 -1
index bde5fd480f..d43c98115c 100644
@@ -2534,7 +2534,7 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
-F: .gitlab-ci-edk2.yml
+F: .gitlab-ci.d/edk2.yml
 F: .gitlab-ci.d/edk2/
 
 Usermode Emulation
Signature · 1 line
Alex Bennée
Reviewed-by on v1 3/7
#5 lore raw

Re: [PATCH 3/7] GitLab CI: avoid calling before_scripts on unintended jobs

In reply to Thomas Huth
Quoted 5 lines · Thomas Huth

From: Cleber Rosa <[email protected]>

At this point it seems that all jobs depend on those steps, with maybe the EDK2 jobs as exceptions.

Quoted 7 lines

The jobs that will be added later will not want those scripts to be run, so let's move these steps to the appropriate jobs, while still trying to avoid repetition.

[thuth: Rebased to current master branch, use separate template]
Signature · 1 line
Alex Bennée
#18 lore raw

Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

In reply to Alex Bennée
Quoted 58 lines · Alex Bennée
Quoted 48 lines · Thomas Huth
Quoted 47 lines · Laurent Vivier
Quoted 33 lines · Thomas Huth

From: Alex Bennée <[email protected]>

Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:

linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes ee94743034bf
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)
linux-user/elfload.c +1 -0
index 01a9323a63..ebc663ea0b 100644
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS

linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086

Quoted 10 lines
linux-user/elfload.c +1 -0
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
     /*

Philippe sent the same patch:

Quoted 4 lines
Quoted 1 line

https://www.mail-archive.com/[email protected]/msg699796.html

Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?

Richard was more inclined to suppress the warning:

Quoted 7 lines

Subject: Re: [PATCH v2] linux-user: limit check to HOST_LONG_BITS == 32
From: Richard Henderson <[email protected]>
Message-ID: <[email protected]>
Date: Thu, 21 May 2020 20:15:51 -0700

One reason I dropped the f32 patch from my last PR was because this wasn't the only warning the latest clang picks up.

... but this is currently the only spot that is required to get the gitlab CI going again, so I think we should include this patch until we have a final decision whether to disable the warning or not (and we can still revert this patch after we disabled the warning). Ok?

Thomas

#19 lore raw

Re: [PATCH 7/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

Quoted 64 lines · Thomas Huth
Quoted 63 lines · Alex Bennée
Quoted 48 lines · Thomas Huth
Quoted 47 lines · Laurent Vivier
Quoted 33 lines · Thomas Huth

From: Alex Bennée <[email protected]>

Newer clangs rightly spot that you can never exceed the full address space of 64 bit hosts with:

linux-user/elfload.c:2076:41: error: result of comparison 'unsigned long' > 18446744073709551615 is always false [-Werror,-Wtautological-type-limit-compare]
4685 if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
4686 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes ee94743034bf
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)
linux-user/elfload.c +1 -0
index 01a9323a63..ebc663ea0b 100644
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS

linux-user/elfload.c static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr, · line 2086

Quoted 10 lines
linux-user/elfload.c +1 -0
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
     /*

Philippe sent the same patch:

Quoted 4 lines
Quoted 1 line

https://www.mail-archive.com/[email protected]/msg699796.html

Indeed, but looking more closely, he's using slightly different locations for the #if and #endif ... not sure what's better though...?

Richard was more inclined to suppress the warning:

Subject: Re: [PATCH v2] linux-user: limit check to HOST_LONG_BITS == 32
From: Richard Henderson <[email protected]>
Message-ID: <[email protected]>
Date: Thu, 21 May 2020 20:15:51 -0700

Quoted 7 lines
Quoted 2 lines

One reason I dropped the f32 patch from my last PR was because this wasn't the only warning the latest clang picks up.

... but this is currently the only spot that is required to get the gitlab CI going again, so I think we should include this patch until we have a final decision whether to disable the warning or not (and we can still revert this patch after we disabled the warning). Ok?

I'm certainly happy with that if it gets gitlab working.

My experience with make docker-test-vlang@fedora (with 32) was there where more things to fix. I guess gitlab didn't trigger them.

Signature · 1 line
Alex Bennée
Acked-by on v1 5/7
#10 lore raw

Re: [PATCH 5/7] gitlab-ci: Do not use the standard container images from gitlab

In reply to Thomas Huth
Quoted 10 lines · Thomas Huth

Currently all pipelines of the gitlab CI are failing, except for the "build-user" pipeline. There is an issue with the default container image (likely Debian stable) where they imported something bad in one of the system headers:

/usr/include/linux/swab.h: In function '__swab':
/builds/huth/qemu/include/qemu/bitops.h:20:34: error: "sizeof" is not
defined, evaluates to 0 [-Werror=undef]
#define BITS_PER_LONG (sizeof (unsigned long) * BITS_PER_BYTE)

Quoted 8 lines

We could maybe work-around this issue or wait for the default containers to get fixed, but considering that we use Ubuntu (and thus Debian-style) CI in Travis already to a very large extent, we should consider to use some RPM-based distros in our gitlab CI instead. Thus let's change the failing pipelines to use Fedora and CentOS (and also one Ubuntu 19.10, since 20.04 is broken, too) now.

I will say Fedora/CentOS won't be immune to this failure if they update the kernel headers to somewhere between the breakage and the fix.

Quoted 92 lines
---
 .gitlab-ci.yml | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)
.gitlab-ci.yml +25 -12
index 5208d93ff8..559ec2ab4d 100644
@@ -5,9 +5,17 @@ include:
 .update_apt_template: &before_script_apt
  before_script:
   - apt-get update -qq
-  - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
+  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
+        genisoimage
+
+.update_dnf_template: &before_script_dnf
+ before_script:
+  - dnf update -y
+  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
+        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
 
 build-system1:
+ image: ubuntu:19.10
  <<: *before_script_apt
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
@@ -21,11 +29,12 @@ build-system1:
  - make -j2 check
 
 build-system2:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
-      libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
-      libzstd-dev
+ - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
+       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
+       libzstd-devel
  - mkdir build
  - cd build
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
@@ -35,7 +44,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
  - mkdir build
  - cd build
@@ -50,9 +60,10 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
- <<: *before_script_apt
+ image: centos:8
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq clang libgtk-3-dev libusb-dev
+ - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
  - mkdir build
  - cd build
  - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
@@ -79,10 +90,11 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
-      xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
+ - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
+       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
  - mkdir build
  - cd build
  - ../configure --cc=clang --cxx=clang++ --enable-werror
@@ -92,7 +104,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
- <<: *before_script_apt
+ image: centos:8
+ <<: *before_script_dnf
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
Signature · 1 line
Alex Bennée
Reviewed-by on v1 6/7
#12 lore raw

Re: [PATCH 6/7] gitlab-ci: Determine the number of jobs dynamically

In reply to Thomas Huth
Quoted 6 lines · Thomas Huth

Some people might want to run the gitlab CI pipelines in an environment where multiple CPUs are available to the runners, so let's rather get the number for "-j" from the "nproc" program (increased by 1 to compensate for jobs that wait for I/O) instead of hard-coding it.

<snip>

Quoted 23 lines
.gitlab-ci.yml +4 -4
@@ -25,8 +27,8 @@ build-system1:
  - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
       cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
       mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-system2:
  image: fedora:latest
@@ -40,8 +42,8 @@ build-system2:
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
       microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
       sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-disabled:
  image: fedora:latest

.gitlab-ci.yml build-disabled: · line 62

Quoted 7 lines
.gitlab-ci.yml +2 -2
@@ -56,8 +58,8 @@ build-disabled:
       --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
       --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
       --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j2
- - make -j2 check-qtest SPEED=slow
+ - make -j"$JOBS"
+ - make -j"$JOBS" check-qtest SPEED=slow

I would make all the check jobs use a single core as it otherwise gets hard to figure out exactly where something broke/hung.

<snip>

.gitlab-ci.yml build-clang: · line 106

Quoted 8 lines
.gitlab-ci.yml +2 -2
@@ -100,8 +102,8 @@ build-clang:
  - ../configure --cc=clang --cxx=clang++ --enable-werror
       --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
                      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check

Ditto for this check

.gitlab-ci.yml build-tci: · line 120

Quoted 12 lines
.gitlab-ci.yml +1 -1
 build-tci:
  image: centos:8
@@ -112,7 +114,7 @@ build-tci:
  - cd build
  - ../configure --enable-tcg-interpreter
       --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
- - make -j2
+ - make -j"$JOBS"
  - make run-tcg-tests-x86_64-softmmu
  - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
  - for tg in $TARGETS ; do

Otherwise:

Signature · 1 line
Alex Bennée
#13 lore raw

Re: [PATCH 6/7] gitlab-ci: Determine the number of jobs dynamically

Quoted 30 lines · Alex Bennée
Quoted 6 lines · Thomas Huth

Some people might want to run the gitlab CI pipelines in an environment where multiple CPUs are available to the runners, so let's rather get the number for "-j" from the "nproc" program (increased by 1 to compensate for jobs that wait for I/O) instead of hard-coding it.

<snip>

Quoted 20 lines
.gitlab-ci.yml +4 -4
@@ -25,8 +27,8 @@ build-system1:
  - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
       cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
       mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-system2:
  image: fedora:latest
@@ -40,8 +42,8 @@ build-system2:
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
       microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
       sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
Quoted 13 lines

.gitlab-ci.yml build-disabled: · line 62

Quoted 10 lines
.gitlab-ci.yml +2 -2
 build-disabled:
  image: fedora:latest
@@ -56,8 +58,8 @@ build-disabled:
       --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
       --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
       --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j2
- - make -j2 check-qtest SPEED=slow
+ - make -j"$JOBS"
+ - make -j"$JOBS" check-qtest SPEED=slow

I would make all the check jobs use a single core as it otherwise gets hard to figure out exactly where something broke/hung.

It's a somewhat double-edged sword ... either faster CI test times, or more deterministic output ... so far I didn't suffer the problem with the deterministic output in the gitlab-CI yet (unlike with Travis), so I'd rather keep the -j here for now. We can still remove it later if we hit a bug that is hard to debug otherwise.

Thomas