Improvements for the Travis CI

We will soon need more control over the individual test matrix entries, so we should not limit the matrix to "env" sections, i.e. put the "matrix:" keyword on the top, not the "env:".

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 92 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 56 insertions(+), 36 deletions(-)

.travis.yml +49 -36

index 6c14953..611bbdc 100644
@@ -16,44 +16,64 @@ addons:
       - qemu-system
 git:
   submodules: false
-env:
-  matrix:
-    - CONFIG=""
-      BUILD_DIR="."
-      TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
+
+matrix:
+  include:
+    - env:
+      - CONFIG=""
+      - BUILD_DIR="."
+      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
              vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
-    - CONFIG=""
-      BUILD_DIR="x86-builddir"
-      TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
+
+    - env:
+      - CONFIG=""
+      - BUILD_DIR="x86-builddir"
+      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
              tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
-    - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
-      BUILD_DIR="."
-      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
-    - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
-      BUILD_DIR="arm-buildir"
-      TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
-    - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
-      BUILD_DIR="."
-      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
-    - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
-      BUILD_DIR="arm64-buildir"
-      TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
-    - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
-      BUILD_DIR="."
-      TESTS="spapr_hcall emulator rtas-set-time-of-day"
-      ACCEL="tcg,cap-htm=off"
-    - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
-      BUILD_DIR="ppc64le-buildir"
-      TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
-      ACCEL="tcg,cap-htm=off"
-    - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
-      BUILD_DIR="."
-      TESTS="diag10 diag308"
-      ACCEL="tcg,firmware=s390x/run"
-    - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
-      BUILD_DIR="s390x-builddir"
-      TESTS="sieve"
-      ACCEL="tcg,firmware=s390x/run"
+
+    - env:
+      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
+      - BUILD_DIR="."
+      - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
+
+    - env:
+      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
+      - BUILD_DIR="arm-buildir"
+      - TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
+
+    - env:
+      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
+      - BUILD_DIR="."
+      - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
+
+    - env:
+      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
+      - BUILD_DIR="arm64-buildir"
+      - TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
+
+    - env:
+      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
+      - BUILD_DIR="."
+      - TESTS="spapr_hcall emulator rtas-set-time-of-day"
+      - ACCEL="tcg,cap-htm=off"
+
+    - env:
+      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
+      - BUILD_DIR="ppc64le-buildir"
+      - TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
+      - ACCEL="tcg,cap-htm=off"
+
+    - env:
+      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
+      - BUILD_DIR="."
+      - TESTS="diag10 diag308"

We don't need all cross compiler and QEMU versions for each and every entry in the test matrix, only the ones for the current target architecture. So let's speed up the installation process a little bit by only installing the packages that we really need.

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 53 +++++++++++++++++++++++++++++++----------------------
 1 file changed, 31 insertions(+), 22 deletions(-)

.travis.yml +28 -21

index 611bbdc..3f5b5ee 100644
@@ -1,75 +1,84 @@
 sudo: false
 dist: bionic
 language: c
-compiler:
-  - gcc
 cache: ccache
-addons:
-  apt:
-    packages:
-      # Cross Toolchains
-      - gcc-arm-linux-gnueabihf
-      - gcc-aarch64-linux-gnu
-      - gcc-powerpc64le-linux-gnu
-      - gcc-s390x-linux-gnu
-      # Run dependencies
-      - qemu-system
 git:
   submodules: false
 
 matrix:
   include:
-    - env:
+
+    - addons:
+        apt_packages: gcc qemu-system-x86
+      env:
       - CONFIG=""
       - BUILD_DIR="."
       - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
              vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
 
-    - env:
+    - addons:
+        apt_packages: gcc qemu-system-x86
+      env:
       - CONFIG=""
       - BUILD_DIR="x86-builddir"
       - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
              tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
 
-    - env:
+    - addons:
+        apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
+      env:
       - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
       - BUILD_DIR="."
       - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
 
-    - env:
+    - addons:
+        apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
+      env:
       - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
       - BUILD_DIR="arm-buildir"
       - TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
 
-    - env:
+    - addons:
+        apt_packages: gcc-aarch64-linux-gnu qemu-system-aarch64
+      env:
       - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
       - BUILD_DIR="."
       - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
 
-    - env:
+    - addons:
+        apt_packages: gcc-aarch64-linux-gnu qemu-system-aarch64
+      env:
       - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
       - BUILD_DIR="arm64-buildir"
       - TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
 
-    - env:
+    - addons:
+        apt_packages: gcc-powerpc64le-linux-gnu qemu-system-ppc
+      env:
       - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
       - BUILD_DIR="."
       - TESTS="spapr_hcall emulator rtas-set-time-of-day"
       - ACCEL="tcg,cap-htm=off"
 
-    - env:
+    - addons:
+        apt_packages: gcc-powerpc64le-linux-gnu qemu-system-ppc
+      env:
       - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
       - BUILD_DIR="ppc64le-buildir"
       - TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
       - ACCEL="tcg,cap-htm=off"
 
-    - env:
+    - addons:
+        apt_packages: gcc-s390x-linux-gnu qemu-system-s390x
+      env:
       - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
       - BUILD_DIR="."

Travis nowadays supports KVM in their CI pipelines, so we can finally run the kvm-unit-tests with KVM instead of TCG here. Unfortunately, there are some quirks: First, the QEMU binary has to be running as root, otherwise you get an "permission denied" error here - even if you fix up the access permissions to /dev/kvm first.

Second, not all x86 tests are working in
this environment, so we still have to manually select the test set here (but the amount of tests is definitely higher now than what we were able to run with TCG before).

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

.travis.yml +10 -5

index 3f5b5ee..f91118c 100644
@@ -1,4 +1,4 @@
-sudo: false
+sudo: true
 dist: bionic
 language: c
 cache: ccache

@@ -13,16 +13,21 @@ matrix:
       env:
       - CONFIG=""
       - BUILD_DIR="."
-      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
-             vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
+      - TESTS="access asyncpf debug emulator ept hypercall hyperv_stimer
+               hyperv_synic idt_test intel_iommu ioapic ioapic-split
+               kvmclock_test msr pcid rdpru realmode rmap_chain s3 umip"
+      - ACCEL="kvm"
 
     - addons:
         apt_packages: gcc qemu-system-x86
       env:
       - CONFIG=""
       - BUILD_DIR="x86-builddir"
-      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
-             tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
+      - TESTS="smptest smptest3 tsc tsc_adjust xsave vmexit_cpuid vmexit_vmcall
+               sieve vmexit_inl_pmtimer vmexit_ipi_halt vmexit_mov_from_cr8
+               vmexit_mov_to_cr8 vmexit_ple_round_robin vmexit_tscdeadline
+               vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
+      - ACCEL="kvm"
 
     - addons:
         apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm

Quoted 5 lines · Thomas Huth

Travis nowadays supports KVM in their CI pipelines, so we can finally run the kvm-unit-tests with KVM instead of TCG here. Unfortunately, there are some quirks: First, the QEMU binary has to be running as root, otherwise you get an "permission denied" error here - even if you fix up the access permissions to /dev/kvm first.

Could it be another resource it's trying to access?

After installing gcc-multilib, we can also test the 32-bit builds on Travis.

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

.travis.yml +8 -0

index f91118c..9ceb04d 100644
@@ -29,6 +29,21 @@ matrix:
                vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
       - ACCEL="kvm"
 
+    - addons:
+        apt_packages: gcc gcc-multilib qemu-system-x86
+      env:
+      - CONFIG="--arch=i386"
+      - BUILD_DIR="."
+      - TESTS="eventinj port80 sieve tsc taskswitch umip vmexit_ple_round_robin"
+
+    - addons:

While working on the travis.yml file, I've run into cases where all tests are reported as "SKIP" by the run_test.sh script (e.g. when QEMU could not be started). This should not result in a successful test run, so mark it as failed if not at least one test passed.

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

.travis.yml +0 -0

index 9ceb04d..aacf7d2 100644

Quoted 5 lines · Thomas Huth

While working on the travis.yml file, I've run into cases where all tests are reported as "SKIP" by the run_test.sh script (e.g. when QEMU could not be started). This should not result in a successful test run, so mark it as failed if not at least one test passed.

But doesn't this mean you could have everything fail except one pass and still report success?

Quoted 6 lines

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

.travis.yml script: · line 117

Quoted 8 lines

.travis.yml +0 -0

index 9ceb04d..aacf7d2 100644
@@ -115,3 +115,4 @@ script:
   - make -j3
   - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
   - if grep -q FAIL results.txt ; then exit 1 ; fi

.travis.yml script: · line 118

Quoted 1 line

.travis.yml +1 -0

+  - if ! grep -q PASS results.txt ; then exit 1 ; fi

Quoted 9 lines · Alex Bennée

Quoted 5 lines · Thomas Huth

While working on the travis.yml file, I've run into cases where all tests are reported as "SKIP" by the run_test.sh script (e.g. when QEMU could not be started). This should not result in a successful test run, so mark it as failed if not at least one test passed.

Quoted 2 lines

But doesn't this mean you could have everything fail except one pass and still report success?

The FAILs are already handled one line earlier...

Quoted 7 lines

Quoted 7 lines

Signed-off-by Thomas Huth <[email protected]>

---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

Quoted 8 lines

.travis.yml script: · line 117

Quoted 8 lines

.travis.yml +0 -0

index 9ceb04d..aacf7d2 100644
@@ -115,3 +115,4 @@ script:
   - make -j3
   - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
   - if grep -q FAIL results.txt ; then exit 1 ; fi

... here -----------^

Quoted 1 line

.travis.yml script: · line 118

Quoted 1 line

.travis.yml +1 -0

+  - if ! grep -q PASS results.txt ; then exit 1 ; fi

Maybe it would also be nicer to provide proper exit values in the run_tests.sh script, but the logic there gives me a bad headache... grep'ing for FAIL and PASS in the yml script is way more easy.

Thomas

The first two patches make the test matrix a little bit more flexible, and the fourth patch enables the 32-bit builds on x86.

But the most important patch is likely the third one: It is possible to test with KVM on Travis now, so we can run the tests within a real KVM environment, without TCG! The only caveat is that qemu-system-x86_64 has to run as root ... fixing only the permissions of /dev/kvm did not help here, I still got a "Permission denied" in that case.

Thomas Huth (5):
travis.yml: Re-arrange the test matrix
travis.yml: Install only the required packages for each entry in the
matrix
travis.yml: Test with KVM instead of TCG (on x86) travis.yml: Test the i386 build, too
travis.yml: Expect that at least one test succeeds

 .travis.yml | 155 +++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 104 insertions(+), 51 deletions(-)

The first two patches make the test matrix a little bit more flexible, and the fourth patch enables the 32-bit builds on x86.

But the most important patch is likely the third one: It is possible to test with KVM on Travis now, so we can run the tests within a real KVM environment, without TCG! The only caveat is that qemu-system-x86_64 has to run as root ... fixing only the permissions of /dev/kvm did not help here, I still got a "Permission denied" in that case.

Thomas Huth (5):
travis.yml: Re-arrange the test matrix
travis.yml: Install only the required packages for each entry in the
matrix
travis.yml: Test with KVM instead of TCG (on x86) travis.yml: Test the i386 build, too
travis.yml: Expect that at least one test succeeds

  .travis.yml | 155 +++++++++++++++++++++++++++++++++++-----------------
  1 file changed, 104 insertions(+), 51 deletions(-)