Skip to contentSkip to conversation

Enable Travis builds on arm64, ppc64le and s390x

PATCH In review Thomas Huth 4 participants 19 messages Dec 4, 2019 – Dec 6, 2019

lore
Share & tools
lore
#0 lore raw

v2 cover Enable Travis builds on arm64, ppc64le and s390x

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

v2:
- Added "make check-tcg" and Alex' patch to disable cross-containers
- Explicitely set "dist: xenial" for arm64 and ppc64le since some iotests are crashing on bionic on these hosts.
- Dropped "libcap-dev" from the package list since it will be replaced by libcapng-dev soon.

Alex Bennée (1):
configure: allow disable of cross compilation containers

Thomas Huth (6):
iotests: Provide a function for checking the creation of huge files
iotests: Skip test 060 if it is not possible to create large files
iotests: Skip test 079 if it is not possible to create large files tests/hd-geo-test: Skip test when images can not be created tests/test-util-filemonitor: Skip test on non-x86 Travis containers travis.yml: Enable builds on arm64, ppc64le and s390x

 .travis.yml                   | 86 +++++++++++++++++++++++++++++++++++
 configure                     |  8 +++-
 tests/hd-geo-test.c           | 12 ++++-
 tests/qemu-iotests/005        |  5 +-
 tests/qemu-iotests/060        |  3 ++
 tests/qemu-iotests/079        |  3 ++
 tests/qemu-iotests/220        |  6 +--
 tests/qemu-iotests/common.rc  | 10 ++++
 tests/tcg/configure.sh        |  6 ++-
 tests/test-util-filemonitor.c | 11 +++++
 10 files changed, 138 insertions(+), 12 deletions(-)
Signature · 1 line
2.18.1
#1 lore raw

PATCH v2 1/7 iotests: Provide a function for checking the creation of huge files

Some tests create huge (but sparse) files, and to be able to run those tests in certain limited environments (like CI containers), we have to check for the possibility to create such files first. Thus let's introduce a common function to check for large files, and replace the already existing checks in the iotests 005 and 220 with this function.

---
 tests/qemu-iotests/005       |  5 +----
 tests/qemu-iotests/220       |  6 ++----
 tests/qemu-iotests/common.rc | 10 ++++++++++
 3 files changed, 13 insertions(+), 8 deletions(-)
tests/qemu-iotests/005 +1 -4
index 58442762fe..b6d03ac37d 100755
@@ -59,10 +59,7 @@ fi
 # Sanity check: For raw, we require a file system that permits the creation
 # of a HUGE (but very sparse) file. Check we can create it before continuing.
 if [ "$IMGFMT" = "raw" ]; then
-    if ! truncate --size=5T "$TEST_IMG"; then
-        _notrun "file system on $TEST_DIR does not support large enough files"
-    fi
-    rm "$TEST_IMG"
+    _require_large_file 5T
 fi
 
 echo
tests/qemu-iotests/220 +2 -4
index 2d62c5dcac..15159270d3 100755
@@ -42,10 +42,8 @@ echo "== Creating huge file =="
 
 # Sanity check: We require a file system that permits the creation
 # of a HUGE (but very sparse) file.  tmpfs works, ext4 does not.
-if ! truncate --size=513T "$TEST_IMG"; then
-    _notrun "file system on $TEST_DIR does not support large enough files"
-fi
-rm "$TEST_IMG"
+_require_large_file 513T
+
 IMGOPTS='cluster_size=2M,refcount_bits=1' _make_test_img 513T
 
 echo "== Populating refcounts =="
tests/qemu-iotests/common.rc +10 -0
index 0cc8acc9ed..6f0582c79a 100644
@@ -643,5 +643,15 @@ _require_drivers()
     done
 }
 
+# Check that we have a file system that allows huge (but very sparse) files
+#
+_require_large_file()
+{
+    if ! truncate --size="$1" "$TEST_IMG"; then
+        _notrun "file system on $TEST_DIR does not support large enough files"
+    fi
+    rm "$TEST_IMG"
+}
+
 # make sure this script returns success
 true
Signature · 1 line
2.18.1
Reviewed-by on v2 1/7
#2 lore raw

Re: [PATCH v2 1/7] iotests: Provide a function for checking the creation of huge files

Quoted 54 lines · Thomas Huth

Some tests create huge (but sparse) files, and to be able to run those tests in certain limited environments (like CI containers), we have to check for the possibility to create such files first. Thus let's introduce a common function to check for large files, and replace the already existing checks in the iotests 005 and 220 with this function.

---
  tests/qemu-iotests/005       |  5 +----
  tests/qemu-iotests/220       |  6 ++----
  tests/qemu-iotests/common.rc | 10 ++++++++++
  3 files changed, 13 insertions(+), 8 deletions(-)
tests/qemu-iotests/005 +1 -4
index 58442762fe..b6d03ac37d 100755
@@ -59,10 +59,7 @@ fi
  # Sanity check: For raw, we require a file system that permits the creation
  # of a HUGE (but very sparse) file. Check we can create it before continuing.
  if [ "$IMGFMT" = "raw" ]; then
-    if ! truncate --size=5T "$TEST_IMG"; then
-        _notrun "file system on $TEST_DIR does not support large enough files"
-    fi
-    rm "$TEST_IMG"
+    _require_large_file 5T
  fi
 
  echo
tests/qemu-iotests/220 +2 -4
index 2d62c5dcac..15159270d3 100755
@@ -42,10 +42,8 @@ echo "== Creating huge file =="
 
  # Sanity check: We require a file system that permits the creation
  # of a HUGE (but very sparse) file.  tmpfs works, ext4 does not.
-if ! truncate --size=513T "$TEST_IMG"; then
-    _notrun "file system on $TEST_DIR does not support large enough files"
-fi
-rm "$TEST_IMG"
+_require_large_file 513T
+
  IMGOPTS='cluster_size=2M,refcount_bits=1' _make_test_img 513T
 
  echo "== Populating refcounts =="
tests/qemu-iotests/common.rc +0 -0
index 0cc8acc9ed..6f0582c79a 100644
@@ -643,5 +643,15 @@ _require_drivers()
      done
  }

tests/qemu-iotests/common.rc _require_drivers() · line 654

Quoted 10 lines
tests/qemu-iotests/common.rc +9 -0
 
+# Check that we have a file system that allows huge (but very sparse) files
+#
+_require_large_file()
+{
+    if ! truncate --size="$1" "$TEST_IMG"; then
+        _notrun "file system on $TEST_DIR does not support large enough files"
+    fi
+    rm "$TEST_IMG"
+}

:)

Quoted 4 lines
tests/qemu-iotests/common.rc +1 -0
+
  # make sure this script returns success
  true
Reviewed-by Tested-by on v2 1/7
#3 lore raw

Re: [PATCH v2 1/7] iotests: Provide a function for checking the creation of huge files

In reply to Thomas Huth
Quoted 70 lines · Thomas Huth

Some tests create huge (but sparse) files, and to be able to run those tests in certain limited environments (like CI containers), we have to check for the possibility to create such files first. Thus let's introduce a common function to check for large files, and replace the already existing checks in the iotests 005 and 220 with this function.

---
 tests/qemu-iotests/005       |  5 +----
 tests/qemu-iotests/220       |  6 ++----
 tests/qemu-iotests/common.rc | 10 ++++++++++
 3 files changed, 13 insertions(+), 8 deletions(-)
tests/qemu-iotests/005 +1 -4
index 58442762fe..b6d03ac37d 100755
@@ -59,10 +59,7 @@ fi
 # Sanity check: For raw, we require a file system that permits the creation
 # of a HUGE (but very sparse) file. Check we can create it before continuing.
 if [ "$IMGFMT" = "raw" ]; then
-    if ! truncate --size=5T "$TEST_IMG"; then
-        _notrun "file system on $TEST_DIR does not support large enough files"
-    fi
-    rm "$TEST_IMG"
+    _require_large_file 5T
 fi
 
 echo
tests/qemu-iotests/220 +2 -4
index 2d62c5dcac..15159270d3 100755
@@ -42,10 +42,8 @@ echo "== Creating huge file =="
 
 # Sanity check: We require a file system that permits the creation
 # of a HUGE (but very sparse) file.  tmpfs works, ext4 does not.
-if ! truncate --size=513T "$TEST_IMG"; then
-    _notrun "file system on $TEST_DIR does not support large enough files"
-fi
-rm "$TEST_IMG"
+_require_large_file 513T
+
 IMGOPTS='cluster_size=2M,refcount_bits=1' _make_test_img 513T
 
 echo "== Populating refcounts =="
tests/qemu-iotests/common.rc +10 -0
index 0cc8acc9ed..6f0582c79a 100644
@@ -643,5 +643,15 @@ _require_drivers()
     done
 }
 
+# Check that we have a file system that allows huge (but very sparse) files
+#
+_require_large_file()
+{
+    if ! truncate --size="$1" "$TEST_IMG"; then
+        _notrun "file system on $TEST_DIR does not support large enough files"
+    fi
+    rm "$TEST_IMG"
+}
+
 # make sure this script returns success
 true
Signature · 1 line
2.18.1

This is a good refactor even without considering the CI environment issues it will help to address.

#4 lore raw

PATCH v2 2/7 iotests: Skip test 060 if it is not possible to create large files

In reply to Thomas Huth

Test 060 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. The repair process in test 060 creates a file of 64 GiB, so test first whether such large files are possible and skip the test if that's not the case.

---
 tests/qemu-iotests/060 | 3 +++
 1 file changed, 3 insertions(+)
tests/qemu-iotests/060 +3 -0
index b91d8321bb..d96f17a484 100755
@@ -49,6 +49,9 @@ _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
 
+# The repair process will create a large file - so check for availability first
+_require_large_file 64G
+
 rt_offset=65536  # 0x10000 (XXX: just an assumption)
 rb_offset=131072 # 0x20000 (XXX: just an assumption)
 l1_offset=196608 # 0x30000 (XXX: just an assumption)
Signature · 1 line
2.18.1
Reviewed-by on v2 2/7
#5 lore raw

Re: [PATCH v2 2/7] iotests: Skip test 060 if it is not possible to create large files

Quoted 17 lines · Thomas Huth

Test 060 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. The repair process in test 060 creates a file of 64 GiB, so test first whether such large files are possible and skip the test if that's not the case.

---
  tests/qemu-iotests/060 | 3 +++
  1 file changed, 3 insertions(+)
tests/qemu-iotests/060 +0 -0
index b91d8321bb..d96f17a484 100755

tests/qemu-iotests/060 _supported_fmt qcow2 · line 57

Quoted 9 lines
tests/qemu-iotests/060 +3 -0
@@ -49,6 +49,9 @@ _supported_fmt qcow2
  _supported_proto file
  _supported_os Linux
 
+# The repair process will create a large file - so check for availability first
+_require_large_file 64G
+
  rt_offset=65536  # 0x10000 (XXX: just an assumption)
  rb_offset=131072 # 0x20000 (XXX: just an assumption)
  l1_offset=196608 # 0x30000 (XXX: just an assumption)
Reviewed-by Tested-by on v2 2/7
#6 lore raw

Re: [PATCH v2 2/7] iotests: Skip test 060 if it is not possible to create large files

In reply to Thomas Huth
Quoted 28 lines · Thomas Huth

Test 060 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. The repair process in test 060 creates a file of 64 GiB, so test first whether such large files are possible and skip the test if that's not the case.

---
 tests/qemu-iotests/060 | 3 +++
 1 file changed, 3 insertions(+)
tests/qemu-iotests/060 +3 -0
index b91d8321bb..d96f17a484 100755
@@ -49,6 +49,9 @@ _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
 
+# The repair process will create a large file - so check for availability first
+_require_large_file 64G
+
 rt_offset=65536  # 0x10000 (XXX: just an assumption)
 rb_offset=131072 # 0x20000 (XXX: just an assumption)
 l1_offset=196608 # 0x30000 (XXX: just an assumption)
Signature · 1 line
2.18.1

The behavior and failure is indeed pretty consistent accross those architectures:

- arm64: https://travis-ci.org/clebergnu/qemu/jobs/621238740#L4217
- ppc64le: https://travis-ci.org/clebergnu/qemu/jobs/621238741#L4252
- s390x: https://travis-ci.org/clebergnu/qemu/jobs/621238742#L4265

And with this, 060 gets skipped properly:

- arm64: https://travis-ci.org/clebergnu/qemu/jobs/621248591#L4202
- ppc64le: https://travis-ci.org/clebergnu/qemu/jobs/621248592#L4236
- s390x: https://travis-ci.org/clebergnu/qemu/jobs/621248593#L4250

#7 lore raw

PATCH v2 3/7 iotests: Skip test 079 if it is not possible to create large files

In reply to Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, which is apparently already too big for these containers, so check first whether we can really create such files before executing the test.

---
 tests/qemu-iotests/079 | 3 +++
 1 file changed, 3 insertions(+)
tests/qemu-iotests/079 +3 -0
index 81f0c21f53..78536d3bbf 100755
@@ -39,6 +39,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file nfs
 
+# Some containers (e.g. non-x86 on Travis) do not allow large files
+_require_large_file 4G
+
 echo "=== Check option preallocation and cluster_size ==="
 echo
 cluster_sizes="16384 32768 65536 131072 262144 524288 1048576 2097152 4194304"
Signature · 1 line
2.18.1
Reviewed-by on v2 3/7
#8 lore raw

Re: [PATCH v2 3/7] iotests: Skip test 079 if it is not possible to create large files

Quoted 18 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, which is apparently already too big for these containers, so check first whether we can really create such files before executing the test.

---
  tests/qemu-iotests/079 | 3 +++
  1 file changed, 3 insertions(+)
tests/qemu-iotests/079 +0 -0
index 81f0c21f53..78536d3bbf 100755

tests/qemu-iotests/079 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 47

Quoted 9 lines
tests/qemu-iotests/079 +3 -0
@@ -39,6 +39,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file nfs
 
+# Some containers (e.g. non-x86 on Travis) do not allow large files
+_require_large_file 4G
+
  echo "=== Check option preallocation and cluster_size ==="
  echo
  cluster_sizes="16384 32768 65536 131072 262144 524288 1048576 2097152 4194304"
#9 lore raw

Re: [PATCH v2 3/7] iotests: Skip test 079 if it is not possible to create large files

In reply to Thomas Huth
Quoted 1 line · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers on Travis

For the record: It's working on s390x as noticed by Cleber. It's only failing on arm64 and ppc64le. After fixing the problem with 060 which fails on all three architectures, I did not pay enough attention when I wrote this commit message here and thought that this would be the very same problem, but apparently it is working on s390x. Thus "s390x" should be removed from the commit message.

Thomas

#10 lore raw

PATCH v2 4/7 tests/hd-geo-test: Skip test when images can not be created

In reply to Thomas Huth

In certain environments like restricted containers, we can not create huge test images. To be able to use "make check" in such container environments, too, let's skip the hd-geo-test instead of failing when the test images could not be created.

---
 tests/hd-geo-test.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/hd-geo-test.c +11 -1
index 7e86c5416c..a249800544 100644
@@ -34,8 +34,13 @@ static char *create_test_img(int secs)
     fd = mkstemp(template);
     g_assert(fd >= 0);
     ret = ftruncate(fd, (off_t)secs * 512);
-    g_assert(ret == 0);
     close(fd);
+
+    if (ret) {
+        free(template);
+        template = NULL;
+    }
+
     return template;
 }
 
@@ -934,6 +939,10 @@ int main(int argc, char **argv)
     for (i = 0; i < backend_last; i++) {
         if (img_secs[i] >= 0) {
             img_file_name[i] = create_test_img(img_secs[i]);
+            if (!img_file_name[i]) {
+                g_test_message("Could not create test images.");
+                goto test_add_done;
+            }
         } else {
             img_file_name[i] = NULL;
         }
@@ -965,6 +974,7 @@ int main(int argc, char **argv)
                        "skipping hd-geo/override/* tests");
     }
 
+test_add_done:
     ret = g_test_run();
 
     for (i = 0; i < backend_last; i++) {
Signature · 1 line
2.18.1
Reviewed-by Tested-by on v2 4/7
#11 lore raw

Re: [PATCH v2 4/7] tests/hd-geo-test: Skip test when images can not be created

Quoted 53 lines · Thomas Huth

In certain environments like restricted containers, we can not create huge test images. To be able to use "make check" in such container environments, too, let's skip the hd-geo-test instead of failing when the test images could not be created.

---
 tests/hd-geo-test.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/hd-geo-test.c +11 -1
index 7e86c5416c..a249800544 100644
@@ -34,8 +34,13 @@ static char *create_test_img(int secs)
     fd = mkstemp(template);
     g_assert(fd >= 0);
     ret = ftruncate(fd, (off_t)secs * 512);
-    g_assert(ret == 0);
     close(fd);
+
+    if (ret) {
+        free(template);
+        template = NULL;
+    }
+
     return template;
 }
 
@@ -934,6 +939,10 @@ int main(int argc, char **argv)
     for (i = 0; i < backend_last; i++) {
         if (img_secs[i] >= 0) {
             img_file_name[i] = create_test_img(img_secs[i]);
+            if (!img_file_name[i]) {
+                g_test_message("Could not create test images.");
+                goto test_add_done;
+            }
         } else {
             img_file_name[i] = NULL;
         }
@@ -965,6 +974,7 @@ int main(int argc, char **argv)
                        "skipping hd-geo/override/* tests");
     }
 
+test_add_done:
     ret = g_test_run();
 
     for (i = 0; i < backend_last; i++) {
Signature · 1 line
2.18.1
#12 lore raw

PATCH v2 5/7 tests/test-util-filemonitor: Skip test on non-x86 Travis containers

In reply to Thomas Huth

test-util-filemonitor fails in restricted non-x86 Travis containers since they apparently blacklisted some required system calls there. Let's simply skip the test if we detect such an environment.

---
 tests/test-util-filemonitor.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tests/test-util-filemonitor.c +11 -0
index 301cd2db61..45009c69f4 100644
@@ -406,10 +406,21 @@ test_file_monitor_events(void)
     char *pathdst = NULL;
     QFileMonitorTestData data;
     GHashTable *ids = g_hash_table_new(g_int64_hash, g_int64_equal);
+    char *travis_arch;
 
     qemu_mutex_init(&data.lock);
     data.records = NULL;
 
+    /*
+     * This test does not work on Travis LXD containers since some
+     * syscalls are blocked in that environment.
+     */
+    travis_arch = getenv("TRAVIS_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_test_skip("Test does not work on non-x86 Travis containers.");
+        return;
+    }
+
     /*
      * The file monitor needs the main loop running in
      * order to receive events from inotify. We must
Signature · 1 line
2.18.1
Reviewed-by Tested-by on v2 5/7
#13 lore raw

Re: [PATCH v2 5/7] tests/test-util-filemonitor: Skip test on non-x86 Travis containers

Quoted 40 lines · Thomas Huth

test-util-filemonitor fails in restricted non-x86 Travis containers since they apparently blacklisted some required system calls there. Let's simply skip the test if we detect such an environment.

---
 tests/test-util-filemonitor.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tests/test-util-filemonitor.c +11 -0
index 301cd2db61..45009c69f4 100644
@@ -406,10 +406,21 @@ test_file_monitor_events(void)
     char *pathdst = NULL;
     QFileMonitorTestData data;
     GHashTable *ids = g_hash_table_new(g_int64_hash, g_int64_equal);
+    char *travis_arch;
 
     qemu_mutex_init(&data.lock);
     data.records = NULL;
 
+    /*
+     * This test does not work on Travis LXD containers since some
+     * syscalls are blocked in that environment.
+     */
+    travis_arch = getenv("TRAVIS_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_test_skip("Test does not work on non-x86 Travis containers.");
+        return;
+    }
+
     /*
      * The file monitor needs the main loop running in
      * order to receive events from inotify. We must
Signature · 1 line
2.18.1
#14 lore raw

PATCH v2 6/7 configure: allow disable of cross compilation containers

In reply to Thomas Huth

From: Alex Bennée <[email protected]>

Our docker infrastructure isn't quite as multiarch as we would wish so let's allow the user to disable it if they want. This will allow us to use still run check-tcg on non-x86 CI setups.

---
 configure              | 8 +++++++-
 tests/tcg/configure.sh | 6 ++++--
 2 files changed, 11 insertions(+), 3 deletions(-)
configure +7 -1
index 6099be1d84..fe6d0971f1 100755
@@ -302,6 +302,7 @@ audio_win_int=""
 libs_qga=""
 debug_info="yes"
 stack_protector=""
+use_containers="yes"
 
 if test -e "$source_path/.git"
 then
@@ -1539,6 +1540,10 @@ for opt do
   ;;
   --disable-plugins) plugins="no"
   ;;
+  --enable-containers) use_containers="yes"
+  ;;
+  --disable-containers) use_containers="no"
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1722,6 +1727,7 @@ Advanced options (experts only):
                            track the maximum stack usage of stacks created by qemu_alloc_stack
   --enable-plugins
                            enable plugins via shared library loading
+  --disable-containers     don't use containers for cross-building
 
 Optional features, enabled with --enable-FEATURE and
 disabled with --disable-FEATURE, default is enabled if available:
@@ -8039,7 +8045,7 @@ done
 (for i in $cross_cc_vars; do
   export $i
 done
-export target_list source_path
+export target_list source_path use_containers
 $source_path/tests/tcg/configure.sh)
 
 # temporary config to build submodules
tests/tcg/configure.sh +4 -2
index 6c4a471aea..210e68396f 100755
@@ -36,8 +36,10 @@ TMPC="${TMPDIR1}/qemu-conf.c"
 TMPE="${TMPDIR1}/qemu-conf.exe"
 
 container="no"
-if has "docker" || has "podman"; then
-  container=$($python $source_path/tests/docker/docker.py probe)
+if test $use_containers = "yes"; then
+    if has "docker" || has "podman"; then
+        container=$($python $source_path/tests/docker/docker.py probe)
+    fi
 fi
 
 # cross compilers defaults, can be overridden with --cross-cc-ARCH
Signature · 1 line
2.18.1
Reviewed-by Tested-by on v2 6/7
#15 lore raw

Re: [PATCH v2 6/7] configure: allow disable of cross compilation containers

Quoted 74 lines · Thomas Huth

From: Alex Bennée <[email protected]>

Our docker infrastructure isn't quite as multiarch as we would wish so let's allow the user to disable it if they want. This will allow us to use still run check-tcg on non-x86 CI setups.

---
 configure              | 8 +++++++-
 tests/tcg/configure.sh | 6 ++++--
 2 files changed, 11 insertions(+), 3 deletions(-)
configure +7 -1
index 6099be1d84..fe6d0971f1 100755
@@ -302,6 +302,7 @@ audio_win_int=""
 libs_qga=""
 debug_info="yes"
 stack_protector=""
+use_containers="yes"
 
 if test -e "$source_path/.git"
 then
@@ -1539,6 +1540,10 @@ for opt do
   ;;
   --disable-plugins) plugins="no"
   ;;
+  --enable-containers) use_containers="yes"
+  ;;
+  --disable-containers) use_containers="no"
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1722,6 +1727,7 @@ Advanced options (experts only):
                            track the maximum stack usage of stacks created by qemu_alloc_stack
   --enable-plugins
                            enable plugins via shared library loading
+  --disable-containers     don't use containers for cross-building
 
 Optional features, enabled with --enable-FEATURE and
 disabled with --disable-FEATURE, default is enabled if available:
@@ -8039,7 +8045,7 @@ done
 (for i in $cross_cc_vars; do
   export $i
 done
-export target_list source_path
+export target_list source_path use_containers
 $source_path/tests/tcg/configure.sh)
 
 # temporary config to build submodules
tests/tcg/configure.sh +4 -2
index 6c4a471aea..210e68396f 100755
@@ -36,8 +36,10 @@ TMPC="${TMPDIR1}/qemu-conf.c"
 TMPE="${TMPDIR1}/qemu-conf.exe"
 
 container="no"
-if has "docker" || has "podman"; then
-  container=$($python $source_path/tests/docker/docker.py probe)
+if test $use_containers = "yes"; then
+    if has "docker" || has "podman"; then
+        container=$($python $source_path/tests/docker/docker.py probe)
+    fi
 fi
 
 # cross compilers defaults, can be overridden with --cross-cc-ARCH
Signature · 1 line
2.18.1
#16 lore raw

PATCH v2 7/7 travis.yml: Enable builds on arm64, ppc64le and s390x

In reply to Thomas Huth

Travis recently added the possibility to test on these architectures, too, so let's enable them in our travis.yml file to extend our test coverage.

Unfortunately, the libssh in this Ubuntu version (bionic) is in a pretty unusable Frankenstein state and libspice-server-dev is not available here, so we can not use the global list of packages to install, but have to provide individual package lists instead.

Also, some of the iotests crash when using "dist: bionic" on arm64 and ppc64le, thus these two builders have to use "dist: xenial" until the problem is understood / fixed.

---
 .travis.yml | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
.travis.yml +86 -0
index 445b0646c1..0e6458b0af 100644
@@ -354,6 +354,92 @@ matrix:
         - TEST_CMD="make -j3 check-tcg V=1"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
 
+    - arch: arm64
+      dist: xenial
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - TEST_CMD="make check check-tcg V=1"
+        - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
+
+    - arch: ppc64le
+      dist: xenial
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - TEST_CMD="make check check-tcg V=1"
+        - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},ppc64le-linux-user"
+
+    - arch: s390x
+      dist: bionic
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - TEST_CMD="make check check-tcg V=1"
+        - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},s390x-linux-user"
 
     # Release builds
     # The make-release script expect a QEMU version, so our tag must start with a 'v'.
Signature · 1 line
2.18.1
Acked-by on v2 7/7
#17 lore raw

Re: [PATCH v2 7/7] travis.yml: Enable builds on arm64, ppc64le and s390x

Quoted 9 lines · Thomas Huth

Travis recently added the possibility to test on these architectures, too, so let's enable them in our travis.yml file to extend our test coverage.

Unfortunately, the libssh in this Ubuntu version (bionic) is in a pretty unusable Frankenstein state and libspice-server-dev is not available here, so we can not use the global list of packages to install, but have to provide individual package lists instead.

Quoted 5 lines

Also, some of the iotests crash when using "dist: bionic" on arm64 and ppc64le, thus these two builders have to use "dist: xenial" until the problem is understood / fixed.

Quoted 101 lines
---
 .travis.yml | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
.travis.yml +86 -0
index 445b0646c1..0e6458b0af 100644
@@ -354,6 +354,92 @@ matrix:
         - TEST_CMD="make -j3 check-tcg V=1"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
 
+    - arch: arm64
+      dist: xenial
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - TEST_CMD="make check check-tcg V=1"
+        - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
+
+    - arch: ppc64le
+      dist: xenial
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - TEST_CMD="make check check-tcg V=1"
+        - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},ppc64le-linux-user"
+
+    - arch: s390x
+      dist: bionic
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - TEST_CMD="make check check-tcg V=1"
+        - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},s390x-linux-user"
 
     # Release builds
     # The make-release script expect a QEMU version, so our tag must start with a 'v'.
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#18 lore raw
In reply to Thomas Huth
Quoted 8 lines · Thomas Huth

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

Hi Thomas,

FIY, Avocado[1] has been running checks on those arches for a little over two weeks and in my experience, there are still some reliability issues (besides the other limitations you're already aware).

During the last week I've stopped seeing "machines" that wouldn't boot, or severe networking limitations, but things are still not as smooth as I'd like.

Anyway, I think we should insist on it, and give it a bit more time, so I definitely agree with and appreciate this work.

[1] https://travis-ci.org/avocado-framework/avocado/builds

- Cleber.

Quoted 33 lines

v2:
- Added "make check-tcg" and Alex' patch to disable cross-containers
- Explicitely set "dist: xenial" for arm64 and ppc64le since some iotests are crashing on bionic on these hosts.
- Dropped "libcap-dev" from the package list since it will be replaced by libcapng-dev soon.

Alex Bennée (1):
configure: allow disable of cross compilation containers

Thomas Huth (6):
iotests: Provide a function for checking the creation of huge files
iotests: Skip test 060 if it is not possible to create large files
iotests: Skip test 079 if it is not possible to create large files tests/hd-geo-test: Skip test when images can not be created tests/test-util-filemonitor: Skip test on non-x86 Travis containers travis.yml: Enable builds on arm64, ppc64le and s390x

 .travis.yml                   | 86 +++++++++++++++++++++++++++++++++++
 configure                     |  8 +++-
 tests/hd-geo-test.c           | 12 ++++-
 tests/qemu-iotests/005        |  5 +-
 tests/qemu-iotests/060        |  3 ++
 tests/qemu-iotests/079        |  3 ++
 tests/qemu-iotests/220        |  6 +--
 tests/qemu-iotests/common.rc  | 10 ++++
 tests/tcg/configure.sh        |  6 ++-
 tests/test-util-filemonitor.c | 11 +++++
 10 files changed, 138 insertions(+), 12 deletions(-)
Signature · 1 line
2.18.1