Skip to contentSkip to conversation

Enable Travis builds on arm64, ppc64le and s390x

PATCH In review Thomas Huth 4 participants 24 messages Nov 19, 2019 – Dec 4, 2019

lore
Share & tools
lore
#0 lore raw

cover Enable Travis builds on arm64, ppc64le and s390x

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

I also included Alex' patch for dropping some targets from the MAIN_SOFTMMU_TARGETS config variable, since at least on arm64, the build process seems to be rather slow and sometimes hits the 50 minutes timeout otherwise.

For the curious, here's a test run (non-x86 at the end of the page):

https://travis-ci.com/huth/qemu/jobs/258095792

Thomas

Alex Bennée (1):
travis.yml: drop 32 bit systems from MAIN_SOFTMMU_TARGETS

Thomas Huth (5):
iotests: Skip test 060 if it is not possible to create large files
iotests: Skip test 079 if it is not possible to create large files tests/hd-geo-test: Skip test when images can not be created tests/test-util-filemonitor: Skip test on non-x86 Travis containers travis.yml: Enable builds on arm64, ppc64le and s390x

 .travis.yml                   | 85 ++++++++++++++++++++++++++++++++++-
 tests/hd-geo-test.c           | 12 ++++-
 tests/qemu-iotests/060        |  6 +++
 tests/qemu-iotests/079        |  6 +++
 tests/test-util-filemonitor.c | 11 +++++
 5 files changed, 118 insertions(+), 2 deletions(-)
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/6 iotests: Skip test 060 if it is not possible to create large files

Test 060 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. The repair process in test 060 creates a file of 64 GiB, so test first whether such large files are possible and skip the test if that's not the case.

---
 tests/qemu-iotests/060 | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qemu-iotests/060 +6 -0
index b91d8321bb..e7ee865c55 100755
@@ -49,6 +49,12 @@ _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
 
+# The repair process will create a large file - so check for availability first
+if ! truncate --size=64G "$TEST_IMG"; then
+    _notrun "file system on $TEST_DIR does not support large enough files"
+fi
+rm "$TEST_IMG"
+
 rt_offset=65536  # 0x10000 (XXX: just an assumption)
 rb_offset=131072 # 0x20000 (XXX: just an assumption)
 l1_offset=196608 # 0x30000 (XXX: just an assumption)
Signature · 1 line
2.23.0
Reviewed-by on v1 1/6
#2 lore raw

Re: [PATCH 1/6] iotests: Skip test 060 if it is not possible to create large files

Quoted 6 lines · Thomas Huth

Test 060 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. The repair process in test 060 creates a file of 64 GiB, so test first whether such large files are possible and skip the test if that's not the case.

Quoted 21 lines
---
 tests/qemu-iotests/060 | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qemu-iotests/060 +6 -0
index b91d8321bb..e7ee865c55 100755
@@ -49,6 +49,12 @@ _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
 
+# The repair process will create a large file - so check for availability first
+if ! truncate --size=64G "$TEST_IMG"; then
+    _notrun "file system on $TEST_DIR does not support large enough files"
+fi
+rm "$TEST_IMG"
+
 rt_offset=65536  # 0x10000 (XXX: just an assumption)
 rb_offset=131072 # 0x20000 (XXX: just an assumption)
 l1_offset=196608 # 0x30000 (XXX: just an assumption)
Signature · 1 line
Alex Bennée
#3 lore raw

PATCH 2/6 iotests: Skip test 079 if it is not possible to create large files

In reply to Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
 tests/qemu-iotests/079 | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qemu-iotests/079 +6 -0
index 81f0c21f53..e9b81419b7 100755
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file nfs
 
+# Some containers (e.g. non-x86 on Travis) do not allow large files
+if ! truncate --size=4G "$TEST_IMG"; then
+    _notrun "file system on $TEST_DIR does not support large enough files"
+fi
+rm "$TEST_IMG"
+
 echo "=== Check option preallocation and cluster_size ==="
 echo
 cluster_sizes="16384 32768 65536 131072 262144 524288 1048576 2097152 4194304"
Signature · 1 line
2.23.0
#4 lore raw

Re: [PATCH 2/6] iotests: Skip test 079 if it is not possible to create large files

Quoted 15 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
  tests/qemu-iotests/079 | 6 ++++++
  1 file changed, 6 insertions(+)
tests/qemu-iotests/079 +0 -0
index 81f0c21f53..e9b81419b7 100755

tests/qemu-iotests/079 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 43

Quoted 5 lines
tests/qemu-iotests/079 +2 -0
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file nfs
 
+# Some containers (e.g. non-x86 on Travis) do not allow large files
+if ! truncate --size=4G "$TEST_IMG"; then

Shouldn't we restrict that to Travis-CI by checking some environment var?

Quoted 8 lines
tests/qemu-iotests/079 +4 -0
+    _notrun "file system on $TEST_DIR does not support large enough files"
+fi
+rm "$TEST_IMG"
+
  echo "=== Check option preallocation and cluster_size ==="
  echo
  cluster_sizes="16384 32768 65536 131072 262144 524288 1048576 2097152 4194304"
#5 lore raw

Re: [PATCH 2/6] iotests: Skip test 079 if it is not possible to create large files

Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 10 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
  tests/qemu-iotests/079 | 6 ++++++

  1 file changed, 6 insertions(+)
Quoted 11 lines
Quoted 9 lines
tests/qemu-iotests/079 +0 -0
index 81f0c21f53..e9b81419b7 100755
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15

  _supported_fmt qcow2
  _supported_proto file nfs
  +# Some containers (e.g. non-x86 on Travis) do not allow large files +if ! truncate --size=4G "$TEST_IMG"; then

Shouldn't we restrict that to Travis-CI by checking some environment var?

I'd rather like to keep it independent from Travis environment variables, since somebody might want to run "make check" in other non-Travis containers or on weird filesystems, and then the test should ideally not fail, but simply skip, too.

Thomas

#6 lore raw

Re: [PATCH 2/6] iotests: Skip test 079 if it is not possible to create large files

Quoted 24 lines · Thomas Huth
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 10 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
   tests/qemu-iotests/079 | 6 ++++++
   1 file changed, 6 insertions(+)
Quoted 11 lines

tests/qemu-iotests/079 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 42

Quoted 9 lines
tests/qemu-iotests/079 +1 -0
index 81f0c21f53..e9b81419b7 100755
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
   _supported_fmt qcow2
   _supported_proto file nfs
   +# Some containers (e.g. non-x86 on Travis) do not allow large files
+if ! truncate --size=4G "$TEST_IMG"; then

Shouldn't we restrict that to Travis-CI by checking some environment var?

Quoted 4 lines

I'd rather like to keep it independent from Travis environment variables, since somebody might want to run "make check" in other non-Travis containers or on weird filesystems, and then the test should ideally not fail, but simply skip, too.

I see. But it would be bad if we stop catching normal bugs with this test because we restricted it to contained environments.

Maybe we can add a generic is_contained() routine that only checks for TRAVIS env var, and people using other containers can expand it.

#7 lore raw

Re: [PATCH 2/6] iotests: Skip test 079 if it is not possible to create large files

Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 24 lines · Thomas Huth
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 10 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
   tests/qemu-iotests/079 | 6 ++++++
   1 file changed, 6 insertions(+)
Quoted 11 lines

tests/qemu-iotests/079 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 42

Quoted 9 lines
tests/qemu-iotests/079 +1 -0
index 81f0c21f53..e9b81419b7 100755
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
   _supported_fmt qcow2
   _supported_proto file nfs
   +# Some containers (e.g. non-x86 on Travis) do not allow large files
+if ! truncate --size=4G "$TEST_IMG"; then

Shouldn't we restrict that to Travis-CI by checking some environment var?

Quoted 4 lines

I'd rather like to keep it independent from Travis environment variables, since somebody might want to run "make check" in other non-Travis containers or on weird filesystems, and then the test should ideally not fail, but simply skip, too.

Quoted 5 lines

I see. But it would be bad if we stop catching normal bugs with this test because we restricted it to contained environments.

Maybe we can add a generic is_contained() routine that only checks for TRAVIS env var, and people using other containers can expand it.

"is_contained" is still expressing the environment.

What we need is a way to express features, and be able to switch beteen autodetecting features & mandatory enablement.

eg

if has_feature "large_file"
then
...stuff using large files...
fi

The "has_feature" helper would by default call out to "has_feature_large_file" todo automatic probing so that things "just work" according to whatever env the tests are run inside.

There should, however, be a flag to "./check" which force enables the feature eg "./check --require-feature large_file" will force execution and not attempt to probe for it.

We could have "--require-feature :all" to force enable all optional bits.

Any formal CI systems should use --require-feature to explicitly force testing of features that are expected to always work.

so Travis x86 would use "--require-feature large_large", but the arch64 version would not pass this flag and so do probing which will auto-skip.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw

Re: [PATCH 2/6] iotests: Skip test 079 if it is not possible to create large files

Quoted 60 lines · Daniel P. Berrangé
Quoted 30 lines · Philippe Mathieu-Daudé
Quoted 24 lines · Thomas Huth
Quoted 11 lines · Philippe Mathieu-Daudé
Quoted 10 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
   tests/qemu-iotests/079 | 6 ++++++
   1 file changed, 6 insertions(+)
Quoted 11 lines

tests/qemu-iotests/079 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 42

Quoted 9 lines
tests/qemu-iotests/079 +1 -0
index 81f0c21f53..e9b81419b7 100755
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
   _supported_fmt qcow2
   _supported_proto file nfs
   +# Some containers (e.g. non-x86 on Travis) do not allow large files
+if ! truncate --size=4G "$TEST_IMG"; then

Shouldn't we restrict that to Travis-CI by checking some environment var?

Quoted 4 lines

I'd rather like to keep it independent from Travis environment variables, since somebody might want to run "make check" in other non-Travis containers or on weird filesystems, and then the test should ideally not fail, but simply skip, too.

Quoted 5 lines

I see. But it would be bad if we stop catching normal bugs with this test because we restricted it to contained environments.

Maybe we can add a generic is_contained() routine that only checks for TRAVIS env var, and people using other containers can expand it.

"is_contained" is still expressing the environment.

What we need is a way to express features, and be able to switch beteen autodetecting features & mandatory enablement.

eg

if has_feature "large_file"
then
...stuff using large files...
fi

The "has_feature" helper would by default call out to "has_feature_large_file" todo automatic probing so that things "just work" according to whatever env the tests are run inside.

There should, however, be a flag to "./check" which force enables the feature eg "./check --require-feature large_file" will force execution and not attempt to probe for it.

We could have "--require-feature :all" to force enable all optional bits.

Quoted 6 lines

Any formal CI systems should use --require-feature to explicitly force testing of features that are expected to always work.

so Travis x86 would use "--require-feature large_large", but the arch64 version would not pass this flag and so do probing which will auto-skip.

Maybe a nice idea, but I think this is out of scope for this patch series. We're using the "if ! truncate --size=4G" check in some other tests already, so I'm not adding some new mechanism here. If you want to see some more fine-grained control for the iotests, please send some patches to rework these other tests first.

Thomas

#9 lore raw

Re: [PATCH 2/6] iotests: Skip test 079 if it is not possible to create large files

In reply to Thomas Huth
Quoted 15 lines · Thomas Huth

Test 079 fails in the arm64, s390x and ppc64le LXD containers, which apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, so check first whether we can really create such files before executing the test.

---
 tests/qemu-iotests/079 | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qemu-iotests/079 +0 -0
index 81f0c21f53..e9b81419b7 100755

tests/qemu-iotests/079 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 47

Quoted 9 lines
tests/qemu-iotests/079 +6 -0
@@ -39,6 +39,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file nfs
 
+# Some containers (e.g. non-x86 on Travis) do not allow large files
+if ! truncate --size=4G "$TEST_IMG"; then
+    _notrun "file system on $TEST_DIR does not support large enough files"
+fi
+rm "$TEST_IMG"
+

Hmm we are repeating ourselves here. Maybe the test should be wrapped up as __supported_filesize and the test can just do:

__supported_filesize 4G

along with the other tests above.

Signature · 1 line
Alex Bennée
#10 lore raw

PATCH 3/6 tests/hd-geo-test: Skip test when images can not be created

In reply to Thomas Huth

In certain environments like restricted containers, we can not create huge test images. To be able to use "make check" in such container environments, too, let's skip the hd-geo-test instead of failing when the test images could not be created.

---
 tests/hd-geo-test.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/hd-geo-test.c +11 -1
index 7e86c5416c..a249800544 100644
@@ -34,8 +34,13 @@ static char *create_test_img(int secs)
     fd = mkstemp(template);
     g_assert(fd >= 0);
     ret = ftruncate(fd, (off_t)secs * 512);
-    g_assert(ret == 0);
     close(fd);
+
+    if (ret) {
+        free(template);
+        template = NULL;
+    }
+
     return template;
 }
 
@@ -934,6 +939,10 @@ int main(int argc, char **argv)
     for (i = 0; i < backend_last; i++) {
         if (img_secs[i] >= 0) {
             img_file_name[i] = create_test_img(img_secs[i]);
+            if (!img_file_name[i]) {
+                g_test_message("Could not create test images.");
+                goto test_add_done;
+            }
         } else {
             img_file_name[i] = NULL;
         }
@@ -965,6 +974,7 @@ int main(int argc, char **argv)
                        "skipping hd-geo/override/* tests");
     }
 
+test_add_done:
     ret = g_test_run();
 
     for (i = 0; i < backend_last; i++) {
Signature · 1 line
2.23.0
Reviewed-by on v1 3/6
#11 lore raw

Re: [PATCH 3/6] tests/hd-geo-test: Skip test when images can not be created

Quoted 42 lines · Thomas Huth

In certain environments like restricted containers, we can not create huge test images. To be able to use "make check" in such container environments, too, let's skip the hd-geo-test instead of failing when the test images could not be created.

---
  tests/hd-geo-test.c | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)
tests/hd-geo-test.c +10 -1
index 7e86c5416c..a249800544 100644
@@ -34,8 +34,13 @@ static char *create_test_img(int secs)
      fd = mkstemp(template);
      g_assert(fd >= 0);
      ret = ftruncate(fd, (off_t)secs * 512);
-    g_assert(ret == 0);
      close(fd);
+
+    if (ret) {
+        free(template);
+        template = NULL;
+    }
+
      return template;
  }
 
@@ -934,6 +939,10 @@ int main(int argc, char **argv)
      for (i = 0; i < backend_last; i++) {
          if (img_secs[i] >= 0) {
              img_file_name[i] = create_test_img(img_secs[i]);
+            if (!img_file_name[i]) {
+                g_test_message("Could not create test images.");
+                goto test_add_done;
+            }
          } else {
              img_file_name[i] = NULL;
          }

tests/hd-geo-test.c int main(int argc, char **argv) · line 980

Quoted 7 lines
tests/hd-geo-test.c +1 -0
@@ -965,6 +974,7 @@ int main(int argc, char **argv)
                         "skipping hd-geo/override/* tests");
      }
 
+test_add_done:
      ret = g_test_run();
 
      for (i = 0; i < backend_last; i++) {
Reviewed-by on v1 3/6
#12 lore raw

Re: [PATCH 3/6] tests/hd-geo-test: Skip test when images can not be created

In reply to Thomas Huth
Quoted 41 lines · Thomas Huth

In certain environments like restricted containers, we can not create huge test images. To be able to use "make check" in such container environments, too, let's skip the hd-geo-test instead of failing when the test images could not be created.

---
 tests/hd-geo-test.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/hd-geo-test.c +10 -1
index 7e86c5416c..a249800544 100644
@@ -34,8 +34,13 @@ static char *create_test_img(int secs)
     fd = mkstemp(template);
     g_assert(fd >= 0);
     ret = ftruncate(fd, (off_t)secs * 512);
-    g_assert(ret == 0);
     close(fd);
+
+    if (ret) {
+        free(template);
+        template = NULL;
+    }
+
     return template;
 }
 
@@ -934,6 +939,10 @@ int main(int argc, char **argv)
     for (i = 0; i < backend_last; i++) {
         if (img_secs[i] >= 0) {
             img_file_name[i] = create_test_img(img_secs[i]);
+            if (!img_file_name[i]) {
+                g_test_message("Could not create test images.");
+                goto test_add_done;
+            }
         } else {
             img_file_name[i] = NULL;
         }

tests/hd-geo-test.c int main(int argc, char **argv) · line 978

Quoted 5 lines
tests/hd-geo-test.c +1 -0
@@ -965,6 +974,7 @@ int main(int argc, char **argv)
                        "skipping hd-geo/override/* tests");
     }
 
+test_add_done:
     ret = g_test_run();

It does seem a bit odd to call g_test_run if we have explicitly not set any up. Personally I'd hoist all the test creation into a new function so you could do:

  if (setup_images()) {
     setup_tests();
     ret = run_tests();
  } else {
     ret = 0; /* pass if we have no images */
  }

  cleanup_images();

but that's just me going above and beyond to avoid goto's ;-)

Quoted 2 lines
tests/hd-geo-test.c +0 -0
     for (i = 0; i < backend_last; i++) {
Signature · 1 line
Alex Bennée
#13 lore raw

PATCH 4/6 tests/test-util-filemonitor: Skip test on non-x86 Travis containers

In reply to Thomas Huth

test-util-filemonitor fails in restricted non-x86 Travis containers since they apparently blacklisted some required system calls there. Let's simply skip the test if we detect such an environment.

---
 tests/test-util-filemonitor.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tests/test-util-filemonitor.c +11 -0
index 301cd2db61..45009c69f4 100644
@@ -406,10 +406,21 @@ test_file_monitor_events(void)
     char *pathdst = NULL;
     QFileMonitorTestData data;
     GHashTable *ids = g_hash_table_new(g_int64_hash, g_int64_equal);
+    char *travis_arch;
 
     qemu_mutex_init(&data.lock);
     data.records = NULL;
 
+    /*
+     * This test does not work on Travis LXD containers since some
+     * syscalls are blocked in that environment.
+     */
+    travis_arch = getenv("TRAVIS_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_test_skip("Test does not work on non-x86 Travis containers.");
+        return;
+    }
+
     /*
      * The file monitor needs the main loop running in
      * order to receive events from inotify. We must
Signature · 1 line
2.23.0
Reviewed-by on v1 4/6
#14 lore raw

Re: [PATCH 4/6] tests/test-util-filemonitor: Skip test on non-x86 Travis containers

Quoted 19 lines · Thomas Huth

test-util-filemonitor fails in restricted non-x86 Travis containers since they apparently blacklisted some required system calls there. Let's simply skip the test if we detect such an environment.

---
  tests/test-util-filemonitor.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
tests/test-util-filemonitor.c +1 -0
index 301cd2db61..45009c69f4 100644
@@ -406,10 +406,21 @@ test_file_monitor_events(void)
      char *pathdst = NULL;
      QFileMonitorTestData data;
      GHashTable *ids = g_hash_table_new(g_int64_hash, g_int64_equal);
+    char *travis_arch;
 

tests/test-util-filemonitor.c test_file_monitor_events(void) · line 420

Quoted 10 lines
tests/test-util-filemonitor.c +7 -0
      qemu_mutex_init(&data.lock);
      data.records = NULL;
 
+    /*
+     * This test does not work on Travis LXD containers since some
+     * syscalls are blocked in that environment.
+     */
+    travis_arch = getenv("TRAVIS_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_test_skip("Test does not work on non-x86 Travis containers.");

We might need to refactor this hunk to a more generic place some day.

Quoted 7 lines
tests/test-util-filemonitor.c +3 -0
+        return;
+    }
+
      /*
       * The file monitor needs the main loop running in
       * order to receive events from inotify. We must
Reviewed-by on v1 4/6
#15 lore raw

Re: [PATCH 4/6] tests/test-util-filemonitor: Skip test on non-x86 Travis containers

In reply to Thomas Huth
Quoted 5 lines · Thomas Huth

test-util-filemonitor fails in restricted non-x86 Travis containers since they apparently blacklisted some required system calls there. Let's simply skip the test if we detect such an environment.

Quoted 30 lines
---
 tests/test-util-filemonitor.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tests/test-util-filemonitor.c +11 -0
index 301cd2db61..45009c69f4 100644
@@ -406,10 +406,21 @@ test_file_monitor_events(void)
     char *pathdst = NULL;
     QFileMonitorTestData data;
     GHashTable *ids = g_hash_table_new(g_int64_hash, g_int64_equal);
+    char *travis_arch;
 
     qemu_mutex_init(&data.lock);
     data.records = NULL;
 
+    /*
+     * This test does not work on Travis LXD containers since some
+     * syscalls are blocked in that environment.
+     */
+    travis_arch = getenv("TRAVIS_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_test_skip("Test does not work on non-x86 Travis containers.");
+        return;
+    }
+
     /*
      * The file monitor needs the main loop running in
      * order to receive events from inotify. We must
Signature · 1 line
Alex Bennée
#16 lore raw

PATCH 5/6 travis.yml: drop 32 bit systems from MAIN_SOFTMMU_TARGETS

In reply to Thomas Huth

From: Alex Bennée <[email protected]>

The older clangs are still struggling to build and run everything withing the 50 minute timeout so lets lighten the load a bit more. We still have coverage for GCC and hopefully no obscure 32 bit guest only breakages slip through the cracks.

---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.travis.yml +1 -1
index b9a026c8ee..c09b6a0014 100644
@@ -79,7 +79,7 @@ env:
     - BASE_CONFIG="--disable-docs --disable-tools"
     - TEST_CMD="make check V=1"
     # This is broadly a list of "mainline" softmmu targets which have support across the major distros
-    - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
+    - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
     - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
     - CCACHE_MAXSIZE=1G
 
Signature · 1 line
2.23.0
Reviewed-by on v1 5/6
#17 lore raw

Re: [PATCH 5/6] travis.yml: drop 32 bit systems from MAIN_SOFTMMU_TARGETS

Quoted 19 lines · Thomas Huth

From: Alex Bennée <[email protected]>

The older clangs are still struggling to build and run everything withing the 50 minute timeout so lets lighten the load a bit more. We still have coverage for GCC and hopefully no obscure 32 bit guest only breakages slip through the cracks.

---
  .travis.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
.travis.yml +0 -0
index b9a026c8ee..c09b6a0014 100644

.travis.yml env: · line 85

Quoted 8 lines
.travis.yml +1 -1
@@ -79,7 +79,7 @@ env:
      - BASE_CONFIG="--disable-docs --disable-tools"
      - TEST_CMD="make check V=1"
      # This is broadly a list of "mainline" softmmu targets which have support across the major distros
-    - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
+    - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
      - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
      - CCACHE_MAXSIZE=1G
 
#18 lore raw

PATCH 6/6 travis.yml: Enable builds on arm64, ppc64le and s390x

In reply to Thomas Huth

Travis recently added the possibility to test on these architectures, too, so let's enable them in our travis.yml file to extend our test coverage.

Unfortunately, the libssh in this Ubuntu version (bionic) is in a pretty unusable Frankenstein state and libspice-server-dev is not available here, so we can not use the global list of packages to install, but have to provide individual package lists instead.

---
 .travis.yml | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
.travis.yml +83 -0
index c09b6a0014..cf48ee452c 100644
@@ -360,6 +360,89 @@ matrix:
         - TEST_CMD="make -j3 check-tcg V=1"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
 
+    - arch: arm64
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS},x86_64-linux-user"
+
+    - arch: ppc64le
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS},x86_64-linux-user"
+
+    - arch: s390x
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS},x86_64-linux-user"
 
     # Release builds
     # The make-release script expect a QEMU version, so our tag must start with a 'v'.
Signature · 1 line
2.23.0
#19 lore raw

Re: [PATCH 6/6] travis.yml: Enable builds on arm64, ppc64le and s390x

Quoted 3 lines · Thomas Huth

Travis recently added the possibility to test on these architectures, too, so let's enable them in our travis.yml file to extend our test coverage.

This is good as far as it goes but it would be nice to exercise the respective TCG backends. If added two commits to:

https://github.com/stsquad/qemu/commits/review/multiarch-testing

which allow for that. I'll know if they worked properly in a hour or two once the testing has finished.

Quoted 105 lines

Unfortunately, the libssh in this Ubuntu version (bionic) is in a pretty unusable Frankenstein state and libspice-server-dev is not available here, so we can not use the global list of packages to install, but have to provide individual package lists instead.

---
 .travis.yml | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
.travis.yml +83 -0
index c09b6a0014..cf48ee452c 100644
@@ -360,6 +360,89 @@ matrix:
         - TEST_CMD="make -j3 check-tcg V=1"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
 
+    - arch: arm64
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS},x86_64-linux-user"
+
+    - arch: ppc64le
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS},x86_64-linux-user"
+
+    - arch: s390x
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+      env:
+        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS},x86_64-linux-user"
 
     # Release builds
     # The make-release script expect a QEMU version, so our tag must start with a 'v'.
Signature · 1 line
Alex Bennée
#20 lore raw
In reply to Thomas Huth
Quoted 7 lines · Thomas Huth

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

<snip>

Quoted 2 lines

iotests: Skip test 060 if it is not possible to create large files
iotests: Skip test 079 if it is not possible to create large files

It seems like 161 is also failing:

https://travis-ci.org/stsquad/qemu/jobs/615672478

Quoted 10 lines

tests/hd-geo-test: Skip test when images can not be created tests/test-util-filemonitor: Skip test on non-x86 Travis containers travis.yml: Enable builds on arm64, ppc64le and s390x

 .travis.yml                   | 85 ++++++++++++++++++++++++++++++++++-
 tests/hd-geo-test.c           | 12 ++++-
 tests/qemu-iotests/060        |  6 +++
 tests/qemu-iotests/079        |  6 +++
 tests/test-util-filemonitor.c | 11 +++++
 5 files changed, 118 insertions(+), 2 deletions(-)
Signature · 1 line
Alex Bennée
#21 lore raw
Quoted 9 lines · Alex Bennée
Quoted 7 lines · Thomas Huth

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

Quoted 7 lines

<snip>

Quoted 2 lines

iotests: Skip test 060 if it is not possible to create large files
iotests: Skip test 079 if it is not possible to create large files

It seems like 161 is also failing:

https://travis-ci.org/stsquad/qemu/jobs/615672478

And sometimes 249

Quoted 12 lines
Quoted 12 lines

tests/hd-geo-test: Skip test when images can not be created tests/test-util-filemonitor: Skip test on non-x86 Travis containers travis.yml: Enable builds on arm64, ppc64le and s390x

 .travis.yml                   | 85 ++++++++++++++++++++++++++++++++++-
 tests/hd-geo-test.c           | 12 ++++-
 tests/qemu-iotests/060        |  6 +++
 tests/qemu-iotests/079        |  6 +++
 tests/test-util-filemonitor.c | 11 +++++
 5 files changed, 118 insertions(+), 2 deletions(-)
Signature · 1 line
Alex Bennée
#22 lore raw
Quoted 13 lines · Alex Bennée
Quoted 10 lines · Alex Bennée
Quoted 7 lines · Thomas Huth

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

<snip>

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

iotests: Skip test 060 if it is not possible to create large files
iotests: Skip test 079 if it is not possible to create large files

It seems like 161 is also failing:

   https://travis-ci.org/stsquad/qemu/jobs/615672478

And sometimes 249

These must be intermittent problems ... I've seen 161 failing once at the very beginning of my tests, but then never again, so I assumed that it was a quirk with the test system that got fixed later. Seems like that was a wrong assumption. I've never seen 249 failing so far... I'll try to do some more tests when I've got some spare time...

Thomas

#23 lore raw
Quoted 23 lines · Thomas Huth
Quoted 22 lines · Alex Bennée
Quoted 18 lines · Alex Bennée
Quoted 9 lines · Thomas Huth

Travis recently added build hosts for arm64, ppc64le and s390x, so this is a welcome addition to our Travis testing matrix.

Unfortunately, the builds are running in quite restricted LXD containers
there, for example it is not possible to create huge files there (even if they are just sparse), and certain system calls are blocked. So we have to change some tests first to stop them failing in such environments.

<snip>

Quoted 2 lines

   iotests: Skip test 060 if it is not possible to create large files
   iotests: Skip test 079 if it is not possible to create large files

It seems like 161 is also failing:

   https://travis-ci.org/stsquad/qemu/jobs/615672478

Quoted 7 lines
Quoted 1 line

And sometimes 249

These must be intermittent problems ... I've seen 161 failing once at the very beginning of my tests, but then never again, so I assumed that it was a quirk with the test system that got fixed later. Seems like that was a wrong assumption. I've never seen 249 failing so far... I'll try to do some more tests when I've got some spare time...

It's not intermittent, it's a problem with "dist: bionic". It seems to work fine with "dist: xenial", as far as I can tell. So I think we should simply stick with "dist: xenial" for the ppc64le builder unless someone has a ppc64le bionic system at hand for debugging.

Thomas