Skip to contentSkip to conversation

IDE failures under netbsd unit tests

PATCH In review John Snow 6 participants 14 messages Aug 24 – Sep 3

lore
Share & tools
lore

John Snow

author
#0 lore raw

Heya Denis;

In the process of upgrading Python to 3.12, I found that our netbsd tests were a little broken - I will post patches soon - but if you add "bison" to the list of deps in the netbsd vm test file, you can get far enough to see this:

▶ 163/1096 ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion
failed: (ret == 0) ERROR
163/1096 qtest+qtest-i386 - qemu:qtest-i386/ide-test
ERROR 0.01s killed by signal 6
SIGABRT

Quoted 1 line
Quoted 1 line
Quoted 1 line

G_TEST_DBUS_DAEMON=/home/qemu/qemu-test.kgtky7/src/tests/dbus-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MALLOC_PERTURB_=224 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_BINARY=./qemu-system-i386 PYTHON=/home/qemu/qemu-test.kgtky7/build/pyvenv/bin/python3.13 QTEST_QEMU_IMG=./qemu-img MESON_TEST_ITERATION=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 RUST_BACKTRACE=1 /home/qemu/qemu-test.kgtky7/build/tests/qtest/ide-test --tap -k

Maybe something that cropped up due to recent changes? I am not sure, but figured I'd give you a heads up.

--js

#1 lore raw

cover tests/qtest: fix the disk tests on a host with a small /tmp

John reported ide-test dying at startup in the NetBSD VM:

ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
(ret == 0)

The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.

The series is designed to do two things:
* reduce the amount of space needed switching to QCOW2
* tolerate the failure skipping the test if there is not enough space

Patches 1-4 skip instead of aborting, the way hd-geo-test has since a0d6d7454a. Patches 5-7 build the images with qemu-img where only the virtual size matters: peak scratch space goes from 129 MiB to 1.6 MiB for ide-test and from 4126 MiB to 1.5 MiB for hd-geo-test.

Hope this should help. John, can you give a try to this in the test? I do not have NetBSD handy and will start with this tomorrow if you will share guest config.

Denis V. Lunev (7):
tests/qtest/libqos: let mkqcow2() report failure tests/qtest/ide-test: skip when the scratch files cannot be created tests/qtest/ahci-test: skip when the scratch files cannot be created tests/qtest/hd-geo-test: skip when the scratch file cannot be created tests/qtest/libqtest: create images with a byte-precise size tests/qtest/ide-test: build the shared disks with qemu-img tests/qtest/hd-geo-test: build the test images with qemu-img

 tests/qtest/ahci-test.c        |  48 ++++++++---
 tests/qtest/hd-geo-test.c      | 145 ++++++++++++++++++++++-----------
 tests/qtest/ide-test.c         |  96 +++++++++++++++++-----
 tests/qtest/libqos/libqos.c    |   4 +-
 tests/qtest/libqos/libqos.h    |   2 +-
 tests/qtest/libqtest.c         |   9 +-
 tests/qtest/libqtest.h         |  12 +++
 tests/qtest/virtio-scsi-test.c |   5 +-
 8 files changed, 237 insertions(+), 84 deletions(-)
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp

Hi Denis!

Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.

Quoted 9 lines · Denis V. Lunev

John reported ide-test dying at startup in the NetBSD VM:

   ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
   (ret == 0)

The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.

Does NetBSD have another file like /var/tmp that might be friendlier to sparse files? If so, maybe that should be used instead?

OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?

...> Hope this should help. John, can you give a try to this in the test?

Quoted 2 lines

I do not have NetBSD handy and will start with this tomorrow if you will share guest config.

Simply type "make vm-build-netbsd" from your build folder, this should take care of everything automatically. No need to install NetBSD on your own.

Thomas

#3 lore raw

Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp

Quoted 17 lines · Thomas Huth

Hi Denis!

Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.

Quoted 9 lines · Denis V. Lunev

John reported ide-test dying at startup in the NetBSD VM:

   ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
   (ret == 0)

The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.

Quoted 5 lines

Does NetBSD have another file like /var/tmp that might be friendlier to sparse files? If so, maybe that should be used instead?

OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?

Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?

We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.

IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.

If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
#4 lore raw

Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp

Quoted 41 lines · Daniel P. Berrangé
Quoted 15 lines · Thomas Huth

Hi Denis!

Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.

Quoted 9 lines · Denis V. Lunev

John reported ide-test dying at startup in the NetBSD VM:

   ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
   (ret == 0)

The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.

Quoted 5 lines

Does NetBSD have another file like /var/tmp that might be friendlier to sparse files? If so, maybe that should be used instead?

OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?

Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?

We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.

IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.

If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?

Quoted 2 lines

With regards,
Daniel

Guys,

will somebody take a look into what was done?
This makes sense anyway - unified same class error tracking plus requirements reduction.

Or if nobody care I could just push along with another test fix?

Thank you in advance,
    Den

#5 lore raw

Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp

Quoted 42 lines · Denis V. Lunev
Quoted 41 lines · Daniel P. Berrangé
Quoted 15 lines · Thomas Huth

Hi Denis!

Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.

Quoted 9 lines · Denis V. Lunev

John reported ide-test dying at startup in the NetBSD VM:

   ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
   (ret == 0)

The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.

Quoted 5 lines

Does NetBSD have another file like /var/tmp that might be friendlier to sparse files? If so, maybe that should be used instead?

OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?

Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?

We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.

IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.

If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?

Quoted 8 lines
Quoted 2 lines

With regards,
Daniel

Guys,

will somebody take a look into what was done?
This makes sense anyway - unified same class error tracking plus requirements reduction.

I cannot reproduce this. I also don't think ENOSPC on 64MB is something worth the churn at all.

Quoted 5 lines

Or if nobody care I could just push along with another test fix?

Thank you in advance,
    Den

#6 lore raw

Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp

Quoted 46 lines · Fabiano Rosas
Quoted 44 lines · Denis V. Lunev
Quoted 43 lines · Daniel P. Berrangé
Quoted 15 lines · Thomas Huth

Hi Denis!

Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.

Quoted 9 lines · Denis V. Lunev

John reported ide-test dying at startup in the NetBSD VM:

   ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
   (ret == 0)

The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.

Quoted 5 lines

Does NetBSD have another file like /var/tmp that might be friendlier to sparse files? If so, maybe that should be used instead?

OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?

Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?

We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.

IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.

If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?

With regards,
Daniel

Quoted 8 lines
Quoted 6 lines

Guys,

will somebody take a look into what was done?
This makes sense anyway - unified same class error tracking plus requirements reduction.

I cannot reproduce this. I also don't think ENOSPC on 64MB is something worth the churn at all.

Great.

Clearly postponed, dropping this from watch list.

Thanks a lot for your time.

Den

#7 lore raw

PATCH 1/7 tests/qtest/libqos: let mkqcow2() report failure

In reply to John Snow

From: Denis V. Lunev <[email protected]>

mkqcow2() asserts that qemu-img succeeded, so a caller that has qemu-img but cannot use it aborts the binary rather than skipping. Return the result instead. virtio-scsi-test takes the skip path it already has for a missing qemu-img; ahci-test grows one two patches on.

---
 tests/qtest/libqos/libqos.c    | 4 ++--
 tests/qtest/libqos/libqos.h    | 2 +-
 tests/qtest/virtio-scsi-test.c | 5 ++++-
 3 files changed, 7 insertions(+), 4 deletions(-)
tests/qtest/libqos/libqos.c +2 -2
index 9b49d0d4dd..ff0d595346 100644
@@ -138,9 +138,9 @@ void migrate(QOSState *from, QOSState *to, const char *uri)
     migrate_allocator(&from->alloc, &to->alloc);
 }
 
-void mkqcow2(const char *file, unsigned size_mb)
+bool mkqcow2(const char *file, unsigned size_mb)
 {
-    g_assert_true(mkimg(file, "qcow2", size_mb));
+    return mkimg(file, "qcow2", size_mb);
 }
 
 void prepare_blkdebug_script(const char *debug_fn, const char *event)
tests/qtest/libqos/libqos.h +1 -1
index c04950e2b1..8390d4fa98 100644
@@ -27,7 +27,7 @@ QOSState *qtest_boot(QOSOps *ops, const char *cmdline_fmt, ...)
     G_GNUC_PRINTF(2, 3);
 void qtest_common_shutdown(QOSState *qs);
 void qtest_shutdown(QOSState *qs);
-void mkqcow2(const char *file, unsigned size_mb);
+bool mkqcow2(const char *file, unsigned size_mb);
 void migrate(QOSState *from, QOSState *to, const char *uri);
 void prepare_blkdebug_script(const char *debug_fn, const char *event);
 void generate_pattern(void *buffer, size_t len, size_t cycle_len);
tests/qtest/virtio-scsi-test.c +4 -1
index e2350c52f6..bce6c11813 100644
@@ -292,7 +292,10 @@ static void test_iothread_attach_node(void *obj, void *data,
         goto fail;
     }
 
-    mkqcow2(tmp_path, 64);
+    if (!mkqcow2(tmp_path, 64)) {
+        g_test_message("Could not create the qcow2 overlay");
+        goto fail;
+    }
 
     /* Attach the overlay to the null0 node */
     qtest_qmp_assert_success(scsi_pci->pci_vdev.pdev->bus->qts,
Signature · 1 line
2.53.0
#8 lore raw

PATCH 2/7 tests/qtest/ide-test: skip when the scratch files cannot be created

In reply to John Snow

From: Denis V. Lunev <[email protected]>

main() creates a blkdebug script and two 64 MiB images and asserts that every step worked. The images are sparse, which is free on Linux but not everywhere: NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length the moment it is extended, so the ftruncate() returns ENOSPC and the binary dies before a test has run:

ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
(ret == 0)

A file system with no room left fails the g_mkstemp() calls too, the blkdebug script first of all.

Skip instead, as hd-geo-test does since commit a0d6d7454a ("tests/hd-geo-test: Skip test when images can not be created"): move g_test_init() ahead of the setup, report the errno and register no tests. The cleanup loop has to cope with a path never filled in.

---
 tests/qtest/ide-test.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)
tests/qtest/ide-test.c +23 -6
index b36b0f8875..805afb9461 100644
@@ -1231,6 +1231,7 @@ int main(int argc, char **argv)
     int i;
     int fd;
     int ret;
+    int err;
 
     /*
      * "base" stores the starting point where we create temporary files.
@@ -1245,25 +1246,37 @@ int main(int argc, char **argv)
     base = ".";
 #endif
 
+    g_test_init(&argc, &argv, NULL);
+
     /* Create temporary blkdebug instructions */
     debug_path = g_strdup_printf("%s/qtest-blkdebug.XXXXXX", base);
     fd = g_mkstemp(debug_path);
-    g_assert(fd >= 0);
+    if (fd < 0) {
+        g_test_message("Could not create %s: %s", debug_path,
+                       strerror(errno));
+        goto test_add_done;
+    }
     close(fd);
 
     /* Create a temporary raw image */
     for (i = 0; i < 2; ++i) {
         tmp_path[i] = g_strdup_printf("%s/qtest.XXXXXX", base);
         fd = g_mkstemp(tmp_path[i]);
-        g_assert(fd >= 0);
+        if (fd < 0) {
+            g_test_message("Could not create %s: %s", tmp_path[i],
+                           strerror(errno));
+            goto test_add_done;
+        }
         ret = ftruncate(fd, TEST_IMAGE_SIZE);
-        g_assert(ret == 0);
+        err = errno;
         close(fd);
+        if (ret < 0) {
+            g_test_message("Could not size %s: %s", tmp_path[i],
+                           strerror(err));
+            goto test_add_done;
+        }
     }
 
-    /* Run the tests */
-    g_test_init(&argc, &argv, NULL);
-
     qtest_add_func("/ide/read_native", test_specify);
 
     qtest_add_func("/ide/identify", test_identify);
@@ -1288,10 +1301,14 @@ int main(int argc, char **argv)
     qtest_add_func("/ide/cdrom/pio_raw", test_cdrom_pio_raw);
     qtest_add_func("/ide/cdrom/dma_raw", test_cdrom_dma_raw);
 
+test_add_done:
     ret = g_test_run();
 
     /* Cleanup */
     for (i = 0; i < 2; ++i) {
+        if (!tmp_path[i]) {
+            continue;
+        }
         unlink(tmp_path[i]);
         g_free(tmp_path[i]);
     }
Signature · 1 line
2.53.0
#9 lore raw

PATCH 3/7 tests/qtest/ahci-test: skip when the scratch files cannot be created

In reply to John Snow

From: Denis V. Lunev <[email protected]>

The fallback ahci-test takes when QTEST_QEMU_IMG is unset extends a raw image to 64 MiB with ftruncate() and asserts it worked, which is the assert ide-test just lost. The temporary files are asserted on too, and a file system with no room left fails g_mkstemp() and g_file_open_tmp() as surely as the ftruncate().

Report the reason and register no tests instead. The cleanup path has to cope with the files now created after the point of no return.

Cc Fabiano Rosas <[email protected]>
Cc John Snow <[email protected]>
Cc Laurent Vivier <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Thomas Huth <[email protected]>
---
 tests/qtest/ahci-test.c | 48 +++++++++++++++++++++++++++++++----------
 1 file changed, 37 insertions(+), 11 deletions(-)
tests/qtest/ahci-test.c +37 -11
index 71d23fe56a..d6af5b30a0 100644
@@ -2110,8 +2110,10 @@ static void create_ahci_io_test(enum IOMode type, enum AddrMode addr,
 
 int main(int argc, char **argv)
 {
+    g_autoptr(GError) error = NULL;
     const char *arch, *base;
     int ret;
+    int err;
     int fd;
     int c;
     int i, j, k, m;
@@ -2164,30 +2166,49 @@ int main(int argc, char **argv)
     /* Create a temporary image */
     tmp_path = g_strdup_printf("%s/qtest.XXXXXX", base);
     fd = g_mkstemp(tmp_path);
-    g_assert(fd >= 0);
+    if (fd < 0) {
+        g_test_message("Could not create %s: %s", tmp_path, strerror(errno));
+        goto test_add_done;
+    }
     if (have_qemu_img()) {
         imgfmt = "qcow2";
         test_image_size_mb = TEST_IMAGE_SIZE_MB_LARGE;
-        mkqcow2(tmp_path, TEST_IMAGE_SIZE_MB_LARGE);
+        close(fd);
+        if (!mkqcow2(tmp_path, TEST_IMAGE_SIZE_MB_LARGE)) {
+            g_test_message("Could not create %s with qemu-img", tmp_path);
+            goto test_add_done;
+        }
     } else {
         g_test_message("QTEST_QEMU_IMG not set or qemu-img missing; "
                        "skipping LBA48 high-sector tests");
         imgfmt = "raw";
         test_image_size_mb = TEST_IMAGE_SIZE_MB_SMALL;
         ret = ftruncate(fd, test_image_size_mb * 1024 * 1024);
-        g_assert(ret == 0);
+        err = errno;
+        close(fd);
+        if (ret < 0) {
+            g_test_message("Could not size %s: %s", tmp_path, strerror(err));
+            goto test_add_done;
+        }
     }
-    close(fd);
 
     /* Create temporary blkdebug instructions */
     debug_path = g_strdup_printf("%s/qtest-blkdebug.XXXXXX", base);
     fd = g_mkstemp(debug_path);
-    g_assert(fd >= 0);
+    if (fd < 0) {
+        g_test_message("Could not create %s: %s", debug_path,
+                       strerror(errno));
+        goto test_add_done;
+    }
     close(fd);
 
     /* Reserve a hollow file to use as a socket for migration tests */
-    fd = g_file_open_tmp("qtest-migration.XXXXXX", &mig_socket, NULL);
-    g_assert(fd >= 0);
+    fd = g_file_open_tmp("qtest-migration.XXXXXX", &mig_socket, &error);
+    if (fd < 0) {
+        g_test_message("Could not create a temporary file: %s",
+                       error->message);
+        goto test_add_done;
+    }
     close(fd);
 
     /* Run the tests */
@@ -2244,15 +2265,20 @@ int main(int argc, char **argv)
     qtest_add_func("/ahci/cdrom/drain/pio", test_atapi_drain_pio);
     qtest_add_func("/ahci/cdrom/drain/dma", test_atapi_drain_dma);
 
+test_add_done:
     ret = g_test_run();
 
     /* Cleanup */
     unlink(tmp_path);
     g_free(tmp_path);
-    unlink(debug_path);
-    g_free(debug_path);
-    unlink(mig_socket);
-    g_free(mig_socket);
+    if (debug_path) {
+        unlink(debug_path);
+        g_free(debug_path);
+    }
+    if (mig_socket) {
+        unlink(mig_socket);
+        g_free(mig_socket);
+    }
 
     return ret;
 }
Signature · 1 line
2.53.0
#10 lore raw

PATCH 4/7 tests/qtest/hd-geo-test: skip when the scratch file cannot be created

In reply to John Snow

From: Denis V. Lunev <[email protected]>

Commit a0d6d7454a ("tests/hd-geo-test: Skip test when images can not be created") skips when the 4 GiB image cannot be sized, but the temporary file itself is still asserted on, and a file system with no room left fails g_file_open_tmp() as surely as the ftruncate().

Skip there too and say why in both cases, from inside create_test_img() where the GError is. Unlink the image that could not be sized while here.

---
 tests/qtest/hd-geo-test.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
tests/qtest/hd-geo-test.c +12 -4
index 41481a5e09..e3b4a4db31 100644
@@ -27,15 +27,24 @@
 
 static char *create_test_img(int secs)
 {
+    g_autoptr(GError) error = NULL;
     char *template;
-    int fd, ret;
+    int fd, ret, err;
+
+    fd = g_file_open_tmp("qtest.XXXXXX", &template, &error);
+    if (fd < 0) {
+        g_test_message("Could not create a temporary file: %s",
+                       error->message);
+        return NULL;
+    }
 
-    fd = g_file_open_tmp("qtest.XXXXXX", &template, NULL);
-    g_assert(fd >= 0);
     ret = ftruncate(fd, (off_t)secs * 512);
+    err = errno;
     close(fd);
 
     if (ret) {
+        g_test_message("Could not size %s: %s", template, strerror(err));
+        unlink(template);
         g_free(template);
         template = NULL;
     }
@@ -1059,7 +1068,6 @@ int main(int argc, char **argv)
         if (img_secs[i] >= 0) {
             img_file_name[i] = create_test_img(img_secs[i]);
             if (!img_file_name[i]) {
-                g_test_message("Could not create test images.");
                 goto test_add_done;
             }
         } else {
Signature · 1 line
2.53.0
#11 lore raw

PATCH 5/7 tests/qtest/libqtest: create images with a byte-precise size

In reply to John Snow

From: Denis V. Lunev <[email protected]>

mkimg() takes megabytes, which is all its callers have needed so far. ide-test wants an image whose length is an exact multiple of its CHS geometry, 130 * 16 * 63 sectors, and that is not a whole number of megabytes. Add mkimg_bytes() and let mkimg() go through it.

---
 tests/qtest/libqtest.c |  9 +++++++--
 tests/qtest/libqtest.h | 12 ++++++++++++
 2 files changed, 19 insertions(+), 2 deletions(-)
tests/qtest/libqtest.c +7 -2
index bec37c71b8..1ae004e6f3 100644
@@ -2115,6 +2115,11 @@ bool have_qemu_img(void)
 }
 
 bool mkimg(const char *file, const char *fmt, unsigned size_mb)
+{
+    return mkimg_bytes(file, fmt, (uint64_t)size_mb * 1024 * 1024);
+}
+
+bool mkimg_bytes(const char *file, const char *fmt, uint64_t size)
 {
     gchar *cli;
     bool ret;
@@ -2133,8 +2138,8 @@ bool mkimg(const char *file, const char *fmt, unsigned size_mb)
         return false;
     }
 
-    cli = g_strdup_printf("%s create -f %s %s %uM", qemu_img_abs_path,
-                          fmt, file, size_mb);
+    cli = g_strdup_printf("%s create -f %s %s %" PRIu64, qemu_img_abs_path,
+                          fmt, file, size);
     ret = g_spawn_command_line_sync(cli, &out, &out2, &rc, &err);
     if (err || !g_spawn_check_exit_status(rc, &err)) {
         fprintf(stderr, "%s\n", err->message);
tests/qtest/libqtest.h +12 -0
index 45217fb8dc..6720ca2de2 100644
@@ -1186,6 +1186,18 @@ bool have_qemu_img(void);
  */
 bool mkimg(const char *file, const char *fmt, unsigned size_mb);
 
+/**
+ * mkimg_bytes:
+ * @file: File name of the image that should be created
+ * @fmt: Format, e.g. "qcow2" or "raw"
+ * @size: Size of the image in bytes
+ *
+ * As mkimg(), for images whose size is not a whole number of megabytes.
+ *
+ * Returns: true if the image has been created successfully.
+ */
+bool mkimg_bytes(const char *file, const char *fmt, uint64_t size);
+
 /**
  * qtest_verbose:
  * @domain: The logging domain
Signature · 1 line
2.53.0
#12 lore raw

PATCH 6/7 tests/qtest/ide-test: build the shared disks with qemu-img

In reply to John Snow

From: Denis V. Lunev <[email protected]>

The two 64 MiB disks are raw and are grown with ftruncate(), which is free only where a sparse file is free. Build them as qcow2 when qemu-img is available, the way ahci-test and hd-geo-test do, keeping the raw ftruncate() as the fallback. Only the virtual size matters to their users: the CHS geometry READ NATIVE MAX ADDRESS reports and the end-of-disk sector the TRIM range check refuses.

Three tests do care what sits below the guest and get a raw 1 MiB image of their own:

- the CD-ROM tests write their ATAPI pattern in with fopen()
- bmdma/trim_reset needs the guest TRIM to reach blkdebug, where a 1 ms delay lets the reset catch the request in flight. qcow2 stops the discard at the format layer, the clusters never having been allocated, and the test would pass while racing nothing
- the two flush tests trip a blkdebug event. Both pass over qcow2, but on raw the event reported is the flush the guest issued

Peak scratch space drops from 129 MiB to 1.6 MiB.

---
 tests/qtest/ide-test.c | 69 +++++++++++++++++++++++++++++++++---------
 1 file changed, 54 insertions(+), 15 deletions(-)
tests/qtest/ide-test.c +54 -15
index 805afb9461..89590e66cc 100644
@@ -37,6 +37,12 @@
 /* Specified by ATA (physical) CHS geometry for ~64 MiB device.  */
 #define TEST_IMAGE_SIZE ((130 * 16 * 63) * 512)
 
+/*
+ * The tests that reach the image behind QEMU's back, or that need an
+ * event to reach blkdebug, get a raw one of their own.
+ */
+#define RAW_IMAGE_SIZE (1 * 1024 * 1024)
+
 #define IDE_PCI_DEV     1
 #define IDE_PCI_FUNC    1
 
@@ -131,7 +137,9 @@ static QPCIBus *pcibus = NULL;
 static QGuestAllocator guest_malloc;
 
 static char *tmp_path[2];
+static char *raw_path;
 static char *debug_path;
+static const char *imgfmt = "raw";
 
 G_GNUC_PRINTF(1, 2)
 static QTestState *ide_test_start(const char *cmdline_fmt, ...)
@@ -348,9 +356,9 @@ static QTestState *test_bmdma_setup(void)
     QTestState *qts;
 
     qts = ide_test_start(
-        "-drive file=%s,if=ide,cache=writeback,format=raw "
+        "-drive file=%s,if=ide,cache=writeback,format=%s "
         "-global ide-hd.serial=%s -global ide-hd.ver=%s",
-        tmp_path[0], "testdisk", "version");
+        tmp_path[0], imgfmt, "testdisk", "version");
     qtest_irq_intercept_in(qts, "ioapic");
 
     return qts;
@@ -506,7 +514,7 @@ static void test_bmdma_trim_reset(void)
         "inject-error.0.event=none,inject-error.0.iotype=discard,"
         "inject-error.0.errno=0,inject-error.0.delay-ns=1000000 "
         "-device ide-hd,drive=dbg,bus=ide.0",
-        tmp_path[0]);
+        raw_path);
     qtest_irq_intercept_in(qts, "ioapic");
 
     guest_buf = guest_alloc(&guest_malloc, len);
@@ -664,9 +672,9 @@ static void test_specify(void)
     uint8_t heads, spt;
 
     qts = ide_test_start(
-        "-blockdev driver=file,node-name=hda,filename=%s "
+        "-drive if=none,id=hda,format=%s,file=%s "
         "-device ide-hd,drive=hda,bus=ide.0,unit=0 ",
-        tmp_path[0]);
+        imgfmt, tmp_path[0]);
 
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
 
@@ -706,9 +714,9 @@ static void test_identify(void)
     int ret;
 
     qts = ide_test_start(
-        "-drive file=%s,if=ide,cache=writeback,format=raw "
+        "-drive file=%s,if=ide,cache=writeback,format=%s "
         "-global ide-hd.serial=%s -global ide-hd.ver=%s",
-        tmp_path[0], "testdisk", "version");
+        tmp_path[0], imgfmt, "testdisk", "version");
 
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
 
@@ -756,11 +764,11 @@ static void test_diagnostic(void)
     uint8_t data;
 
     qts = ide_test_start(
-        "-blockdev driver=file,node-name=hda,filename=%s "
-        "-blockdev driver=file,node-name=hdb,filename=%s "
+        "-drive if=none,id=hda,format=%s,file=%s "
+        "-drive if=none,id=hdb,format=%s,file=%s "
         "-device ide-hd,drive=hda,bus=ide.0,unit=0 "
         "-device ide-hd,drive=hdb,bus=ide.0,unit=1 ",
-        tmp_path[0], tmp_path[1]);
+        imgfmt, tmp_path[0], imgfmt, tmp_path[1]);
 
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
 
@@ -826,7 +834,7 @@ static void test_flush(void)
 
     qts = ide_test_start(
         "-drive file=blkdebug::%s,if=ide,cache=writeback,format=raw",
-        tmp_path[0]);
+        raw_path);
 
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
 
@@ -877,7 +885,7 @@ static void test_pci_retry_flush(void)
     qts = ide_test_start(
         "-drive file=blkdebug:%s:%s,if=ide,cache=writeback,format=raw,"
         "rerror=stop,werror=stop",
-        debug_path, tmp_path[0]);
+        debug_path, raw_path);
 
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
 
@@ -1100,14 +1108,14 @@ static void cdrom_read_impl(int nblocks, unsigned flags)
 
     /* Prepopulate the CDROM with an interesting pattern */
     generate_pattern(pattern, patt_len, ATAPI_BLOCK_SIZE);
-    fh = fopen(tmp_path[0], "wb+");
+    fh = fopen(raw_path, "wb+");
     ret = fwrite(pattern, ATAPI_BLOCK_SIZE, patt_blocks, fh);
     g_assert_cmpint(ret, ==, patt_blocks);
     fclose(fh);
 
     qts = ide_test_start(
             "-drive if=none,file=%s,media=cdrom,format=raw,id=sr0,index=0 "
-            "-device ide-cd,drive=sr0,bus=ide.0", tmp_path[0]);
+            "-device ide-cd,drive=sr0,bus=ide.0", raw_path);
     dev = get_pci_device(qts, &bmdma_bar, &ide_bar);
     qtest_irq_intercept_in(qts, "ioapic");
 
@@ -1258,7 +1266,11 @@ int main(int argc, char **argv)
     }
     close(fd);
 
-    /* Create a temporary raw image */
+    if (have_qemu_img()) {
+        imgfmt = "qcow2";
+    }
+
+    /* Create the disks the tests share */
     for (i = 0; i < 2; ++i) {
         tmp_path[i] = g_strdup_printf("%s/qtest.XXXXXX", base);
         fd = g_mkstemp(tmp_path[i]);
@@ -1267,6 +1279,15 @@ int main(int argc, char **argv)
                            strerror(errno));
             goto test_add_done;
         }
+        if (!strcmp(imgfmt, "qcow2")) {
+            close(fd);
+            if (!mkimg_bytes(tmp_path[i], imgfmt, TEST_IMAGE_SIZE)) {
+                g_test_message("Could not create %s with qemu-img",
+                               tmp_path[i]);
+                goto test_add_done;
+            }
+            continue;
+        }
         ret = ftruncate(fd, TEST_IMAGE_SIZE);
         err = errno;
         close(fd);
@@ -1277,6 +1298,20 @@ int main(int argc, char **argv)
         }
     }
 
+    raw_path = g_strdup_printf("%s/qtest.XXXXXX", base);
+    fd = g_mkstemp(raw_path);
+    if (fd < 0) {
+        g_test_message("Could not create %s: %s", raw_path, strerror(errno));
+        goto test_add_done;
+    }
+    ret = ftruncate(fd, RAW_IMAGE_SIZE);
+    err = errno;
+    close(fd);
+    if (ret < 0) {
+        g_test_message("Could not size %s: %s", raw_path, strerror(err));
+        goto test_add_done;
+    }
+
     qtest_add_func("/ide/read_native", test_specify);
 
     qtest_add_func("/ide/identify", test_identify);
@@ -1312,6 +1347,10 @@ test_add_done:
         unlink(tmp_path[i]);
         g_free(tmp_path[i]);
     }
+    if (raw_path) {
+        unlink(raw_path);
+        g_free(raw_path);
+    }
     unlink(debug_path);
     g_free(debug_path);
 
Signature · 1 line
2.53.0
#13 lore raw

PATCH 7/7 tests/qtest/hd-geo-test: build the test images with qemu-img

In reply to John Snow

From: Denis V. Lunev <[email protected]>

The two backing images are raw and are grown with ftruncate(), one to 30 MiB and one to 4 GiB, so on a host where sparse files are not free the test skips and the BIOS geometry goes untested.

The override tests already build their images with qemu-img, so use the same route here. The convert and resize pair moves into mkqcow2_with_mbr(), which both users call and which reports a failed qemu-img rather than asserting. setup_mbr() stamps the MBR by rebuilding the image, that being the only way to reach sector 0 of a qcow2 from outside QEMU. Only the setup running before any test can skip, so the rest keep asserting. Without qemu-img the images stay raw.

Peak scratch space drops from 4126 MiB to 1.5 MiB.

---
 tests/qtest/hd-geo-test.c | 129 +++++++++++++++++++++++++-------------
 1 file changed, 86 insertions(+), 43 deletions(-)
tests/qtest/hd-geo-test.c +86 -43
index e3b4a4db31..160bdc7216 100644
@@ -25,9 +25,69 @@
 
 #define ARGV_SIZE 256
 
+#define MBR_SIZE 512
+
+static const char *imgfmt = "raw";
+
+static bool mkqcow2_with_mbr(const char *path, const uint8_t *mbr,
+                             uint64_t size)
+{
+    g_autoptr(GError) error = NULL;
+    g_autofree char *raw_path = NULL;
+    char cmd[100 + 2 * PATH_MAX];
+    char *qemu_img_path;
+    char *qemu_img_abs_path = NULL;
+    bool ok = false;
+    int fd, ret;
+
+    fd = g_file_open_tmp("qtest.XXXXXX", &raw_path, &error);
+    if (fd < 0) {
+        g_test_message("Could not create a temporary file: %s",
+                       error->message);
+        return false;
+    }
+    ret = write(fd, mbr, MBR_SIZE);
+    g_assert(ret == MBR_SIZE);
+    close(fd);
+
+    qemu_img_path = getenv("QTEST_QEMU_IMG");
+    g_assert(qemu_img_path);
+    qemu_img_abs_path = realpath(qemu_img_path, NULL);
+    g_assert(qemu_img_abs_path);
+
+    ret = snprintf(cmd, sizeof(cmd),
+                   "%s convert -f raw -O qcow2 %s %s > /dev/null",
+                   qemu_img_abs_path,
+                   raw_path, path);
+    g_assert((0 < ret) && (ret <= sizeof(cmd)));
+    if (system(cmd) != 0) {
+        g_test_message("Could not create %s with qemu-img", path);
+        goto out;
+    }
+
+    ret = snprintf(cmd, sizeof(cmd),
+                   "%s resize %s %" PRIu64 " > /dev/null",
+                   qemu_img_abs_path,
+                   path, size);
+    g_assert((0 < ret) && (ret <= sizeof(cmd)));
+    if (system(cmd) != 0) {
+        g_test_message("Could not size %s with qemu-img", path);
+        goto out;
+    }
+
+    ok = true;
+
+out:
+    free(qemu_img_abs_path);
+    unlink(raw_path);
+
+    return ok;
+}
+
 static char *create_test_img(int secs)
 {
     g_autoptr(GError) error = NULL;
+    uint8_t blank_mbr[MBR_SIZE] = {};
     char *template;
     int fd, ret, err;
 
@@ -38,6 +98,16 @@ static char *create_test_img(int secs)
         return NULL;
     }
 
+    if (!strcmp(imgfmt, "qcow2")) {
+        close(fd);
+        if (!mkqcow2_with_mbr(template, blank_mbr, (uint64_t)secs * 512)) {
+            unlink(template);
+            g_free(template);
+            return NULL;
+        }
+        return template;
+    }
+
     ret = ftruncate(fd, (off_t)secs * 512);
     err = errno;
     close(fd);
@@ -207,7 +277,7 @@ static void setup_mbr(int img_idx, MBRcontents mbr)
         /* chs 0,1,1 (lba 63) to chs 7,15,63 (8001 sectors) */
         0x80, 1, 1, 0, 6,  15, 63, 7, 63, 0, 0, 0, 0x41, 0x1F, 0, 0,
     };
-    uint8_t buf[512];
+    uint8_t buf[MBR_SIZE];
     int fd, ret;
 
     memset(buf, 0, sizeof(buf));
@@ -218,6 +288,12 @@ static void setup_mbr(int img_idx, MBRcontents mbr)
         memcpy(buf + 0x1BE, mbr == mbr_lba ? part_lba : part_chs, 16);
     }
 
+    if (!strcmp(imgfmt, "qcow2")) {
+        g_assert_true(mkqcow2_with_mbr(img_file_name[img_idx], buf,
+                                       (uint64_t)img_secs[img_idx] * 512));
+        return;
+    }
+
     fd = open(img_file_name[img_idx], O_WRONLY);
     g_assert(fd >= 0);
     ret = write(fd, buf, sizeof(buf));
@@ -237,7 +313,8 @@ static int setup_ide(int argc, char *argv[], int argv_sz,
 
     if (img_secs[img_idx] >= 0) {
         setup_mbr(img_idx, mbr);
-        s3 = g_strdup_printf(",format=raw,file=%s", img_file_name[img_idx]);
+        s3 = g_strdup_printf(",format=%s,file=%s", imgfmt,
+                             img_file_name[img_idx]);
     } else {
         s3 = g_strdup(",media=cdrom");
     }
@@ -431,15 +508,10 @@ static MBRpartitions empty_mbr = { {false, 0, 0, 0, 0, 0, 0, 0, 0},
 
 static char *create_qcow2_with_mbr(MBRpartitions mbr, uint64_t sectors)
 {
-    g_autofree char *raw_path = NULL;
     char *qcow2_path;
-    char cmd[100 + 2 * PATH_MAX];
-    uint8_t buf[512] = {};
-    int i, ret, fd, offset;
-    uint64_t qcow2_size = sectors * 512;
+    uint8_t buf[MBR_SIZE] = {};
+    int i, fd, offset;
     uint8_t status, parttype, head, sector, cyl;
-    char *qemu_img_path;
-    char *qemu_img_abs_path;
 
     offset = 0xbe;
 
@@ -476,44 +548,11 @@ static char *create_qcow2_with_mbr(MBRpartitions mbr, uint64_t sectors)
         offset += 0x10;
     }
 
-    fd = g_file_open_tmp("qtest.XXXXXX", &raw_path, NULL);
-    g_assert(fd >= 0);
-    close(fd);
-
-    fd = open(raw_path, O_WRONLY);
-    g_assert(fd >= 0);
-    ret = write(fd, buf, sizeof(buf));
-    g_assert(ret == sizeof(buf));
-    close(fd);
-
     fd = g_file_open_tmp("qtest.XXXXXX", &qcow2_path, NULL);
     g_assert(fd >= 0);
     close(fd);
 
-    qemu_img_path = getenv("QTEST_QEMU_IMG");
-    g_assert(qemu_img_path);
-    qemu_img_abs_path = realpath(qemu_img_path, NULL);
-    g_assert(qemu_img_abs_path);
-
-    ret = snprintf(cmd, sizeof(cmd),
-                   "%s convert -f raw -O qcow2 %s %s > /dev/null",
-                   qemu_img_abs_path,
-                   raw_path, qcow2_path);
-    g_assert((0 < ret) && (ret <= sizeof(cmd)));
-    ret = system(cmd);
-    g_assert(ret == 0);
-
-    ret = snprintf(cmd, sizeof(cmd),
-                   "%s resize %s %" PRIu64 " > /dev/null",
-                   qemu_img_abs_path,
-                   qcow2_path, qcow2_size);
-    g_assert((0 < ret) && (ret <= sizeof(cmd)));
-    ret = system(cmd);
-    g_assert(ret == 0);
-
-    free(qemu_img_abs_path);
-
-    unlink(raw_path);
+    g_assert_true(mkqcow2_with_mbr(qcow2_path, buf, sectors * 512));
 
     return qcow2_path;
 }
@@ -1064,6 +1103,10 @@ int main(int argc, char **argv)
 
     g_test_init(&argc, &argv, NULL);
 
+    if (have_qemu_img()) {
+        imgfmt = "qcow2";
+    }
+
     for (i = 0; i < backend_last; i++) {
         if (img_secs[i] >= 0) {
             img_file_name[i] = create_test_img(img_secs[i]);
Signature · 1 line
2.53.0