IDE failures under netbsd unit tests
Heya Denis;
In the process of upgrading Python to 3.12, I found that our netbsd tests were a little broken - I will post patches soon - but if you add "bison" to the list of deps in the netbsd vm test file, you can get far enough to see this:
▶ 163/1096 ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion
failed: (ret == 0) ERROR
163/1096 qtest+qtest-i386 - qemu:qtest-i386/ide-test
ERROR 0.01s killed by signal 6
SIGABRT
Quoted 1 line
Quoted 1 line
Quoted 1 line
G_TEST_DBUS_DAEMON=/home/qemu/qemu-test.kgtky7/src/tests/dbus-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MALLOC_PERTURB_=224 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_BINARY=./qemu-system-i386 PYTHON=/home/qemu/qemu-test.kgtky7/build/pyvenv/bin/python3.13 QTEST_QEMU_IMG=./qemu-img MESON_TEST_ITERATION=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 RUST_BACKTRACE=1
/home/qemu/qemu-test.kgtky7/build/tests/qtest/ide-test--tap -k
Maybe something that cropped up due to recent changes? I am not sure, but figured I'd give you a heads up.
--js
cover tests/qtest: fix the disk tests on a host with a small /tmp
John reported ide-test dying at startup in the NetBSD VM:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
(ret == 0)
The hypothesys is that the ftruncate() of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.
The series is designed to do two things:
* reduce the amount of space needed switching to QCOW2
* tolerate the failure skipping the test if there is not enough space
Patches 1-4 skip instead of aborting, the way hd-geo-test has since a0d6d7454a. Patches 5-7 build the images with qemu-img where only the virtual size matters: peak scratch space goes from 129 MiB to 1.6 MiB for ide-test and from 4126 MiB to 1.5 MiB for hd-geo-test.
Hope this should help. John, can you give a try to this in the test? I do not have NetBSD handy and will start with this tomorrow if you will share guest config.
Denis V. Lunev (7):
tests/qtest/libqos: let mkqcow2() report failure tests/qtest/ide-test: skip when the scratch files cannot be created tests/qtest/ahci-test: skip when the scratch files cannot be created tests/qtest/hd-geo-test: skip when the scratch file cannot be created tests/qtest/libqtest: create images with a byte-precise size tests/qtest/ide-test: build the shared disks with qemu-img tests/qtest/hd-geo-test: build the test images with qemu-img
tests/qtest/ahci-test.c | 48 ++++++++--- tests/qtest/hd-geo-test.c | 145 ++++++++++++++++++++++----------- tests/qtest/ide-test.c | 96 +++++++++++++++++----- tests/qtest/libqos/libqos.c | 4 +- tests/qtest/libqos/libqos.h | 2 +- tests/qtest/libqtest.c | 9 +- tests/qtest/libqtest.h | 12 +++ tests/qtest/virtio-scsi-test.c | 5 +- 8 files changed, 237 insertions(+), 84 deletions(-)
Signature · 1 line
2.53.0
Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp
Hi Denis!
Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.
Quoted 9 lines · Denis V. Lunev
John reported ide-test dying at startup in the NetBSD VM:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed: (ret == 0)The hypothesys is that the
ftruncate()of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.
Does NetBSD have another file like /var/tmp that might be friendlier to sparse files? If so, maybe that should be used instead?
OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?
...> Hope this should help. John, can you give a try to this in the test?
Quoted 2 lines
I do not have NetBSD handy and will start with this tomorrow if you will share guest config.
Simply type "make vm-build-netbsd" from your build folder, this should take care of everything automatically. No need to install NetBSD on your own.
Thomas
Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp
Quoted 17 lines · Thomas Huth
Hi Denis!
Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.
Quoted 9 lines · Denis V. Lunev
John reported ide-test dying at startup in the NetBSD VM:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed: (ret == 0)The hypothesys is that the
ftruncate()of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.
Quoted 5 lines
Does NetBSD have another file like
/var/tmpthat might be friendlier to sparse files? If so, maybe that should be used instead?OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?
Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?
We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.
IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.
If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp
Quoted 41 lines · Daniel P. Berrangé
Quoted 15 lines · Thomas Huth
Hi Denis!
Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.
Quoted 9 lines · Denis V. Lunev
John reported ide-test dying at startup in the NetBSD VM:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed: (ret == 0)The hypothesys is that the
ftruncate()of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.Quoted 5 lines
Does NetBSD have another file like
/var/tmpthat might be friendlier to sparse files? If so, maybe that should be used instead?OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?
Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?
We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.
IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.
If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?
Quoted 2 lines
With regards,
Daniel
Guys,
will somebody take a look into what was done?
This makes sense anyway - unified same class error tracking plus requirements reduction.
Or if nobody care I could just push along with another test fix?
Thank you in advance,
Den
Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp
Quoted 42 lines · Denis V. Lunev
Quoted 41 lines · Daniel P. Berrangé
Quoted 15 lines · Thomas Huth
Hi Denis!
Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.
Quoted 9 lines · Denis V. Lunev
John reported ide-test dying at startup in the NetBSD VM:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed: (ret == 0)The hypothesys is that the
ftruncate()of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.Quoted 5 lines
Does NetBSD have another file like
/var/tmpthat might be friendlier to sparse files? If so, maybe that should be used instead?OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?
Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?
We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.
IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.
If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?
Quoted 8 lines
Quoted 2 lines
With regards,
DanielGuys,
will somebody take a look into what was done?
This makes sense anyway - unified same class error tracking plus requirements reduction.
I cannot reproduce this. I also don't think ENOSPC on 64MB is something worth the churn at all.
Quoted 5 lines
Or if nobody care I could just push along with another test fix?
Thank you in advance,
Den
Re: [PATCH 0/7] tests/qtest: fix the disk tests on a host with a small /tmp
Quoted 46 lines · Fabiano Rosas
Quoted 44 lines · Denis V. Lunev
Quoted 43 lines · Daniel P. Berrangé
Quoted 15 lines · Thomas Huth
Hi Denis!
Please make sure to CC: the qtest maintainer (Fabiano Rosas) on series like this.
Quoted 9 lines · Denis V. Lunev
John reported ide-test dying at startup in the NetBSD VM:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed: (ret == 0)The hypothesys is that the
ftruncate()of a 64 MiB scratch image. NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length themoment it is extended, so a sparse image is not free there and the call returns ENOSPC. The assert dates to 2013, nothing regressed.Quoted 5 lines
Does NetBSD have another file like
/var/tmpthat might be friendlier to sparse files? If so, maybe that should be used instead?OTOH, our NetBSD VM in tests/vm/ uses 4G of RAM, so a temporary file with just 64 MiB should really not be a problem...?
Or this is a concurrency scaling problem, or racing with another test that also uses stuff ?
We hard code memory to 4 GB, but -smp we scale to "$NUM-CPUs / 2" for the "make vm-build-DIST" commands.
IOW, regardless of whether QEMU is launched with -smp 1 or -smp 20, we only give it 4 GB to play with. We could be using a lot of RAM for concurrent build jobs leaving almost nothing for the tmpfs for the test.
If we were that close to exhaustion I'd expected to see out of memory errors, but I'm unclear what NetBSD's behaviour is in this respect ? Maybe normal RAM usage can be pushed to swap (of which I see another 4 GB) while tmpfs can't be pushed to swap ?
With regards,
Daniel
Quoted 8 lines
Quoted 6 lines
Guys,
will somebody take a look into what was done?
This makes sense anyway - unified same class error tracking plus requirements reduction.I cannot reproduce this. I also don't think ENOSPC on 64MB is something worth the churn at all.
Great.
Clearly postponed, dropping this from watch list.
Thanks a lot for your time.
Den
PATCH 1/7 tests/qtest/libqos: let mkqcow2() report failure
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
mkqcow2() asserts that qemu-img succeeded, so a caller that has qemu-img but cannot use it aborts the binary rather than skipping. Return the result instead. virtio-scsi-test takes the skip path it already has for a missing qemu-img; ahci-test grows one two patches on.
--- tests/qtest/libqos/libqos.c | 4 ++-- tests/qtest/libqos/libqos.h | 2 +- tests/qtest/virtio-scsi-test.c | 5 ++++- 3 files changed, 7 insertions(+), 4 deletions(-)
tests/qtest/libqos/libqos.c +2 -2
@@ -138,9 +138,9 @@ void migrate(QOSState *from, QOSState *to, const char *uri)migrate_allocator (&from->alloc, &to->alloc); } -void mkqcow2 (const char *file,unsigned size_mb) +bool mkqcow2 (const char *file,unsigned size_mb) { -g_assert_true (mkimg (file,"qcow2" , size_mb)); +return mkimg (file,"qcow2" , size_mb); }void prepare_blkdebug_script (const char *debug_fn,const char *event)
tests/qtest/libqos/libqos.h +1 -1
@@ -27,7 +27,7 @@ QOSState *qtest_boot(QOSOps *ops, const char *cmdline_fmt, ...)G_GNUC_PRINTF (2 ,3 );void qtest_common_shutdown (QOSState *qs);void qtest_shutdown (QOSState *qs); -void mkqcow2 (const char *file,unsigned size_mb); +bool mkqcow2 (const char *file,unsigned size_mb);void migrate (QOSState *from,QOSState *to,const char *uri);void prepare_blkdebug_script (const char *debug_fn,const char *event);void generate_pattern (void *buffer,size_t len,size_t cycle_len);
tests/qtest/virtio-scsi-test.c +4 -1
@@ -292,7 +292,10 @@ static void test_iothread_attach_node(void *obj, void *data, goto fail; } -mkqcow2 (tmp_path,64 ); +if (!mkqcow2 (tmp_path,64 )) { +g_test_message ("Could not create the qcow2 overlay" ); + goto fail; + }/* Attach the overlay to the null0 node */ qtest_qmp_assert_success(scsi_pci->pci_vdev.pdev->bus->qts,
Signature · 1 line
2.53.0
PATCH 2/7 tests/qtest/ide-test: skip when the scratch files cannot be created
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
main() creates a blkdebug script and two 64 MiB images and asserts that every step worked. The images are sparse, which is free on Linux but not everywhere: NetBSD mounts /tmp as a tmpfs sized at 25% of RAM and charges a file its full length the moment it is extended, so the ftruncate() returns ENOSPC and the binary dies before a test has run:
ERROR:../src/tests/qtest/ide-test.c:1260:main: assertion failed:
(ret == 0)
A file system with no room left fails the g_mkstemp() calls too, the blkdebug script first of all.
Skip instead, as hd-geo-test does since commit a0d6d7454a ("tests/hd-geo-test: Skip test when images can not be created"): move g_test_init() ahead of the setup, report the errno and register no tests. The cleanup loop has to cope with a path never filled in.
--- tests/qtest/ide-test.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-)
tests/qtest/ide-test.c +23 -6
@@ -1231,6 +1231,7 @@ int main(int argc, char **argv)int i;int fd;int ret; +int err;/* * "base" stores the starting point where we create temporary files.
@@ -1245,25 +1246,37 @@ int main(int argc, char **argv) base ="." ;#endif +g_test_init (&argc, &argv,NULL ); +/* Create temporary blkdebug instructions */ debug_path =g_strdup_printf ("%s/qtest-blkdebug.XXXXXX" , base); fd =g_mkstemp (debug_path); -g_assert (fd >=0 ); +if (fd <0 ) { +g_test_message ("Could not create %s: %s" , debug_path, +strerror (errno)); + goto test_add_done; + }close (fd);/* Create a temporary raw image */ for (i =0 ; i <2 ; ++i) { tmp_path[i] =g_strdup_printf ("%s/qtest.XXXXXX" , base); fd =g_mkstemp (tmp_path[i]); -g_assert (fd >=0 ); +if (fd <0 ) { +g_test_message ("Could not create %s: %s" , tmp_path[i], +strerror (errno)); + goto test_add_done; + } ret =ftruncate (fd,TEST_IMAGE_SIZE ); -g_assert (ret ==0 ); + err = errno;close (fd); +if (ret <0 ) { +g_test_message ("Could not size %s: %s" , tmp_path[i], +strerror (err)); + goto test_add_done; + } } -/* Run the tests */ -g_test_init (&argc, &argv,NULL ); -qtest_add_func ("/ide/read_native" , test_specify);qtest_add_func ("/ide/identify" , test_identify);
@@ -1288,10 +1301,14 @@ int main(int argc, char **argv)qtest_add_func ("/ide/cdrom/pio_raw" , test_cdrom_pio_raw);qtest_add_func ("/ide/cdrom/dma_raw" , test_cdrom_dma_raw); +test_add_done: ret =g_test_run ();/* Cleanup */ for (i =0 ; i <2 ; ++i) { +if (!tmp_path[i]) { +continue ; + }unlink (tmp_path[i]);g_free (tmp_path[i]); }
Signature · 1 line
2.53.0
PATCH 3/7 tests/qtest/ahci-test: skip when the scratch files cannot be created
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
The fallback ahci-test takes when QTEST_QEMU_IMG is unset extends a raw image to 64 MiB with ftruncate() and asserts it worked, which is the assert ide-test just lost. The temporary files are asserted on too, and a file system with no room left fails g_mkstemp() and g_file_open_tmp() as surely as the ftruncate().
Report the reason and register no tests instead. The cleanup path has to cope with the files now created after the point of no return.
--- tests/qtest/ahci-test.c | 48 +++++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 11 deletions(-)
tests/qtest/ahci-test.c +37 -11
@@ -2110,8 +2110,10 @@ static void create_ahci_io_test(enum IOMode type, enum AddrMode addr,int main (int argc,char **argv) { +g_autoptr (GError) error =NULL ;const char *arch, *base;int ret; +int err;int fd;int c;int i, j, k, m;
@@ -2164,30 +2166,49 @@ int main(int argc, char **argv)/* Create a temporary image */ tmp_path =g_strdup_printf ("%s/qtest.XXXXXX" , base); fd =g_mkstemp (tmp_path); -g_assert (fd >=0 ); +if (fd <0 ) { +g_test_message ("Could not create %s: %s" , tmp_path,strerror (errno)); + goto test_add_done; + }if (have_qemu_img ()) { imgfmt ="qcow2" ; test_image_size_mb =TEST_IMAGE_SIZE_MB_LARGE ; -mkqcow2 (tmp_path,TEST_IMAGE_SIZE_MB_LARGE ); +close (fd); +if (!mkqcow2 (tmp_path,TEST_IMAGE_SIZE_MB_LARGE )) { +g_test_message ("Could not create %s with qemu-img" , tmp_path); + goto test_add_done; + } }else {g_test_message ("QTEST_QEMU_IMG not set or qemu-img missing; " "skipping LBA48 high-sector tests" ); imgfmt ="raw" ; test_image_size_mb =TEST_IMAGE_SIZE_MB_SMALL ; ret =ftruncate (fd, test_image_size_mb *1024 *1024 ); -g_assert (ret ==0 ); + err = errno; +close (fd); +if (ret <0 ) { +g_test_message ("Could not size %s: %s" , tmp_path,strerror (err)); + goto test_add_done; + } } -close (fd);/* Create temporary blkdebug instructions */ debug_path =g_strdup_printf ("%s/qtest-blkdebug.XXXXXX" , base); fd =g_mkstemp (debug_path); -g_assert (fd >=0 ); +if (fd <0 ) { +g_test_message ("Could not create %s: %s" , debug_path, +strerror (errno)); + goto test_add_done; + }close (fd);/* Reserve a hollow file to use as a socket for migration tests */ - fd =g_file_open_tmp ("qtest-migration.XXXXXX" , &mig_socket,NULL ); -g_assert (fd >=0 ); + fd =g_file_open_tmp ("qtest-migration.XXXXXX" , &mig_socket, &error); +if (fd <0 ) { +g_test_message ("Could not create a temporary file: %s" , + error->message); + goto test_add_done; + }close (fd);/* Run the tests */
@@ -2244,15 +2265,20 @@ int main(int argc, char **argv)qtest_add_func ("/ahci/cdrom/drain/pio" , test_atapi_drain_pio);qtest_add_func ("/ahci/cdrom/drain/dma" , test_atapi_drain_dma); +test_add_done: ret =g_test_run ();/* Cleanup */ unlink (tmp_path);g_free (tmp_path); -unlink (debug_path); -g_free (debug_path); -unlink (mig_socket); -g_free (mig_socket); +if (debug_path) { +unlink (debug_path); +g_free (debug_path); + } +if (mig_socket) { +unlink (mig_socket); +g_free (mig_socket); + }return ret; }
Signature · 1 line
2.53.0
PATCH 4/7 tests/qtest/hd-geo-test: skip when the scratch file cannot be created
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
Commit a0d6d7454a ("tests/hd-geo-test: Skip test when images can not be created") skips when the 4 GiB image cannot be sized, but the temporary file itself is still asserted on, and a file system with no room left fails g_file_open_tmp() as surely as the ftruncate().
Skip there too and say why in both cases, from inside create_test_img() where the GError is. Unlink the image that could not be sized while here.
--- tests/qtest/hd-geo-test.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
tests/qtest/hd-geo-test.c +12 -4
@@ -27,15 +27,24 @@static char *create_test_img (int secs) { +g_autoptr (GError) error =NULL ;char *template; -int fd, ret; +int fd, ret, err; + + fd =g_file_open_tmp ("qtest.XXXXXX" , &template, &error); +if (fd <0 ) { +g_test_message ("Could not create a temporary file: %s" , + error->message); +return NULL ; + } - fd =g_file_open_tmp ("qtest.XXXXXX" , &template,NULL ); -g_assert (fd >=0 ); ret =ftruncate (fd, (off_t )secs *512 ); + err = errno;close (fd);if (ret) { +g_test_message ("Could not size %s: %s" , template,strerror (err)); +unlink (template);g_free (template); template =NULL ; }
@@ -1059,7 +1068,6 @@ int main(int argc, char **argv)if (img_secs[i] >=0 ) { img_file_name[i] =create_test_img (img_secs[i]);if (!img_file_name[i]) { -g_test_message ("Could not create test images." ); goto test_add_done; } }else {
Signature · 1 line
2.53.0
PATCH 5/7 tests/qtest/libqtest: create images with a byte-precise size
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
mkimg() takes megabytes, which is all its callers have needed so far. ide-test wants an image whose length is an exact multiple of its CHS geometry, 130 * 16 * 63 sectors, and that is not a whole number of megabytes. Add mkimg_bytes() and let mkimg() go through it.
--- tests/qtest/libqtest.c | 9 +++++++-- tests/qtest/libqtest.h | 12 ++++++++++++ 2 files changed, 19 insertions(+), 2 deletions(-)
tests/qtest/libqtest.c +7 -2
@@ -2115,6 +2115,11 @@ bool have_qemu_img(void) }bool mkimg (const char *file,const char *fmt,unsigned size_mb) +{ +return mkimg_bytes (file, fmt, (uint64_t )size_mb *1024 *1024 ); +} + +bool mkimg_bytes (const char *file,const char *fmt,uint64_t size) {gchar *cli;bool ret;
@@ -2133,8 +2138,8 @@ bool mkimg(const char *file, const char *fmt, unsigned size_mb)return false; } - cli =g_strdup_printf ("%s create -f %s %s %uM" , qemu_img_abs_path, - fmt, file, size_mb); + cli =g_strdup_printf ("%s create -f %s %s %" PRIu64, qemu_img_abs_path, + fmt, file, size); ret =g_spawn_command_line_sync (cli, &out, &out2, &rc, &err);if (err || !g_spawn_check_exit_status (rc, &err)) {fprintf (stderr,"%s\n" , err->message);
tests/qtest/libqtest.h +12 -0
@@ -1186,6 +1186,18 @@ bool have_qemu_img(void);*/ bool mkimg (const char *file,const char *fmt,unsigned size_mb); +/** +* mkimg_bytes: +* @file: File name of the image that should be created +* @fmt: Format, e.g. "qcow2" or "raw" +* @size: Size of the image in bytes +* +* As mkimg(), for images whose size is not a whole number of megabytes. +* +* Returns: true if the image has been created successfully. +*/ +bool mkimg_bytes (const char *file,const char *fmt,uint64_t size); +/** * qtest_verbose: * @domain: The logging domain
Signature · 1 line
2.53.0
PATCH 6/7 tests/qtest/ide-test: build the shared disks with qemu-img
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
The two 64 MiB disks are raw and are grown with ftruncate(), which is free only where a sparse file is free. Build them as qcow2 when qemu-img is available, the way ahci-test and hd-geo-test do, keeping the raw ftruncate() as the fallback. Only the virtual size matters to their users: the CHS geometry READ NATIVE MAX ADDRESS reports and the end-of-disk sector the TRIM range check refuses.
Three tests do care what sits below the guest and get a raw 1 MiB image of their own:
- the CD-ROM tests write their ATAPI pattern in with fopen()
- bmdma/trim_reset needs the guest TRIM to reach blkdebug, where a 1 ms delay lets the reset catch the request in flight. qcow2 stops the discard at the format layer, the clusters never having been allocated, and the test would pass while racing nothing
- the two flush tests trip a blkdebug event. Both pass over qcow2, but on raw the event reported is the flush the guest issued
Peak scratch space drops from 129 MiB to 1.6 MiB.
--- tests/qtest/ide-test.c | 69 +++++++++++++++++++++++++++++++++--------- 1 file changed, 54 insertions(+), 15 deletions(-)
tests/qtest/ide-test.c +54 -15
@@ -37,6 +37,12 @@/* Specified by ATA (physical) CHS geometry for ~64 MiB device. */ #define TEST_IMAGE_SIZE ((130 * 16 * 63) * 512) +/* +* The tests that reach the image behind QEMU's back, or that need an +* event to reach blkdebug, get a raw one of their own. +*/ +#define RAW_IMAGE_SIZE (1 * 1024 * 1024) +#define IDE_PCI_DEV 1#define IDE_PCI_FUNC 1
@@ -131,7 +137,9 @@ static QPCIBus *pcibus = NULL;static QGuestAllocator guest_malloc;static char *tmp_path[2 ]; +static char *raw_path;static char *debug_path; +static const char *imgfmt ="raw" ; G_GNUC_PRINTF(1, 2)static QTestState *ide_test_start (const char *cmdline_fmt, ...)
@@ -348,9 +356,9 @@ static QTestState *test_bmdma_setup(void)QTestState *qts; qts =ide_test_start ( -"-drive file=%s,if=ide,cache=writeback,format=raw " +"-drive file=%s,if=ide,cache=writeback,format=%s " "-global ide-hd.serial=%s -global ide-hd.ver=%s" , - tmp_path[0 ],"testdisk" ,"version" ); + tmp_path[0 ], imgfmt,"testdisk" ,"version" );qtest_irq_intercept_in (qts,"ioapic" );return qts;
@@ -506,7 +514,7 @@ static void test_bmdma_trim_reset(void)"inject-error.0.event=none,inject-error.0.iotype=discard," "inject-error.0.errno=0,inject-error.0.delay-ns=1000000 " "-device ide-hd,drive=dbg,bus=ide.0" , - tmp_path[0]); + raw_path);qtest_irq_intercept_in (qts,"ioapic" ); guest_buf =guest_alloc (&guest_malloc, len);
@@ -664,9 +672,9 @@ static void test_specify(void)uint8_t heads, spt; qts =ide_test_start ( -"-blockdev driver=file,node-name=hda,filename=%s " +"-drive if=none,id=hda,format=%s,file=%s " "-device ide-hd,drive=hda,bus=ide.0,unit=0 " , - tmp_path[0 ]); + imgfmt, tmp_path[0 ]); dev =get_pci_device (qts, &bmdma_bar, &ide_bar);
@@ -706,9 +714,9 @@ static void test_identify(void)int ret; qts =ide_test_start ( -"-drive file=%s,if=ide,cache=writeback,format=raw " +"-drive file=%s,if=ide,cache=writeback,format=%s " "-global ide-hd.serial=%s -global ide-hd.ver=%s" , - tmp_path[0 ],"testdisk" ,"version" ); + tmp_path[0 ], imgfmt,"testdisk" ,"version" ); dev =get_pci_device (qts, &bmdma_bar, &ide_bar);
@@ -756,11 +764,11 @@ static void test_diagnostic(void)uint8_t data; qts =ide_test_start ( -"-blockdev driver=file,node-name=hda,filename=%s " -"-blockdev driver=file,node-name=hdb,filename=%s " +"-drive if=none,id=hda,format=%s,file=%s " +"-drive if=none,id=hdb,format=%s,file=%s " "-device ide-hd,drive=hda,bus=ide.0,unit=0 " "-device ide-hd,drive=hdb,bus=ide.0,unit=1 " , - tmp_path[0 ], tmp_path[1 ]); + imgfmt, tmp_path[0 ], imgfmt, tmp_path[1 ]); dev =get_pci_device (qts, &bmdma_bar, &ide_bar);
@@ -826,7 +834,7 @@ static void test_flush(void) qts =ide_test_start ("-drive file=blkdebug::%s,if=ide,cache=writeback,format=raw" , - tmp_path[0 ]); + raw_path); dev =get_pci_device (qts, &bmdma_bar, &ide_bar);
@@ -877,7 +885,7 @@ static void test_pci_retry_flush(void) qts =ide_test_start ("-drive file=blkdebug:%s:%s,if=ide,cache=writeback,format=raw," "rerror=stop,werror=stop" , - debug_path, tmp_path[0 ]); + debug_path, raw_path); dev =get_pci_device (qts, &bmdma_bar, &ide_bar);
@@ -1100,14 +1108,14 @@ static void cdrom_read_impl(int nblocks, unsigned flags)/* Prepopulate the CDROM with an interesting pattern */ generate_pattern (pattern, patt_len,ATAPI_BLOCK_SIZE ); - fh =fopen (tmp_path[0 ],"wb+" ); + fh =fopen (raw_path,"wb+" ); ret =fwrite (pattern,ATAPI_BLOCK_SIZE , patt_blocks, fh); g_assert_cmpint(ret, ==, patt_blocks);fclose (fh); qts =ide_test_start ("-drive if=none,file=%s,media=cdrom,format=raw,id=sr0,index=0 " -"-device ide-cd,drive=sr0,bus=ide.0" , tmp_path[0 ]); +"-device ide-cd,drive=sr0,bus=ide.0" , raw_path); dev =get_pci_device (qts, &bmdma_bar, &ide_bar);qtest_irq_intercept_in (qts,"ioapic" );
@@ -1258,7 +1266,11 @@ int main(int argc, char **argv) }close (fd); -/* Create a temporary raw image */ +if (have_qemu_img ()) { + imgfmt ="qcow2" ; + } + +/* Create the disks the tests share */ for (i =0 ; i <2 ; ++i) { tmp_path[i] =g_strdup_printf ("%s/qtest.XXXXXX" , base); fd =g_mkstemp (tmp_path[i]);
@@ -1267,6 +1279,15 @@ int main(int argc, char **argv) strerror(errno)); goto test_add_done; } +if (!strcmp (imgfmt,"qcow2" )) { +close (fd); +if (!mkimg_bytes (tmp_path[i], imgfmt,TEST_IMAGE_SIZE )) { +g_test_message ("Could not create %s with qemu-img" , + tmp_path[i]); + goto test_add_done; + } +continue ; + } ret =ftruncate (fd,TEST_IMAGE_SIZE ); err = errno;close (fd);
@@ -1277,6 +1298,20 @@ int main(int argc, char **argv) } } + raw_path =g_strdup_printf ("%s/qtest.XXXXXX" , base); + fd =g_mkstemp (raw_path); +if (fd <0 ) { +g_test_message ("Could not create %s: %s" , raw_path,strerror (errno)); + goto test_add_done; + } + ret =ftruncate (fd,RAW_IMAGE_SIZE ); + err = errno; +close (fd); +if (ret <0 ) { +g_test_message ("Could not size %s: %s" , raw_path,strerror (err)); + goto test_add_done; + } +qtest_add_func ("/ide/read_native" , test_specify);qtest_add_func ("/ide/identify" , test_identify);
@@ -1312,6 +1347,10 @@ test_add_done:unlink (tmp_path[i]);g_free (tmp_path[i]); } +if (raw_path) { +unlink (raw_path); +g_free (raw_path); + }unlink (debug_path);g_free (debug_path);
Signature · 1 line
2.53.0
PATCH 7/7 tests/qtest/hd-geo-test: build the test images with qemu-img
In reply to John SnowFrom: Denis V. Lunev <[email protected]>
The two backing images are raw and are grown with ftruncate(), one to 30 MiB and one to 4 GiB, so on a host where sparse files are not free the test skips and the BIOS geometry goes untested.
The override tests already build their images with qemu-img, so use the same route here. The convert and resize pair moves into mkqcow2_with_mbr(), which both users call and which reports a failed qemu-img rather than asserting. setup_mbr() stamps the MBR by rebuilding the image, that being the only way to reach sector 0 of a qcow2 from outside QEMU. Only the setup running before any test can skip, so the rest keep asserting. Without qemu-img the images stay raw.
Peak scratch space drops from 4126 MiB to 1.5 MiB.
--- tests/qtest/hd-geo-test.c | 129 +++++++++++++++++++++++++------------- 1 file changed, 86 insertions(+), 43 deletions(-)
tests/qtest/hd-geo-test.c +86 -43
@@ -25,9 +25,69 @@#define ARGV_SIZE 256 +#define MBR_SIZE 512 + +static const char *imgfmt ="raw" ; + +static bool mkqcow2_with_mbr (const char *path,const uint8_t *mbr, +uint64_t size) +{ +g_autoptr (GError) error =NULL ; + g_autofree char *raw_path = NULL; +char cmd[100 +2 *PATH_MAX ]; +char *qemu_img_path; +char *qemu_img_abs_path =NULL ; +bool ok = false; +int fd, ret; + + fd =g_file_open_tmp ("qtest.XXXXXX" , &raw_path, &error); +if (fd <0 ) { +g_test_message ("Could not create a temporary file: %s" , + error->message); +return false; + } + ret =write (fd, mbr,MBR_SIZE ); +g_assert (ret ==MBR_SIZE ); +close (fd); + + qemu_img_path =getenv ("QTEST_QEMU_IMG" ); +g_assert (qemu_img_path); + qemu_img_abs_path =realpath (qemu_img_path,NULL ); +g_assert (qemu_img_abs_path); + + ret =snprintf (cmd,sizeof (cmd), +"%s convert -f raw -O qcow2 %s %s > /dev/null" , + qemu_img_abs_path, + raw_path, path); +g_assert ((0 < ret) && (ret <=sizeof (cmd))); +if (system (cmd) !=0 ) { +g_test_message ("Could not create %s with qemu-img" , path); + goto out; + } + + ret =snprintf (cmd,sizeof (cmd), +"%s resize %s %" PRIu64" > /dev/null" , + qemu_img_abs_path, + path, size); +g_assert ((0 < ret) && (ret <=sizeof (cmd))); +if (system (cmd) !=0 ) { +g_test_message ("Could not size %s with qemu-img" , path); + goto out; + } + + ok = true; + +out: +free (qemu_img_abs_path); +unlink (raw_path); + +return ok; +} +static char *create_test_img (int secs) {g_autoptr (GError) error =NULL ; +uint8_t blank_mbr[MBR_SIZE ] = {};char *template;int fd, ret, err;
@@ -38,6 +98,16 @@ static char *create_test_img(int secs)return NULL ; } +if (!strcmp (imgfmt,"qcow2" )) { +close (fd); +if (!mkqcow2_with_mbr (template, blank_mbr, (uint64_t )secs *512 )) { +unlink (template); +g_free (template); +return NULL ; + } +return template; + } + ret =ftruncate (fd, (off_t )secs *512 ); err = errno;close (fd);
@@ -207,7 +277,7 @@ static void setup_mbr(int img_idx, MBRcontents mbr)/* chs 0,1,1 (lba 63) to chs 7,15,63 (8001 sectors) */ 0x80, 1, 1, 0, 6, 15, 63, 7, 63, 0, 0, 0, 0x41, 0x1F, 0, 0, }; -uint8_t buf[512 ]; +uint8_t buf[MBR_SIZE ];int fd, ret;memset (buf,0 ,sizeof (buf));
@@ -218,6 +288,12 @@ static void setup_mbr(int img_idx, MBRcontents mbr)memcpy (buf +0x1BE , mbr == mbr_lba ? part_lba : part_chs,16 ); } +if (!strcmp (imgfmt,"qcow2" )) { +g_assert_true (mkqcow2_with_mbr (img_file_name[img_idx], buf, + (uint64_t )img_secs[img_idx] *512 )); +return ; + } + fd =open (img_file_name[img_idx],O_WRONLY );g_assert (fd >=0 ); ret =write (fd, buf,sizeof (buf));
@@ -237,7 +313,8 @@ static int setup_ide(int argc, char *argv[], int argv_sz,if (img_secs[img_idx] >=0 ) {setup_mbr (img_idx, mbr); - s3 =g_strdup_printf (",format=raw,file=%s" , img_file_name[img_idx]); + s3 =g_strdup_printf (",format=%s,file=%s" , imgfmt, + img_file_name[img_idx]); }else { s3 =g_strdup (",media=cdrom" ); }
@@ -431,15 +508,10 @@ static MBRpartitions empty_mbr = { {false, 0, 0, 0, 0, 0, 0, 0, 0},static char *create_qcow2_with_mbr (MBRpartitions mbr,uint64_t sectors) { - g_autofree char *raw_path = NULL;char *qcow2_path; -char cmd[100 +2 *PATH_MAX ]; -uint8_t buf[512 ] = {}; -int i, ret, fd, offset; -uint64_t qcow2_size = sectors *512 ; +uint8_t buf[MBR_SIZE ] = {}; +int i, fd, offset;uint8_t status, parttype, head, sector, cyl; -char *qemu_img_path; -char *qemu_img_abs_path; offset =0xbe ;
@@ -476,44 +548,11 @@ static char *create_qcow2_with_mbr(MBRpartitions mbr, uint64_t sectors) offset +=0x10 ; } - fd =g_file_open_tmp ("qtest.XXXXXX" , &raw_path,NULL ); -g_assert (fd >=0 ); -close (fd); - - fd =open (raw_path,O_WRONLY ); -g_assert (fd >=0 ); - ret =write (fd, buf,sizeof (buf)); -g_assert (ret ==sizeof (buf)); -close (fd); - fd =g_file_open_tmp ("qtest.XXXXXX" , &qcow2_path,NULL );g_assert (fd >=0 );close (fd); - qemu_img_path =getenv ("QTEST_QEMU_IMG" ); -g_assert (qemu_img_path); - qemu_img_abs_path =realpath (qemu_img_path,NULL ); -g_assert (qemu_img_abs_path); - - ret =snprintf (cmd,sizeof (cmd), -"%s convert -f raw -O qcow2 %s %s > /dev/null" , - qemu_img_abs_path, - raw_path, qcow2_path); -g_assert ((0 < ret) && (ret <=sizeof (cmd))); - ret =system (cmd); -g_assert (ret ==0 ); - - ret =snprintf (cmd,sizeof (cmd), -"%s resize %s %" PRIu64" > /dev/null" , - qemu_img_abs_path, - qcow2_path, qcow2_size); -g_assert ((0 < ret) && (ret <=sizeof (cmd))); - ret =system (cmd); -g_assert (ret ==0 ); - -free (qemu_img_abs_path); - -unlink (raw_path); +g_assert_true (mkqcow2_with_mbr (qcow2_path, buf, sectors *512 ));return qcow2_path; }
@@ -1064,6 +1103,10 @@ int main(int argc, char **argv)g_test_init (&argc, &argv,NULL ); +if (have_qemu_img ()) { + imgfmt ="qcow2" ; + } +for (i =0 ; i < backend_last; i++) {if (img_secs[i] >=0 ) { img_file_name[i] =create_test_img (img_secs[i]);
Signature · 1 line
2.53.0