Enable more iotests during "make check-block"
v4 cover Enable more iotests during "make check-block"
As discussed here:
https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html
and here:
https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html
it would be good to have some more valuable iotests enabled in the "auto" group to get better iotest coverage during "make check".
Since these Python-based tests require a QEMU that features a 'virtio-blk' device, we can only run the Python tests if this device is available. With binaries like qemu-system-tricore, the Python-based tests will be skipped.
v4:
- The check for 'virtio-blk' is now done in the tests/qemu-iotests/check script instead of tests/check-block.sh (to avoid to duplicate the code that searches for the right QEMU binary - and we can also still run the shell-based tests this way).
- Added the new patch to check for the availability of virtio devices in the iotests 127 and 267.
- The patch that drops test 130 from the "auto" group has already been merged and thus been dropped from this series.
v3:
- Test 183 fails on Patchew, so I removed it from the "auto" group again
v2:
- Checked the iotests with NetBSD, too (now that Eduardo has re-activated Gerd's patches for creating NetBSD VM images)
- Use 'openbsd' instead of 'openbsd6'
- Use 'grep -q' instead of 'grep' for grep'ing silently
- Added the patch to disable 130 from the "auto" group
John Snow (1):
iotests: remove 'linux' from default supported platforms
Thomas Huth (5):
iotests: Test 041 only works on certain systems
iotests: Test 183 does not work on macOS and OpenBSD
iotests: Check for the availability of the required devices in 267 and
127
iotests: Skip Python-based tests if QEMU does not support virtio-blk
iotests: Enable more tests in the 'auto' group to improve test
coverage
tests/qemu-iotests/041 | 3 ++- tests/qemu-iotests/127 | 2 ++ tests/qemu-iotests/183 | 1 + tests/qemu-iotests/267 | 2 ++ tests/qemu-iotests/check | 8 ++++++-- tests/qemu-iotests/common.rc | 14 ++++++++++++++ tests/qemu-iotests/group | 14 +++++++------- tests/qemu-iotests/iotests.py | 16 +++++++++++----- 8 files changed, 45 insertions(+), 15 deletions(-)
Signature · 1 line
2.18.1
PATCH v4 1/6 iotests: remove 'linux' from default supported platforms
From: John Snow <[email protected]>
verify_platform will check an explicit whitelist and blacklist instead. The default will now be assumed to be allowed to run anywhere.
For tests that do not specify their platforms explicitly, this has the effect of enabling these tests on non-linux platforms. For tests that always specified linux explicitly, there is no change.
For Python tests on FreeBSD at least; only seven python tests fail:
045 147 149 169 194 199 211
045 and 149 appear to be misconfigurations, 147 and 194 are the AF_UNIX path too long error, 169 and 199 are bitmap migration bugs, and 211 is a bug that shows up on Linux platforms, too.
This is at least good evidence that these tests are not Linux-only. If they aren't suitable for other platforms, they should be disabled on a per-platform basis as appropriate.
Therefore, let's switch these on and deal with the failures.
--- tests/qemu-iotests/iotests.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
tests/qemu-iotests/iotests.py +11 -5
@@ -896,9 +896,14 @@ def verify_protocol(supported=[], unsupported=[]): if not_sup or (imgproto in unsupported): notrun('not suitable for this protocol: %s' % imgproto) -def verify_platform(supported_oses=['linux']): - if True not in [sys.platform.startswith(x) for x in supported_oses]: - notrun('not suitable for this OS: %s' % sys.platform) +def verify_platform(supported=None, unsupported=None): + if unsupported is not None: + if any((sys.platform.startswith(x) for x in unsupported)): + notrun('not suitable for this OS: %s' % sys.platform) + + if supported is not None: + if not any((sys.platform.startswith(x) for x in supported)): + notrun('not suitable for this OS: %s' % sys.platform) def verify_cache_mode(supported_cache_modes=[]): if supported_cache_modes and (cachemode not in supported_cache_modes):
@@ -989,7 +994,8 @@ def execute_unittest(output, verbosity, debug): sys.stderr.write(out) def execute_test(test_function=None, - supported_fmts=[], supported_oses=['linux'], + supported_fmts=[], + supported_platforms=None, supported_cache_modes=[], unsupported_fmts=[], supported_protocols=[], unsupported_protocols=[]): """Run either unittest or script-style tests."""
@@ -1006,7 +1012,7 @@ def execute_test(test_function=None, verbosity = 1 verify_image_format(supported_fmts, unsupported_fmts) verify_protocol(supported_protocols, unsupported_protocols) - verify_platform(supported_oses) + verify_platform(supported=supported_platforms) verify_cache_mode(supported_cache_modes) if debug:
Signature · 1 line
2.18.1
PATCH v4 2/6 iotests: Test 041 only works on certain systems
In reply to Thomas Huth041 works fine on Linux, FreeBSD, NetBSD and OpenBSD, but fails on macOS. Let's mark it as only supported on the systems where we know that it is working fine.
--- tests/qemu-iotests/041 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tests/qemu-iotests/041 +2 -1
@@ -1123,4 +1123,5 @@ class TestOrphanedSource(iotests.QMPTestCase): if __name__ == '__main__': iotests.main(supported_fmts=['qcow2', 'qed'], - supported_protocols=['file']) + supported_protocols=['file'], + supported_platforms=['linux', 'freebsd', 'netbsd', 'openbsd'])
Signature · 1 line
2.18.1
Re: [PATCH v4 2/6] iotests: Test 041 only works on certain systems
Quoted 8 lines · Thomas Huth
041 works fine on Linux, FreeBSD, NetBSD and OpenBSD, but fails on macOS. Let's mark it as only supported on the systems where we know that it is working fine.
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/041 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
PATCH v4 3/6 iotests: Test 183 does not work on macOS and OpenBSD
In reply to Thomas HuthIn the long term, we might want to add test 183 to the "auto" group (but it still fails occasionally, so we cannot do that yet). However, when running 183 in Cirrus-CI on macOS, or with our vm-build-openbsd target, it currently always fails with an "Timeout waiting for return on handle 0" error.
Let's mark it as supported only on systems where the test is working fine (i.e. Linux, FreeBSD and NetBSD).
--- tests/qemu-iotests/183 | 1 + 1 file changed, 1 insertion(+)
tests/qemu-iotests/183 +1 -0
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter . ./common.qemu +_supported_os Linux FreeBSD NetBSD _supported_fmt qcow2 raw qed quorum _supported_proto file
Signature · 1 line
2.18.1
Re: [PATCH v4 3/6] iotests: Test 183 does not work on macOS and OpenBSD
Quoted 19 lines · Thomas Huth
In the long term, we might want to add test 183 to the "auto" group (but it still fails occasionally, so we cannot do that yet). However, when running 183 in Cirrus-CI on macOS, or with our vm-build-openbsd target, it currently always fails with an "Timeout waiting for return on handle 0" error.
Let's mark it as supported only on systems where the test is working fine (i.e. Linux, FreeBSD and NetBSD).
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/183 | 1 + 1 file changed, 1 insertion(+)tests/qemu-iotests/183 +0
-0
tests/qemu-iotests/183 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 45
Quoted 4 lines
tests/qemu-iotests/183 +1
-0@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter . ./common.qemu +_supported_os Linux FreeBSD NetBSD
I don’t suppose you have data on OpenBSD?
(And maybe it would be best to let _unsupported_os just check whether $HOSTOS contains any of these strings (ideally in a case-insensitive way) instead of checking for equality, but that’s for another series to decide.)
Re: [PATCH v4 3/6] iotests: Test 183 does not work on macOS and OpenBSD
Quoted 23 lines · Max Reitz
Quoted 22 lines · Thomas Huth
In the long term, we might want to add test 183 to the "auto" group (but it still fails occasionally, so we cannot do that yet). However, when running 183 in Cirrus-CI on macOS, or with our vm-build-openbsd target, it currently always fails with an "Timeout waiting for return on handle 0" error.
Let's mark it as supported only on systems where the test is working fine (i.e. Linux, FreeBSD and NetBSD).
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/183 | 1 + 1 file changed, 1 insertion(+)tests/qemu-iotests/183 +0
-0@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter . ./common.qemu
Quoted 3 lines
tests/qemu-iotests/183 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 45
Quoted 1 line
tests/qemu-iotests/183 +1
-0+_supported_os Linux FreeBSD NetBSDI don’t suppose you have data on OpenBSD?
I've checked it with our "make vm-build-openbsd" target (see patch description) and it was failing there in the same way as on macOS.
Quoted 1 line
Reviewed-by Max Reitz <[email protected]>
Thanks!
Thomas
PATCH v4 4/6 iotests: Check for the availability of the required devices in 267 and 127
In reply to Thomas HuthWe are going to enable 127 in the "auto" group, but it only works if virtio-scsi and scsi-hd are available - which is not the case with QEMU binaries like qemu-system-tricore for example, so we need a proper check for the availability of these devices here.
A very similar problem exists in iotest 267 - it has been added to the "auto" group already, but requires virtio-blk and thus currently fails with qemu-system-tricore for example. Let's also add aproper check there.
--- tests/qemu-iotests/127 | 2 ++ tests/qemu-iotests/267 | 2 ++ tests/qemu-iotests/common.rc | 14 ++++++++++++++ 3 files changed, 18 insertions(+)
tests/qemu-iotests/127 +2 -0
@@ -43,6 +43,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file +_require_devices virtio-scsi scsi-hd + IMG_SIZE=64K _make_test_img $IMG_SIZE
tests/qemu-iotests/267 +2 -0
@@ -44,6 +44,8 @@ _supported_os Linux # Internal snapshots are (currently) impossible with refcount_bits=1 _unsupported_imgopts 'refcount_bits=1[^0-9]' +_require_devices virtio-blk + do_run_qemu() { echo Testing: "$@"
tests/qemu-iotests/common.rc +14 -0
@@ -643,5 +643,19 @@ _require_drivers() done } +# Check that a set of devices is available in the QEMU binary +# +_require_devices() +{ + available=$($QEMU -M none -device help | \ + grep ^name | sed -e 's/^name "//' -e 's/".*$//') + for device + do + if ! echo "$available" | grep -q "$device" ; then + _notrun "$device not available" + fi + done +} + # make sure this script returns success true
Signature · 1 line
2.18.1
Re: [PATCH v4 4/6] iotests: Check for the availability of the required devices in 267 and 127
Quoted 10 lines · Thomas Huth
We are going to enable 127 in the "auto" group, but it only works if virtio-scsi and scsi-hd are available - which is not the case with QEMU binaries like qemu-system-tricore for example, so we need a proper check for the availability of these devices here.
A very similar problem exists in iotest 267 - it has been added to the "auto" group already, but requires virtio-blk and thus currently fails with qemu-system-tricore for example. Let's also add aproper check there.
Quoted 6 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/127 | 2 ++ tests/qemu-iotests/267 | 2 ++ tests/qemu-iotests/common.rc | 14 ++++++++++++++ 3 files changed, 18 insertions(+)
PATCH v4 5/6 iotests: Skip Python-based tests if QEMU does not support virtio-blk
In reply to Thomas HuthWe are going to enable some of the python-based tests in the "auto" group, and these tests require virtio-blk to work properly. Running iotests without virtio-blk likely does not make too much sense anyway, so instead of adding a check for the availability of virtio-blk to each and every test (which does not sound very appealing), let's rather add a check for this a central spot in the "check" script instead (so that it is still possible to run "make check" for qemu-system-tricore for example).
--- tests/qemu-iotests/check | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
tests/qemu-iotests/check +6 -2
@@ -642,7 +642,11 @@ fi python_usable=false if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)' then - python_usable=true + # Our python framework also requires virtio-blk + if "$QEMU_PROG" -M none -device help | grep -q virtio-blk >/dev/null 2>&1 + then + python_usable=true + fi fi default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
@@ -830,7 +834,7 @@ do run_command="$PYTHON $seq" else run_command="false" - echo "Unsupported Python version" > $seq.notrun + echo "Unsupported Python version or missing virtio-blk" > $seq.notrun fi else run_command="./$seq"
Signature · 1 line
2.18.1
Re: [PATCH v4 5/6] iotests: Skip Python-based tests if QEMU does not support virtio-blk
Quoted 31 lines · Thomas Huth
We are going to enable some of the python-based tests in the "auto" group, and these tests require virtio-blk to work properly. Running iotests without virtio-blk likely does not make too much sense anyway, so instead of adding a check for the availability of virtio-blk to each and every test (which does not sound very appealing), let's rather add a check for this a central spot in the "check" script instead (so that it is still possible to run "make check" for qemu-system-tricore for example).
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/check | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)tests/qemu-iotests/check +5
-1@@ -642,7 +642,11 @@ fi python_usable=false if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)' then - python_usable=true + # Our python framework also requires virtio-blk + if "$QEMU_PROG" -M none -device help | grep -q virtio-blk >/dev/null 2>&1 + then + python_usable=true + fi fi default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
tests/qemu-iotests/check do · line 837
Quoted 5 lines
tests/qemu-iotests/check +1
-1@@ -830,7 +834,7 @@ do run_command="$PYTHON $seq" else run_command="false" - echo "Unsupported Python version" > $seq.notrun + echo "Unsupported Python version or missing virtio-blk" > $seq.notrun
A $python_unusable_because might be helpful (set in to-be-added else branches for the ifs that set python_usable to true), but either way:
Quoted 4 lines
tests/qemu-iotests/check +0
-0fi else run_command="./$seq"
Re: [PATCH v4 5/6] iotests: Skip Python-based tests if QEMU does not support virtio-blk
Quoted 30 lines · Max Reitz
Quoted 29 lines · Thomas Huth
We are going to enable some of the python-based tests in the "auto" group, and these tests require virtio-blk to work properly. Running iotests without virtio-blk likely does not make too much sense anyway, so instead of adding a check for the availability of virtio-blk to each and every test (which does not sound very appealing), let's rather add a check for this a central spot in the "check" script instead (so that it is still possible to run "make check" for qemu-system-tricore for example).
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/check | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)tests/qemu-iotests/check +5
-1@@ -642,7 +642,11 @@ fi python_usable=false if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)' then - python_usable=true + # Our python framework also requires virtio-blk + if "$QEMU_PROG" -M none -device help | grep -q virtio-blk >/dev/null 2>&1 + then + python_usable=true + fi fi
Quoted 10 lines
tests/qemu-iotests/check do · line 837
Quoted 7 lines
tests/qemu-iotests/check +1
-1default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')@@ -830,7 +834,7 @@ do run_command="$PYTHON $seq" else run_command="false" - echo "Unsupported Python version" > $seq.notrun + echo "Unsupported Python version or missing virtio-blk" > $seq.notrunA $python_unusable_because might be helpful (set in to-be-added else branches for the ifs that set
python_usableto true), but either way:
Sounds like a good idea, I'll give it a try and send a v5.
Thanks,
Thomas
PATCH v4 6/6 iotests: Enable more tests in the 'auto' group to improve test coverage
In reply to Thomas HuthAccording to Kevin, tests 030, 040 and 041 are among the most valuable tests that we have, so we should always run them if possible, even if they take a little bit longer.
According to Max, it would be good to have a test for iothreads and migration. 127 and 256 seem to be good candidates for iothreads. For migration, let's enable 181 and 203 (which also tests iothreads). (091 would be a good candidate for migration, too, but Alex Bennée reported that this test fails on ZFS file systems, so it can't be included yet)
--- tests/qemu-iotests/group | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
tests/qemu-iotests/group +7 -7
@@ -51,7 +51,7 @@ 027 rw auto quick 028 rw backing quick 029 rw auto quick -030 rw backing +030 rw auto backing 031 rw auto quick 032 rw auto quick 033 rw auto quick
@@ -61,8 +61,8 @@ 037 rw auto backing quick 038 rw auto backing quick 039 rw auto quick -040 rw -041 rw backing +040 rw auto +041 rw auto backing 042 rw auto quick 043 rw auto backing 044 rw
@@ -148,7 +148,7 @@ 124 rw backing 125 rw 126 rw auto backing -127 rw backing quick +127 rw auto backing quick 128 rw quick 129 rw quick 130 rw quick
@@ -197,7 +197,7 @@ 177 rw auto quick 178 img 179 rw auto quick -181 rw migration +181 rw auto migration 182 rw quick 183 rw migration 184 rw auto quick
@@ -218,7 +218,7 @@ 200 rw 201 rw migration 202 rw quick -203 rw migration +203 rw auto migration 204 rw quick 205 rw quick 206 rw
@@ -270,7 +270,7 @@ 253 rw quick 254 rw backing quick 255 rw quick -256 rw quick +256 rw auto quick 257 rw 258 rw quick 260 rw quick
Signature · 1 line
2.18.1
Re: [PATCH v4 6/6] iotests: Enable more tests in the 'auto' group to improve test coverage
tests/qemu-iotests/group
Quoted 21 lines · Thomas Huth
According to Kevin, tests 030, 040 and 041 are among the most valuable tests that we have, so we should always run them if possible, even if they take a little bit longer.
According to Max, it would be good to have a test for iothreads and migration. 127 and 256 seem to be good candidates for iothreads. For migration, let's enable 181 and 203 (which also tests iothreads). (091 would be a good candidate for migration, too, but Alex Bennée reported that this test fails on ZFS file systems, so it can't be included yet)
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/group | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)tests/qemu-iotests/group +0
-0@@ -51,7 +51,7 @@
Quoted 15 lines · Thomas Huth
As discussed here:
https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg00697.html
and here:
https://lists.gnu.org/archive/html/qemu-devel/2019-10/msg01388.html
it would be good to have some more valuable iotests enabled in the "auto" group to get better iotest coverage during "make check".
Since these Python-based tests require a QEMU that features a 'virtio-blk' device, we can only run the Python tests if this device is available. With binaries like qemu-system-tricore, the Python-based tests will be skipped.
Quoted 9 lines
v4:
- The check for 'virtio-blk' is now done in the tests/qemu-iotests/check script instead oftests/check-block.sh(to avoid to duplicate the code that searches for the right QEMU binary - and we can also still run the shell-based tests this way).
- Added the new patch to check for the availability of virtio devices in the iotests 127 and 267.
- The patch that drops test 130 from the "auto" group has already been merged and thus been dropped from this series.
Looks good to me, the only question is whether you’d like to add a $python_unusable_because in patch 5 or not.
Max