iotests: Check for enabled drivers before testing them
PATCH 1/1 iotests: Check for enabled drivers before testing them
It is possible to enable only a subset of the block drivers with the "--block-drv-rw-whitelist" option of the "configure" script. All other drivers are marked as unusable (or only included as read-only with the "--block-drv-ro-whitelist" option). If an iotest is now using such a disabled block driver, it is failing - which is bad, since at least the tests in the "auto" group should be able to deal with this situation. Thus let's introduce a "_require_drivers" function that can be used by the shell tests to check for the availability of certain drivers first, and marks the test as "not run" if one of the drivers is missing.
--- tests/qemu-iotests/071 | 1 + tests/qemu-iotests/081 | 1 + tests/qemu-iotests/099 | 1 + tests/qemu-iotests/184 | 1 + tests/qemu-iotests/common.rc | 13 +++++++++++++ 5 files changed, 17 insertions(+)
tests/qemu-iotests/071 +1 -0
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file +_require_drivers blkdebug blkverify do_run_qemu() {
tests/qemu-iotests/081 +1 -0
@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw _supported_proto file _supported_os Linux +_require_drivers quorum do_run_qemu() {
tests/qemu-iotests/099 +1 -0
@@ -42,6 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc _supported_proto file _supported_os Linux +_require_drivers blkdebug blkverify _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \ "subformat=twoGbMaxExtentSparse"
tests/qemu-iotests/184 +1 -0
@@ -33,6 +33,7 @@ trap "exit \$status" 0 1 2 3 15 . ./common.filter _supported_os Linux +_require_drivers throttle do_run_qemu() {
tests/qemu-iotests/common.rc +13 -0
@@ -520,5 +520,18 @@ _require_command() [ -x "$c" ] || _notrun "$1 utility required, skipped this test" } +# Check that a set of drivers has been whitelisted in the QEMU binary +# +_require_drivers() +{ + available=$($QEMU -drive format=help | grep 'Supported formats:') + for driver + do + if ! echo "$available" | grep -q "$driver"; then + _notrun "$driver not available" + fi + done +} + # make sure this script returns success true
Signature · 1 line
2.18.1
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed build test on s390x host. Please find the details below.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with # HEAD pointing to a commit that has the patches applied on top of "base"
# branch set -e
echo
echo "=== ENV ==="
env
echo
echo "=== PACKAGES ==="
rpm -qa
echo
echo "=== UNAME ==="
uname -a
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1 make install === TEST SCRIPT END ===
CC mips-softmmu/trace/control-target.o CC mips-softmmu/trace/generated-helpers.o LINK mips-softmmu/qemu-system-mips
collect2: error: ld returned 1 exit status make[1]: *** [Makefile:209: qemu-system-mipsel] Error 1
make: *** [Makefile:472: mipsel-softmmu/all] Error 2
make: *** Waiting for unfinished jobs....
---
CC hppa-softmmu/hw/block/vhost-user-blk.o CC microblaze-softmmu/balloon.o CC hppa-softmmu/hw/block/dataplane/virtio-blk.o
collect2: error: ld returned 1 exit status make[1]: *** [Makefile:209: qemu-system-mips] Error 1
make: *** [Makefile:472: mips-softmmu/all] Error 2
CC hppa-softmmu/hw/char/virtio-serial-bus.o
---
CC microblaze-softmmu/trace/generated-helpers.o LINK microblaze-softmmu/qemu-system-microblaze LINK hppa-softmmu/qemu-system-hppa
collect2: error: ld returned 1 exit status make[1]: *** [Makefile:209: qemu-system-microblaze] Error 1
make: *** [Makefile:472: microblaze-softmmu/all] Error 2
collect2: error: ld returned 1 exit status make[1]: *** [Makefile:209: qemu-system-hppa] Error 1
make: *** [Makefile:472: hppa-softmmu/all] Error 2 === OUTPUT END ===
The full log is available at
http://patchew.org/logs/[email protected]/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Quoted 9 lines · Thomas Huth
It is possible to enable only a subset of the block drivers with the "--block-drv-rw-whitelist" option of the "configure" script. All other drivers are marked as unusable (or only included as read-only with the "--block-drv-ro-whitelist" option). If an iotest is now using such a disabled block driver, it is failing - which is bad, since at least the tests in the "auto" group should be able to deal with this situation. Thus let's introduce a "
_require_drivers" function that can be used by the shell tests to check for the availability of certain drivers first, and marks the test as "not run" if one of the drivers is missing.
Well, the reasoning for generally not making blkdebug/blkverify explicit requirements was that you should just have both enabled when running iotests.
Of course, that no longer works as an argument now that we unconditionally run some iotests in make check.
But still, the question is how strict you want to be. If blkdebug cannot be assumed to be present, what about null-co? What about raw?
Quoted 14 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/071 | 1 + tests/qemu-iotests/081 | 1 + tests/qemu-iotests/099 | 1 + tests/qemu-iotests/184 | 1 + tests/qemu-iotests/common.rc | 13 +++++++++++++ 5 files changed, 17 insertions(+)tests/qemu-iotests/071 +0
-0
tests/qemu-iotests/071 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 41
Quoted 4 lines
tests/qemu-iotests/071 +1
-0@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file +_require_drivers blkdebug blkverify
Because this test also requires the raw driver.
tests/qemu-iotests/081 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 44
Quoted 12 lines
tests/qemu-iotests/071 +0
-0do_run_qemu() {tests/qemu-iotests/081 +1
-0@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw _supported_proto file _supported_os Linux +_require_drivers quorum
This test has already a check whether quorum is supported, that should be removed now.
(Also, this test requires the raw driver.)
Quoted 2 lines
tests/qemu-iotests/081 +0
-0do_run_qemu() {
[...]
tests/qemu-iotests/184 trap "exit \$status" 0 1 2 3 15 · line 36
Quoted 9 lines
tests/qemu-iotests/184 +1
-0@@ -33,6 +33,7 @@ trap "exit \$status" 0 1 2 3 15 . ./common.filter _supported_os Linux +_require_drivers throttle
This test also requires null-co.
tests/qemu-iotests/184 trap "exit \$status" 0 1 2 3 15 · line 38
Quoted 2 lines
tests/qemu-iotests/184 +0
-0do_run_qemu() {
I found two more check-block tests that may or may not require use of _require_drivers (depending on which drivers we deem absolutely essential):
- 120: Needs raw
- 186: Needs null-co
tests/qemu-iotests/common.rc _require_command() · line 527
Quoted 13 lines
tests/qemu-iotests/common.rc +5
-0@@ -520,5 +520,18 @@ _require_command() [ -x "$c" ] || _notrun "$1 utility required, skipped this test" } +# Check that a set of drivers has been whitelisted in the QEMU binary +# +_require_drivers() +{ + available=$($QEMU -drive format=help | grep 'Supported formats:')
Seems a bit shortcut-y to not remove the “Supported formats:” prefix, but I don’t suppose we’ll ever have block drivers with either name...
tests/qemu-iotests/common.rc _require_command() · line 530
Quoted 3 lines
tests/qemu-iotests/common.rc +3
-0+ for driver + do + if ! echo "$available" | grep -q "$driver"; then
162 greps like this:
Quoted 1 line
grep '^Supported formats:.* ssh\( \|$\)'
Maybe the same should be done here, i.e. grep -q " $driver\( \|\$\)"? I can well imagine that something like “ssh” might appear as a substring in some other driver.
(Speaking of which, why not change 162 to using this new function? Yes, it isn’t in auto, but still...)
Max
Quoted 8 lines
tests/qemu-iotests/common.rc +5
-0+ _notrun "$driver not available" + fi + done +} + # make sure this script returns success true
Quoted 11 lines · Max Reitz
Quoted 9 lines · Thomas Huth
It is possible to enable only a subset of the block drivers with the "--block-drv-rw-whitelist" option of the "configure" script. All other drivers are marked as unusable (or only included as read-only with the "--block-drv-ro-whitelist" option). If an iotest is now using such a disabled block driver, it is failing - which is bad, since at least the tests in the "auto" group should be able to deal with this situation. Thus let's introduce a "
_require_drivers" function that can be used by the shell tests to check for the availability of certain drivers first, and marks the test as "not run" if one of the drivers is missing.
Quoted 3 lines
Well, the reasoning for generally not making blkdebug/blkverify explicit requirements was that you should just have both enabled when running iotests.
Well, we disable blkverify in our downstream RHEL version of QEMU - so it would be great if the iotests could at least adapt to that missing driver.
Quoted 5 lines
Of course, that no longer works as an argument now that we unconditionally run some iotests in make check.
But still, the question is how strict you want to be. If blkdebug cannot be assumed to be present, what about null-co? What about raw?
I tried to disable everything beside qcow2 - but that causes so many things to fail that it hardly makes sense to try to get that working. I think we can assume that at least null-co, qcow2 and raw are enabled. (If anybody still wants to try to run "make check" with one of these drivers disabled, I think we should rather add a superior check to tests/check-block.sh or tests/qemu-iotests/check instead and skip the iotests completely in that case).
Quoted 15 lines
Quoted 15 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/071 | 1 + tests/qemu-iotests/081 | 1 + tests/qemu-iotests/099 | 1 + tests/qemu-iotests/184 | 1 + tests/qemu-iotests/common.rc | 13 +++++++++++++ 5 files changed, 17 insertions(+)tests/qemu-iotests/071 +0
-0@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
Quoted 5 lines
tests/qemu-iotests/071 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 41
Quoted 3 lines
tests/qemu-iotests/071 +1
-0_supported_fmt qcow2 _supported_proto file +_require_drivers blkdebug blkverifyBecause this test also requires the raw driver.
The test also works for me when I configured QEMU with:
--block-drv-rw-whitelist="qcow2 file null-co blkdebug blkverify"
i.e. the raw driver should be disabled in that case?
Quoted 15 lines
tests/qemu-iotests/081 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 44
Quoted 12 lines
tests/qemu-iotests/071 +0
-0do_run_qemu() {tests/qemu-iotests/081 +1
-0@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw _supported_proto file _supported_os Linux +_require_drivers quorumThis test has already a check whether quorum is supported, that should be removed now.
Hmm, true ... apparently that test was not working for my case ... could it be that qemu-img ignores the whitelist and simply says that all drivers are supported?
Quoted 1 line
(Also, this test requires the raw driver.)
Agreed, this test indeed does not work without 'raw'. But it is already marked with "_supported_fmt raw", so you can indeed only run it with "raw". And running a raw-only test with a qemu binary where raw is disabled could be considered as user error, I guess ;-)
Quoted 12 lines
tests/qemu-iotests/184 trap "exit \$status" 0 1 2 3 15 · line 36
Quoted 9 lines
tests/qemu-iotests/184 +1
-0@@ -33,6 +33,7 @@ trap "exit \$status" 0 1 2 3 15 . ./common.filter _supported_os Linux +_require_drivers throttleThis test also requires null-co.
Quoted 7 lines
tests/qemu-iotests/184 trap "exit \$status" 0 1 2 3 15 · line 38
Quoted 2 lines
tests/qemu-iotests/184 +0
-0do_run_qemu() {I found two more check-block tests that may or may not require use of
_require_drivers(depending on which drivers we deem absolutely essential):
- 120: Needs raw
- 186: Needs null-co
I think we really have to assume that null-co is available, otherwise too many things break... (also some qtests are using null-co).
But I could for sure add a check for raw in 120 if desired.
Quoted 8 lines
Quoted 8 lines
tests/qemu-iotests/common.rc +0
-0@@ -520,5 +520,18 @@ _require_command() [ -x "$c" ] || _notrun "$1 utility required, skipped this test" }
Quoted 7 lines
tests/qemu-iotests/common.rc _require_command() · line 527
Quoted 5 lines
tests/qemu-iotests/common.rc +5
-0+# Check that a set of drivers has been whitelisted in the QEMU binary +# +_require_drivers() +{ + available=$($QEMU -drive format=help | grep 'Supported formats:')Seems a bit shortcut-y to not remove the “Supported formats:” prefix, but I don’t suppose we’ll ever have block drivers with either name...
I can remove it, just to be sure.
Quoted 11 lines
tests/qemu-iotests/common.rc _require_command() · line 530
Quoted 3 lines
tests/qemu-iotests/common.rc +3
-0+ for driver + do + if ! echo "$available" | grep -q "$driver"; then162 greps like this:
Quoted 1 line
grep '^Supported formats:.* ssh\( \|$\)'
Maybe the same should be done here, i.e. grep -q " $driver\( \|\$\)"? I can well imagine that something like “ssh” might appear as a substring in some other driver.
Fair, I'll add the spaces.
Quoted 2 lines
(Speaking of which, why not change 162 to using this new function? Yes, it isn’t in auto, but still...)
Yes, I can add that.
Thomas
Quoted 12 lines · Thomas Huth
Quoted 11 lines · Max Reitz
Quoted 9 lines · Thomas Huth
It is possible to enable only a subset of the block drivers with the "--block-drv-rw-whitelist" option of the "configure" script. All other drivers are marked as unusable (or only included as read-only with the "--block-drv-ro-whitelist" option). If an iotest is now using such a disabled block driver, it is failing - which is bad, since at least the tests in the "auto" group should be able to deal with this situation. Thus let's introduce a "
_require_drivers" function that can be used by the shell tests to check for the availability of certain drivers first, and marks the test as "not run" if one of the drivers is missing.
Quoted 7 lines
Quoted 3 lines
Well, the reasoning for generally not making blkdebug/blkverify explicit requirements was that you should just have both enabled when running iotests.
Well, we disable blkverify in our downstream RHEL version of QEMU - so it would be great if the iotests could at least adapt to that missing driver.
I would like to say that RHEL is not a gold standard, but then I have this series of selfish patches that specifically addresses RHEL whitelisting issues.
It feels a bit weird to me to say “blkverify is not essential, because RHEL disables it, but null-co is” – even though there is no reason why anyone would need null-co except for testing either.
Quoted 8 lines
Quoted 5 lines
Of course, that no longer works as an argument now that we unconditionally run some iotests in make check.
But still, the question is how strict you want to be. If blkdebug cannot be assumed to be present, what about null-co? What about raw?
I tried to disable everything beside qcow2 - but that causes so many things to fail that it hardly makes sense to try to get that working.
Hm, really? I just whitelisted qcow2 and file and running the auto group worked rather well (except for the failing tests you address here, and the two others I mentioned).
Quoted 5 lines
I think we can assume that at least null-co, qcow2 and raw are enabled. (If anybody still wants to try to run "make check" with one of these drivers disabled, I think we should rather add a superior check to
tests/check-block.shor tests/qemu-iotests/check instead and skip the iotests completely in that case).
I’m OK either way: We can add a global check, or we just make a decision on what users definitely have to have enabled or the qemu build would be a bit boring.
Assuming file, qcow2, and raw to be enabled seems reasonable. I’m not so sure about null-co.
(I mean, I personally don’t really care. I never added such checks myself, even a bit purposefully so, because it was my opinion that you should probably have all block drivers whitelisted before running the iotests. But then came CI and make check-block integration...)
((Also, you’ll notice that I was really inconsequential about adding null-co checks in my “selfish patches” series, precisely because I assumed everyone would have whitelisted null-co. So I’m indeed OK with just making it an implicit prerequisite.))
Quoted 21 lines
Quoted 15 lines
Quoted 15 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/qemu-iotests/071 | 1 + tests/qemu-iotests/081 | 1 + tests/qemu-iotests/099 | 1 + tests/qemu-iotests/184 | 1 + tests/qemu-iotests/common.rc | 13 +++++++++++++ 5 files changed, 17 insertions(+)tests/qemu-iotests/071 +0
-0@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15Quoted 5 lines
tests/qemu-iotests/071 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 41
Quoted 3 lines
tests/qemu-iotests/071 +1
-0_supported_fmt qcow2 _supported_proto file +_require_drivers blkdebug blkverifyBecause this test also requires the raw driver.
Quoted 5 lines
The test also works for me when I configured QEMU with:
--block-drv-rw-whitelist="qcow2 file null-co blkdebug blkverify"
i.e. the raw driver should be disabled in that case?
Ah, it’s just used by qemu-io, which of course ignores whitelisting.
Quoted 16 lines
Quoted 15 lines
tests/qemu-iotests/081 trap "_cleanup; exit \$status" 0 1 2 3 15 · line 44
Quoted 12 lines
tests/qemu-iotests/071 +0
-0do_run_qemu() {tests/qemu-iotests/081 +1
-0@@ -41,6 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw _supported_proto file _supported_os Linux +_require_drivers quorumThis test has already a check whether quorum is supported, that should be removed now.
Quoted 3 lines
Hmm, true ... apparently that test was not working for my case ... could it be that qemu-img ignores the whitelist and simply says that all drivers are supported?
Ah, yeah. The whitelist is relevant only for the system emulator.
I forgot why we even had the existing check, then. Maybe just a mistake to use qemu-img.
Quoted 6 lines
Quoted 1 line
(Also, this test requires the raw driver.)
Agreed, this test indeed does not work without 'raw'. But it is already marked with "
_supported_fmtraw", so you can indeed only run it with "raw". And running a raw-only test with a qemu binary where raw is disabled could be considered as user error, I guess ;-)
Oops, didn’t even notice somehow. O:-)
Quoted 20 lines
Quoted 12 lines
tests/qemu-iotests/184 trap "exit \$status" 0 1 2 3 15 · line 36
Quoted 9 lines
tests/qemu-iotests/184 +1
-0@@ -33,6 +33,7 @@ trap "exit \$status" 0 1 2 3 15 . ./common.filter _supported_os Linux +_require_drivers throttleThis test also requires null-co.
Quoted 7 lines
tests/qemu-iotests/184 trap "exit \$status" 0 1 2 3 15 · line 38
Quoted 2 lines
tests/qemu-iotests/184 +0
-0do_run_qemu() {I found two more check-block tests that may or may not require use of
_require_drivers(depending on which drivers we deem absolutely essential):
- 120: Needs raw
- 186: Needs null-co
Quoted 4 lines
I think we really have to assume that null-co is available, otherwise too many things break... (also some qtests are using null-co).
But I could for sure add a check for raw in 120 if desired.
If we assume that null-co is there, raw is definitely going to be there.
Max
Quoted 1 line · Max Reitz
On 20.08.19 18:01, Thomas Huth wrote:
[...]
Quoted 5 lines
Quoted 3 lines
Well, we disable blkverify in our downstream RHEL version of QEMU - so it would be great if the iotests could at least adapt to that missing driver.
I would like to say that RHEL is not a gold standard
Well, let's put it this way: The less changes we have to carry along downstream (and thus review each time we rebase the downstream tree), the more time we have to work on upstream.
Quoted 3 lines
It feels a bit weird to me to say “blkverify is not essential, because RHEL disables it, but null-co is” – even though there is no reason why anyone would need null-co except for testing either.
Ok, fine for me, too, if we also declare "null-co" as optional for the iotests - let's make sure that the tests in the "auto" group also work without them.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Of course, that no longer works as an argument now that we unconditionally run some iotests in make check.
But still, the question is how strict you want to be. If blkdebug cannot be assumed to be present, what about null-co? What about raw?
Quoted 6 lines
Quoted 2 lines
I tried to disable everything beside qcow2 - but that causes so many things to fail that it hardly makes sense to try to get that working.
Hm, really? I just whitelisted qcow2 and file and running the auto group worked rather well (except for the failing tests you address here, and the two others I mentioned).
IIRC I tried to run all qcow2 tests when I disabled null-co and saw lots of failures ... but anyway, let's just focus on the "auto" tests right now, that should be doable.
Thomas
Quoted 7 lines · Thomas Huth
Quoted 1 line · Max Reitz
On 20.08.19 18:01, Thomas Huth wrote:
[...]
Quoted 4 lines
Quoted 3 lines
Well, we disable blkverify in our downstream RHEL version of QEMU - so it would be great if the iotests could at least adapt to that missing driver.
Quoted 5 lines
Quoted 1 line
I would like to say that RHEL is not a gold standard
Well, let's put it this way: The less changes we have to carry along downstream (and thus review each time we rebase the downstream tree), the more time we have to work on upstream.
As I said, I’m guilty myself.
Quoted 7 lines
Quoted 3 lines
It feels a bit weird to me to say “blkverify is not essential, because RHEL disables it, but null-co is” – even though there is no reason why anyone would need null-co except for testing either.
Ok, fine for me, too, if we also declare "null-co" as optional for the iotests - let's make sure that the tests in the "auto" group also work without them.
Well, should we or not? You said there are other tests (outside of the iotests) that break without null-co. If so, I don’t think there’s any point in making it optional here.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Of course, that no longer works as an argument now that we unconditionally run some iotests in make check.
But still, the question is how strict you want to be. If blkdebug cannot be assumed to be present, what about null-co? What about raw?
I tried to disable everything beside qcow2 - but that causes so many things to fail that it hardly makes sense to try to get that working.
Quoted 7 lines
Quoted 3 lines
Hm, really? I just whitelisted qcow2 and file and running the auto group worked rather well (except for the failing tests you address here, and the two others I mentioned).
IIRC I tried to run all qcow2 tests when I disabled null-co and saw lots of failures ... but anyway, let's just focus on the "auto" tests right now, that should be doable.
OK, I didn’t bother running all. :-)
Max