tests/qemu-iotests: Run basic iotests during "make check"
v3 cover tests/qemu-iotests: Run basic iotests during "make check"
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet.
This patch series now changes the "auto" group to only include tests that should be safe to run in all CI environments (including FreeBSD and macOS!). Thus these iotests can now always be run during "make check" automatically, too.
v3:
- Rebase to master, solved trivial conflict with new iotest 249 (and the patch to fix the "qemu-io" output has already been merged)
- Removed some more tests from the "auto" group that failed in certain environments
- Added Reviewed-by tags
v2:
- Use "auto" group instead of adding a new "ci" group
- Adjusted the tests that are run automatically (after doing lots of CI runs on cirrus-ci.com, gitlab and travis)
- Added patch to fix the current regression with the "qemu-io:" prefix in the master branch
- Added a fix for iotest 005 on ext4 filesystems (skip it there)
Thomas Huth (7):
tests/qemu-iotests/005: Add a sanity check for large sparse file
support
tests/qemu-iotests/check: Pick a default machine if necessary tests/qemu-iotests: Do not hard-code the path to bash cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD tests/qemu-iotests: Remove the "_supported_os Linux" line from many
tests
tests/qemu-iotests/group: Re-use the "auto" group for tests that can
always run
tests: Run the iotests during "make check" again
.cirrus.yml | 4 +- .travis.yml | 1 + tests/Makefile.include | 8 +- tests/check-block.sh | 44 +++++++-- tests/qemu-iotests-quick.sh | 8 -- tests/qemu-iotests/001 | 1 - tests/qemu-iotests/002 | 1 - tests/qemu-iotests/003 | 1 - tests/qemu-iotests/004 | 1 - tests/qemu-iotests/005 | 9 ++ tests/qemu-iotests/007 | 1 - tests/qemu-iotests/008 | 1 - tests/qemu-iotests/009 | 1 - tests/qemu-iotests/010 | 1 - tests/qemu-iotests/011 | 1 - tests/qemu-iotests/012 | 1 - tests/qemu-iotests/015 | 1 - tests/qemu-iotests/017 | 1 - tests/qemu-iotests/020 | 1 - tests/qemu-iotests/021 | 1 - tests/qemu-iotests/022 | 1 - tests/qemu-iotests/025 | 1 - tests/qemu-iotests/026 | 1 - tests/qemu-iotests/027 | 1 - tests/qemu-iotests/029 | 1 - tests/qemu-iotests/031 | 1 - tests/qemu-iotests/032 | 1 - tests/qemu-iotests/033 | 1 - tests/qemu-iotests/035 | 1 - tests/qemu-iotests/036 | 1 - tests/qemu-iotests/037 | 1 - tests/qemu-iotests/042 | 1 - tests/qemu-iotests/043 | 1 - tests/qemu-iotests/046 | 1 - tests/qemu-iotests/047 | 1 - tests/qemu-iotests/049 | 1 - tests/qemu-iotests/050 | 1 - tests/qemu-iotests/051 | 1 - tests/qemu-iotests/052 | 1 - tests/qemu-iotests/053 | 1 - tests/qemu-iotests/054 | 1 - tests/qemu-iotests/062 | 1 - tests/qemu-iotests/063 | 1 - tests/qemu-iotests/066 | 1 - tests/qemu-iotests/067 | 1 - tests/qemu-iotests/068 | 1 - tests/qemu-iotests/069 | 1 - tests/qemu-iotests/071 | 1 - tests/qemu-iotests/072 | 1 - tests/qemu-iotests/073 | 1 - tests/qemu-iotests/079 | 1 - tests/qemu-iotests/082 | 1 - tests/qemu-iotests/085 | 1 - tests/qemu-iotests/089 | 1 - tests/qemu-iotests/090 | 1 - tests/qemu-iotests/094 | 1 - tests/qemu-iotests/095 | 1 - tests/qemu-iotests/098 | 1 - tests/qemu-iotests/102 | 1 - tests/qemu-iotests/103 | 1 - tests/qemu-iotests/104 | 1 - tests/qemu-iotests/105 | 1 - tests/qemu-iotests/107 | 1 - tests/qemu-iotests/110 | 1 - tests/qemu-iotests/111 | 1 - tests/qemu-iotests/112 | 1 - tests/qemu-iotests/114 | 1 - tests/qemu-iotests/115 | 1 - tests/qemu-iotests/117 | 1 - tests/qemu-iotests/120 | 1 - tests/qemu-iotests/125 | 1 - tests/qemu-iotests/126 | 1 - tests/qemu-iotests/127 | 1 - tests/qemu-iotests/133 | 1 - tests/qemu-iotests/134 | 1 - tests/qemu-iotests/142 | 1 - tests/qemu-iotests/143 | 1 - tests/qemu-iotests/144 | 1 - tests/qemu-iotests/145 | 1 - tests/qemu-iotests/153 | 1 - tests/qemu-iotests/156 | 1 - tests/qemu-iotests/157 | 1 - tests/qemu-iotests/158 | 1 - tests/qemu-iotests/159 | 1 - tests/qemu-iotests/162 | 1 - tests/qemu-iotests/170 | 1 - tests/qemu-iotests/173 | 1 - tests/qemu-iotests/182 | 1 - tests/qemu-iotests/183 | 1 - tests/qemu-iotests/186 | 1 - tests/qemu-iotests/187 | 1 - tests/qemu-iotests/190 | 1 - tests/qemu-iotests/191 | 1 - tests/qemu-iotests/192 | 1 - tests/qemu-iotests/195 | 1 - tests/qemu-iotests/197 | 1 - tests/qemu-iotests/200 | 1 - tests/qemu-iotests/214 | 1 - tests/qemu-iotests/215 | 1 - tests/qemu-iotests/217 | 1 - tests/qemu-iotests/227 | 1 - tests/qemu-iotests/231 | 1 - tests/qemu-iotests/233 | 1 - tests/qemu-iotests/239 | 2 +- tests/qemu-iotests/240 | 3 +- tests/qemu-iotests/241 | 2 +- tests/qemu-iotests/243 | 2 +- tests/qemu-iotests/244 | 2 +- tests/qemu-iotests/247 | 1 - tests/qemu-iotests/check | 13 ++- tests/qemu-iotests/group | 177 +++++++++++++++++++----------------- 111 files changed, 162 insertions(+), 211 deletions(-) delete mode 100755 tests/qemu-iotests-quick.sh
Signature · 1 line
2.21.0
PATCH v3 2/7 tests/qemu-iotests/check: Pick a default machine if necessary
qemu-system-arm, qemu-system-aarch64 and qemu-system-tricore do not have a default machine, so when running the qemu-iotests with such a binary, lots of tests are failing. Fix it by picking a default machine in the "check" script instead.
--- tests/qemu-iotests/check | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
tests/qemu-iotests/check +12 -1
@@ -130,7 +130,6 @@ export CACHEMODE="writeback" export QEMU_IO_OPTIONS="" export QEMU_IO_OPTIONS_NO_FMT="" export CACHEMODE_IS_DEFAULT=true -export QEMU_OPTIONS="-nodefaults -machine accel=qtest" export VALGRIND_QEMU= export IMGKEYSECRET= export IMGOPTSSYNTAX=false
@@ -564,6 +563,18 @@ then fi export QEMU_PROG="$(type -p "$QEMU_PROG")" +case "$QEMU_PROG" in + *qemu-system-arm|*qemu-system-aarch64) + export QEMU_OPTIONS="-nodefaults -machine virt,accel=qtest" + ;; + *qemu-system-tricore) + export QEMU_OPTIONS="-nodefaults -machine tricore_testboard,accel=qtest" + ;; + *) + export QEMU_OPTIONS="-nodefaults -machine accel=qtest" + ;; +esac + if [ -z "$QEMU_IMG_PROG" ]; then if [ -x "$build_iotests/qemu-img" ]; then export QEMU_IMG_PROG="$build_iotests/qemu-img"
Signature · 1 line
2.21.0
PATCH v3 1/7 tests/qemu-iotests/005: Add a sanity check for large sparse file support
In reply to Thomas Huth"check -raw 005" fails when running on certain filesystems - these do not support such large sparse files. Use the same check as in test 220 to skip the test in this case.
--- tests/qemu-iotests/005 | 9 +++++++++ 1 file changed, 9 insertions(+)
tests/qemu-iotests/005 +9 -0
@@ -55,6 +55,15 @@ if [ "$IMGPROTO" = "sheepdog" ]; then _notrun "image protocol $IMGPROTO does not support large image sizes" fi +# Sanity check: For raw, we require a file system that permits the creation +# of a HUGE (but very sparse) file. Check we can create it before continuing. +if [ "$IMGFMT" = "raw" ]; then + if ! truncate --size=5T "$TEST_IMG"; then + _notrun "file system on $TEST_DIR does not support large enough files" + fi + rm "$TEST_IMG" +fi + echo echo "creating large image" _make_test_img 5000G
Signature · 1 line
2.21.0
PATCH v3 3/7 tests/qemu-iotests: Do not hard-code the path to bash
In reply to Thomas Huthbash is installed in a different directory on non-Linux systems like FreeBSD. Do not hard-code /bin/bash here so that the tests can run there, too.
--- tests/qemu-iotests/239 | 2 +- tests/qemu-iotests/240 | 2 +- tests/qemu-iotests/241 | 2 +- tests/qemu-iotests/243 | 2 +- tests/qemu-iotests/244 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
tests/qemu-iotests/239 +1 -1
@@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for dmg #
tests/qemu-iotests/240 +1 -1
@@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Test hot plugging and unplugging with iothreads #
tests/qemu-iotests/241 +1 -1
@@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-nbd vs. unaligned images #
tests/qemu-iotests/243 +1 -1
@@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 preallocation #
tests/qemu-iotests/244 +1 -1
@@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 with external data files #
Signature · 1 line
2.21.0
PATCH v3 4/7 cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD
In reply to Thomas HuthWe are going to enable the qemu-iotests during "make check" again, and for running the iotests, we need bash and gnu-sed.
--- .cirrus.yml | 4 ++-- .travis.yml | 1 + 2 files changed, 3 insertions(+), 2 deletions(-)
.cirrus.yml +2 -2
@@ -7,7 +7,7 @@ freebsd_12_task: cpu: 8 memory: 8G install_script: pkg install -y - bison curl cyrus-sasl git glib gmake gnutls + bash bison curl cyrus-sasl git glib gmake gnutls gsed nettle perl5 pixman pkgconf png usbredir script: - mkdir build
@@ -20,7 +20,7 @@ macos_task: osx_instance: image: mojave-base install_script: - - brew install pkg-config python glib pixman make sdl2 + - brew install pkg-config python gnu-sed glib pixman make sdl2 script: - ./configure --python=/usr/local/bin/python3 || { cat config.log; exit 1; } - gmake -j$(sysctl -n hw.ncpu)
.travis.yml +1 -0
@@ -42,6 +42,7 @@ addons: packages: - glib - pixman + - gnu-sed # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
Signature · 1 line
2.21.0
PATCH v3 6/7 tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
In reply to Thomas HuthCurrently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).
--- tests/qemu-iotests/group | 177 +++++++++++++++++++++------------------ 1 file changed, 95 insertions(+), 82 deletions(-)
tests/qemu-iotests/group +95 -82
@@ -1,8 +1,21 @@ # # QA groups control file # Defines test groups +# +# Some notes about the groups: +# # - do not start group names with a digit # +# - quick : Tests in this group should finish within some few seconds. +# +# - img : Tests in this group can be used to excercise the qemu-img tool. +# +# - auto : Tests in this group are used during "make check" and should be +# runnable in any case. That means they should run with every QEMU binary +# (also non-x86), with every QEMU configuration (i.e. must not fail if +# an optional feature is not compiled in - but reporting a "skip" is ok), +# and work all kind of host filesystems and users (e.g. "nobody" or "root"). +# # # test-group association ... one line per test
@@ -32,11 +45,11 @@ 023 rw auto 024 rw backing auto quick 025 rw auto quick -026 rw blkdbg auto +026 rw blkdbg 027 rw auto quick -028 rw backing auto quick +028 rw backing quick 029 rw auto quick -030 rw auto backing +030 rw backing 031 rw auto quick 032 rw auto quick 033 rw auto quick
@@ -46,35 +59,35 @@ 037 rw auto backing quick 038 rw auto backing quick 039 rw auto quick -040 rw auto -041 rw auto backing +040 rw +041 rw backing 042 rw auto quick 043 rw auto backing -044 rw auto -045 rw auto quick +044 rw +045 rw quick 046 rw auto aio quick 047 rw auto quick 048 img auto quick 049 rw auto 050 rw auto backing quick -051 rw auto +051 rw 052 rw auto backing quick 053 rw auto quick 054 rw auto quick -055 rw auto -056 rw auto backing -057 rw auto -058 rw auto quick +055 rw +056 rw backing +057 rw +058 rw quick 059 rw auto quick 060 rw auto quick 061 rw auto 062 rw auto quick 063 rw auto quick 064 rw auto quick -065 rw auto quick +065 rw quick 066 rw auto quick -067 rw auto quick -068 rw auto quick +067 rw quick +068 rw quick 069 rw auto quick 070 rw auto quick 071 rw auto quick
@@ -91,18 +104,18 @@ 082 rw auto quick 083 rw auto 084 img auto quick -085 rw auto +085 rw 086 rw auto quick -087 rw auto quick +087 rw quick 088 rw auto quick 089 rw auto quick 090 rw auto quick 091 rw auto migration 092 rw auto quick -093 auto +093 throttle 094 rw auto quick -095 rw auto quick -096 rw auto quick +095 rw quick +096 rw quick 097 rw auto backing 098 rw auto backing quick 099 rw auto quick
@@ -118,60 +131,60 @@ 109 rw auto 110 rw auto backing quick 111 rw auto quick -112 rw auto +112 rw 113 rw auto quick 114 rw auto quick -115 rw auto +115 rw 116 rw auto quick 117 rw auto -118 rw auto +118 rw 119 rw auto quick 120 rw auto quick -121 rw auto +121 rw 122 rw auto 123 rw auto quick -124 rw auto backing -125 rw auto +124 rw backing +125 rw 126 rw auto backing -127 rw auto backing quick +127 rw backing quick 128 rw auto quick -129 rw auto quick +129 rw quick 130 rw auto quick 131 rw auto quick -132 rw auto quick +132 rw quick 133 auto quick 134 rw auto quick 135 rw auto -136 rw auto +136 rw 137 rw auto 138 rw auto quick -139 rw auto quick +139 rw quick 140 rw auto quick 141 rw auto quick 142 auto 143 auto quick -144 rw auto quick -145 auto quick +144 rw quick +145 quick 146 auto quick -147 auto -148 rw auto quick -149 rw auto sudo +147 img +148 rw quick +149 rw sudo 150 rw auto quick -151 rw auto -152 rw auto quick -153 rw auto quick +151 rw +152 rw quick +153 rw quick 154 rw auto backing quick -155 rw auto +155 rw 156 rw auto quick -157 auto +157 quick 158 rw auto quick 159 rw auto quick 160 rw auto quick 161 rw auto quick -162 auto quick -163 rw auto -165 rw auto quick -169 rw auto quick migration +162 quick +163 rw +165 rw quick +169 rw quick migration 170 rw auto quick 171 rw auto quick 172 auto
@@ -180,72 +193,72 @@ 175 auto quick 176 rw auto backing 177 rw auto quick -178 auto +178 img 179 rw auto quick 181 rw auto migration -182 rw auto quick -183 rw auto migration +182 rw quick +183 rw migration 184 rw auto quick -185 rw auto +185 rw 186 rw auto 187 rw auto -188 rw auto quick -189 rw auto +188 rw quick +189 rw 190 rw auto quick 191 rw auto 192 rw auto quick -194 rw auto migration quick +194 rw migration quick 195 rw auto quick -196 rw auto quick migration +196 rw quick migration 197 rw auto quick -198 rw auto -199 rw auto migration -200 rw auto +198 rw +199 rw migration +200 rw 201 rw auto migration -202 rw auto quick -203 rw auto migration -204 rw auto quick -205 rw auto quick -206 rw auto +202 rw quick +203 rw migration +204 rw quick +205 rw quick +206 rw 207 rw auto -208 rw auto quick -209 rw auto quick +208 rw quick +209 rw quick 210 rw auto 211 rw auto quick 212 rw auto quick 213 rw auto quick 214 rw auto 215 rw auto quick -216 rw auto quick +216 rw quick 217 rw auto quick -218 rw auto quick -219 rw auto +218 rw quick +219 rw 220 rw auto 221 rw auto quick -222 rw auto quick -223 rw auto quick -224 rw auto quick +222 rw quick +223 rw quick +224 rw quick 225 rw auto quick 226 auto quick -227 auto quick -228 rw auto quick +227 quick +228 rw quick 229 auto quick 231 auto quick -232 auto quick +232 quick 233 auto quick -234 auto quick migration -235 auto quick -236 auto quick +234 quick migration +235 quick +236 quick 237 rw auto quick -238 auto quick +238 quick 239 rw auto quick -240 auto quick +240 quick 241 rw auto quick -242 rw auto quick +242 rw quick 243 rw auto quick 244 rw auto quick -245 rw auto -246 rw auto quick -247 rw auto quick -248 rw auto quick +245 rw +246 rw quick +247 rw quick +248 rw quick 249 rw auto quick
Signature · 1 line
2.21.0
PATCH v3 7/7 tests: Run the iotests during "make check" again
In reply to Thomas HuthPeople often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the check-block.sh script.
Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using check-block.sh now again, this patch also removes the qemu-iotests-quick.sh script since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).
--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.sh
tests/Makefile.include +4 -4
@@ -140,7 +140,7 @@ check-unit-y += tests/test-uuid$(EXESUF) check-unit-y += tests/ptimer-test$(EXESUF) check-unit-y += tests/test-qapi-util$(EXESUF) -check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh +check-block-$(CONFIG_POSIX) += tests/check-block.sh # All QTests for now are POSIX-only, but the dependencies are # really in libqtest, not in the testcases themselves.
@@ -1096,8 +1096,8 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu-iotests/socket_scm_helper$(EXESUF) -.PHONY: check-tests/qemu-iotests-quick.sh -check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) +.PHONY: check-tests/check-block.sh +check-tests/check-block.sh: tests/check-block.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) $< .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
@@ -1168,7 +1168,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) check-qapi-schema: $(patsubst %,check-%, $(check-qapi-schema-y)) check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y)) -check: check-qapi-schema check-unit check-softfloat check-qtest check-decodetree +check: check-qapi-schema check-unit check-softfloat check-qtest check-decodetree check-block check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
tests/check-block.sh +34 -10
58 lines not on this page · raw message
tests/qemu-iotests-quick.sh deleted +0 -8
@@ -1,8 +0,0 @@ -#!/bin/sh - -cd tests/qemu-iotests - -ret=0 -TEST_DIR=${TEST_DIR:-/tmp/qemu-iotests-quick-$$} ./check -T -qcow2 -g quick || ret=1 - -exit $ret
Signature · 1 line
2.21.0
PATCH v3 5/7 tests/qemu-iotests: Remove the "_supported_os Linux" line from many tests
In reply to Thomas HuthA lot of tests run fine on FreeBSD and macOS, too - the limitation to Linux here was likely just copied-and-pasted from other tests. Thus remove the "_supported_os Linux" line from tests that run successful in our CI pipelines on FreeBSD and macOS.
--- tests/qemu-iotests/001 | 1 - tests/qemu-iotests/002 | 1 - tests/qemu-iotests/003 | 1 - tests/qemu-iotests/004 | 1 - tests/qemu-iotests/007 | 1 - tests/qemu-iotests/008 | 1 - tests/qemu-iotests/009 | 1 - tests/qemu-iotests/010 | 1 - tests/qemu-iotests/011 | 1 - tests/qemu-iotests/012 | 1 - tests/qemu-iotests/015 | 1 - tests/qemu-iotests/017 | 1 - tests/qemu-iotests/020 | 1 - tests/qemu-iotests/021 | 1 - tests/qemu-iotests/022 | 1 - tests/qemu-iotests/025 | 1 - tests/qemu-iotests/026 | 1 - tests/qemu-iotests/027 | 1 - tests/qemu-iotests/029 | 1 - tests/qemu-iotests/031 | 1 - tests/qemu-iotests/032 | 1 - tests/qemu-iotests/033 | 1 - tests/qemu-iotests/035 | 1 - tests/qemu-iotests/036 | 1 - tests/qemu-iotests/037 | 1 - tests/qemu-iotests/042 | 1 - tests/qemu-iotests/043 | 1 - tests/qemu-iotests/046 | 1 - tests/qemu-iotests/047 | 1 - tests/qemu-iotests/049 | 1 - tests/qemu-iotests/050 | 1 - tests/qemu-iotests/051 | 1 - tests/qemu-iotests/052 | 1 - tests/qemu-iotests/053 | 1 - tests/qemu-iotests/054 | 1 - tests/qemu-iotests/062 | 1 - tests/qemu-iotests/063 | 1 - tests/qemu-iotests/066 | 1 - tests/qemu-iotests/067 | 1 - tests/qemu-iotests/068 | 1 - tests/qemu-iotests/069 | 1 - tests/qemu-iotests/071 | 1 - tests/qemu-iotests/072 | 1 - tests/qemu-iotests/073 | 1 - tests/qemu-iotests/079 | 1 - tests/qemu-iotests/082 | 1 - tests/qemu-iotests/085 | 1 - tests/qemu-iotests/089 | 1 - tests/qemu-iotests/090 | 1 - tests/qemu-iotests/094 | 1 - tests/qemu-iotests/095 | 1 - tests/qemu-iotests/098 | 1 - tests/qemu-iotests/102 | 1 - tests/qemu-iotests/103 | 1 - tests/qemu-iotests/104 | 1 - tests/qemu-iotests/105 | 1 - tests/qemu-iotests/107 | 1 - tests/qemu-iotests/110 | 1 - tests/qemu-iotests/111 | 1 - tests/qemu-iotests/112 | 1 - tests/qemu-iotests/114 | 1 - tests/qemu-iotests/115 | 1 - tests/qemu-iotests/117 | 1 - tests/qemu-iotests/120 | 1 - tests/qemu-iotests/125 | 1 - tests/qemu-iotests/126 | 1 - tests/qemu-iotests/127 | 1 - tests/qemu-iotests/133 | 1 - tests/qemu-iotests/134 | 1 - tests/qemu-iotests/142 | 1 - tests/qemu-iotests/143 | 1 - tests/qemu-iotests/144 | 1 - tests/qemu-iotests/145 | 1 - tests/qemu-iotests/153 | 1 - tests/qemu-iotests/156 | 1 - tests/qemu-iotests/157 | 1 - tests/qemu-iotests/158 | 1 - tests/qemu-iotests/159 | 1 - tests/qemu-iotests/162 | 1 - tests/qemu-iotests/170 | 1 - tests/qemu-iotests/173 | 1 - tests/qemu-iotests/182 | 1 - tests/qemu-iotests/183 | 1 - tests/qemu-iotests/186 | 1 - tests/qemu-iotests/187 | 1 - tests/qemu-iotests/190 | 1 - tests/qemu-iotests/191 | 1 - tests/qemu-iotests/192 | 1 - tests/qemu-iotests/195 | 1 - tests/qemu-iotests/197 | 1 - tests/qemu-iotests/200 | 1 - tests/qemu-iotests/214 | 1 - tests/qemu-iotests/215 | 1 - tests/qemu-iotests/217 | 1 - tests/qemu-iotests/227 | 1 - tests/qemu-iotests/231 | 1 - tests/qemu-iotests/233 | 1 - tests/qemu-iotests/240 | 1 - tests/qemu-iotests/247 | 1 - 99 files changed, 99 deletions(-)
tests/qemu-iotests/001 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/002 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/003 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=128M offset=67M
tests/qemu-iotests/004 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw qcow qcow2 qed vdi vmdk vhdx luks _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/007 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # currently only qcow2 allows for consistency checks using qemu-img _supported_fmt qcow2 _supported_proto generic -_supported_os Linux # refcount_bits must be at least 4 so we can create ten internal snapshots # (1 bit supports none, 2 bits support two, 4 bits support 14) _unsupported_imgopts 'refcount_bits=\(1\|2\)[^0-9]'
tests/qemu-iotests/008 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/009 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=6G
tests/qemu-iotests/010 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=6G
tests/qemu-iotests/011 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=6G
tests/qemu-iotests/012 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux # Remove once all tests are fixed to use TEST_IMG_FILE # correctly and common.rc sets it unconditionally
tests/qemu-iotests/015 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # actually any format that supports snapshots _supported_fmt qcow2 _supported_proto generic -_supported_os Linux # Internal snapshots are (currently) impossible with refcount_bits=1 _unsupported_imgopts 'refcount_bits=1[^0-9]'
tests/qemu-iotests/017 +0 -1
@@ -41,7 +41,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 vmdk qed _supported_proto generic _unsupported_proto vxhs -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" TEST_OFFSETS="0 4294967296"
tests/qemu-iotests/020 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Any format supporting backing files _supported_fmt qcow qcow2 vmdk qed _supported_proto file -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" \ "subformat=twoGbMaxExtentFlat" \ "subformat=twoGbMaxExtentSparse"
tests/qemu-iotests/021 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/022 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Any format that supports snapshots _supported_fmt qcow2 _supported_proto generic -_supported_os Linux TEST_OFFSETS="10485760 4294967296" CLUSTER_SIZE="4096"
tests/qemu-iotests/025 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw qcow2 qed luks _supported_proto file sheepdog rbd nfs -_supported_os Linux echo "=== Creating image" echo
tests/qemu-iotests/026 +0 -1
@@ -41,7 +41,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Currently only qcow2 supports rebasing _supported_fmt qcow2 _supported_proto file -_supported_os Linux _default_cache_mode "writethrough" _supported_cache_modes "writethrough" "none" # The refcount table tests expect a certain minimum width for refcount entries
tests/qemu-iotests/027 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt vmdk qcow qcow2 qed _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/029 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _unsupported_proto vxhs -_supported_os Linux # Internal snapshots are (currently) impossible with refcount_bits=1 _unsupported_imgopts 'refcount_bits=1[^0-9]'
tests/qemu-iotests/031 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This tests qcow2-specific low-level functionality _supported_fmt qcow2 _supported_proto file -_supported_os Linux CLUSTER_SIZE=65536
tests/qemu-iotests/032 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This works for any image format (though unlikely to segfault for raw) _supported_fmt generic _supported_proto generic -_supported_os Linux echo echo === Prepare image ===
tests/qemu-iotests/033 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/035 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic -_supported_os Linux size=6G
tests/qemu-iotests/036 +0 -1
@@ -43,7 +43,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This tests qcow2-specific low-level functionality _supported_fmt qcow2 _supported_proto file -_supported_os Linux # Only qcow2v3 and later supports feature bits IMGOPTS="compat=1.1"
tests/qemu-iotests/037 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 vmdk qed _supported_proto file -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" \ "subformat=twoGbMaxExtentFlat" \ "subformat=twoGbMaxExtentSparse"
tests/qemu-iotests/042 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 qed _supported_proto file -_supported_os Linux echo echo "== Creating zero size image =="
tests/qemu-iotests/043 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Any format supporting backing files _supported_fmt qcow2 qed _supported_proto file -_supported_os Linux size=128M
tests/qemu-iotests/046 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux CLUSTER_SIZE=64k size=128M
tests/qemu-iotests/047 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic -_supported_os Linux size=128M
tests/qemu-iotests/049 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux filter_test_dir() {
tests/qemu-iotests/050 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 qed _supported_proto file -_supported_os Linux if test "$IMGFMT" = qcow2 && test $IMGOPTS = ""; then IMGOPTS=compat=1.1
tests/qemu-iotests/051 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # A compat=0.10 image is created in this test which does not support anything # other than refcount_bits=16 _unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
tests/qemu-iotests/052 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux # Don't do O_DIRECT on tmpfs _supported_cache_modes "writeback" "writethrough" "unsafe"
tests/qemu-iotests/053 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 qcow _supported_proto file -_supported_os Linux echo echo "== Creating single sector image =="
tests/qemu-iotests/054 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux echo echo "creating too large image (1 EB)"
tests/qemu-iotests/062 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This tests qocw2-specific low-level functionality _supported_fmt qcow2 _supported_proto generic -_supported_os Linux IMGOPTS="compat=1.1" IMG_SIZE=64M
tests/qemu-iotests/063 +0 -1
@@ -41,7 +41,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 vmdk qed raw _supported_proto file -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" \ "subformat=twoGbMaxExtentFlat" \ "subformat=twoGbMaxExtentSparse"
tests/qemu-iotests/066 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This tests qocw2-specific low-level functionality _supported_fmt qcow2 _supported_proto generic -_supported_os Linux # Intentionally create an unaligned image IMGOPTS="compat=1.1"
tests/qemu-iotests/067 +0 -1
@@ -32,7 +32,6 @@ status=1 # failure is the default! _supported_fmt qcow2 _supported_proto file -_supported_os Linux # Because anything other than 16 would change the output of query-block _unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
tests/qemu-iotests/068 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This tests qocw2-specific low-level functionality _supported_fmt qcow2 _supported_proto generic -_supported_os Linux IMGOPTS="compat=1.1" IMG_SIZE=128K
tests/qemu-iotests/069 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qed qcow qcow2 vmdk _supported_proto file -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" IMG_SIZE=128K
tests/qemu-iotests/071 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux do_run_qemu() {
tests/qemu-iotests/072 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow _supported_proto file -_supported_os Linux IMG_SIZE=64M
tests/qemu-iotests/073 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _unsupported_proto vxhs -_supported_os Linux CLUSTER_SIZE=64k size=128M
tests/qemu-iotests/079 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs -_supported_os Linux echo "=== Check option preallocation and cluster_size ===" echo
tests/qemu-iotests/082 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs -_supported_os Linux run_qemu_img() {
tests/qemu-iotests/085 +0 -1
@@ -56,7 +56,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # ${1}: unique identifier for the snapshot filename
tests/qemu-iotests/089 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # Because anything other than 16 would change the output of qemu_io -c info _unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
tests/qemu-iotests/090 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs -_supported_os Linux IMG_SIZE=128K
tests/qemu-iotests/094 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto nbd -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" _make_test_img 64M
tests/qemu-iotests/095 +0 -1
@@ -44,7 +44,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux size_smaller=5M size_larger=100M
tests/qemu-iotests/098 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux IMGOPTS="compat=1.1"
tests/qemu-iotests/102 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux IMG_SIZE=64K
tests/qemu-iotests/103 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs -_supported_os Linux # Internal snapshots are (currently) impossible with refcount_bits=1 _unsupported_imgopts 'refcount_bits=1[^0-9]'
tests/qemu-iotests/104 +0 -1
@@ -34,7 +34,6 @@ trap "exit \$status" 0 1 2 3 15 _supported_fmt raw qcow qcow2 qed vdi vmdk vhdx _supported_proto generic -_supported_os Linux echo "=== Check qemu-img info output ===" echo
tests/qemu-iotests/105 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 vmdk vhdx qed _supported_proto generic -_supported_os Linux _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \ "subformat=twoGbMaxExtentSparse"
tests/qemu-iotests/107 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file nfs -_supported_os Linux IMG_SIZE=64K
tests/qemu-iotests/110 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Any format supporting backing files _supported_fmt qed qcow qcow2 vmdk _supported_proto file -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" TEST_IMG_REL=$(basename "$TEST_IMG")
tests/qemu-iotests/111 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qed qcow qcow2 vmdk _supported_proto file -_supported_os Linux _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" $QEMU_IMG create -f $IMGFMT -b "$TEST_IMG.inexistent" "$TEST_IMG" 2>&1 \
tests/qemu-iotests/112 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This tests qcow2-specific low-level functionality _supported_fmt qcow2 _supported_proto file -_supported_os Linux # This test will set refcount_bits on its own which would conflict with the # manual setting; compat will be overridden as well _unsupported_imgopts refcount_bits 'compat=0.10'
tests/qemu-iotests/114 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto generic _unsupported_proto vxhs -_supported_os Linux TEST_IMG="$TEST_IMG.base" _make_test_img 64M
tests/qemu-iotests/115 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # This test relies on refcounts being 64 bits wide (which does not work with # compat=0.10) _unsupported_imgopts 'refcount_bits=\([^6]\|.\([^4]\|$\)\)' 'compat=0.10'
tests/qemu-iotests/117 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux _make_test_img 64k
tests/qemu-iotests/120 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux _unsupported_fmt luks _make_test_img 64M
tests/qemu-iotests/125 +0 -1
@@ -44,7 +44,6 @@ get_image_size_on_host() _supported_fmt qcow2 _supported_proto file -_supported_os Linux if [ -z "$TEST_IMG_FILE" ]; then TEST_IMG_FILE=$TEST_IMG
tests/qemu-iotests/126 +0 -1
@@ -37,7 +37,6 @@ _supported_fmt qcow qcow2 qed vmdk # colons which separate a protocol prefix from the rest and colons which are # just part of the filename, so we cannot test protocols which require a prefix) _supported_proto file -_supported_os Linux echo echo '=== Testing plain files ==='
tests/qemu-iotests/127 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux IMG_SIZE=64K
tests/qemu-iotests/133 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux TEST_IMG="$TEST_IMG.base" _make_test_img 64M _make_test_img -b "$TEST_IMG.base"
tests/qemu-iotests/134 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 _supported_proto generic _unsupported_proto vxhs -_supported_os Linux size=128M
tests/qemu-iotests/142 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # We test all cache modes anyway, but O_DIRECT needs to be supported _default_cache_mode none
tests/qemu-iotests/143 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux keep_stderr=y \ _launch_qemu 2> >(_filter_nbd)
tests/qemu-iotests/144 +0 -1
@@ -46,7 +46,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux size=512M
tests/qemu-iotests/145 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto generic -_supported_os Linux _make_test_img 1M
tests/qemu-iotests/153 +0 -1
@@ -62,7 +62,6 @@ _check_ofd || _notrun "OFD lock not available" _supported_fmt qcow2 _supported_proto file -_supported_os Linux _run_cmd() {
tests/qemu-iotests/156 +0 -1
@@ -49,7 +49,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 qed _supported_proto generic _unsupported_proto vxhs -_supported_os Linux # Create source disk TEST_IMG="$TEST_IMG.backing" _make_test_img 1M
tests/qemu-iotests/157 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux do_run_qemu() {
tests/qemu-iotests/158 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow qcow2 _supported_proto generic _unsupported_proto vxhs -_supported_os Linux size=128M
tests/qemu-iotests/159 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux _unsupported_fmt luks TEST_SIZES="5 512 1024 1999 1K 64K 1M"
tests/qemu-iotests/162 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common.filter _supported_fmt generic -_supported_os Linux test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh\( \|$\)') [ "$test_ssh" = "" ] && _notrun "ssh support required"
tests/qemu-iotests/170 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux _unsupported_fmt luks echo
tests/qemu-iotests/173 +0 -1
@@ -41,7 +41,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto nfs -_supported_os Linux size=100M
tests/qemu-iotests/182 +0 -1
@@ -41,7 +41,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux size=32M
tests/qemu-iotests/183 +0 -1
@@ -44,7 +44,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 raw qed quorum _supported_proto file -_supported_os Linux size=64M _make_test_img $size
tests/qemu-iotests/186 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then _notrun "Requires a PC machine"
tests/qemu-iotests/187 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux size=64M _make_test_img $size
tests/qemu-iotests/190 +0 -1
@@ -41,7 +41,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # See 178 for more extensive tests across more formats _supported_fmt qcow2 _supported_proto file -_supported_os Linux echo "== Huge file ==" echo
tests/qemu-iotests/191 +0 -1
@@ -43,7 +43,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux size=64M
tests/qemu-iotests/192 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then _notrun "Requires a PC machine"
tests/qemu-iotests/195 +0 -1
@@ -39,7 +39,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux do_run_qemu() {
tests/qemu-iotests/197 +0 -1
@@ -51,7 +51,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Test is supported for any backing file; but we force qcow2 for our wrapper. _supported_fmt generic _supported_proto generic -_supported_os Linux # LUKS support may be possible, but it complicates things. _unsupported_fmt luks
tests/qemu-iotests/200 +0 -1
@@ -42,7 +42,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 qed _supported_proto file -_supported_os Linux BACKING_IMG="${TEST_DIR}/backing.img" TEST_IMG="${TEST_DIR}/test.img"
tests/qemu-iotests/214 +0 -1
@@ -36,7 +36,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # Repairing the corrupted image requires qemu-img check to store a # refcount up to 3, which requires at least two refcount bits.
tests/qemu-iotests/215 +0 -1
@@ -48,7 +48,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Test is supported for any backing file; but we force qcow2 for our wrapper. _supported_fmt generic _supported_proto generic -_supported_os Linux # LUKS support may be possible, but it complicates things. _unsupported_fmt luks
tests/qemu-iotests/217 +0 -1
@@ -37,7 +37,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # This test is specific to qcow2 _supported_fmt qcow2 _supported_proto file -_supported_os Linux # This test needs clusters with at least a refcount of 2 so that # OFLAG_COPIED is not set. refcount_bits=1 is therefore unsupported.
tests/qemu-iotests/227 +0 -1
@@ -38,7 +38,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto file -_supported_os Linux do_run_qemu() {
tests/qemu-iotests/231 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt generic _supported_proto rbd -_supported_os Linux BOGUS_CONF=${TEST_DIR}/ceph-$$.conf touch "${BOGUS_CONF}"
tests/qemu-iotests/233 +0 -1
@@ -47,7 +47,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt raw qcow2 _supported_proto file # If porting to non-Linux, consider using socat instead of ss in common.nbd -_supported_os Linux _require_command QEMU_NBD nbd_server_set_tcp_port
tests/qemu-iotests/240 +0 -1
@@ -33,7 +33,6 @@ status=1 # failure is the default! _supported_fmt generic _supported_proto generic -_supported_os Linux do_run_qemu() {
tests/qemu-iotests/247 +0 -1
@@ -40,7 +40,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # Requires backing files and .bdrv_change_backing_file support _supported_fmt qcow2 qed _supported_proto file -_supported_os Linux size=128M
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v3 1/7] tests/qemu-iotests/005: Add a sanity check for large sparse file support
In reply to Thomas HuthQuoted 7 lines · Thomas Huth
"check -raw 005" fails when running on certain filesystems - these do not support such large sparse files. Use the same check as in test 220 to skip the test in this case.
Suggested-by Eric Blake <[email protected]>Reviewed-by Eric Blake <[email protected]>Signed-off-by Thomas Huth <[email protected]>
Quoted 14 lines
--- tests/qemu-iotests/005 | 9 +++++++++ 1 file changed, 9 insertions(+)tests/qemu-iotests/005 +2
-0@@ -55,6 +55,15 @@ if [ "$IMGPROTO" = "sheepdog" ]; then _notrun "image protocol $IMGPROTO does not support large image sizes" fi +# Sanity check: For raw, we require a file system that permits the creation +# of a HUGE (but very sparse) file. Check we can create it before continuing.
tests/qemu-iotests/005 if [ "$IMGPROTO" = "sheepdog" ]; then · line 69
Quoted 10 lines
tests/qemu-iotests/005 +7
-0+if [ "$IMGFMT" = "raw" ]; then + if ! truncate --size=5T "$TEST_IMG"; then + _notrun "file system on $TEST_DIR does not support large enough files" + fi + rm "$TEST_IMG" +fi + echo echo "creating large image" _make_test_img 5000G
--
Alex Bennée
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
In reply to Thomas HuthQuoted 109 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shtests/Makefile.include +4
-4@@ -140,7 +140,7 @@ check-unit-y += tests/test-uuid$(EXESUF) check-unit-y += tests/ptimer-test$(EXESUF) check-unit-y += tests/test-qapi-util$(EXESUF) -check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh +check-block-$(CONFIG_POSIX) += tests/check-block.sh # All QTests for now are POSIX-only, but the dependencies are # really in libqtest, not in the testcases themselves.@@ -1096,8 +1096,8 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu-iotests/socket_scm_helper$(EXESUF) -.PHONY: check-tests/qemu-iotests-quick.sh -check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) +.PHONY: check-tests/check-block.sh +check-tests/check-block.sh: tests/check-block.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) $< .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))@@ -1168,7 +1168,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) check-qapi-schema: $(patsubst %,check-%, $(check-qapi-schema-y)) check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y)) -check: check-qapi-schema check-unit check-softfloat check-qtest check-decodetree +check: check-qapi-schema check-unit check-softfloat check-qtest check-decodetree check-block check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))tests/check-block.sh +31
-8@@ -1,24 +1,48 @@ #!/bin/sh -FORMAT_LIST="raw qcow2 qed vmdk vpc" +# Honor the SPEED environment variable, just like we do it for the qtests. +if [ "$SPEED" = "slow" ]; then + format_list="raw qcow2" + group= +elif [ "$SPEED" = "thorough" ]; then + format_list="raw qcow2 qed vmdk vpc" + group= +else + format_list=qcow2 + group="-g auto" +fi + if [ "$#" -ne 0 ]; then - FORMAT_LIST="$@" + format_list="$@" +fi + +if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then + echo "GPROF is enabled ==> Not running the qemu-iotests." + exit 0 fi -export QEMU_PROG="$PWD/x86_64-softmmu/qemu-system-x86_64" -export QEMU_IMG_PROG="$PWD/qemu-img" -export QEMU_IO_PROG="$PWD/qemu-io" +if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then + echo "No qemu-system binary available ==> Not running the qemu-iotests." + exit 0 +fi + +if ! command -v bash >/dev/null 2>&1 ; then + echo "bash not available ==> Not running the qemu-iotests." + exit 0 +fi -if [ ! -x $QEMU_PROG ]; then - echo "'make check-block' requires qemu-system-x86_64" - exit 1 +if ! (sed --version | grep 'GNU sed') > /dev/null 2>&1 ; then + if ! command -v gsed >/dev/null 2>&1; then + echo "GNU sed not available ==> Not running the qemu-iotests." + exit 0
tests/check-block.sh line 45
Quoted 10 lines
tests/check-block.sh +3
-2+ fi fi cd tests/qemu-iotests ret=0 -for FMT in $FORMAT_LIST ; do - ./check -T -nocache -$FMT || ret=1 +for fmt in $format_list ; do + ./check -$fmt $group || ret=1
This is all looking good and my only remaining objection is aesthetic. If you run "make check -jN" you end up with the with the block test output intermingled with the rest of the output which is now fairly uniform.
Any chance we could hide the verbosity unless requested and have something like:
TEST check-block: test xxx
emitted for each passing test? It's OK to show more in the failing case to help with diagnosis but the passing tests should be easy on the eye when scanning through the logs.
Quoted 9 lines
tests/check-block.sh +0
-0done exit $rettests/qemu-iotests-quick.sh deleted +0
-0
tests/qemu-iotests-quick.sh line 8
Quoted 8 lines
tests/qemu-iotests-quick.sh deleted +0
-8@@ -1,8 +0,0 @@ -#!/bin/sh - -cd tests/qemu-iotests - -ret=0 -TEST_DIR=${TEST_DIR:-/tmp/qemu-iotests-quick-$$} ./check -T -qcow2 -g quick || ret=1 - -exit $ret
--
Alex Bennée
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
Quoted 128 lines · Alex Bennée
Quoted 109 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shtests/Makefile.include +4
-4@@ -140,7 +140,7 @@ check-unit-y += tests/test-uuid$(EXESUF) check-unit-y += tests/ptimer-test$(EXESUF) check-unit-y += tests/test-qapi-util$(EXESUF) -check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh +check-block-$(CONFIG_POSIX) += tests/check-block.sh # All QTests for now are POSIX-only, but the dependencies are # really in libqtest, not in the testcases themselves.@@ -1096,8 +1096,8 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu-iotests/socket_scm_helper$(EXESUF) -.PHONY: check-tests/qemu-iotests-quick.sh -check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) +.PHONY: check-tests/check-block.sh +check-tests/check-block.sh: tests/check-block.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) $< .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))@@ -1168,7 +1168,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) check-qapi-schema: $(patsubst %,check-%, $(check-qapi-schema-y)) check-tests/qapi-schema/doc-good.texi check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS)) check-block: $(patsubst %,check-%, $(check-block-y)) -check: check-qapi-schema check-unit check-softfloat check-qtest check-decodetree +check: check-qapi-schema check-unit check-softfloat check-qtest check-decodetree check-block check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))tests/check-block.sh +31
-8@@ -1,24 +1,48 @@ #!/bin/sh -FORMAT_LIST="raw qcow2 qed vmdk vpc" +# Honor the SPEED environment variable, just like we do it for the qtests. +if [ "$SPEED" = "slow" ]; then + format_list="raw qcow2" + group= +elif [ "$SPEED" = "thorough" ]; then + format_list="raw qcow2 qed vmdk vpc" + group= +else + format_list=qcow2 + group="-g auto" +fi + if [ "$#" -ne 0 ]; then - FORMAT_LIST="$@" + format_list="$@" +fi + +if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then + echo "GPROF is enabled ==> Not running the qemu-iotests." + exit 0 fi -export QEMU_PROG="$PWD/x86_64-softmmu/qemu-system-x86_64" -export QEMU_IMG_PROG="$PWD/qemu-img" -export QEMU_IO_PROG="$PWD/qemu-io" +if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then + echo "No qemu-system binary available ==> Not running the qemu-iotests." + exit 0 +fi + +if ! command -v bash >/dev/null 2>&1 ; then + echo "bash not available ==> Not running the qemu-iotests." + exit 0 +fi -if [ ! -x $QEMU_PROG ]; then - echo "'make check-block' requires qemu-system-x86_64" - exit 1 +if ! (sed --version | grep 'GNU sed') > /dev/null 2>&1 ; then + if ! command -v gsed >/dev/null 2>&1; then + echo "GNU sed not available ==> Not running the qemu-iotests." + exit 0tests/check-block.sh line 45
Quoted 10 lines
tests/check-block.sh +3
-2+ fi fi cd tests/qemu-iotests ret=0 -for FMT in $FORMAT_LIST ; do - ./check -T -nocache -$FMT || ret=1 +for fmt in $format_list ; do + ./check -$fmt $group || ret=1This is all looking good and my only remaining objection is aesthetic. If you run "make check -jN" you end up with the with the block test output intermingled with the rest of the output which is now fairly uniform.
Quoted 6 lines
Any chance we could hide the verbosity unless requested and have something like:
TEST check-block: test xxx
emitted for each passing test?
Yeah, I noticed this already, too. I was already thinking of changing the tests/qemu-iotests/check script a little bit (maybe even teaching it the TAP protocol?), but I felt it was too much for this series. Considering that the iotests have been broken two times already after the 4.1 tree has been opened, I think we should get this merged as soon as possible to avoid at least partly future regressions. Clean up of the test output can and will be done in a separate patch later.
Thomas
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
In reply to Thomas HuthQuoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Quoted 14 lines · Markus Armbruster
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).
Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
003 1s ...
007 2s ...
013 5s ...
014 15s ...
015 9s ...
022 1s ...
023 18s ...
043 0s ...
049 3s ...
061 4s ...
079 2s ...
080 4s ...
091 1s ...
097 2s ...
104 0s ...
117 0s ...
122 8s ...
126 1s ...
137 1s ...
142 5s ...
172 3s ...
174 0s ...
176 5s ...
181 2s ...
186 2s ...
187 0s ...
191 5s ...
201 2s ...
214 0s ...
That looks reasonable to me.
Thomas
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Quoted 18 lines · Thomas Huth
Quoted 14 lines · Markus Armbruster
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...
Should these be moved to quick?
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...
Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 1 line
022 1s ...
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
etc.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Quoted 27 lines · Eric Blake
Quoted 18 lines · Thomas Huth
Quoted 14 lines · Markus Armbruster
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
Quoted 6 lines
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...Should these be moved to quick?
I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...?
Quoted 10 lines
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 7 lines
Quoted 1 line
022 1s ...
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later.
Thomas
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
In reply to Eric BlakeQuoted 46 lines · Eric Blake
Quoted 18 lines · Thomas Huth
Quoted 14 lines · Markus Armbruster
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...Should these be moved to quick?
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 1 line
022 1s ...
Quoted 7 lines
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
etc.
Kevin, Max?
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
In reply to Thomas HuthQuoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).
Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.sh
Can I interest you in a Makefile target that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.
Max
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.sh
Quoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.
Can't you simply run
./check -qcow2 -x auto
instead?
Thomas
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
In reply to Thomas HuthQuoted 51 lines · Thomas Huth
Quoted 27 lines · Eric Blake
Quoted 18 lines · Thomas Huth
Quoted 14 lines · Markus Armbruster
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
Quoted 6 lines
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...Should these be moved to quick?
I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...?
Quoted 12 lines
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 1 line
022 1s ...
Quoted 8 lines
Quoted 5 lines
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later.
Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group.
Thomas
PATCH v4 1/1 tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), run with qcow2 and file protocol, run reasonable fast, and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).
---
v4:
- Dropped the tests from the "auto" group that run longer than 5s
- Dropped also the tests that do not support the qcow2 format
Alex, could you replace the patch in your patch series, too, please?
tests/qemu-iotests/group | 262 +++++++++++++++++++++------------------ 1 file changed, 138 insertions(+), 124 deletions(-)
tests/qemu-iotests/group +138 -124
@@ -1,8 +1,22 @@ # # QA groups control file # Defines test groups +# +# Some notes about the groups: +# # - do not start group names with a digit # +# - quick : Tests in this group should finish within some few seconds. +# +# - img : Tests in this group can be used to excercise the qemu-img tool. +# +# - auto : Tests in this group are used during "make check" and should be +# runnable in any case. That means they should run with every QEMU binary +# (also non-x86), with every QEMU configuration (i.e. must not fail if +# an optional feature is not compiled in - but reporting a "skip" is ok), +# work with the qcow2 format and file protocol, run reasonably fast (< 10s) +# and work all kind of host filesystems and users (e.g. "nobody" or "root"). +# # # test-group association ... one line per test
@@ -20,8 +34,8 @@ 011 rw auto quick 012 auto quick 013 rw auto -014 rw auto -015 rw snapshot auto +014 rw +015 rw snapshot # 016 was removed, do not reuse 017 rw backing auto quick 018 rw backing auto quick
@@ -29,14 +43,14 @@ 020 rw backing auto quick 021 io auto quick 022 rw snapshot auto -023 rw auto +023 rw 024 rw backing auto quick 025 rw auto quick -026 rw blkdbg auto +026 rw blkdbg 027 rw auto quick -028 rw backing auto quick +028 rw backing quick 029 rw auto quick -030 rw auto backing +030 rw backing 031 rw auto quick 032 rw auto quick 033 rw auto quick
@@ -46,206 +60,206 @@ 037 rw auto backing quick 038 rw auto backing quick 039 rw auto quick -040 rw auto -041 rw auto backing +040 rw +041 rw backing 042 rw auto quick 043 rw auto backing -044 rw auto -045 rw auto quick +044 rw +045 rw quick 046 rw auto aio quick 047 rw auto quick 048 img auto quick 049 rw auto 050 rw auto backing quick -051 rw auto +051 rw 052 rw auto backing quick 053 rw auto quick 054 rw auto quick -055 rw auto -056 rw auto backing -057 rw auto -058 rw auto quick -059 rw auto quick +055 rw +056 rw backing +057 rw +058 rw quick +059 rw quick 060 rw auto quick 061 rw auto 062 rw auto quick 063 rw auto quick -064 rw auto quick -065 rw auto quick +064 rw quick +065 rw quick 066 rw auto quick -067 rw auto quick -068 rw auto quick +067 rw quick +068 rw quick 069 rw auto quick -070 rw auto quick +070 rw quick 071 rw auto quick 072 rw auto quick 073 rw auto quick 074 rw auto quick -075 rw auto quick -076 auto -077 rw auto quick -078 rw auto quick +075 rw quick +076 io +077 rw quick +078 rw quick 079 rw auto 080 rw auto -081 rw auto quick +081 rw quick 082 rw auto quick -083 rw auto -084 img auto quick -085 rw auto +083 rw +084 img quick +085 rw 086 rw auto quick -087 rw auto quick -088 rw auto quick +087 rw quick +088 rw quick 089 rw auto quick 090 rw auto quick 091 rw auto migration -092 rw auto quick -093 auto -094 rw auto quick -095 rw auto quick -096 rw auto quick +092 rw quick +093 throttle +094 rw quick +095 rw quick +096 rw quick 097 rw auto backing 098 rw auto backing quick 099 rw auto quick # 100 was removed, do not reuse -101 rw auto quick +101 rw quick 102 rw auto quick 103 rw auto quick 104 rw auto 105 rw auto quick -106 rw auto quick +106 rw quick 107 rw auto quick 108 rw auto quick -109 rw auto +109 rw 110 rw auto backing quick 111 rw auto quick -112 rw auto -113 rw auto quick +112 rw +113 rw quick 114 rw auto quick -115 rw auto -116 rw auto quick +115 rw +116 rw quick 117 rw auto -118 rw auto -119 rw auto quick +118 rw +119 rw quick 120 rw auto quick -121 rw auto +121 rw 122 rw auto -123 rw auto quick -124 rw auto backing -125 rw auto +123 rw quick +124 rw backing +125 rw 126 rw auto backing -127 rw auto backing quick -128 rw auto quick -129 rw auto quick +127 rw backing quick +128 rw quick +129 rw quick 130 rw auto quick -131 rw auto quick -132 rw auto quick +131 rw quick +132 rw quick 133 auto quick 134 rw auto quick -135 rw auto -136 rw auto +135 rw +136 rw 137 rw auto 138 rw auto quick -139 rw auto quick +139 rw quick 140 rw auto quick 141 rw auto quick 142 auto 143 auto quick -144 rw auto quick -145 auto quick -146 auto quick -147 auto -148 rw auto quick -149 rw auto sudo +144 rw quick +145 quick +146 quick +147 img +148 rw quick +149 rw sudo 150 rw auto quick -151 rw auto -152 rw auto quick -153 rw auto quick +151 rw +152 rw quick +153 rw quick 154 rw auto backing quick -155 rw auto +155 rw 156 rw auto quick -157 auto +157 quick 158 rw auto quick 159 rw auto quick -160 rw auto quick +160 rw quick 161 rw auto quick -162 auto quick -163 rw auto -165 rw auto quick -169 rw auto quick migration +162 quick +163 rw +165 rw quick +169 rw quick migration 170 rw auto quick -171 rw auto quick +171 rw quick 172 auto -173 rw auto +173 rw 174 auto -175 auto quick +175 quick 176 rw auto backing 177 rw auto quick -178 auto +178 img 179 rw auto quick 181 rw auto migration -182 rw auto quick -183 rw auto migration +182 rw quick +183 rw migration 184 rw auto quick -185 rw auto +185 rw 186 rw auto 187 rw auto -188 rw auto quick -189 rw auto +188 rw quick +189 rw 190 rw auto quick 191 rw auto 192 rw auto quick -194 rw auto migration quick +194 rw migration quick 195 rw auto quick -196 rw auto quick migration +196 rw quick migration 197 rw auto quick -198 rw auto -199 rw auto migration -200 rw auto +198 rw +199 rw migration +200 rw 201 rw auto migration -202 rw auto quick -203 rw auto migration -204 rw auto quick -205 rw auto quick -206 rw auto -207 rw auto -208 rw auto quick -209 rw auto quick -210 rw auto -211 rw auto quick -212 rw auto quick -213 rw auto quick +202 rw quick +203 rw migration +204 rw quick +205 rw quick +206 rw +207 rw +208 rw quick +209 rw quick +210 rw +211 rw quick +212 rw quick +213 rw quick 214 rw auto 215 rw auto quick -216 rw auto quick +216 rw quick 217 rw auto quick -218 rw auto quick -219 rw auto +218 rw quick +219 rw 220 rw auto -221 rw auto quick -222 rw auto quick -223 rw auto quick -224 rw auto quick -225 rw auto quick +221 rw quick +222 rw quick +223 rw quick +224 rw quick +225 rw quick 226 auto quick -227 auto quick -228 rw auto quick +227 quick +228 rw quick 229 auto quick -231 auto quick -232 auto quick +231 quick +232 quick 233 auto quick -234 auto quick migration -235 auto quick -236 auto quick -237 rw auto quick -238 auto quick -239 rw auto quick -240 auto quick -241 rw auto quick -242 rw auto quick +234 quick migration +235 quick +236 quick +237 rw quick +238 quick +239 rw quick +240 quick +241 rw quick +242 rw quick 243 rw auto quick 244 rw auto quick -245 rw auto -246 rw auto quick -247 rw auto quick -248 rw auto quick +245 rw +246 rw quick +247 rw quick +248 rw quick 249 rw auto quick
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
In reply to Thomas HuthQuoted 4 lines · Thomas Huth
Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group.
Appreciated!
I'm all for automated testing, but "make check" is already slow enough to make certain kinds of work painful. Not quite slow enough to make me renege on commitments and go on a quest to speed it up.
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
In reply to Thomas HuthQuoted 32 lines · Thomas Huth
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shQuoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.
Quoted 5 lines
Can't you simply run
./check -qcow2 -x auto
instead?
I don’t run just qcow2 tests. I run qcow2, qcow2 with compat=0.10, qcow2 with refcount_bits=1, raw, nbd, qed, vmdk, vhdx, ... A lot.
So for which of the protocol/format combinations do I exclude the auto group? check-block.sh says it runs raw, qcow2, qed, vmdk, and vpc. But may that not be subject to change?
Max
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
Quoted 37 lines · Max Reitz
Quoted 36 lines · Thomas Huth
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shQuoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.Can't you simply run
./check -qcow2 -x auto
Quoted 8 lines
Quoted 1 line
instead?
I don’t run just qcow2 tests. I run qcow2, qcow2 with compat=0.10, qcow2 with refcount_bits=1, raw, nbd, qed, vmdk, vhdx, ... A lot.
So for which of the protocol/format combinations do I exclude the auto group?
check-block.shsays it runs raw, qcow2, qed, vmdk, and vpc. But may that not be subject to change?
With my patch series, the auto group is only used for qcow2.
Thomas
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
In reply to Max ReitzQuoted 37 lines · Max Reitz
Quoted 36 lines · Thomas Huth
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shQuoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.Can't you simply run
./check -qcow2 -x auto
Quoted 8 lines
Quoted 1 line
instead?
I don’t run just qcow2 tests. I run qcow2, qcow2 with compat=0.10, qcow2 with refcount_bits=1, raw, nbd, qed, vmdk, vhdx, ... A lot.
So for which of the protocol/format combinations do I exclude the auto group?
check-block.shsays it runs raw, qcow2, qed, vmdk, and vpc. But may that not be subject to change?
Oh, and furthermore I prefer to run the tests on tmpfs.
Max
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
In reply to Thomas HuthQuoted 40 lines · Thomas Huth
Quoted 39 lines · Max Reitz
Quoted 37 lines · Thomas Huth
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shQuoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.Can't you simply run
./check -qcow2 -x auto
instead?
Quoted 8 lines
Quoted 6 lines
I don’t run just qcow2 tests. I run qcow2, qcow2 with compat=0.10, qcow2 with refcount_bits=1, raw, nbd, qed, vmdk, vhdx, ... A lot.
So for which of the protocol/format combinations do I exclude the auto group?
check-block.shsays it runs raw, qcow2, qed, vmdk, and vpc. But may that not be subject to change?With my patch series, the auto group is only used for qcow2.
And that is not subject to change? Like, maybe someone wants to add nbd in the future?
I mean, I have a test branch anyway which collects a number of patches on top of master that make everything pass more or less reliably for me (11 patches currently...). I suppose I can just revert your patch on top of that. But that doesn’t feel right.
Max
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
In reply to Thomas HuthAm 10.05.2019 um 10:55 hat Thomas Huth geschrieben:
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Eric Blake
Quoted 27 lines · Thomas Huth
Quoted 18 lines · Markus Armbruster
Quoted 14 lines
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
Quoted 6 lines
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...Should these be moved to quick?
I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...?
Quoted 17 lines
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 1 line
022 1s ...
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
Quoted 7 lines
Quoted 2 lines
I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later.
Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group.
I don't think time is everything. We should also consider how much the tests contribute to basic code coverage. There is no point in removing a test from the list because it takes 10 seconds, but if I split it in two tests taking each 5 seconds, you would include both halves.
For example, 030, 040 and 041 are not that quick (14/11/42 seconds, respectively), but they are the most important tests for block jobs and covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at least.
Kevin
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Quoted 73 lines · Kevin Wolf
Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben:
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Eric Blake
Quoted 27 lines · Thomas Huth
Quoted 18 lines · Markus Armbruster
Quoted 14 lines
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
Quoted 6 lines
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...Should these be moved to quick?
I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...?
Quoted 17 lines
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 1 line
022 1s ...
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
Quoted 7 lines
Quoted 2 lines
I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later.
Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group.
I don't think time is everything. We should also consider how much the tests contribute to basic code coverage. There is no point in removing a test from the list because it takes 10 seconds, but if I split it in two tests taking each 5 seconds, you would include both halves.
Quoted 4 lines
For example, 030, 040 and 041 are not that quick (14/11/42 seconds, respectively), but they are the most important tests for block jobs and covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at least.
Yes, we want block jobs covered. However, 42 seconds is a lot indeed. Can you think of ways to get a useful part of the full coverage in five seconds or less?
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Am 10.05.2019 um 17:29 hat Markus Armbruster geschrieben:
Quoted 75 lines
Quoted 73 lines · Kevin Wolf
Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben:
Quoted 58 lines · Thomas Huth
Quoted 57 lines · Eric Blake
Quoted 27 lines · Thomas Huth
Quoted 18 lines · Markus Armbruster
Quoted 14 lines
Quoted 11 lines · Thomas Huth
Currently, all tests are in the "auto" group. This is a little bit pointless. OTOH, we need a group for the tests that we can automatically run during "make check" each time, too. Tests in this new group are supposed to run with every possible QEMU configuration, for example they must run with every QEMU binary (also non-x86), without failing when an optional features is missing (but reporting "skip" is ok), and be able to run on all kind of host filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial list has been determined by running the iotests with non-x86 QEMU targets and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including macOS and FreeBSD).Quoted 2 lines
I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
Quoted 7 lines
I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok?
When I run the tests from the auto group that are not in the quick group, I currently get:
Quoted 6 lines
My personal threshold is about 5 seconds for quick, so:
Quoted 2 lines
003 1s ...
007 2s ...Should these be moved to quick?
I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...?
Quoted 17 lines
Quoted 1 line
013 5s ...
this one is borderline
Quoted 2 lines
014 15s ...
015 9s ...Definitely not quick, but if you think they are still okay for auto, I can live with that.
Quoted 1 line
022 1s ...
Another candidate for quick?
Quoted 1 line
023 18s ...
Even longer than 14. Okay for auto?
Quoted 7 lines
Quoted 2 lines
I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later.
Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group.
I don't think time is everything. We should also consider how much the tests contribute to basic code coverage. There is no point in removing a test from the list because it takes 10 seconds, but if I split it in two tests taking each 5 seconds, you would include both halves.
Quoted 8 lines
Quoted 4 lines
For example, 030, 040 and 041 are not that quick (14/11/42 seconds, respectively), but they are the most important tests for block jobs and covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at least.
Yes, we want block jobs covered. However, 42 seconds is a lot indeed. Can you think of ways to get a useful part of the full coverage in five seconds or less?
I'm not even sure which part of the test is taking so long. Maybe it's only one of the 88 test functions that's taking long. Or we're duplicating a lot of tests because we test all combinations of setups where testing only certain combinations would give enough coverage.
Answering this will probably take some effort checking what the exact cases tested are and how long each one takes.
Kevin
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
In reply to Max ReitzQuoted 44 lines · Max Reitz
Quoted 43 lines · Thomas Huth
Quoted 42 lines · Max Reitz
Quoted 32 lines · Thomas Huth
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shQuoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.Quoted 5 lines
Can't you simply run
./check -qcow2 -x auto
instead?
I don’t run just qcow2 tests. I run qcow2, qcow2 with compat=0.10, qcow2 with refcount_bits=1, raw, nbd, qed, vmdk, vhdx, ... A lot.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
So for which of the protocol/format combinations do I exclude the auto group?
check-block.shsays it runs raw, qcow2, qed, vmdk, and vpc. But may that not be subject to change?With my patch series, the auto group is only used for qcow2.
And that is not subject to change? Like, maybe someone wants to add nbd in the future?
The current set of qcow2 auto tests takes already quite a while, so I don't think that this will change soon.
And if the "normal" users want to run more tests, they can simply use "make check SPEED=slow" or SPEED=thorough, so IMHO no need to extend the quick default list right now.
Quoted 4 lines
I mean, I have a test branch anyway which collects a number of patches on top of master that make everything pass more or less reliably for me (11 patches currently...). I suppose I can just revert your patch on top of that. But that doesn’t feel right.
Hmm, sure, non-upstream patches are always a bad thing. But I still don't see why you really need an extra Makefile target for this. In the worst case, you could also simply change your script to run something this instead:
make $(grep ^check: tests/Makefile.include | sed s/check-block//)
?
Thomas
Re: [Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again
Quoted 65 lines · Thomas Huth
Quoted 44 lines · Max Reitz
Quoted 43 lines · Thomas Huth
Quoted 42 lines · Max Reitz
Quoted 32 lines · Thomas Huth
Quoted 27 lines · Max Reitz
Quoted 18 lines · Thomas Huth
People often forget to run the iotests before submitting patches or pull requests - this is likely due to the fact that we do not run the tests during our mandatory "make check" tests yet. Now that we've got a proper "auto" group of iotests that should be fine to run in every environment, we can enable the iotests during "make check" again by running the "auto" tests by default from the
check-block.shscript.Some cases still need to be checked first, though: iotests need bash and GNU sed (otherwise they fail), and if gprof is enabled, it spoils the output of some test cases causing them to fail. So if we detect that one of the required programs is missing or that gprof is enabled, we still have to skip the iotests to avoid failures.
And finally, since we are using
check-block.shnow again, this patch also removes theqemu-iotests-quick.shscript since we do not need that anymore (and having two shell wrapper scripts around the block tests seem rather confusing than helpful).Quoted 7 lines
Signed-off-by Thomas Huth <[email protected]>--- tests/Makefile.include | 8 +++---- tests/check-block.sh | 44 ++++++++++++++++++++++++++++--------- tests/qemu-iotests-quick.sh | 8 ------- 3 files changed, 38 insertions(+), 22 deletions(-) delete mode 100755 tests/qemu-iotests-quick.shQuoted 3 lines
Can I interest you in a
Makefiletarget that explicitly excludes check-block? I run the iotests anyway, but I also run make check. Running some iotests twice would be a bit pointless.Quoted 5 lines
Can't you simply run
./check -qcow2 -x auto
instead?
I don’t run just qcow2 tests. I run qcow2, qcow2 with compat=0.10, qcow2 with refcount_bits=1, raw, nbd, qed, vmdk, vhdx, ... A lot.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
So for which of the protocol/format combinations do I exclude the auto group?
check-block.shsays it runs raw, qcow2, qed, vmdk, and vpc. But may that not be subject to change?With my patch series, the auto group is only used for qcow2.
And that is not subject to change? Like, maybe someone wants to add nbd in the future?
The current set of qcow2 auto tests takes already quite a while, so I don't think that this will change soon.
And if the "normal" users want to run more tests, they can simply use "make check SPEED=slow" or SPEED=thorough, so IMHO no need to extend the quick default list right now.Quoted 4 lines
I mean, I have a test branch anyway which collects a number of patches on top of master that make everything pass more or less reliably for me (11 patches currently...). I suppose I can just revert your patch on top of that. But that doesn’t feel right.
Quoted 8 lines
Hmm, sure, non-upstream patches are always a bad thing. But I still don't see why you really need an extra
Makefiletarget for this. In the worst case, you could also simply change your script to run something this instead:make $(grep ^check: tests/Makefile.include | sed s/check-block//)
?
Hm. I didn’t think of that. Thanks. :-)
Max