Rework iotests/check
Subsystems (2)
v8 cover Rework iotests/check
v8:
about linters:
I didn't modify 297, as Max already staged 297 modifications to test all files.
Also, now I have two complains:
+************* Module testenv
+testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) +************* Module testrunner
+testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements)
Success: no issues found in 5 source files
And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can do some refactoring as a follow-up.
findtests: grammar, wording, add Eric's r-b
testenv: commit msg adjusted
wording, grammar
missed env_variables added
add logic for searching qemu-system-*, if qemu-system-{arch} is not exist
handle valgrind and MALLOC_PERTURB_
fix s/vmkd/vmdk/
testrunner: commit msg adjusted
wording
fix extra new lines in diff output s/0/fd/
use class instead of namedtuple
don't handle MALLOC_PERTURB_
declare self._stack in __init__
change thistime to be float
handle KeyboardInterrupt
casenotrun init to ''
fix "output mismatch .. {f_bad}" to be f-string smarter terminal color handling
check: wording, grammar, sheepdoG, exClude_groups
change valgrind to be simple boolean option and pass
it to TestEnv
handle group.local
check virtio-blk in iotests.py
add --color option
Vladimir Sementsov-Ogievskiy (5):
iotests: add findtests.py
iotests: add testenv.py
iotests: add testrunner.py
iotests: rewrite check into python
iotests: rename and move 169 and 199 tests
docs/devel/testing.rst | 50 +-
Makefile | 1 -
tests/check-block.sh | 2 +-
tests/qemu-iotests/check | 1095 ++---------------
tests/qemu-iotests/common.env.in | 3 -
tests/qemu-iotests/findtests.py | 159 +++
tests/qemu-iotests/group | 321 -----
tests/qemu-iotests/iotests.py | 8 +
tests/qemu-iotests/meson.build | 3 -
tests/qemu-iotests/testenv.py | 278 +++++
tests/qemu-iotests/testrunner.py | 366 ++++++
.../migrate-bitmaps-postcopy-test} | 0
.../migrate-bitmaps-postcopy-test.out} | 0
.../{169 => tests/migrate-bitmaps-test} | 0
.../migrate-bitmaps-test.out} | 0
15 files changed, 990 insertions(+), 1296 deletions(-)
delete mode 100644 tests/qemu-iotests/common.env.in
create mode 100644 tests/qemu-iotests/findtests.py
delete mode 100644 tests/qemu-iotests/group
create mode 100644 tests/qemu-iotests/testenv.py
create mode 100644 tests/qemu-iotests/testrunner.py
rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (100%)
rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%)
rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (100%)
rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)Signature · 1 line
2.29.2
PATCH v8 1/5 iotests: add findtests.py
Add python script with new logic of searching for tests:
Current ./check behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong to any group, like 142)
Behavior of findtests.py:
- group file is dropped
- tests are all files in tests/ subdirectory (except for .out files), so it's not needed more to "register the test", just create it with appropriate name in tests/ subdirectory. Old names like [0-9][0-9][0-9] (in root iotests directory) are supported too, but not recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of commenting tests in old 'group' file you now can add them to disabled group with help of 'group.local' file
- selecting test ranges like 5-15 are not supported more (to support restarting failed ./check command from the middle of the process, new argument is added: --start-from)
Benefits:
- no rebase conflicts in group file on patch porting from branch to branch
- no conflicts in upstream, when different series want to occupy same test number
- meaningful names for test files
For example, with digital number, when some person wants to add some test about block-stream, he most probably will just create a new test. But if there would be test-block-stream test already, he will at first look at it and may be just add a test-case into it. And anyway meaningful names are better.
This commit doesn't update check behavior (which will be done in further commit), still, the documentation changed like new behavior is already here. Let's live with this small inconsistency for the following few commits, until final change.
--- docs/devel/testing.rst | 50 +++++++++- tests/qemu-iotests/findtests.py | 159 ++++++++++++++++++++++++++++++++ 2 files changed, 208 insertions(+), 1 deletion(-) create mode 100644 tests/qemu-iotests/findtests.py
docs/devel/testing.rst +49 -1
@@ -111,7 +111,7 @@ check-block ----------- ``make check-block`` runs a subset of the block layer iotests (the tests that -are in the "auto" group in ``tests/qemu-iotests/group``). +are in the "auto" group). See the "QEMU iotests" section below for more information. GCC gcov support
@@ -224,6 +224,54 @@ another application on the host may have locked the file, possibly leading to a test failure. If using such devices are explicitly desired, consider adding ``locking=off`` option to disable image locking. +Test case groups +---------------- + +"Tests may belong to one or more test groups, which are defined in the form +of a comment in the test source file. By convention, test groups are listed +in the second line of the test file, after the "#!/..." line, like this: + +.. code:: + + #!/usr/bin/env python3 + # group: auto quick + # + ... + +Another way of defining groups is creating the tests/qemu-iotests/group.local +file. This should be used only for downstream (this file should never appear +in upstream). This file may be used for defining some downstream test groups +or for temporarily disabling tests, like this: + +.. code:: + + # groups for some company downstream process + # + # ci - tests to run on build + # down - our downstream tests, not for upstream + # + # Format of each line is: + # TEST_NAME TEST_GROUP [TEST_GROUP ]... + + 013 ci + 210 disabled + 215 disabled + our-ugly-workaround-test down ci + +Note that the following group names have a special meaning: + +- quick: Tests in this group should finish within a few seconds. + +- auto: Tests in this group are used during "make check" and should be + runnable in any case. That means they should run with every QEMU binary + (also non-x86), with every QEMU configuration (i.e. must not fail if + an optional feature is not compiled in - but reporting a "skip" is ok), + work at least with the qcow2 file format, work with all kind of host + filesystems and users (e.g. "nobody" or "root") and must not take too + much memory and disk space (since CI pipelines tend to fail otherwise). + +- disabled: Tests in this group are disabled and ignored by check. + .. _docker-ref: Docker based tests
tests/qemu-iotests/findtests.py new file +159 -0
159 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v8 2/5 iotests: add testenv.py
In reply to Vladimir Sementsov-OgievskiyAdd TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 tests/qemu-iotests/testenv.py
tests/qemu-iotests/testenv.py new file +278 -0
@@ -0,0 +1,278 @@ +# TestEnv class to manage test environment variables. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import sys +import tempfile +from pathlib import Path +import shutil +import collections +import random +import subprocess +import glob +from contextlib import AbstractContextManager +from typing import Dict, Any, Optional + + +def get_default_machine(qemu_prog: str) -> str: + outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True, + universal_newlines=True, + stdout=subprocess.PIPE).stdout + + machines = outp.split('\n') + default_machine = next(m for m in machines if m.endswith(' (default)')) + default_machine = default_machine.split(' ', 1)[0] + + alias_suf = ' (alias of {})'.format(default_machine) + alias = next((m for m in machines if m.endswith(alias_suf)), None) + if alias is not None: + default_machine = alias.split(' ', 1)[0] + + return default_machine + + +class TestEnv(AbstractContextManager['TestEnv']): + """ + Manage system environment for running tests + + The following variables are supported/provided. They are represented by + lower-cased TestEnv attributes. + """ + + # We store environment variables as instance attributes, and there are a + # lot of them. Silence pylint: + # pylint: disable=too-many-instance-attributes + + env_variables = ['PYTHONPATH', 'TEST_DIR', 'SOCK_DIR', 'SAMPLE_IMG_DIR', + 'OUTPUT_DIR', 'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG', + 'QEMU_IO_PROG', 'QEMU_NBD_PROG', 'QSD_PROG', + 'SOCKET_SCM_HELPER', 'QEMU_OPTIONS', 'QEMU_IMG_OPTIONS', + 'QEMU_IO_OPTIONS', 'QEMU_IO_OPTIONS_NO_FMT', + 'QEMU_NBD_OPTIONS', 'IMGOPTS', 'IMGFMT', 'IMGPROTO', + 'AIOMODE', 'CACHEMODE', 'VALGRIND_QEMU', + 'CACHEMODE_IS_DEFAULT', 'IMGFMT_GENERIC', 'IMGOPTSSYNTAX', + 'IMGKEYSECRET', 'QEMU_DEFAULT_MACHINE', 'MALLOC_PERTURB_'] + + def get_env(self) -> Dict[str, str]: + env = {} + for v in self.env_variables: + val = getattr(self, v.lower(), None) + if val is not None: + env[v] = val + + return env + + def init_directories(self): + """Init directory variables: + PYTHONPATH + TEST_DIR + SOCK_DIR + SAMPLE_IMG_DIR + OUTPUT_DIR + """ + self.pythonpath = os.getenv('PYTHONPATH') + if self.pythonpath: + self.pythonpath = self.source_iotests + os.pathsep + \ + self.pythonpath + else: + self.pythonpath = self.source_iotests + + self.test_dir = os.getenv('TEST_DIR', + os.path.join(os.getcwd(), 'scratch')) + Path(self.test_dir).mkdir(parents=True, exist_ok=True) + + self.sock_dir = os.getenv('SOCK_DIR') + self.tmp_sock_dir = False + if self.sock_dir: + Path(self.test_dir).mkdir(parents=True, exist_ok=True) + else: + self.sock_dir = tempfile.mkdtemp() + self.tmp_sock_dir = True + + self.sample_img_dir = os.getenv('SAMPLE_IMG_DIR', + os.path.join(self.source_iotests, + 'sample_images')) + + self.output_dir = os.getcwd() # OUTPUT_DIR + + def init_binaries(self): + """Init binary path variables: + PYTHON (for bash tests) + QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG + SOCKET_SCM_HELPER + """ + self.python = sys.executable + + def root(*names): + return os.path.join(self.build_root, *names) + + arch = os.uname().machine + if 'ppc64' in arch: + arch = 'ppc64' + + self.qemu_prog = os.getenv('QEMU_PROG', root(f'qemu-system-{arch}')) + if not os.path.exists(self.qemu_prog): + pattern = root('qemu-system-*') + progs = glob.glob(pattern) + if not progs: + sys.exit(f"Not found any Qemu binary by pattern '{pattern}'") + if len(progs) > 1: + progs_list = ', '.join(progs) + sys.exit(f"Several non '{arch}' qemu binaries found: " + f"{progs_list}, please set QEMU_PROG environment " + "variable") + self.qemu_prog = progs[0] + + self.qemu_img_prog = os.getenv('QEMU_IMG_PROG', root('qemu-img')) + self.qemu_io_prog = os.getenv('QEMU_IO_PROG', root('qemu-io')) + self.qemu_nbd_prog = os.getenv('QEMU_NBD_PROG', root('qemu-nbd')) + self.qsd_prog = os.getenv('QSD_PROG', root('storage-daemon', + 'qemu-storage-daemon')) + + for b in [self.qemu_img_prog, self.qemu_io_prog, self.qemu_nbd_prog, + self.qemu_prog, self.qsd_prog]: + if not os.path.exists(b): + sys.exit('No such file: ' + b) + if not os.access(b, os.X_OK): + sys.exit('Not executable: ' + b) + + helper_path = os.path.join(self.build_iotests, 'socket_scm_helper') + if os.access(helper_path, os.X_OK): + self.socket_scm_helper = helper_path # SOCKET_SCM_HELPER + + def __init__(self, imgfmt: str, imgproto: str, aiomode: str, + cachemode: Optional[str] = None, + imgopts: Optional[str] = None, + misalign: bool = False, + debug: bool = False, + valgrind: bool = False) -> None: + self.imgfmt = imgfmt + self.imgproto = imgproto + self.aiomode = aiomode + self.imgopts = imgopts + self.misalign = misalign + self.debug = debug + + if valgrind: + self.valgrind_qemu = 'y' + + if cachemode is None: + self.cachemode_is_default = 'true' + self.cachemode = 'writeback' + else: + self.cachemode_is_default = 'false' + self.cachemode = cachemode + + # Initialize generic paths: build_root, build_iotests, source_iotests, + # which are needed to initialize some environment variables. They are + # used by init_*() functions as well. + + if os.path.islink(sys.argv[0]): + # called from the build tree + self.source_iotests = os.path.dirname(os.readlink(sys.argv[0])) + self.build_iotests = os.path.dirname(os.path.abspath(sys.argv[0])) + else: + # called from the source tree + self.source_iotests = os.getcwd() + self.build_iotests = self.source_iotests + + self.build_root = os.path.join(self.build_iotests, '..', '..') + + self.init_directories() + self.init_binaries() + + self.malloc_perturb_ = os.getenv('MALLOC_PERTURB_', + str(random.randrange(1, 255))) + + # QEMU_OPTIONS + self.qemu_options = '-nodefaults -display none -accel qtest' + machine_map = ( + ('arm', 'virt'), + ('aarch64', 'virt'), + ('avr', 'mega2560'), + ('rx', 'gdbsim-r5f562n8'), + ('tricore', 'tricore_testboard') + ) + for suffix, machine in machine_map: + if self.qemu_prog.endswith(f'qemu-system-{suffix}'): + self.qemu_options += f' -machine {machine}' + + # QEMU_DEFAULT_MACHINE + self.qemu_default_machine = get_default_machine(self.qemu_prog) + + self.qemu_img_options = os.getenv('QEMU_IMG_OPTIONS') + self.qemu_nbd_options = os.getenv('QEMU_NBD_OPTIONS') + + is_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg'] + self.imgfmt_generic = 'true' if is_generic else 'false' + + self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}' + if self.misalign: + self.qemu_io_options += ' --misalign' + + self.qemu_io_options_no_fmt = self.qemu_io_options + + if self.imgfmt == 'luks': + self.imgoptssyntax = 'true' + self.imgkeysecret = '123456' + if not self.imgopts: + self.imgopts = 'iter-time=10' + elif 'iter-time=' not in self.imgopts: + self.imgopts += ',iter-time=10' + else: + self.imgoptssyntax = 'false' + self.qemu_io_options += ' -f ' + self.imgfmt + + if self.imgfmt == 'vmdk': + if not self.imgopts: + self.imgopts = 'zeroed_grain=on' + elif 'zeroed_grain=' not in self.imgopts: + self.imgopts += ',zeroed_grain=on' + + def close(self) -> None: + if self.tmp_sock_dir: + shutil.rmtree(self.sock_dir) + + def __enter__(self) -> 'TestEnv': + return self + + def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: + self.close() + + def print_env(self) -> None: + template = """\ +QEMU -- "{QEMU_PROG}" {QEMU_OPTIONS} +QEMU_IMG -- "{QEMU_IMG_PROG}" {QEMU_IMG_OPTIONS} +QEMU_IO -- "{QEMU_IO_PROG}" {QEMU_IO_OPTIONS} +QEMU_NBD -- "{QEMU_NBD_PROG}" {QEMU_NBD_OPTIONS} +IMGFMT -- {IMGFMT}{imgopts} +IMGPROTO -- {IMGPROTO} +PLATFORM -- {platform} +TEST_DIR -- {TEST_DIR} +SOCK_DIR -- {SOCK_DIR} +SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}""" + + args = collections.defaultdict(str, self.get_env()) + + if 'IMGOPTS' in args: + args['imgopts'] = f" ({args['IMGOPTS']})" + + u = os.uname() + args['platform'] = f'{u.sysname}/{u.machine} {u.nodename} {u.release}' + + print(template.format_map(args))
Signature · 1 line
2.29.2
Re: [PATCH v8 2/5] iotests: add testenv.py
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
Add TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
Signed-off-by Vladimir Sementsov-Ogievskiy<[email protected]>--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+)create mode 100644
tests/qemu-iotests/testenv.pytests/qemu-iotests/testenv.py new file +0
-0
[..]
Quoted 17 lines
tests/qemu-iotests/testenv.py new file +17
-0+ def init_binaries(self): + """Init binary path variables: + PYTHON (for bash tests) + QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG + SOCKET_SCM_HELPER + """ + self.python = sys.executable + + def root(*names): + return os.path.join(self.build_root, *names) + + arch = os.uname().machine + if 'ppc64' in arch: + arch = 'ppc64' + + self.qemu_prog = os.getenv('QEMU_PROG', root(f'qemu-system-{arch}')) + if not os.path.exists(self.qemu_prog):
tests/qemu-iotests/testenv.py
Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10
-0+ pattern = root('qemu-system-*') + progs = glob.glob(pattern) + if not progs: + sys.exit(f"Not found any Qemu binary by pattern '{pattern}'") + if len(progs) > 1: + progs_list = ', '.join(progs) + sys.exit(f"Several non '{arch}' qemu binaries found: " + f"{progs_list}, please set QEMU_PROG environment " + "variable") + self.qemu_prog = progs[0]
squash-in:
tests/qemu-iotests/testenv.py +8 -9
@@ -129,14 +129,14 @@ class TestEnv(AbstractContextManager['TestEnv']): if not os.path.exists(self.qemu_prog): pattern = root('qemu-system-*') progs = glob.glob(pattern) - if not progs: - sys.exit(f"Not found any Qemu binary by pattern '{pattern}'") - if len(progs) > 1: - progs_list = ', '.join(progs) - sys.exit(f"Several non '{arch}' qemu binaries found: " - f"{progs_list}, please set QEMU_PROG environment " - "variable") - self.qemu_prog = progs[0] + found = False + for p in progs: + if os.access(p, os.X_OK): + self.qemu_prog = p + found = True + if not found: + sys.exit("Not found any Qemu executable binary by pattern " + f"'{pattern}'") --
Best regards,
Vladimir
Re: [PATCH v8 2/5] iotests: add testenv.py
In reply to Vladimir Sementsov-OgievskiyAm 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 56 lines
Add TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 tests/qemu-iotests/testenv.pytests/qemu-iotests/testenv.py new file +39
-0@@ -0,0 +1,278 @@ +# TestEnv class to manage test environment variables. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import sys +import tempfile +from pathlib import Path +import shutil +import collections +import random +import subprocess +import glob +from contextlib import AbstractContextManager +from typing import Dict, Any, Optional + + +def get_default_machine(qemu_prog: str) -> str: + outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True, + universal_newlines=True, + stdout=subprocess.PIPE).stdout + + machines = outp.split('\n') + default_machine = next(m for m in machines if m.endswith(' (default)')) + default_machine = default_machine.split(' ', 1)[0]
tests/qemu-iotests/testenv.py line 49
Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10
-0+ + alias_suf = ' (alias of {})'.format(default_machine) + alias = next((m for m in machines if m.endswith(alias_suf)), None) + if alias is not None: + default_machine = alias.split(' ', 1)[0] + + return default_machine + + +class TestEnv(AbstractContextManager['TestEnv']):
I'm getting CI failures here:
Traceback (most recent call last):
File "./check", line 23, in <module>
from testenv import TestEnv
File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 49, in <module>
class TestEnv(AbstractContextManager['TestEnv']):
TypeError: 'ABCMeta' object is not subscriptable
On the other hand, if I make it just AbstractContextManager without giving the type parameter, mypy complains:
testenv.py:49: error: Missing type parameters for generic type "ContextManager"
I guess I need to have another look into this tomorrow.
By the way, mypy --strict still finds a few errors. I think we want to address at least the warnings about missing type annotatings and calling untyped functions.
Kevin
Re: [PATCH v8 2/5] iotests: add testenv.py
Quoted 75 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 56 lines
Add TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+)create mode 100644
tests/qemu-iotests/testenv.pytests/qemu-iotests/testenv.py new file +39
-0@@ -0,0 +1,278 @@ +# TestEnv class to manage test environment variables. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import sys +import tempfile +from pathlib import Path +import shutil +import collections +import random +import subprocess +import glob +from contextlib import AbstractContextManager +from typing import Dict, Any, Optional + + +def get_default_machine(qemu_prog: str) -> str: + outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True, + universal_newlines=True, + stdout=subprocess.PIPE).stdout + + machines = outp.split('\n') + default_machine = next(m for m in machines if m.endswith(' (default)')) + default_machine = default_machine.split(' ', 1)[0]tests/qemu-iotests/testenv.py line 49
Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10
-0+ + alias_suf = ' (alias of {})'.format(default_machine) + alias = next((m for m in machines if m.endswith(alias_suf)), None) + if alias is not None: + default_machine = alias.split(' ', 1)[0] + + return default_machine + + +class TestEnv(AbstractContextManager['TestEnv']):I'm getting CI failures here:
Traceback (most recent call last):
File "./check", line 23, in <module> from testenv import TestEnv File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 49, in <module> class TestEnv(AbstractContextManager['TestEnv']):
Quoted 8 lines
TypeError: 'ABCMeta' object is not subscriptable
On the other hand, if I make it just AbstractContextManager without giving the type parameter, mypy complains:
testenv.py:49: error: Missing type parameters for generic type "ContextManager"
I guess I need to have another look into this tomorrow.
It may help to use typing.ContextManager instead of AbstractContextManager. mypy is OK with it, probably CI will be OK too..
Quoted 5 lines
By the way, mypy --strict still finds a few errors. I think we want to address at least the warnings about missing type annotatings and calling untyped functions.
OK
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v8 2/5] iotests: add testenv.py
Am 26.01.2021 um 09:28 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 81 lines
Quoted 80 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 56 lines
Add TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+)create mode 100644
tests/qemu-iotests/testenv.pytests/qemu-iotests/testenv.py new file +39
-0@@ -0,0 +1,278 @@ +# TestEnv class to manage test environment variables. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import sys +import tempfile +from pathlib import Path +import shutil +import collections +import random +import subprocess +import glob +from contextlib import AbstractContextManager +from typing import Dict, Any, Optional + + +def get_default_machine(qemu_prog: str) -> str: + outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True, + universal_newlines=True, + stdout=subprocess.PIPE).stdout + + machines = outp.split('\n') + default_machine = next(m for m in machines if m.endswith(' (default)')) + default_machine = default_machine.split(' ', 1)[0]tests/qemu-iotests/testenv.py line 49
Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10
-0+ + alias_suf = ' (alias of {})'.format(default_machine) + alias = next((m for m in machines if m.endswith(alias_suf)), None) + if alias is not None: + default_machine = alias.split(' ', 1)[0] + + return default_machine + + +class TestEnv(AbstractContextManager['TestEnv']):I'm getting CI failures here:
Traceback (most recent call last):
File "./check", line 23, in <module> from testenv import TestEnv File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 49, in <module> class TestEnv(AbstractContextManager['TestEnv']):TypeError: 'ABCMeta' object is not subscriptable
On the other hand, if I make it just AbstractContextManager without giving the type parameter, mypy complains:
Quoted 7 lines
Quoted 3 lines
testenv.py:49: error: Missing type parameters for generic type "ContextManager"
I guess I need to have another look into this tomorrow.
It may help to use typing.ContextManager instead of AbstractContextManager. mypy is OK with it, probably CI will be OK too..
Okay, I'm trying now if this change works (on top of v9, of course). If it does, I'll just squash it in. I also silenced some of the mypy warnings, so that I'm not testing with the following patch squashed in.
Kevin
tests/qemu-iotests/testenv.py +5 -6
@@ -25,8 +25,7 @@ import collections import random import subprocess import glob -from contextlib import AbstractContextManager -from typing import Dict, Any, Optional +from typing import ContextManager, Dict, Any, Optional def isxfile(path: str) -> bool:
@@ -50,7 +49,7 @@ def get_default_machine(qemu_prog: str) -> str: return default_machine -class TestEnv(AbstractContextManager['TestEnv']): +class TestEnv(ContextManager['TestEnv']): """ Manage system environment for running tests
@@ -81,7 +80,7 @@ class TestEnv(AbstractContextManager['TestEnv']): return env - def init_directories(self): + def init_directories(self) -> None: """Init directory variables: PYTHONPATH TEST_DIR
@@ -114,7 +113,7 @@ class TestEnv(AbstractContextManager['TestEnv']): self.output_dir = os.getcwd() # OUTPUT_DIR - def init_binaries(self): + def init_binaries(self) -> None: """Init binary path variables: PYTHON (for bash tests) QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG
@@ -122,7 +121,7 @@ class TestEnv(AbstractContextManager['TestEnv']): """ self.python = sys.executable - def root(*names): + def root(*names: str) -> str: return os.path.join(self.build_root, *names) arch = os.uname().machine
Re: [PATCH v8 2/5] iotests: add testenv.py
Quoted 140 lines · Kevin Wolf
Am 26.01.2021 um 09:28 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 81 lines
Quoted 80 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 56 lines
Add TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+)create mode 100644
tests/qemu-iotests/testenv.pytests/qemu-iotests/testenv.py new file +39
-0@@ -0,0 +1,278 @@ +# TestEnv class to manage test environment variables. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import sys +import tempfile +from pathlib import Path +import shutil +import collections +import random +import subprocess +import glob +from contextlib import AbstractContextManager +from typing import Dict, Any, Optional + + +def get_default_machine(qemu_prog: str) -> str: + outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True, + universal_newlines=True, + stdout=subprocess.PIPE).stdout + + machines = outp.split('\n') + default_machine = next(m for m in machines if m.endswith(' (default)')) + default_machine = default_machine.split(' ', 1)[0]tests/qemu-iotests/testenv.py line 49
Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10
-0+ + alias_suf = ' (alias of {})'.format(default_machine) + alias = next((m for m in machines if m.endswith(alias_suf)), None) + if alias is not None: + default_machine = alias.split(' ', 1)[0] + + return default_machine + + +class TestEnv(AbstractContextManager['TestEnv']):I'm getting CI failures here:
Traceback (most recent call last):
File "./check", line 23, in <module> from testenv import TestEnv File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 49, in <module> class TestEnv(AbstractContextManager['TestEnv']):TypeError: 'ABCMeta' object is not subscriptable
On the other hand, if I make it just AbstractContextManager without giving the type parameter, mypy complains:
Quoted 7 lines
Quoted 3 lines
testenv.py:49: error: Missing type parameters for generic type "ContextManager"
I guess I need to have another look into this tomorrow.
It may help to use typing.ContextManager instead of AbstractContextManager. mypy is OK with it, probably CI will be OK too..
Okay, I'm trying now if this change works (on top of v9, of course). If it does, I'll just squash it in. I also silenced some of the mypy warnings, so that I'm not testing with the following patch squashed in.
Kevin
tests/qemu-iotests/testenv.py +4
-5@@ -25,8 +25,7 @@ import collections import random import subprocess import glob -from contextlib import AbstractContextManager -from typing import Dict, Any, Optional +from typing import ContextManager, Dict, Any, Optional def isxfile(path: str) -> bool:@@ -50,7 +49,7 @@ def get_default_machine(qemu_prog: str) -> str: return default_machine -class TestEnv(AbstractContextManager['TestEnv']): +class TestEnv(ContextManager['TestEnv']): """ Manage system environment for running tests@@ -81,7 +80,7 @@ class TestEnv(AbstractContextManager['TestEnv']): return env - def init_directories(self): + def init_directories(self) -> None: """Init directory variables: PYTHONPATH TEST_DIR@@ -114,7 +113,7 @@ class TestEnv(AbstractContextManager['TestEnv']): self.output_dir = os.getcwd() # OUTPUT_DIR - def init_binaries(self): + def init_binaries(self) -> None: """Init binary path variables: PYTHON (for bash tests) QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG
tests/qemu-iotests/testenv.py class TestEnv(AbstractContextManager['TestEnv']): · line 127
Quoted 8 lines
tests/qemu-iotests/testenv.py +1
-1@@ -122,7 +121,7 @@ class TestEnv(AbstractContextManager['TestEnv']): """ self.python = sys.executable - def root(*names): + def root(*names: str) -> str: return os.path.join(self.build_root, *names) arch = os.uname().machine
Strange, that CI doesn't complain AbstractContextManager['...'] in testrunner.py.. Anyway, I think we should consistently use typing.ContextManager, if it works.
I've fixed a bit more mypy complains, I'll post squash-ins in v9 thread
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v8 2/5] iotests: add testenv.py
Am 26.01.2021 um 11:08 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 147 lines
Quoted 146 lines · Kevin Wolf
Am 26.01.2021 um 09:28 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 81 lines
Quoted 80 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 56 lines
Add TestEnv class, which will handle test environment in a new python iotests running framework.
Don't add compat=1.1 for qcow2 IMGOPTS, as v3 is default anyway.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/testenv.py | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+)create mode 100644
tests/qemu-iotests/testenv.pytests/qemu-iotests/testenv.py new file +39
-0@@ -0,0 +1,278 @@ +# TestEnv class to manage test environment variables. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import sys +import tempfile +from pathlib import Path +import shutil +import collections +import random +import subprocess +import glob +from contextlib import AbstractContextManager +from typing import Dict, Any, Optional + + +def get_default_machine(qemu_prog: str) -> str: + outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True, + universal_newlines=True, + stdout=subprocess.PIPE).stdout + + machines = outp.split('\n') + default_machine = next(m for m in machines if m.endswith(' (default)')) + default_machine = default_machine.split(' ', 1)[0]tests/qemu-iotests/testenv.py line 49
Quoted 10 lines
tests/qemu-iotests/testenv.py new file +10
-0+ + alias_suf = ' (alias of {})'.format(default_machine) + alias = next((m for m in machines if m.endswith(alias_suf)), None) + if alias is not None: + default_machine = alias.split(' ', 1)[0] + + return default_machine + + +class TestEnv(AbstractContextManager['TestEnv']):I'm getting CI failures here:
Traceback (most recent call last):
File "./check", line 23, in <module> from testenv import TestEnv File "/builds/.../qemu/tests/qemu-iotests/testenv.py", line 49, in <module> class TestEnv(AbstractContextManager['TestEnv']):TypeError: 'ABCMeta' object is not subscriptable
On the other hand, if I make it just AbstractContextManager without giving the type parameter, mypy complains:
Quoted 7 lines
Quoted 3 lines
testenv.py:49: error: Missing type parameters for generic type "ContextManager"
I guess I need to have another look into this tomorrow.
It may help to use typing.ContextManager instead of AbstractContextManager. mypy is OK with it, probably CI will be OK too..
Okay, I'm trying now if this change works (on top of v9, of course). If it does, I'll just squash it in. I also silenced some of the mypy warnings, so that I'm not testing with the following patch squashed in.
Kevin
tests/qemu-iotests/testenv.py +5
-6@@ -25,8 +25,7 @@ import collections import random import subprocess import glob -from contextlib import AbstractContextManager -from typing import Dict, Any, Optional +from typing import ContextManager, Dict, Any, Optional def isxfile(path: str) -> bool:@@ -50,7 +49,7 @@ def get_default_machine(qemu_prog: str) -> str: return default_machine -class TestEnv(AbstractContextManager['TestEnv']): +class TestEnv(ContextManager['TestEnv']): """ Manage system environment for running tests@@ -81,7 +80,7 @@ class TestEnv(AbstractContextManager['TestEnv']): return env - def init_directories(self): + def init_directories(self) -> None: """Init directory variables: PYTHONPATH TEST_DIR@@ -114,7 +113,7 @@ class TestEnv(AbstractContextManager['TestEnv']): self.output_dir = os.getcwd() # OUTPUT_DIR - def init_binaries(self): + def init_binaries(self) -> None: """Init binary path variables: PYTHON (for bash tests) QEMU_PROG, QEMU_IMG_PROG, QEMU_IO_PROG, QEMU_NBD_PROG, QSD_PROG@@ -122,7 +121,7 @@ class TestEnv(AbstractContextManager['TestEnv']): """ self.python = sys.executable - def root(*names): + def root(*names: str) -> str: return os.path.join(self.build_root, *names)
Quoted 5 lines
tests/qemu-iotests/testenv.py class TestEnv(AbstractContextManager['TestEnv']): · line 127
Quoted 1 line
tests/qemu-iotests/testenv.py +0
-0arch = os.uname().machineStrange, that CI doesn't complain AbstractContextManager['...'] in
testrunner.py.. Anyway, I think we should consistently use typing.ContextManager, if it works.
Ah, it probably does. The runs just take so long that I haven't got results yet. So I'll probably have to start another run.
Kevin
PATCH v8 3/5 iotests: add testrunner.py
In reply to Vladimir Sementsov-OgievskiyAdd TestRunner class, which will run tests in a new python iotests running framework.
There are some differences with current ./check behavior, most significant are:
- Consider all tests self-executable, just run them, don't run python by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- Instead of using "diff -w" which ignores all whitespace differences, manually strip whitespace at line end then use python difflib, which no longer ignores spacing mid-line
--- tests/qemu-iotests/testrunner.py | 366 +++++++++++++++++++++++++++++++ 1 file changed, 366 insertions(+) create mode 100644 tests/qemu-iotests/testrunner.py
tests/qemu-iotests/testrunner.py new file +366 -0
@@ -0,0 +1,366 @@ +# Class for actually running tests. +# +# Copyright (c) 2020-2021 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +from pathlib import Path +import datetime +import time +import difflib +import subprocess +import contextlib +import json +import termios +import sys +from contextlib import contextmanager +from contextlib import AbstractContextManager +from typing import List, Optional, Iterator, Any, Sequence + +from testenv import TestEnv + + +def silent_unlink(path: Path) -> None: + try: + path.unlink() + except OSError: + pass + + +def file_diff(file1: str, file2: str) -> List[str]: + with open(file1) as f1, open(file2) as f2: + # We want to ignore spaces at line ends. There are a lot of mess about + # it in iotests. + # TODO: fix all tests to not produce extra spaces, fix all .out files + # and use strict diff here! + seq1 = [line.rstrip() for line in f1] + seq2 = [line.rstrip() for line in f2] + res = [line.rstrip() + for line in difflib.unified_diff(seq1, seq2, file1, file2)] + return res + + +# We want to save current tty settings during test run, +# since an aborting qemu call may leave things screwed up. +@contextmanager +def savetty() -> Iterator[None]: + isterm = sys.stdin.isatty() + if isterm: + fd = sys.stdin.fileno() + attr = termios.tcgetattr(fd) + + try: + yield + finally: + if isterm: + termios.tcsetattr(fd, termios.TCSADRAIN, attr) + + +class LastElapsedTime(AbstractContextManager['LastElapsedTime']): + """ Cache for elapsed time for tests, to show it during new test run + + It is safe to use get() at any time. To use update(), you must either + use it inside with-block or use save() after update(). + """ + def __init__(self, cache_file: str, env: TestEnv) -> None: + self.env = env + self.cache_file = cache_file + + try: + with open(cache_file) as f: + self.cache = json.load(f) + except (OSError, ValueError): + self.cache = {} + + def get(self, test: str, + default: Optional[float] = None) -> Optional[float]: + if test not in self.cache: + return default + + if self.env.imgproto not in self.cache[test]: + return default + + return self.cache[test][self.env.imgproto].get(self.env.imgfmt, + default) + + def update(self, test: str, elapsed: float) -> None: + d = self.cache.setdefault(test, {}) + d = d.setdefault(self.env.imgproto, {}) + d[self.env.imgfmt] = elapsed + + def save(self) -> None: + with open(self.cache_file, 'w') as f: + json.dump(self.cache, f) + + def __enter__(self) -> 'LastElapsedTime': + return self + + def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: + self.save() + + +class TestResult: + def __init__(self, status: str, description: str = '', + elapsed: Optional[float] = None, diff: Sequence[str] = (), + casenotrun: str = '', interrupted: bool = False) -> None: + self.status = status + self.description = description + self.elapsed = elapsed + self.diff = diff + self.casenotrun = casenotrun + self.interrupted = interrupted + + +class TestRunner(AbstractContextManager['TestRunner']): + def __init__(self, env: TestEnv, makecheck: bool = False, + color: str = 'auto') -> None: + self.env = env + self.test_run_env = self.env.get_env() + self.makecheck = makecheck + self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env) + + assert color in ('auto', 'on', 'off') + self.color = (color == 'on') or (color == 'auto' and + sys.stdout.isatty()) + + self._stack: contextlib.ExitStack + + def __enter__(self) -> 'TestRunner': + self._stack = contextlib.ExitStack() + self._stack.enter_context(self.env) + self._stack.enter_context(self.last_elapsed) + self._stack.enter_context(savetty()) + return self + + def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: + self._stack.close() + + def test_print_one_line(self, test: str, starttime: str, + endtime: Optional[str] = None, status: str = '...', + lasttime: Optional[float] = None, + thistime: Optional[float] = None, + description: str = '', + test_field_width: Optional[int] = None, + end: str = '\n') -> None: + """ Print short test info before/after test run """ + test = os.path.basename(test) + + if test_field_width is None: + test_field_width = 8 + + if self.makecheck and status != '...': + if status and status != 'pass': + status = f' [{status}]' + else: + status = '' + + print(f' TEST iotest-{self.env.imgfmt}: {test}{status}') + return + + if lasttime: + lasttime_s = f' (last: {lasttime:.1f}s)' + else: + lasttime_s = '' + if thistime: + thistime_s = f'{thistime:.1f}s' + else: + thistime_s = '...' + + if endtime: + endtime = f'[{endtime}]' + else: + endtime = '' + + if self.color: + if status == 'pass': + col = '\033[32m' + elif status == 'fail': + col = '\033[1m\033[31m' + elif status == 'not run': + col = '\033[33m' + else: + col = '' + + col_end = '\033[0m' + else: + col = '' + col_end = '' + + print(f'{test:{test_field_width}} {col}{status:10}{col_end} ' + f'[{starttime}] {endtime:13}{thistime_s:5} {lasttime_s:14} ' + f'{description}', end=end) + + def find_reference(self, test: str) -> str: + if self.env.cachemode == 'none': + ref = f'{test}.out.nocache' + if os.path.isfile(ref): + return ref + + ref = f'{test}.out.{self.env.imgfmt}' + if os.path.isfile(ref): + return ref + + ref = f'{test}.{self.env.qemu_default_machine}.out' + if os.path.isfile(ref): + return ref + + return f'{test}.out' + + def do_run_test(self, test: str) -> TestResult: + f_test = Path(test) + f_bad = Path(f_test.name + '.out.bad') + f_notrun = Path(f_test.name + '.notrun') + f_casenotrun = Path(f_test.name + '.casenotrun') + f_reference = Path(self.find_reference(test)) + + if not f_test.exists(): + return TestResult(status='fail', + description=f'No such test file: {f_test}') + + if not os.access(str(f_test), os.X_OK): + sys.exit(f'Not executable: {f_test}') + + if not f_reference.exists(): + return TestResult(status='fail', + description='No qualified output ' + f'(expected {f_reference})') + + for p in (f_bad, f_notrun, f_casenotrun): + silent_unlink(p) + + args = [str(f_test.resolve())] + if self.env.debug: + args.append('-d') + + with f_test.open() as f: + try: + if f.readline() == '#!/usr/bin/env python3': + args.insert(0, self.env.python) + except UnicodeDecodeError: # binary test? for future. + pass + + env = os.environ.copy() + env.update(self.test_run_env) + + t0 = time.time() + with f_bad.open('w') as f: + proc = subprocess.Popen(args, cwd=str(f_test.parent), env=env, + stdout=f, stderr=subprocess.STDOUT) + try: + proc.wait() + except KeyboardInterrupt: + proc.terminate() + proc.wait() + return TestResult(status='not run', + description='Interrupted by user', + interrupted=True) + ret = proc.returncode + + elapsed = round(time.time() - t0, 1) + + if ret != 0: + return TestResult(status='fail', elapsed=elapsed, + description=f'failed, exit status {ret}', + diff=file_diff(str(f_reference), str(f_bad))) + + if f_notrun.exists(): + return TestResult(status='not run', elapsed=elapsed, + description=f_notrun.read_text().strip()) + + casenotrun = '' + if f_casenotrun.exists(): + casenotrun = f_casenotrun.read_text() + + diff = file_diff(str(f_reference), str(f_bad)) + if diff: + return TestResult(status='fail', elapsed=elapsed, + description=f'output mismatch (see {f_bad})', + diff=diff, casenotrun=casenotrun) + else: + f_bad.unlink() + self.last_elapsed.update(test, elapsed) + return TestResult(status='pass', elapsed=elapsed, + casenotrun=casenotrun) + + def run_test(self, test: str, + test_field_width: Optional[int] = None) -> TestResult: + last_el = self.last_elapsed.get(test) + start = datetime.datetime.now().strftime('%H:%M:%S') + + self.test_print_one_line(test=test, starttime=start, lasttime=last_el, + end='\r', test_field_width=test_field_width) + + res = self.do_run_test(test) + + end = datetime.datetime.now().strftime('%H:%M:%S') + self.test_print_one_line(test=test, status=res.status, + starttime=start, endtime=end, + lasttime=last_el, thistime=res.elapsed, + description=res.description, + test_field_width=test_field_width) + + if res.casenotrun: + print(res.casenotrun) + + return res + + def run_tests(self, tests: List[str]) -> None: + n_run = 0 + failed = [] + notrun = [] + casenotrun = [] + + if not self.makecheck: + self.env.print_env() + print() + + test_field_width = max(len(os.path.basename(t)) for t in tests) + 2 + + for t in tests: + name = os.path.basename(t) + res = self.run_test(t, test_field_width=test_field_width) + + assert res.status in ('pass', 'fail', 'not run') + + if res.casenotrun: + casenotrun.append(t) + + if res.status != 'not run': + n_run += 1 + + if res.status == 'fail': + failed.append(name) + if self.makecheck: + self.env.print_env() + if res.diff: + print('\n'.join(res.diff)) + elif res.status == 'not run': + notrun.append(name) + + if res.interrupted: + break + + if notrun: + print('Not run:', ' '.join(notrun)) + + if casenotrun: + print('Some cases not run in:', ' '.join(casenotrun)) + + if failed: + print('Failures:', ' '.join(failed)) + print(f'Failed {len(failed)} of {n_run} iotests') + else: + print(f'Passed all {n_run} iotests')
Signature · 1 line
2.29.2
PATCH v8 4/5 iotests: rewrite check into python
In reply to Vladimir Sementsov-OgievskiyJust use classes introduced in previous three commits. Behavior difference is described in these three commits.
Drop group file, as it becomes unused.
Drop common.env: now check is in python, and for tests we use same python interpreter that runs the check itself. Use build environment PYTHON in check-block instead, to keep "make check" use the same python.
Checking for virtio-blk moved to iotests.py, as it actually iotests.py dependency. Actually not all python iotests depend on it, so in future it may be refactored to checked only when really needed.
--- Makefile | 1 - tests/check-block.sh | 2 +- tests/qemu-iotests/check | 1095 ++++-------------------------- tests/qemu-iotests/common.env.in | 3 - tests/qemu-iotests/group | 321 --------- tests/qemu-iotests/iotests.py | 8 + tests/qemu-iotests/meson.build | 3 - 7 files changed, 138 insertions(+), 1295 deletions(-) delete mode 100644 tests/qemu-iotests/common.env.in delete mode 100644 tests/qemu-iotests/group
Makefile +0 -1
@@ -236,7 +236,6 @@ distclean: clean rm -f config-host.mak config-host.h* rm -f tests/tcg/config-*.mak rm -f config-all-disas.mak config.status - rm -f tests/qemu-iotests/common.env rm -f roms/seabios/config.mak roms/vgabios/config.mak rm -f qemu-plugins-ld.symbols qemu-plugins-ld64.symbols rm -f *-config-target.h *-config-devices.mak *-config-devices.h
tests/check-block.sh +1 -1
@@ -76,7 +76,7 @@ export QEMU_CHECK_BLOCK_AUTO=1 ret=0 for fmt in $format_list ; do - ./check -makecheck -$fmt $group || ret=1 + ${PYTHON} ./check -makecheck -$fmt $group || ret=1 done exit $ret
tests/qemu-iotests/check +129 -966
1,103 lines not on this page · raw message
tests/qemu-iotests/common.env.in deleted +0 -3
@@ -1,3 +0,0 @@ -# Automatically generated by configure - do not modify - -export PYTHON='@PYTHON@'
tests/qemu-iotests/group deleted +0 -321
321 lines not on this page · raw message
tests/qemu-iotests/iotests.py +8 -0
@@ -1120,6 +1120,13 @@ def _verify_formats(required_formats: Sequence[str] = ()) -> None: if usf_list: notrun(f'formats {usf_list} are not whitelisted') + +def _verify_virtio_blk() -> None: + out = qemu_pipe('-M', 'none', '-device', 'help') + if 'virtio-blk' not in out: + notrun('Missing virtio-blk in QEMU binary') + + def supports_quorum(): return 'quorum' in qemu_img_pipe('--help')
@@ -1305,6 +1312,7 @@ def execute_setup_common(supported_fmts: Sequence[str] = (), _verify_cache_mode(supported_cache_modes) _verify_aio_mode(supported_aio_modes) _verify_formats(required_fmts) + _verify_virtio_blk() return debug
tests/qemu-iotests/meson.build +0 -3
@@ -3,6 +3,3 @@ if 'CONFIG_LINUX' in config_host else socket_scm_helper = [] endif -configure_file(output: 'common.env', - input: files('common.env.in'), - configuration: {'PYTHON': python.full_path()})
Signature · 1 line
2.29.2
PATCH v8 5/5 iotests: rename and move 169 and 199 tests
In reply to Vladimir Sementsov-OgievskiyRename bitmaps migration tests and move them to tests subdirectory to demonstrate new human-friendly test naming.
---
tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} | 0
.../{199.out => tests/migrate-bitmaps-postcopy-test.out} | 0
tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} | 0
tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} | 0
4 files changed, 0 insertions(+), 0 deletions(-)
rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (100%)
rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%)
rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (100%)
rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test renamed from tests/qemu-iotests/199 +0 -0
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test.out renamed from tests/qemu-iotests/199.out +0 -0
tests/qemu-iotests/tests/migrate-bitmaps-test renamed from tests/qemu-iotests/169 +0 -0
tests/qemu-iotests/tests/migrate-bitmaps-test.out renamed from tests/qemu-iotests/169.out +0 -0
Signature · 1 line
2.29.2
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 13 lines
v8:
about linters:
I didn't modify 297, as Max already staged 297 modifications to test all files.
Also, now I have two complains:
(quoted diff) +4
-0+************* Module testenv +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) +************* Module testrunner +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) Success: no issues found in 5 source files
Quoted 2 lines
And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can do some refactoring as a follow-up.
I don't think these warning are very helpful, I would agree with disabling them (even globally).
When testing this with the other image formats, I found some problems.
1. The first one probably means that we have changed the order of some
checks: 150 and 178 have reference outputs for raw and qcow2, but no other formats.
Previously, the _supported_fmt line in the test would just skip the test:
$ build/check -vhdx 150 178 150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx 178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx
Now we seem to test first if a reference output exists and fail:
150 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/150.out) 178 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/178.out)
2. Test case 146 for vpc passed previously, it fails now. This seems to be because of whitespace that is checked now.
3. Skipped tests display either "..." or "0.1s" as the elapsed time. The old check implementation didn't display any time for them. I don't really mind either of the three ways, but a consistent result would be nice.
Kevin
Quoted 37 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 13 lines
v8:
about linters:
I didn't modify 297, as Max already staged 297 modifications to test all files.
Also, now I have two complains:
(quoted diff) +4
-0+************* Module testenv +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) +************* Module testrunner +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) Success: no issues found in 5 source filesQuoted 2 lines
And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can do some refactoring as a follow-up.
I don't think these warning are very helpful, I would agree with disabling them (even globally).
When testing this with the other image formats, I found some problems.
1. The first one probably means that we have changed the order of some
checks: 150 and 178 have reference outputs for raw and qcow2, but no other formats.Previously, the
_supported_fmtline in the test would just skip the test:$ build/check -vhdx 150 178
150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx
178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdxNow we seem to test first if a reference output exists and fail:
150 fail [16:49:18] [16:49:18] ... No qualified output (expected
/home/kwolf/source/qemu/tests/qemu-iotests/150.out)
178 fail [16:49:18] [16:49:18] ... No qualified output (expected/home/kwolf/source/qemu/tests/qemu-iotests/178.out)
Quoted 8 lines
2. Test case 146 for vpc passed previously, it fails now. This seems to be because of whitespace that is checked now.
3. Skipped tests display either "..." or "0.1s" as the elapsed time. The old check implementation didn't display any time for them. I don't really mind either of the three ways, but a consistent result would be nice.
Thanks for testing! I should have done it myself actually :/
Will fix soon.
Signature · 2 lines
Best regards, Vladimir
Quoted 28 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 13 lines
v8:
about linters:
I didn't modify 297, as Max already staged 297 modifications to test all files.
Also, now I have two complains:
(quoted diff) +4
-0+************* Module testenv +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) +************* Module testrunner +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) Success: no issues found in 5 source filesQuoted 2 lines
And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can do some refactoring as a follow-up.
I don't think these warning are very helpful, I would agree with disabling them (even globally).
When testing this with the other image formats, I found some problems.
1. The first one probably means that we have changed the order of some
checks: 150 and 178 have reference outputs for raw and qcow2, but no other formats.Previously, the
_supported_fmtline in the test would just skip the test:
Quoted 8 lines
$ build/check -vhdx 150 178
150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx
178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdxNow we seem to test first if a reference output exists and fail:
150 fail [16:49:18] [16:49:18] ... No qualified output (expected
/home/kwolf/source/qemu/tests/qemu-iotests/150.out)
178 fail [16:49:18] [16:49:18] ... No qualified output (expected/home/kwolf/source/qemu/tests/qemu-iotests/178.out)
Hmm. Still, I do think that new order is better: no reason to run the test, when we don't have corresponding .out file. So, may be just change it into "not run", with same "No qualified output (expected ..)" message, what do you think?
Quoted 11 lines
2. Test case 146 for vpc passed previously, it fails now. This seems to be because of whitespace that is checked now.
3. Skipped tests display either "..." or "0.1s" as the elapsed time. The old check implementation didn't display any time for them. I don't really mind either of the three ways, but a consistent result would be nice.
Kevin
Signature · 2 lines
Best regards, Vladimir
Am 25.01.2021 um 17:36 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 35 lines
Quoted 34 lines · Kevin Wolf
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 13 lines
v8:
about linters:
I didn't modify 297, as Max already staged 297 modifications to test all files.
Also, now I have two complains:
(quoted diff) +4
-0+************* Module testenv +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) +************* Module testrunner +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) Success: no issues found in 5 source filesQuoted 2 lines
And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can do some refactoring as a follow-up.
I don't think these warning are very helpful, I would agree with disabling them (even globally).
When testing this with the other image formats, I found some problems.
1. The first one probably means that we have changed the order of some
checks: 150 and 178 have reference outputs for raw and qcow2, but no other formats.Previously, the
_supported_fmtline in the test would just skip the test:$ build/check -vhdx 150 178
150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx
178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdxNow we seem to test first if a reference output exists and fail:
Quoted 8 lines
Quoted 2 lines
150 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/150.out) 178 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/178.out)Hmm. Still, I do think that new order is better: no reason to run the test, when we don't have corresponding .out file. So, may be just change it into "not run", with same "No qualified output (expected ..)" message, what do you think?
Works for me.
(There would actually be a reason to run the test, namely for creating the reference output when you add the test. But this didn't leave a .bad file behind before either, and just doing 'touch 123.out' first is easy enough anyway.)
Kevin