Skip to contentSkip to conversation

iotests: Fix 129 and expand 297’s reach

PATCH In review Max Reitz 3 participants 17 messages Jan 18, 2021 – Jan 27, 2021

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v5 cover iotests: Fix 129 and expand 297’s reach

Cover letters:
v1: https://lists.nongnu.org/archive/html/qemu-block/2021-01/msg00254.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2021-01/msg00296.html
v3: https://lists.nongnu.org/archive/html/qemu-block/2021-01/msg00371.html
v4: https://lists.nongnu.org/archive/html/qemu-block/2021-01/msg00371.html

git:
https://github.com/XanClic/qemu.git fix-129-2-v5
https://git.xanclic.moe/XanClic/qemu.git fix-129-2-v5

Change in v5 (from v4):
- Patch 2:
- Construct PYTHONPATH/MYPYPATH platform-independently
- Use os.pathsep instead of hard-coding ':'
- Use os.path.dirname(__file__) as the basis for '../../python' instead of adding a relative path to PYTHONPATH
- Use os.path.join() instead of joining with '/'

git-backport-diff of v4 <-> v5:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/10:[----] [--] 'iotests.py: Assume a couple of variables as given' 002/10:[0006] [FC] 'iotests/297: Rewrite in Python and extend reach' 003/10:[----] [--] 'iotests: Move try_remove to iotests.py' 004/10:[----] [--] 'iotests/129: Remove test images in tearDown()' 005/10:[----] [--] 'iotests/129: Do not check @busy' 006/10:[----] [--] 'iotests/129: Use throttle node' 007/10:[----] [--] 'iotests/129: Actually test a commit job' 008/10:[----] [--] 'iotests/129: Limit mirror job's buffer size' 009/10:[----] [--] 'iotests/129: Clean up pylint and mypy complaints' 010/10:[----] [--] 'iotests/300: Clean up pylint and mypy complaints'

Max Reitz (10):
iotests.py: Assume a couple of variables as given iotests/297: Rewrite in Python and extend reach
iotests: Move try_remove to iotests.py iotests/129: Remove test images in tearDown() iotests/129: Do not check @busy iotests/129: Use throttle node iotests/129: Actually test a commit job iotests/129: Limit mirror job's buffer size iotests/129: Clean up pylint and mypy complaints iotests/300: Clean up pylint and mypy complaints

 tests/qemu-iotests/124        |   8 +--
 tests/qemu-iotests/129        |  72 +++++++++++++---------
 tests/qemu-iotests/297        | 112 +++++++++++++++++++++++++++-------
 tests/qemu-iotests/297.out    |   5 +-
 tests/qemu-iotests/300        |  19 ++++--
 tests/qemu-iotests/iotests.py |  37 +++++------
 6 files changed, 171 insertions(+), 82 deletions(-)
Signature · 1 line
2.29.2
author
#1 lore raw

PATCH v5 1/10 iotests.py: Assume a couple of variables as given

There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and qemu_default_machine to be set, and aborts if they are not).

Using .get() this way has the disadvantage of returning an Optional[str] type, which mypy will complain about when tests just assume these values to be str.

Use [] instead, which raises a KeyError for environment variables that are not set. When this exception is raised, catch it and move the abort code from execute_setup_common() there.

Drop the 'assert iotests.sock_dir is not None' from iotest 300, because that sort of thing is precisely what this patch wants to prevent.

---
 tests/qemu-iotests/300        |  1 -
 tests/qemu-iotests/iotests.py | 26 +++++++++++++-------------
 2 files changed, 13 insertions(+), 14 deletions(-)
tests/qemu-iotests/300 +0 -1
index 5b75121b84..b864a21d5e 100755
@@ -27,7 +27,6 @@ import qemu
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
 
-assert iotests.sock_dir is not None
 mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
 
 
tests/qemu-iotests/iotests.py +13 -13
index dcdcd0387f..52facb8e04 100644
@@ -75,12 +75,20 @@ qemu_opts = os.environ.get('QEMU_OPTIONS', '').strip().split(' ')
 
 imgfmt = os.environ.get('IMGFMT', 'raw')
 imgproto = os.environ.get('IMGPROTO', 'file')
-test_dir = os.environ.get('TEST_DIR')
-sock_dir = os.environ.get('SOCK_DIR')
 output_dir = os.environ.get('OUTPUT_DIR', '.')
-cachemode = os.environ.get('CACHEMODE')
-aiomode = os.environ.get('AIOMODE')
-qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
+
+try:
+    test_dir = os.environ['TEST_DIR']
+    sock_dir = os.environ['SOCK_DIR']
+    cachemode = os.environ['CACHEMODE']
+    aiomode = os.environ['AIOMODE']
+    qemu_default_machine = os.environ['QEMU_DEFAULT_MACHINE']
+except KeyError:
+    # We are using these variables as proxies to indicate that we're
+    # not being run via "check". There may be other things set up by
+    # "check" that individual test cases rely on.
+    sys.stderr.write('Please run this test via the "check" script\n')
+    sys.exit(os.EX_USAGE)
 
 socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
 
@@ -1294,14 +1302,6 @@ def execute_setup_common(supported_fmts: Sequence[str] = (),
     """
     # Note: Python 3.6 and pylint do not like 'Collection' so use 'Sequence'.
 
-    # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
-    # indicate that we're not being run via "check". There may be
-    # other things set up by "check" that individual test cases rely
-    # on.
-    if test_dir is None or qemu_default_machine is None:
-        sys.stderr.write('Please run this test via the "check" script\n')
-        sys.exit(os.EX_USAGE)
-
     debug = '-d' in sys.argv
     if debug:
         sys.argv.remove('-d')
Signature · 1 line
2.29.2
Reviewed-by on v5 1/10
#2 lore raw

Re: [PATCH v5 01/10] iotests.py: Assume a couple of variables as given

Quoted 6 lines · Max Reitz

There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and qemu_default_machine to be set, and aborts if they are not).

Quoted 8 lines

Using .get() this way has the disadvantage of returning an Optional[str] type, which mypy will complain about when tests just assume these values to be str.

Use [] instead, which raises a KeyError for environment variables that are not set. When this exception is raised, catch it and move the abort code from execute_setup_common() there.

Good idea.

Quoted 6 lines

Drop the 'assert iotests.sock_dir is not None' from iotest 300, because that sort of thing is precisely what this patch wants to prevent.

author
#3 lore raw

PATCH v5 2/10 iotests/297: Rewrite in Python and extend reach

In reply to Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted. This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment. Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments. TODO is fine, we do not need 297 to complain about such comments.

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
 tests/qemu-iotests/297     | 112 +++++++++++++++++++++++++++++--------
 tests/qemu-iotests/297.out |   5 +-
 2 files changed, 92 insertions(+), 25 deletions(-)
tests/qemu-iotests/297 +90 -22
index 5c5420712b..e3db3e061e 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
 #
 # Copyright (C) 2020 Red Hat, Inc.
 #
@@ -15,30 +15,98 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
 
-status=1	# failure is the default!
+import iotests
 
-# get standard environment
-. ./common.rc
 
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
 
-pylint-3 --score=n iotests.py
 
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
 
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None
+        except UnicodeDecodeError:  # Ignore binary files
+            return False
+
+
+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+             if is_python_file(filename)]
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))
+
+    print('=== pylint ===')
+    sys.stdout.flush()
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)
+    env = os.environ.copy()
+    qemu_module_path = os.path.join(os.path.dirname(__file__),
+                                    '..', '..', 'python')
+    try:
+        env['PYTHONPATH'] += os.pathsep + qemu_module_path
+    except KeyError:
+        env['PYTHONPATH'] = qemu_module_path
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+                   env=env, check=False)
+
+    print('=== mypy ===')
+    sys.stdout.flush()
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT,
+                           universal_newlines=True)
+
+        if p.returncode != 0:
+            print(p.stdout)
+
+
+for linter in ('pylint-3', 'mypy'):
+    if shutil.which(linter) is None:
+        iotests.notrun(f'{linter} not found')
+
+iotests.script_main(run_linters)
tests/qemu-iotests/297.out +2 -3
index 6acc843649..f2e1314d10 100644
@@ -1,3 +1,2 @@
-QA output created by 297
-Success: no issues found in 1 source file
-*** done
+=== pylint ===
+=== mypy ===
Signature · 1 line
2.29.2
Reviewed-by on v5 2/10
#4 lore raw

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

Quoted 27 lines · Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted. This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment. Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments. TODO is fine, we do not need 297 to complain about such comments.

Quoted 7 lines

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v5 2/10
#5 lore raw

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

In reply to Max Reitz
Quoted 5 lines · Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.)

Chiming in to say that I tried to tackle this before; I wrote some preliminary cleanups and sent to the list as an "WIP RFC" or something like that in earlyish 2020. I paid attention to qed.py and the other non-numerical files.

Maybe badly rotted by now, I don't know.

Quoted 4 lines

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

I agree. Stop the bleeding first and worry about the rest after.

Quoted 6 lines

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

Quoted 8 lines

- For the pylint invocation, PYTHONPATH is adjusted. This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment. Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

Does this solve an observed problem, or is it preventative? I ran into trouble once by pointing mypy to my system python libraries; it seemed to have a check that explicitly warned me against such tricks.

I guess for now, if it works, it works. :o)

Quoted 4 lines

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments. TODO is fine, we do not need 297 to complain about such comments.

Agreed. You can also edit pylintrc to choose which keywords trigger the check -- "TODO" is probably fine, but "FIXME" is maybe a shade worse. Season to taste.

Quoted 17 lines

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
  tests/qemu-iotests/297     | 112 +++++++++++++++++++++++++++++--------
  tests/qemu-iotests/297.out |   5 +-
  2 files changed, 92 insertions(+), 25 deletions(-)
tests/qemu-iotests/297 +0 -0
index 5c5420712b..e3db3e061e 100755

tests/qemu-iotests/297 line 3

Quoted 4 lines
tests/qemu-iotests/297 +1 -1
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
  #
  # Copyright (C) 2020 Red Hat, Inc.

You could bump it up, if you wanted.

Quoted 78 lines
tests/qemu-iotests/297 +44 -21
  #
@@ -15,30 +15,98 @@
  # You should have received a copy of the GNU General Public License
  # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
 
-status=1	# failure is the default!
+import iotests
 
-# get standard environment
-. ./common.rc
 
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
 
-pylint-3 --score=n iotests.py
 
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
 
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None
+        except UnicodeDecodeError:  # Ignore binary files
+            return False
+
+
 
 
+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+             if is_python_file(filename)]
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))
+
+    print('=== pylint ===')
+    sys.stdout.flush()
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)

tests/qemu-iotests/297 line 77

Quoted 10 lines
tests/qemu-iotests/297 +10 -0
+    env = os.environ.copy()
+    qemu_module_path = os.path.join(os.path.dirname(__file__),
+                                    '..', '..', 'python')
+    try:
+        env['PYTHONPATH'] += os.pathsep + qemu_module_path
+    except KeyError:
+        env['PYTHONPATH'] = qemu_module_path
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+                   env=env, check=False)
+

Is there a reason to use 'pylint-3' here? I realize that this is not something you are introducing, but it jogged the question loose for me.

'pylint-3' is, I believe, a fedora-ism. The pip package for pylint installs only a 'pylint' script. It might be better to just use it without the suffix.

If we are concerned about accidentally invoking a python 2 version of pylint that might be installed on a system, I'd then recommend doing something like this:

'python3 -m pylint [...]'

Quoted 41 lines
tests/qemu-iotests/297 +35 -0
+    print('=== mypy ===')
+    sys.stdout.flush()
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT,
+                           universal_newlines=True)
+
+        if p.returncode != 0:
+            print(p.stdout)
+
+
+for linter in ('pylint-3', 'mypy'):
+    if shutil.which(linter) is None:
+        iotests.notrun(f'{linter} not found')
+
+iotests.script_main(run_linters)
tests/qemu-iotests/297.out +0 -0
index 6acc843649..f2e1314d10 100644

tests/qemu-iotests/297.out line 2

Quoted 5 lines
tests/qemu-iotests/297.out +2 -3
@@ -1,3 +1,2 @@
-QA output created by 297
-Success: no issues found in 1 source file
-*** done
+=== pylint ===
+=== mypy ===
author
#6 lore raw

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

Quoted 27 lines · John Snow
Quoted 5 lines · Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory.  Of course, most of them do not pass, so there is an extensive skip list for now.  (The only files that do pass are 209, 254, 283, and iotests.py.)

Chiming in to say that I tried to tackle this before; I wrote some preliminary cleanups and sent to the list as an "WIP RFC" or something like that in earlyish 2020. I paid attention to qed.py and the other non-numerical files.

Maybe badly rotted by now, I don't know.

Quoted 4 lines

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

I agree. Stop the bleeding first and worry about the rest after.

Quoted 9 lines

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted.  This mirrors setting MYPYPATH for mypy.

Quoted 7 lines
Quoted 5 lines

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment.  Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

Does this solve an observed problem, or is it preventative? I ran into trouble once by pointing mypy to my system python libraries; it seemed to have a check that explicitly warned me against such tricks.

Yes, that happens when you derive it from sys.path. (Which I tried at one point, and then ran into that exact problem.)

PYTHONPATH generally doesn’t include the system libraries, though, generally it shouldn’t even be set for the iotests. So the only thing that’s put in there is ../../python/, and we need that in MYPYPATH, too.

(As I wrote, perhaps in the future the check script will add ../../python/, so we don’t need to do that anywhere in the iotests, and then it makes more sense why MYPYPATH should just be $PYTHONPATH.)

Quoted 8 lines

I guess for now, if it works, it works. :o)

Quoted 4 lines

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments.  TODO is fine, we do not need 297 to complain about such comments.

Agreed. You can also edit pylintrc to choose which keywords trigger the check -- "TODO" is probably fine, but "FIXME" is maybe a shade worse. Season to taste.

Yes, definitely a matter of taste. I kind of like pylint to complain about TODO when I’m running it explicitly, so, well.

Quoted 12 lines
Quoted 12 lines

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
  tests/qemu-iotests/297     | 112 +++++++++++++++++++++++++++++-------- tests/qemu-iotests/297.out |   5 +-

  2 files changed, 92 insertions(+), 25 deletions(-)
Quoted 11 lines

tests/qemu-iotests/297 line 3

Quoted 9 lines
tests/qemu-iotests/297 +1 -1
index 5c5420712b..e3db3e061e 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
   #
   # Copyright (C) 2020 Red Hat, Inc.

You could bump it up, if you wanted.

Do I, though? :)

Quoted 94 lines
Quoted 88 lines
tests/qemu-iotests/297 +34 -22
   #
@@ -15,30 +15,98 @@
   # You should have received a copy of the GNU General Public License
   # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
-status=1    # failure is the default!
+import iotests
-# get standard environment
-. ./common.rc
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', 
 '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', 
 '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', 
 '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', 
 '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', 
 '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', 
 '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', 
 '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
-pylint-3 --score=n iotests.py
-MYPYPATH=../../python/ mypy --warn-unused-configs 
--disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None
+        except UnicodeDecodeError:  # Ignore binary files
+            return False
+
+
 
 
+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - 

set(SKIP_FILES))
+             if is_python_file(filename)]

tests/qemu-iotests/297 +17 -0
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))
+
+    print('=== pylint ===')
+    sys.stdout.flush()
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)
+    env = os.environ.copy()
+    qemu_module_path = os.path.join(os.path.dirname(__file__),
+                                    '..', '..', 'python')
+    try:
+        env['PYTHONPATH'] += os.pathsep + qemu_module_path
+    except KeyError:
+        env['PYTHONPATH'] = qemu_module_path
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', 
Quoted 3 lines

*files),
+                   env=env, check=False)
+

Is there a reason to use 'pylint-3' here? I realize that this is not something you are introducing, but it jogged the question loose for me.

Quoted 5 lines

'pylint-3' is, I believe, a fedora-ism. The pip package for pylint installs only a 'pylint' script. It might be better to just use it without the suffix.

If we are concerned about accidentally invoking a python 2 version of pylint that might be installed on a system, I'd then recommend doing something like this:

'python3 -m pylint [...]'

Sounds reasonable to me (can’t find pylint-3 on Arch, for example). As a follow-up, that is, because I’m afraid I’ve already put this series in a pull request...

Max

Quoted 48 lines
Quoted 41 lines

+    print('=== mypy ===')
+    sys.stdout.flush()

tests/qemu-iotests/297 +33 -0
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT,
+                           universal_newlines=True)
+
+        if p.returncode != 0:
+            print(p.stdout)
+
+
+for linter in ('pylint-3', 'mypy'):
+    if shutil.which(linter) is None:
+        iotests.notrun(f'{linter} not found')
+
+iotests.script_main(run_linters)
tests/qemu-iotests/297.out +0 -0
index 6acc843649..f2e1314d10 100644

tests/qemu-iotests/297.out line 2

Quoted 5 lines
tests/qemu-iotests/297.out +2 -3
@@ -1,3 +1,2 @@
-QA output created by 297
-Success: no issues found in 1 source file
-*** done
+=== pylint ===
+=== mypy ===
#7 lore raw

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

Quoted 4 lines · Max Reitz

Sounds reasonable to me (can’t find pylint-3 on Arch, for example).  As a follow-up, that is, because I’m afraid I’ve already put this series in a pull request...

Max

Ah, yeah. I checked master but didn't hunt through the maintainer branches. It's not a big deal, since it's an existing problem anyway.

Thanks for working on this!

--js

author
#8 lore raw

PATCH v5 3/10 iotests: Move try_remove to iotests.py

In reply to Max Reitz
---
 tests/qemu-iotests/124        |  8 +-------
 tests/qemu-iotests/iotests.py | 11 +++++++----
 2 files changed, 8 insertions(+), 11 deletions(-)
tests/qemu-iotests/124 +1 -7
index 3705cbb6b3..e40eeb50b9 100755
@@ -22,6 +22,7 @@
 
 import os
 import iotests
+from iotests import try_remove
 
 
 def io_write_patterns(img, patterns):
@@ -29,13 +30,6 @@ def io_write_patterns(img, patterns):
         iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
 
-def try_remove(img):
-    try:
-        os.remove(img)
-    except OSError:
-        pass
-
-
 def transaction_action(action, **kwargs):
     return {
         'type': action,
tests/qemu-iotests/iotests.py +7 -4
index 52facb8e04..a69b4cdc4e 100644
@@ -523,12 +523,15 @@ class FilePath:
         return False
 
 
+def try_remove(img):
+    try:
+        os.remove(img)
+    except OSError:
+        pass
+
 def file_path_remover():
     for path in reversed(file_path_remover.paths):
-        try:
-            os.remove(path)
-        except OSError:
-            pass
+        try_remove(path)
 
 
 def file_path(*names, base_dir=test_dir):
Signature · 1 line
2.29.2
Reviewed-by on v5 3/10
#9 lore raw

Re: [PATCH v5 03/10] iotests: Move try_remove to iotests.py

Quoted 50 lines · Max Reitz
---
  tests/qemu-iotests/124        |  8 +-------
  tests/qemu-iotests/iotests.py | 11 +++++++----
  2 files changed, 8 insertions(+), 11 deletions(-)
tests/qemu-iotests/124 +1 -7
index 3705cbb6b3..e40eeb50b9 100755
@@ -22,6 +22,7 @@
 
  import os
  import iotests
+from iotests import try_remove
 
 
  def io_write_patterns(img, patterns):
@@ -29,13 +30,6 @@ def io_write_patterns(img, patterns):
          iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
 
-def try_remove(img):
-    try:
-        os.remove(img)
-    except OSError:
-        pass
-
-
  def transaction_action(action, **kwargs):
      return {
          'type': action,
tests/qemu-iotests/iotests.py +6 -0
index 52facb8e04..a69b4cdc4e 100644
@@ -523,12 +523,15 @@ class FilePath:
          return False
 
 
+def try_remove(img):
+    try:
+        os.remove(img)
+    except OSError:
+        pass
+

tests/qemu-iotests/iotests.py class FilePath: · line 537

Quoted 10 lines
tests/qemu-iotests/iotests.py +1 -4
  def file_path_remover():
      for path in reversed(file_path_remover.paths):
-        try:
-            os.remove(path)
-        except OSError:
-            pass
+        try_remove(path)
 
 
  def file_path(*names, base_dir=test_dir):

For what it's worth, at the time I wrote this I barely knew Python. I'd use FileNotFoundError instead now, which is a subclass of OSError.

Not your baby, not your problem.

author
#10 lore raw

PATCH v5 4/10 iotests/129: Remove test images in tearDown()

In reply to Max Reitz
---
 tests/qemu-iotests/129 | 2 ++
 1 file changed, 2 insertions(+)
tests/qemu-iotests/129 +2 -0
index 0e13244d85..2fc65ada6a 100755
@@ -47,6 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
                              **params)
         self.vm.shutdown()
+        for img in (self.test_img, self.target_img, self.base_img):
+            iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
Signature · 1 line
2.29.2
author
#11 lore raw

PATCH v5 5/10 iotests/129: Do not check @busy

In reply to Max Reitz

@busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3).

Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information.

---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index 2fc65ada6a..dd23bb2e5a 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
-        self.assert_qmp(result, 'return[0]/busy', True)
+        self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
     def test_drive_mirror(self):
Signature · 1 line
2.29.2
author
#12 lore raw

PATCH v5 6/10 iotests/129: Use throttle node

In reply to Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
 tests/qemu-iotests/129 | 37 +++++++++++++------------------------
 1 file changed, 13 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +13 -24
index dd23bb2e5a..d40e2db24e 100755
@@ -32,20 +32,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
         iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
-        self.vm = iotests.VM().add_drive(self.test_img)
+        self.vm = iotests.VM()
+        self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+        source_drive = 'driver=throttle,' \
+                       'throttle-group=tg0,' \
+                       f'file.driver={iotests.imgfmt},' \
+                       f'file.file.filename={self.test_img}'
+
+        self.vm.add_drive(None, source_drive)
         self.vm.launch()
 
     def tearDown(self):
-        params = {"device": "drive0",
-                  "bps": 0,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
         self.vm.shutdown()
         for img in (self.test_img, self.target_img, self.base_img):
             iotests.try_remove(img)
@@ -53,33 +51,24 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
         The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
-        self.assert_qmp(result, 'return', {})
         result = self.vm.qmp(cmd, **args)
         self.assert_qmp(result, 'return', {})
+
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
+
         self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                           sync="full")
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                           sync="full")
 
     def test_block_commit(self):
Signature · 1 line
2.29.2
author
#13 lore raw

PATCH v5 7/10 iotests/129: Actually test a commit job

In reply to Max Reitz

Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the commit is done (and then specifying the source node as the top node for the commit job).

---
 tests/qemu-iotests/129 | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
tests/qemu-iotests/129 +25 -2
index d40e2db24e..104be6dded 100755
@@ -26,6 +26,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
     base_img = os.path.join(iotests.test_dir, 'base.img')
+    overlay_img = os.path.join(iotests.test_dir, 'overlay.img')
 
     def setUp(self):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
@@ -36,6 +37,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
         source_drive = 'driver=throttle,' \
+                       'node-name=source,' \
                        'throttle-group=tg0,' \
                        f'file.driver={iotests.imgfmt},' \
                        f'file.file.filename={self.test_img}'
@@ -45,7 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        for img in (self.test_img, self.target_img, self.base_img):
+        for img in (self.test_img, self.target_img, self.base_img,
+                    self.overlay_img):
             iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
@@ -72,7 +75,27 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
                           sync="full")
 
     def test_block_commit(self):
-        self.do_test_stop("block-commit", device="drive0")
+        # Add overlay above the source node so that we actually use a
+        # commit job instead of a mirror job
+
+        iotests.qemu_img('create', '-f', iotests.imgfmt, self.overlay_img,
+                         '1G')
+
+        result = self.vm.qmp('blockdev-add', **{
+                                 'node-name': 'overlay',
+                                 'driver': iotests.imgfmt,
+                                 'file': {
+                                     'driver': 'file',
+                                     'filename': self.overlay_img
+                                 }
+                             })
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('blockdev-snapshot',
+                             node='source', overlay='overlay')
+        self.assert_qmp(result, 'return', {})
+
+        self.do_test_stop('block-commit', device='drive0', top_node='source')
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=["qcow2"],
Signature · 1 line
2.29.2
author
#14 lore raw

PATCH v5 8/10 iotests/129: Limit mirror job's buffer size

In reply to Max Reitz

Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like 'stop' would make the job try to complete (which is what 129 should verify not to happen).

We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.)

---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index 104be6dded..80a5db521b 100755
@@ -67,7 +67,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
                           target=self.target_img, format=iotests.imgfmt,
-                          sync="full")
+                          sync="full", buf_size=65536)
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
Signature · 1 line
2.29.2
author
#15 lore raw

PATCH v5 9/10 iotests/129: Clean up pylint and mypy complaints

In reply to Max Reitz

And consequentially drop it from 297's skip list.

---
 tests/qemu-iotests/129 | 4 ++--
 tests/qemu-iotests/297 | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/129 +2 -2
index 80a5db521b..9a56217bf8 100755
@@ -20,7 +20,6 @@
 
 import os
 import iotests
-import time
 
 class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -32,7 +31,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
-        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+                        self.test_img)
         self.vm = iotests.VM()
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
tests/qemu-iotests/297 +1 -1
index e3db3e061e..234e1da2fb 100755
@@ -27,7 +27,7 @@ import iotests
 # TODO: Empty this list!
 SKIP_FILES = (
     '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
-    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '096', '118', '124', '132', '136', '139', '147', '148', '149',
     '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
     '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
     '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
Signature · 1 line
2.29.2
author
#16 lore raw

PATCH v5 10/10 iotests/300: Clean up pylint and mypy complaints

In reply to Max Reitz

And consequentially drop it from 297's skip list.

---
 tests/qemu-iotests/297 |  2 +-
 tests/qemu-iotests/300 | 18 +++++++++++++++---
 2 files changed, 16 insertions(+), 4 deletions(-)
tests/qemu-iotests/297 +1 -1
index 234e1da2fb..eab3d660bb 100755
@@ -33,7 +33,7 @@ SKIP_FILES = (
     '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
     '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
     '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
-    '299', '300', '302', '303', '304', '307',
+    '299', '302', '303', '304', '307',
     'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
 )
 
tests/qemu-iotests/300 +15 -3
index b864a21d5e..4115f90578 100755
@@ -22,7 +22,11 @@ import os
 import random
 import re
 from typing import Dict, List, Optional, Union
+
 import iotests
+
+# Import qemu after iotests.py has amended sys.path
+# pylint: disable=wrong-import-order
 import qemu
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
@@ -110,10 +114,14 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
         If @msg is None, check that there has not been any error.
         """
         self.vm_b.shutdown()
+
+        log = self.vm_b.get_log()
+        assert log is not None  # Loaded after shutdown
+
         if msg is None:
-            self.assertNotIn('qemu-system-', self.vm_b.get_log())
+            self.assertNotIn('qemu-system-', log)
         else:
-            self.assertIn(msg, self.vm_b.get_log())
+            self.assertIn(msg, log)
 
     @staticmethod
     def mapping(node_name: str, node_alias: str,
@@ -445,9 +453,13 @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
 
         # Check for the error in the source's log
         self.vm_a.shutdown()
+
+        log = self.vm_a.get_log()
+        assert log is not None  # Loaded after shutdown
+
         self.assertIn(f"Cannot migrate bitmap '{name}' on node "
                       f"'{self.src_node_name}': Name is longer than 255 bytes",
-                      self.vm_a.get_log())
+                      log)
 
         # Expect abnormal shutdown of the destination VM because of
         # the failed migration
Signature · 1 line
2.29.2