Skip to contentSkip to conversation

iotests: Fix 129 and expand 297’s reach

PATCH Superseded Max Reitz 3 participants 29 messages Jan 14, 2021 – Jan 15, 2021

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v3 cover iotests: Fix 129 and expand 297’s reach

Cover letters:
v1: https://lists.nongnu.org/archive/html/qemu-block/2021-01/msg00254.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2021-01/msg00296.html

git:
https://github.com/XanClic/qemu.git fix-129-2-v3
https://git.xanclic.moe/XanClic/qemu.git fix-129-2-v3

Hi,

See the v1 cover letter above for the main point of this series (it’s just that all patch indices are shifted up by two).

The main change in v2 is the extension of iotest 297 to run pylint and mypy not only on iotests.py, but on every Python file in the qemu-iotests/ directory that isn’t part of a skip list.

The main changes in v3 are that 297 is rewritten in Python, that patch 1 is added (which helps tests to pass mypy scrutiny without having to assert that vital variables such as iotests.test_dir are not None), and that patch 10 is added (because I was already modifying 300 in patch 1, so I thought i might as well).

Changes in v3 (from v2):
- Patch 1: Some vital environment variables such as TEST_DIR are currently fetched with os.environ.get(), which returns an Optional[str] type. Every test wanting to use them has to assert they are not None, or mypy will complain.
This new patch makes iotests.py use os.environ[] for those vital variables instead, which makes them of plain str type.

- Patch 2:
- Rewrite 297 in Python
- As in v2, make it check all Python files in qemu-iotests/ by default, skipping “only” files given in an explicit list. This list is extremely long right now, but at least patches 9 and 10 make it one entry shorter each.
- Drop the list of checked files from the reference output, because that may lead to conflicts when backporting patches etc.

- Patch 9: Drop 'assert iotests.test_dir is not None', because patch 1 makes that unnecessary. (Also, some changes in how 297 is treated, thanks to the changes in patch 2.)

- Patch 10: When writing patch 1, I looked for iotests that contained such 'assert iotests.* is not None' lines, and 300 was the one I found. That made it look to me like it was already written with mypy acceptance in mind. Turned out there were still a couple of things to fix, but then I was already working on it and decided there’s no point in not making a patch out of it.

git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/10:[down] 'iotests.py: Assume a couple of variables as given' 002/10:[down] 'iotests/297: Rewrite in Python and extend reach' 003/10:[----] [--] 'iotests: Move try_remove to iotests.py' 004/10:[----] [--] 'iotests/129: Remove test images in tearDown()' 005/10:[----] [--] 'iotests/129: Do not check @busy' 006/10:[----] [--] 'iotests/129: Use throttle node' 007/10:[----] [--] 'iotests/129: Actually test a commit job' 008/10:[----] [--] 'iotests/129: Limit mirror job's buffer size' 009/10:[0007] [FC] 'iotests/129: Clean up pylint and mypy complaints' 010/10:[down] 'iotests/300: Clean up pylint and mypy complaints'

Max Reitz (10):
iotests.py: Assume a couple of variables as given iotests/297: Rewrite in Python and extend reach
iotests: Move try_remove to iotests.py iotests/129: Remove test images in tearDown() iotests/129: Do not check @busy iotests/129: Use throttle node iotests/129: Actually test a commit job iotests/129: Limit mirror job's buffer size iotests/129: Clean up pylint and mypy complaints iotests/300: Clean up pylint and mypy complaints

 tests/qemu-iotests/124        |   8 +--
 tests/qemu-iotests/129        |  72 +++++++++++++---------
 tests/qemu-iotests/297        | 109 +++++++++++++++++++++++++++-------
 tests/qemu-iotests/297.out    |   5 +-
 tests/qemu-iotests/300        |  19 ++++--
 tests/qemu-iotests/iotests.py |  37 ++++++------
 6 files changed, 168 insertions(+), 82 deletions(-)
Signature · 1 line
2.29.2
author
#1 lore raw

PATCH v3 1/10 iotests.py: Assume a couple of variables as given

There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and qemu_default_machine to be set, and aborts if they are not).

Using .get() this way has the disadvantage of returning an Optional[str] type, which mypy will complain about when tests just assume these values to be str.

Use [] instead, which raises a KeyError for environment variables that are not set. When this exception is raised, catch it and move the abort code from execute_setup_common() there.

Drop the 'assert iotests.sock_dir is not None' from iotest 300, because that sort of thing is precisely what this patch wants to prevent.

---
 tests/qemu-iotests/300        |  1 -
 tests/qemu-iotests/iotests.py | 26 +++++++++++++-------------
 2 files changed, 13 insertions(+), 14 deletions(-)
tests/qemu-iotests/300 +0 -1
index 5b75121b84..b864a21d5e 100755
@@ -27,7 +27,6 @@ import qemu
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
 
-assert iotests.sock_dir is not None
 mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
 
 
tests/qemu-iotests/iotests.py +13 -13
index dcdcd0387f..52facb8e04 100644
@@ -75,12 +75,20 @@ qemu_opts = os.environ.get('QEMU_OPTIONS', '').strip().split(' ')
 
 imgfmt = os.environ.get('IMGFMT', 'raw')
 imgproto = os.environ.get('IMGPROTO', 'file')
-test_dir = os.environ.get('TEST_DIR')
-sock_dir = os.environ.get('SOCK_DIR')
 output_dir = os.environ.get('OUTPUT_DIR', '.')
-cachemode = os.environ.get('CACHEMODE')
-aiomode = os.environ.get('AIOMODE')
-qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
+
+try:
+    test_dir = os.environ['TEST_DIR']
+    sock_dir = os.environ['SOCK_DIR']
+    cachemode = os.environ['CACHEMODE']
+    aiomode = os.environ['AIOMODE']
+    qemu_default_machine = os.environ['QEMU_DEFAULT_MACHINE']
+except KeyError:
+    # We are using these variables as proxies to indicate that we're
+    # not being run via "check". There may be other things set up by
+    # "check" that individual test cases rely on.
+    sys.stderr.write('Please run this test via the "check" script\n')
+    sys.exit(os.EX_USAGE)
 
 socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
 
@@ -1294,14 +1302,6 @@ def execute_setup_common(supported_fmts: Sequence[str] = (),
     """
     # Note: Python 3.6 and pylint do not like 'Collection' so use 'Sequence'.
 
-    # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
-    # indicate that we're not being run via "check". There may be
-    # other things set up by "check" that individual test cases rely
-    # on.
-    if test_dir is None or qemu_default_machine is None:
-        sys.stderr.write('Please run this test via the "check" script\n')
-        sys.exit(os.EX_USAGE)
-
     debug = '-d' in sys.argv
     if debug:
         sys.argv.remove('-d')
Signature · 1 line
2.29.2
Reviewed-by on v3 1/10
#2 lore raw

Re: [PATCH v3 01/10] iotests.py: Assume a couple of variables as given

Quoted 10 lines · Max Reitz

There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and qemu_default_machine to be set, and aborts if they are not).

Using .get() this way has the disadvantage of returning an Optional[str] type, which mypy will complain about when tests just assume these values to be str.

Quoted 8 lines

Use [] instead, which raises a KeyError for environment variables that are not set. When this exception is raised, catch it and move the abort code from execute_setup_common() there.

Drop the 'assert iotests.sock_dir is not None' from iotest 300, because that sort of thing is precisely what this patch wants to prevent.

Signature · 2 lines
Best regards,
Vladimir
author
#3 lore raw

PATCH v3 2/10 iotests/297: Rewrite in Python and extend reach

In reply to Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted. This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment. Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments. TODO is fine, we do not need 297 to complain about such comments.

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
 tests/qemu-iotests/297     | 109 +++++++++++++++++++++++++++++--------
 tests/qemu-iotests/297.out |   5 +-
 2 files changed, 89 insertions(+), 25 deletions(-)
tests/qemu-iotests/297 +87 -22
index 5c5420712b..bfa26d280b 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
 #
 # Copyright (C) 2020 Red Hat, Inc.
 #
@@ -15,30 +15,95 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
 
-status=1	# failure is the default!
+import iotests
 
-# get standard environment
-. ./common.rc
 
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
 
-pylint-3 --score=n iotests.py
 
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
 
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None
+        except UnicodeDecodeError: # Ignore binary files
+            return False
+
+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+             if is_python_file(filename)]
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))
+
+    print('=== pylint ===')
+    sys.stdout.flush()
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)
+    env = os.environ
+    try:
+        env['PYTHONPATH'] += ':../../python/'
+    except KeyError:
+        env['PYTHONPATH'] = '../../python/'
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+                   env=env, check=False)
+
+    print('=== mypy ===')
+    sys.stdout.flush()
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT,
+                           text=True)
+
+        if p.returncode != 0:
+            print(p.stdout)
+
+
+for linter in ('pylint-3', 'mypy'):
+    if shutil.which(linter) is None:
+        iotests.notrun(f'{linter} not found')
+
+iotests.script_main(run_linters)
tests/qemu-iotests/297.out +2 -3
index 6acc843649..f2e1314d10 100644
@@ -1,3 +1,2 @@
-QA output created by 297
-Success: no issues found in 1 source file
-*** done
+=== pylint ===
+=== mypy ===
Signature · 1 line
2.29.2
Reviewed-by on v3 2/10
#4 lore raw

Re: [PATCH v3 02/10] iotests/297: Rewrite in Python and extend reach

Quoted 99 lines · Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted. This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment. Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments. TODO is fine, we do not need 297 to complain about such comments.

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
  tests/qemu-iotests/297     | 109 +++++++++++++++++++++++++++++--------
  tests/qemu-iotests/297.out |   5 +-
  2 files changed, 89 insertions(+), 25 deletions(-)
tests/qemu-iotests/297 +22 -20
index 5c5420712b..bfa26d280b 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
  #
  # Copyright (C) 2020 Red Hat, Inc.
  #
@@ -15,30 +15,95 @@
  # You should have received a copy of the GNU General Public License
  # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
 
-status=1	# failure is the default!
+import iotests
 
-# get standard environment
-. ./common.rc
 
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
 
-pylint-3 --score=n iotests.py
 
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
 
-# success, all done
-echo "*** done"

tests/qemu-iotests/297 line 52

Quoted 10 lines
tests/qemu-iotests/297 +8 -2
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None
+        except UnicodeDecodeError: # Ignore binary files

PEP8 asks for two spaces before inline comment

tests/qemu-iotests/297 line 54

Quoted 2 lines
tests/qemu-iotests/297 +2 -0
+            return False
+

and two empty lines here

(good ALE vim plugin runs flake8, mypy and pylint asynchronously for me and shows these things)

tests/qemu-iotests/297 line 60

Quoted 6 lines
tests/qemu-iotests/297 +6 -0
+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+             if is_python_file(filename)]
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))

O, good.

tests/qemu-iotests/297 line 63

Quoted 3 lines
tests/qemu-iotests/297 +3 -0
+
+    print('=== pylint ===')
+    sys.stdout.flush()

Should not hurt.. But why? Should be flushed anyway as print will put a '\n'.. And why you care about it at all?

tests/qemu-iotests/297 line 73

Quoted 10 lines
tests/qemu-iotests/297 +10 -0
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)
+    env = os.environ
+    try:
+        env['PYTHONPATH'] += ':../../python/'
+    except KeyError:
+        env['PYTHONPATH'] = '../../python/'
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+                   env=env, check=False)

as I understand, you don't need env argument, as os.environ is inherited by default.

Quoted 16 lines
tests/qemu-iotests/297 +16 -0
+
+    print('=== mypy ===')
+    sys.stdout.flush()
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',

tests/qemu-iotests/297 line 99

Quoted 10 lines
tests/qemu-iotests/297 +10 -0
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT,
+                           text=True)

Can you really use "text" ? We require python 3.6 (in check), but text comes from 3.7.. It may break some test environments. I think we need old good universal_newlines=True which is the same.

With s/text/universal_newlines/:

Quoted 21 lines
tests/qemu-iotests/297 +10 -0
+
+        if p.returncode != 0:
+            print(p.stdout)
+
+
+for linter in ('pylint-3', 'mypy'):
+    if shutil.which(linter) is None:
+        iotests.notrun(f'{linter} not found')
+
+iotests.script_main(run_linters)
tests/qemu-iotests/297.out +2 -3
index 6acc843649..f2e1314d10 100644
@@ -1,3 +1,2 @@
-QA output created by 297
-Success: no issues found in 1 source file
-*** done
+=== pylint ===
+=== mypy ===
Signature · 2 lines
Best regards,
Vladimir
author
#5 lore raw

Re: [PATCH v3 02/10] iotests/297: Rewrite in Python and extend reach

Quoted 105 lines · Vladimir Sementsov-Ogievskiy
Quoted 104 lines · Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory.  Of course, most of them do not pass, so there is an extensive skip list for now.  (The only files that do pass are 209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted.  This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment.  Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments.  TODO is fine, we do not need 297 to complain about such comments.

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
  tests/qemu-iotests/297     | 109 +++++++++++++++++++++++++++++-------- tests/qemu-iotests/297.out |   5 +-

  2 files changed, 89 insertions(+), 25 deletions(-)
tests/qemu-iotests/297 +24 -23
index 5c5420712b..bfa26d280b 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
   #
   # Copyright (C) 2020 Red Hat, Inc.
   #
@@ -15,30 +15,95 @@
   # You should have received a copy of the GNU General Public License
   # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
-status=1    # failure is the default!
+import iotests
-# get standard environment
-. ./common.rc
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', 
 '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', 
 '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', 
 '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', 
 '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', 
 '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', 
 '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', 
 '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
-pylint-3 --score=n iotests.py
-MYPYPATH=../../python/ mypy --warn-unused-configs 
--disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
Quoted 8 lines
Quoted 6 lines

+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None +        except UnicodeDecodeError: # Ignore binary files

PEP8 asks for two spaces before inline comment

Wow. I really hate PEP8.

Quoted 6 lines
Quoted 2 lines

+            return False
+

and two empty lines here

(good ALE vim plugin runs flake8, mypy and pylint asynchronously for me and shows these things)

I’d like to argue that that isn’t good, but I need to quench my anger. Perhaps one day I can quench it sufficiently to install ALE myself.

Quoted 10 lines
Quoted 7 lines

+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+             if is_python_file(filename)]
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))

O, good.

Quoted 5 lines
Quoted 3 lines

+
+    print('=== pylint ===')
+    sys.stdout.flush()

Should not hurt.. But why? Should be flushed anyway as print will put a '\n'.. And why you care about it at all?

When pylint fails, I can see its result above the '=== pylint ===' line, even though its output is on stdout. I suspect the Python output indeed isn’t flushed on \n.

(Test it by dropping the flush(), and then also e.g. 041 from the ignore list. Perhaps you get a different result, but for me, the errors appear above the '=== pylint ===' line.)

Quoted 12 lines
Quoted 11 lines

+
+    # Todo notes are fine, but fixme's or xxx's should probably just be +    # fixed (in tests, at least)
+    env = os.environ
+    try:
+        env['PYTHONPATH'] += ':../../python/'
+    except KeyError:
+        env['PYTHONPATH'] = '../../python/'
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+                   env=env, check=False)

Quoted 1 line

as I understand, you don't need env argument, as os.environ is inherited by default.

Yes, but os.environ['PYTHONPATH'] doesn’t include ../../python/, which is why I copy it.

On second though, I don’t copy it. I think the “env = os.environ” line should be “env = os.environ.copy()”, actually, or I’ll modify 297’s own PYTHONPATH.

Quoted 20 lines
Quoted 20 lines
tests/qemu-iotests/297 +20 -0
+
+    print('=== mypy ===')
+    sys.stdout.flush()
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
Quoted 8 lines
Quoted 6 lines

+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE, +                           stderr=subprocess.STDOUT, +                           text=True)

Can you really use "text" ? We require python 3.6 (in check), but text comes from 3.7..

Oh well. My fault for just reading the current docs.

Quoted 1 line

It may break some test environments. I think we need old good universal_newlines=True which is the same.

Good enough for me.

Quoted 2 lines

With s/text/universal_newlines/:

Thanks! I’ll clean up the PEP8 violations, make the “env = os.environ” line an “env = os.environ.copy()”, and use universal_newlines here.

Max

#6 lore raw

Re: [PATCH v3 02/10] iotests/297: Rewrite in Python and extend reach

Quoted 107 lines · Max Reitz
Quoted 97 lines · Vladimir Sementsov-Ogievskiy
Quoted 96 lines · Max Reitz

Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory.  Of course, most of them do not pass, so there is an extensive skip list for now.  (The only files that do pass are 209, 254, 283, and iotests.py.)

(Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.)

Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream.

Besides the technical rewrite, some more things are changed:

- For the pylint invocation, PYTHONPATH is adjusted.  This mirrors setting MYPYPATH for mypy.

- Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment.  Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that.

- Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments.  TODO is fine, we do not need 297 to complain about such comments.

- The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want.

---
  tests/qemu-iotests/297     | 109 +++++++++++++++++++++++++++++-------- tests/qemu-iotests/297.out |   5 +-

  2 files changed, 89 insertions(+), 25 deletions(-)
tests/qemu-iotests/297 +24 -22
index 5c5420712b..bfa26d280b 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
   #
   # Copyright (C) 2020 Red Hat, Inc.
   #
@@ -15,30 +15,95 @@
   # You should have received a copy of the GNU General Public License
   # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-seq=$(basename $0)
-echo "QA output created by $seq"
+import os
+import re
+import shutil
+import subprocess
+import sys
-status=1    # failure is the default!
+import iotests
-# get standard environment
-. ./common.rc
-if ! type -p "pylint-3" > /dev/null; then
-    _notrun "pylint-3 not found"
-fi
-if ! type -p "mypy" > /dev/null; then
-    _notrun "mypy not found"
-fi
+# TODO: Empty this list!
+SKIP_FILES = (
+    '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
+    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
+    '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
+    '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
+    '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
+    '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
+    '299', '300', '302', '303', '304', '307',
+    'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
+)
-pylint-3 --score=n iotests.py
-MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
-    --disallow-any-generics --disallow-incomplete-defs \
-    --disallow-untyped-decorators --no-implicit-optional \
-    --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+def is_python_file(filename):
+    if not os.path.isfile(filename):
+        return False
-# success, all done
-echo "*** done"
-rm -f $seq.full
-status=0
+    if filename.endswith('.py'):
+        return True
Quoted 8 lines

tests/qemu-iotests/297 line 52

Quoted 6 lines
tests/qemu-iotests/297 +6 -0
+
+    with open(filename) as f:
+        try:
+            first_line = f.readline()
+            return re.match('^#!.*python', first_line) is not None
+        except UnicodeDecodeError: # Ignore binary files

PEP8 asks for two spaces before inline comment

Quoted 1 line

Wow.  I really hate PEP8.

Wow, it's unexpected :) I like it since first meet..

Quoted 23 lines
Quoted 7 lines

tests/qemu-iotests/297 line 47

Quoted 3 lines
tests/qemu-iotests/297 +2 -0
+            return False
+

and two empty lines here

(good ALE vim plugin runs flake8, mypy and pylint asynchronously for me and shows these things)

I’d like to argue that that isn’t good, but I need to quench my anger. Perhaps one day I can quench it sufficiently to install ALE myself.

Quoted 13 lines

tests/qemu-iotests/297 line 60

Quoted 6 lines
tests/qemu-iotests/297 +6 -0
+def run_linters():
+    files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
+             if is_python_file(filename)]
+
+    iotests.logger.debug('Files to be checked:')
+    iotests.logger.debug(', '.join(sorted(files)))

O, good.

Quoted 3 lines
tests/qemu-iotests/297 +3 -0
+
+    print('=== pylint ===')
+    sys.stdout.flush()
Quoted 5 lines
Quoted 1 line

Should not hurt.. But why? Should be flushed anyway as print will put a '\n'.. And why you care about it at all?

When pylint fails, I can see its result above the '=== pylint ===' line, even though its output is on stdout.  I suspect the Python output indeed isn’t flushed on \n.

(Test it by dropping the flush(), and then also e.g. 041 from the ignore list.  Perhaps you get a different result, but for me, the errors appear above the '=== pylint ===' line.)

Checked. Hmm, you are right. Interesting. Probably it explains similar issues with failed iotests output I see sometimes..

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
tests/qemu-iotests/297 +10 -0
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)
+    env = os.environ
+    try:
+        env['PYTHONPATH'] += ':../../python/'
+    except KeyError:
+        env['PYTHONPATH'] = '../../python/'
+    subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
+                   env=env, check=False)
Quoted 5 lines
Quoted 1 line

as I understand, you don't need env argument, as os.environ is inherited by default.

Yes, but os.environ['PYTHONPATH'] doesn’t include ../../python/, which is why I copy it.

On second though, I don’t copy it.  I think the “env = os.environ” line should be “env = os.environ.copy()”, actually, or I’ll modify 297’s own PYTHONPATH.

I think modifying os.environ won't hurt in this case.. But copying is more strict of course.

Quoted 32 lines
Quoted 21 lines
Quoted 21 lines
tests/qemu-iotests/297 +20 -0
+
+    print('=== mypy ===')
+    sys.stdout.flush()
+
+    # We have to call mypy separately for each file.  Otherwise, it
+    # will interpret all given files as belonging together (i.e., they
+    # may not both define the same classes, etc.; most notably, they
+    # must not both define the __main__ module).
+    env['MYPYPATH'] = env['PYTHONPATH']
+    for filename in files:
+        p = subprocess.run(('mypy',
+                            '--warn-unused-configs',
+                            '--disallow-subclassing-any',
+                            '--disallow-any-generics',
+                            '--disallow-incomplete-defs',
+                            '--disallow-untyped-decorators',
+                            '--no-implicit-optional',
+                            '--warn-redundant-casts',
+                            '--warn-unused-ignores',
+                            '--no-implicit-reexport',
Quoted 8 lines

tests/qemu-iotests/297 line 99

Quoted 6 lines
tests/qemu-iotests/297 +6 -0
+                            filename),
+                           env=env,
+                           check=False,
+                           stdout=subprocess.PIPE,
+                           stderr=subprocess.STDOUT,
+                           text=True)

Can you really use "text" ? We require python 3.6 (in check), but text comes from 3.7..

Oh well.  My fault for just reading the current docs.

Quoted 9 lines
Quoted 1 line

It may break some test environments. I think we need old good universal_newlines=True which is the same.

Good enough for me.

Quoted 2 lines

With s/text/universal_newlines/:

Thanks!  I’ll clean up the PEP8 violations, make the “env = os.environ” line an “env = os.environ.copy()”, and use universal_newlines here.

OK, thanks, keep my r-b with it.

Signature · 2 lines
Best regards,
Vladimir
author
#7 lore raw

PATCH v3 3/10 iotests: Move try_remove to iotests.py

In reply to Max Reitz
---
 tests/qemu-iotests/124        |  8 +-------
 tests/qemu-iotests/iotests.py | 11 +++++++----
 2 files changed, 8 insertions(+), 11 deletions(-)
tests/qemu-iotests/124 +1 -7
index 3705cbb6b3..e40eeb50b9 100755
@@ -22,6 +22,7 @@
 
 import os
 import iotests
+from iotests import try_remove
 
 
 def io_write_patterns(img, patterns):
@@ -29,13 +30,6 @@ def io_write_patterns(img, patterns):
         iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
 
-def try_remove(img):
-    try:
-        os.remove(img)
-    except OSError:
-        pass
-
-
 def transaction_action(action, **kwargs):
     return {
         'type': action,
tests/qemu-iotests/iotests.py +7 -4
index 52facb8e04..a69b4cdc4e 100644
@@ -523,12 +523,15 @@ class FilePath:
         return False
 
 
+def try_remove(img):
+    try:
+        os.remove(img)
+    except OSError:
+        pass
+
 def file_path_remover():
     for path in reversed(file_path_remover.paths):
-        try:
-            os.remove(path)
-        except OSError:
-            pass
+        try_remove(path)
 
 
 def file_path(*names, base_dir=test_dir):
Signature · 1 line
2.29.2
author
#9 lore raw

PATCH v3 4/10 iotests/129: Remove test images in tearDown()

In reply to Max Reitz
---
 tests/qemu-iotests/129 | 2 ++
 1 file changed, 2 insertions(+)
tests/qemu-iotests/129 +2 -0
index 0e13244d85..2fc65ada6a 100755
@@ -47,6 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
                              **params)
         self.vm.shutdown()
+        for img in (self.test_img, self.target_img, self.base_img):
+            iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
Signature · 1 line
2.29.2
author
#11 lore raw

PATCH v3 5/10 iotests/129: Do not check @busy

In reply to Max Reitz

@busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3).

Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information.

---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index 2fc65ada6a..dd23bb2e5a 100755
@@ -69,7 +69,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
-        self.assert_qmp(result, 'return[0]/busy', True)
+        self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
     def test_drive_mirror(self):
Signature · 1 line
2.29.2
author
#12 lore raw

PATCH v3 6/10 iotests/129: Use throttle node

In reply to Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

---
 tests/qemu-iotests/129 | 37 +++++++++++++------------------------
 1 file changed, 13 insertions(+), 24 deletions(-)
tests/qemu-iotests/129 +13 -24
index dd23bb2e5a..58536bc6ee 100755
@@ -32,20 +32,18 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
         iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
-        self.vm = iotests.VM().add_drive(self.test_img)
+        self.vm = iotests.VM()
+        self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
+
+        source_drive = 'driver=throttle,' \
+                       'throttle-group=tg0,' \
+                      f'file.driver={iotests.imgfmt},' \
+                      f'file.file.filename={self.test_img}'
+
+        self.vm.add_drive(None, source_drive)
         self.vm.launch()
 
     def tearDown(self):
-        params = {"device": "drive0",
-                  "bps": 0,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
         self.vm.shutdown()
         for img in (self.test_img, self.target_img, self.base_img):
             iotests.try_remove(img)
@@ -53,33 +51,24 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
         The 'stop' command shouldn't drain the job"""
-        params = {"device": "drive0",
-                  "bps": 1024,
-                  "bps_rd": 0,
-                  "bps_wr": 0,
-                  "iops": 0,
-                  "iops_rd": 0,
-                  "iops_wr": 0,
-                 }
-        result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
-                             **params)
-        self.assert_qmp(result, 'return', {})
         result = self.vm.qmp(cmd, **args)
         self.assert_qmp(result, 'return', {})
+
         result = self.vm.qmp("stop")
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp("query-block-jobs")
+
         self.assert_qmp(result, 'return[0]/status', 'running')
         self.assert_qmp(result, 'return[0]/ready', False)
 
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                           sync="full")
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
-                          target=self.target_img,
+                          target=self.target_img, format=iotests.imgfmt,
                           sync="full")
 
     def test_block_commit(self):
Signature · 1 line
2.29.2
Reviewed-by on v3 6/10
#13 lore raw

Re: [PATCH v3 06/10] iotests/129: Use throttle node

Quoted 6 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

you forget my :)

Signature · 2 lines
Best regards,
Vladimir
author
#14 lore raw

Re: [PATCH v3 06/10] iotests/129: Use throttle node

Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 6 lines · Max Reitz

Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node.

you forget my :)

Sorry O:)

author
#15 lore raw

PATCH v3 7/10 iotests/129: Actually test a commit job

In reply to Max Reitz

Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead.

Doing so requires adding an overlay above the source node before the commit is done (and then specifying the source node as the top node for the commit job).

---
 tests/qemu-iotests/129 | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
tests/qemu-iotests/129 +25 -2
index 58536bc6ee..7b4b6649f0 100755
@@ -26,6 +26,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
     base_img = os.path.join(iotests.test_dir, 'base.img')
+    overlay_img = os.path.join(iotests.test_dir, 'overlay.img')
 
     def setUp(self):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
@@ -36,6 +37,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
         source_drive = 'driver=throttle,' \
+                       'node-name=source,' \
                        'throttle-group=tg0,' \
                       f'file.driver={iotests.imgfmt},' \
                       f'file.file.filename={self.test_img}'
@@ -45,7 +47,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        for img in (self.test_img, self.target_img, self.base_img):
+        for img in (self.test_img, self.target_img, self.base_img,
+                    self.overlay_img):
             iotests.try_remove(img)
 
     def do_test_stop(self, cmd, **args):
@@ -72,7 +75,27 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
                           sync="full")
 
     def test_block_commit(self):
-        self.do_test_stop("block-commit", device="drive0")
+        # Add overlay above the source node so that we actually use a
+        # commit job instead of a mirror job
+
+        iotests.qemu_img('create', '-f', iotests.imgfmt, self.overlay_img,
+                         '1G')
+
+        result = self.vm.qmp('blockdev-add', **{
+                                 'node-name': 'overlay',
+                                 'driver': iotests.imgfmt,
+                                 'file': {
+                                     'driver': 'file',
+                                     'filename': self.overlay_img
+                                 }
+                             })
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('blockdev-snapshot',
+                             node='source', overlay='overlay')
+        self.assert_qmp(result, 'return', {})
+
+        self.do_test_stop('block-commit', device='drive0', top_node='source')
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=["qcow2"],
Signature · 1 line
2.29.2
author
#16 lore raw

PATCH v3 8/10 iotests/129: Limit mirror job's buffer size

In reply to Max Reitz

Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like 'stop' would make the job try to complete (which is what 129 should verify not to happen).

We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.)

---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
index 7b4b6649f0..6d21470cd7 100755
@@ -67,7 +67,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     def test_drive_mirror(self):
         self.do_test_stop("drive-mirror", device="drive0",
                           target=self.target_img, format=iotests.imgfmt,
-                          sync="full")
+                          sync="full", buf_size=65536)
 
     def test_drive_backup(self):
         self.do_test_stop("drive-backup", device="drive0",
Signature · 1 line
2.29.2
Reviewed-by on v3 8/10
#17 lore raw

Re: [PATCH v3 08/10] iotests/129: Limit mirror job's buffer size

Quoted 15 lines · Max Reitz

Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like 'stop' would make the job try to complete (which is what 129 should verify not to happen).

We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain.

(We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.)

Quoted 7 lines
---
 tests/qemu-iotests/129 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
author
#18 lore raw

PATCH v3 9/10 iotests/129: Clean up pylint and mypy complaints

In reply to Max Reitz
---
 tests/qemu-iotests/129 | 4 ++--
 tests/qemu-iotests/297 | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/129 +2 -2
index 6d21470cd7..201d9e0a0b 100755
@@ -20,7 +20,6 @@
 
 import os
 import iotests
-import time
 
 class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -32,7 +31,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
-        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+                        self.test_img)
         self.vm = iotests.VM()
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
tests/qemu-iotests/297 +1 -1
index bfa26d280b..1dce1d1b1c 100755
@@ -27,7 +27,7 @@ import iotests
 # TODO: Empty this list!
 SKIP_FILES = (
     '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
-    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '096', '118', '124', '132', '136', '139', '147', '148', '149',
     '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
     '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
     '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
Signature · 1 line
2.29.2
Reviewed-by on v3 9/10
#19 lore raw

Re: [PATCH v3 09/10] iotests/129: Clean up pylint and mypy complaints

Quoted 34 lines · Max Reitz
---
 tests/qemu-iotests/129 | 4 ++--
 tests/qemu-iotests/297 | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/129 +2 -2
index 6d21470cd7..201d9e0a0b 100755
@@ -20,7 +20,6 @@
 
 import os
 import iotests
-import time
 
 class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -32,7 +31,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
         iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                          "-b", self.base_img, '-F', iotests.imgfmt)
-        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+                        self.test_img)
         self.vm = iotests.VM()
         self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
tests/qemu-iotests/297 +0 -0
index bfa26d280b..1dce1d1b1c 100755

tests/qemu-iotests/297 import iotests · line 30

Quoted 5 lines
tests/qemu-iotests/297 +1 -1
@@ -27,7 +27,7 @@ import iotests
 # TODO: Empty this list!
 SKIP_FILES = (
     '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
-    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '096', '118', '124', '132', '136', '139', '147', '148', '149',

Is this also part of mypy/pylint cleanup? It seems you are doing more than that here. It would be good to have this listed in the commit message. Despite that,

Quoted 7 lines
tests/qemu-iotests/297 +0 -1
     '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
     '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
     '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
--

2.29.2

author
#20 lore raw

Re: [PATCH v3 09/10] iotests/129: Clean up pylint and mypy complaints

Quoted 30 lines · Willian Rampazzo
Quoted 29 lines · Max Reitz
---
  tests/qemu-iotests/129 | 4 ++--
  tests/qemu-iotests/297 | 2 +-
  2 files changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/129 +2 -2
index 6d21470cd7..201d9e0a0b 100755
@@ -20,7 +20,6 @@
 
  import os
  import iotests
-import time
 
  class TestStopWithBlockJob(iotests.QMPTestCase):
      test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -32,7 +31,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
          iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
          iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                           "-b", self.base_img, '-F', iotests.imgfmt)
-        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+                        self.test_img)
          self.vm = iotests.VM()
          self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
Quoted 14 lines

tests/qemu-iotests/297 import iotests · line 30

Quoted 10 lines
tests/qemu-iotests/297 +1 -1
index bfa26d280b..1dce1d1b1c 100755
@@ -27,7 +27,7 @@ import iotests
  # TODO: Empty this list!
  SKIP_FILES = (
      '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
-    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '096', '118', '124', '132', '136', '139', '147', '148', '149',

Is this also part of mypy/pylint cleanup? It seems you are doing more than that here. It would be good to have this listed in the commit message.

Sure, why not. Something like “And consequentially drop it from 297's skip list.”?

Though I think making a test pass pylint+mypy complaints basically means exactly to remove it from 297's skip list and then making 297 pass, so I’m not entirely sure it’s necessary. But it can’t hurt, so.

Quoted 3 lines

Despite that,

Thanks!

Max

Quoted 8 lines
Quoted 7 lines
tests/qemu-iotests/297 +0 -1
      '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
      '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
      '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
--

2.29.2

#21 lore raw

Re: [PATCH v3 09/10] iotests/129: Clean up pylint and mypy complaints

Quoted 47 lines · Max Reitz
Quoted 30 lines · Willian Rampazzo
Quoted 29 lines · Max Reitz
---
  tests/qemu-iotests/129 | 4 ++--
  tests/qemu-iotests/297 | 2 +-
  2 files changed, 3 insertions(+), 3 deletions(-)
tests/qemu-iotests/129 +2 -2
index 6d21470cd7..201d9e0a0b 100755
@@ -20,7 +20,6 @@
 
  import os
  import iotests
-import time
 
  class TestStopWithBlockJob(iotests.QMPTestCase):
      test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -32,7 +31,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
          iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
          iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img,
                           "-b", self.base_img, '-F', iotests.imgfmt)
-        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M',
+                        self.test_img)
          self.vm = iotests.VM()
          self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024')
 
Quoted 14 lines

tests/qemu-iotests/297 import iotests · line 30

Quoted 10 lines
tests/qemu-iotests/297 +1 -1
index bfa26d280b..1dce1d1b1c 100755
@@ -27,7 +27,7 @@ import iotests
  # TODO: Empty this list!
  SKIP_FILES = (
      '030', '040', '041', '044', '045', '055', '056', '057', '065', '093',
-    '096', '118', '124', '129', '132', '136', '139', '147', '148', '149',
+    '096', '118', '124', '132', '136', '139', '147', '148', '149',

Is this also part of mypy/pylint cleanup? It seems you are doing more than that here. It would be good to have this listed in the commit message.

Quoted 2 lines

Sure, why not. Something like “And consequentially drop it from 297's skip list.”?

+1 Thanks!

Quoted 22 lines

Though I think making a test pass pylint+mypy complaints basically means exactly to remove it from 297's skip list and then making 297 pass, so I’m not entirely sure it’s necessary. But it can’t hurt, so.

Quoted 3 lines

Despite that,

Thanks!

Max

Quoted 8 lines
Quoted 7 lines
tests/qemu-iotests/297 +0 -1
      '151', '152', '155', '163', '165', '169', '194', '196', '199', '202',
      '203', '205', '206', '207', '208', '210', '211', '212', '213', '216',
      '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
--

2.29.2

author
#23 lore raw

PATCH v3 10/10 iotests/300: Clean up pylint and mypy complaints

In reply to Max Reitz
---
 tests/qemu-iotests/297 |  2 +-
 tests/qemu-iotests/300 | 18 +++++++++++++++---
 2 files changed, 16 insertions(+), 4 deletions(-)
tests/qemu-iotests/297 +1 -1
index 1dce1d1b1c..03d8604538 100755
@@ -33,7 +33,7 @@ SKIP_FILES = (
     '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
     '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
     '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
-    '299', '300', '302', '303', '304', '307',
+    '299', '302', '303', '304', '307',
     'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
 )
 
tests/qemu-iotests/300 +15 -3
index b864a21d5e..64d388a8fa 100755
@@ -22,7 +22,11 @@ import os
 import random
 import re
 from typing import Dict, List, Optional, Union
+
 import iotests
+
+# Import qemu after iotests.py has amended the PYTHONPATH
+# pylint: disable=wrong-import-order
 import qemu
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
@@ -110,10 +114,14 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
         If @msg is None, check that there has not been any error.
         """
         self.vm_b.shutdown()
+
+        log = self.vm_b.get_log()
+        assert log is not None # Loaded after shutdown
+
         if msg is None:
-            self.assertNotIn('qemu-system-', self.vm_b.get_log())
+            self.assertNotIn('qemu-system-', log)
         else:
-            self.assertIn(msg, self.vm_b.get_log())
+            self.assertIn(msg, log)
 
     @staticmethod
     def mapping(node_name: str, node_alias: str,
@@ -445,9 +453,13 @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
 
         # Check for the error in the source's log
         self.vm_a.shutdown()
+
+        log = self.vm_a.get_log()
+        assert log is not None # Loaded after shutdown
+
         self.assertIn(f"Cannot migrate bitmap '{name}' on node "
                       f"'{self.src_node_name}': Name is longer than 255 bytes",
-                      self.vm_a.get_log())
+                      log)
 
         # Expect abnormal shutdown of the destination VM because of
         # the failed migration
Signature · 1 line
2.29.2
Reviewed-by on v3 10/10
#24 lore raw

Re: [PATCH v3 10/10] iotests/300: Clean up pylint and mypy complaints

Quoted 24 lines · Max Reitz
---
  tests/qemu-iotests/297 |  2 +-
  tests/qemu-iotests/300 | 18 +++++++++++++++---
  2 files changed, 16 insertions(+), 4 deletions(-)
tests/qemu-iotests/297 +1 -1
index 1dce1d1b1c..03d8604538 100755
@@ -33,7 +33,7 @@ SKIP_FILES = (
      '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
      '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
      '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
-    '299', '300', '302', '303', '304', '307',
+    '299', '302', '303', '304', '307',
      'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py'
  )
 
tests/qemu-iotests/300 +0 -0
index b864a21d5e..64d388a8fa 100755

tests/qemu-iotests/300 import os · line 28

Quoted 7 lines
tests/qemu-iotests/300 +3 -0
@@ -22,7 +22,11 @@ import os
  import random
  import re
  from typing import Dict, List, Optional, Union
+
  import iotests
+
+# Import qemu after iotests.py has amended the PYTHONPATH

you mean os.path, exactly,

sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))

yes?

Quoted 27 lines
tests/qemu-iotests/300 +8 -2
+# pylint: disable=wrong-import-order
  import qemu
 
  BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
@@ -110,10 +114,14 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
          If @msg is None, check that there has not been any error.
          """
          self.vm_b.shutdown()
+
+        log = self.vm_b.get_log()
+        assert log is not None # Loaded after shutdown
+
          if msg is None:
-            self.assertNotIn('qemu-system-', self.vm_b.get_log())
+            self.assertNotIn('qemu-system-', log)
          else:
-            self.assertIn(msg, self.vm_b.get_log())
+            self.assertIn(msg, log)
 
      @staticmethod
      def mapping(node_name: str, node_alias: str,
@@ -445,9 +453,13 @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
 
          # Check for the error in the source's log
          self.vm_a.shutdown()
+

tests/qemu-iotests/300 class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration): · line 465

Quoted 10 lines
tests/qemu-iotests/300 +4 -1
+        log = self.vm_a.get_log()
+        assert log is not None # Loaded after shutdown
+
          self.assertIn(f"Cannot migrate bitmap '{name}' on node "
                        f"'{self.src_node_name}': Name is longer than 255 bytes",
-                      self.vm_a.get_log())
+                      log)
 
          # Expect abnormal shutdown of the destination VM because of
          # the failed migration

Your new comments are the only PEP8 complains in the 300 iotest:

flake8 300
300:119:31: E261 at least two spaces before inline comment 300:458:31: E261 at least two spaces before inline comment

So, I'd fix them.

anyway:

Signature · 2 lines
Best regards,
Vladimir
author
#25 lore raw

Re: [PATCH v3 10/10] iotests/300: Clean up pylint and mypy complaints

Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 29 lines · Max Reitz

---
  tests/qemu-iotests/297 |  2 +-
  tests/qemu-iotests/300 | 18 +++++++++++++++---

  2 files changed, 16 insertions(+), 4 deletions(-)
tests/qemu-iotests/297 +1 -1
index 1dce1d1b1c..03d8604538 100755
@@ -33,7 +33,7 @@ SKIP_FILES = (
       '218', '219', '222', '224', '228', '234', '235', '236', '237', '238',
       '240', '242', '245', '246', '248', '255', '256', '257', '258', '260',
       '262', '264', '266', '274', '277', '280', '281', '295', '296', '298',
-    '299', '300', '302', '303', '304', '307',
+    '299', '302', '303', '304', '307',
 

      'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py' )

tests/qemu-iotests/300 +0 -0
index b864a21d5e..64d388a8fa 100755
@@ -22,7 +22,11 @@ import os

  import random
  import re
  from typing import Dict, List, Optional, Union
+
  import iotests

tests/qemu-iotests/300

Quoted 2 lines
tests/qemu-iotests/300 +2 -0
+
+# Import qemu after iotests.py has amended the PYTHONPATH

you mean os.path, exactly,

Quoted 4 lines

 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..',
'python'))

yes?

Indeed.

Quoted 38 lines
Quoted 33 lines

+# pylint: disable=wrong-import-order
  import qemu
  BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]

tests/qemu-iotests/300 +0 -0
@@ -110,10 +114,14 @@ class 

TestDirtyBitmapMigration(iotests.QMPTestCase):
          If @msg is None, check that there has not been any error. """
          self.vm_b.shutdown()

tests/qemu-iotests/300 +4 -0
+
+        log = self.vm_b.get_log()
+        assert log is not None # Loaded after shutdown
+

          if msg is None:
-            self.assertNotIn('qemu-system-', self.vm_b.get_log()) +            self.assertNotIn('qemu-system-', log) else:
-            self.assertIn(msg, self.vm_b.get_log())
+            self.assertIn(msg, log)
      @staticmethod
      def mapping(node_name: str, node_alias: str,

tests/qemu-iotests/300 +0 -0
@@ -445,9 +453,13 @@ class 

TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
          # Check for the error in the source's log self.vm_a.shutdown()

tests/qemu-iotests/300 +4 -0
+
+        log = self.vm_a.get_log()
+        assert log is not None # Loaded after shutdown
+

          self.assertIn(f"Cannot migrate bitmap '{name}' on node " f"'{self.src_node_name}': Name is longer than 255 bytes", -                      self.vm_a.get_log())
+                      log)
          # Expect abnormal shutdown of the destination VM because of

Quoted 2 lines
          # the failed migration

Your new comments are the only PEP8 complains in the 300 iotest:

Quoted 5 lines

flake8 300
300:119:31: E261 at least two spaces before inline comment 300:458:31: E261 at least two spaces before inline comment

So, I'd fix them.

Haha, sure :)

Quoted 2 lines

anyway:

Thanks!

#26 lore raw

[PATCH v3 11/10] iotests: add flake8 linter

In reply to Max Reitz

pylint is good, but doesn't cover the PEP8. Let's add flake8, to be sure that our code sutisfy PEP8. Add new linter and fix some code style in checked files.

---

Hi!

Here is my small addition to Max's series, hope you like it!

Note, that this is not the first occurrence of 'flake8' in Qemu:

    # git grep flake8
    python/qemu/.flake8:[flake8]
    scripts/qapi/.flake8:[flake8]
    scripts/qapi/.flake8:extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 tests/qemu-iotests/129        |  6 ++-
 tests/qemu-iotests/254        |  2 +-
 tests/qemu-iotests/297        | 21 ++++++---
 tests/qemu-iotests/297.out    |  1 +
 tests/qemu-iotests/300        |  4 +-
 tests/qemu-iotests/iotests.py | 88 +++++++++++++++++++++++++++++++++--
 6 files changed, 106 insertions(+), 16 deletions(-)
tests/qemu-iotests/129 +4 -2
index 201d9e0a0b..28e6666c1d 100755
@@ -21,6 +21,7 @@
 import os
 import iotests
 
+
 class TestStopWithBlockJob(iotests.QMPTestCase):
     test_img = os.path.join(iotests.test_dir, 'test.img')
     target_img = os.path.join(iotests.test_dir, 'target.img')
@@ -39,8 +40,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         source_drive = 'driver=throttle,' \
                        'node-name=source,' \
                        'throttle-group=tg0,' \
-                      f'file.driver={iotests.imgfmt},' \
-                      f'file.file.filename={self.test_img}'
+                       f'file.driver={iotests.imgfmt},' \
+                       f'file.file.filename={self.test_img}'
 
         self.vm.add_drive(None, source_drive)
         self.vm.launch()
@@ -97,6 +98,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
 
         self.do_test_stop('block-commit', device='drive0', top_node='source')
 
+
 if __name__ == '__main__':
     iotests.main(supported_fmts=["qcow2"],
                  supported_protocols=["file"])
tests/qemu-iotests/254 +1 -1
index 150e58be8e..798f316e36 100755
@@ -74,7 +74,7 @@ log("query-block: device = {}, node-name = {}, dirty-bitmaps:".format(
     result['device'], result['inserted']['node-name']))
 log(result['dirty-bitmaps'], indent=2)
 log("\nbitmaps in backing image:")
-log(result['inserted']['image']['backing-image']['format-specific'] \
+log(result['inserted']['image']['backing-image']['format-specific']
     ['data']['bitmaps'], indent=2)
 
 vm.shutdown()
tests/qemu-iotests/297 +14 -7
index 03d8604538..b79c341a3c 100755
@@ -49,9 +49,10 @@ def is_python_file(filename):
         try:
             first_line = f.readline()
             return re.match('^#!.*python', first_line) is not None
-        except UnicodeDecodeError: # Ignore binary files
+        except UnicodeDecodeError:  # Ignore binary files
             return False
 
+
 def run_linters():
     files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES))
              if is_python_file(filename)]
@@ -59,16 +60,22 @@ def run_linters():
     iotests.logger.debug('Files to be checked:')
     iotests.logger.debug(', '.join(sorted(files)))
 
-    print('=== pylint ===')
-    sys.stdout.flush()
-
-    # Todo notes are fine, but fixme's or xxx's should probably just be
-    # fixed (in tests, at least)
     env = os.environ
     try:
         env['PYTHONPATH'] += ':../../python/'
     except KeyError:
         env['PYTHONPATH'] = '../../python/'
+
+    print('=== flake8 ===')
+    sys.stdout.flush()
+
+    subprocess.run(('flake8', *files), env=env, check=False)
+
+    print('=== pylint ===')
+    sys.stdout.flush()
+
+    # Todo notes are fine, but fixme's or xxx's should probably just be
+    # fixed (in tests, at least)
     subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files),
                    env=env, check=False)
 
@@ -102,7 +109,7 @@ def run_linters():
             print(p.stdout)
 
 
-for linter in ('pylint-3', 'mypy'):
+for linter in ('flake8', 'pylint-3', 'mypy'):
     if shutil.which(linter) is None:
         iotests.notrun(f'{linter} not found')
 
tests/qemu-iotests/297.out +1 -0
index f2e1314d10..46bf3e665d 100644
@@ -1,2 +1,3 @@
+=== flake8 ===
 === pylint ===
 === mypy ===
tests/qemu-iotests/300 +2 -2
index 64d388a8fa..bbd227ff73 100755
@@ -116,7 +116,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
         self.vm_b.shutdown()
 
         log = self.vm_b.get_log()
-        assert log is not None # Loaded after shutdown
+        assert log is not None  # Loaded after shutdown
 
         if msg is None:
             self.assertNotIn('qemu-system-', log)
@@ -455,7 +455,7 @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
         self.vm_a.shutdown()
 
         log = self.vm_a.get_log()
-        assert log is not None # Loaded after shutdown
+        assert log is not None  # Loaded after shutdown
 
         self.assertIn(f"Cannot migrate bitmap '{name}' on node "
                       f"'{self.src_node_name}': Name is longer than 255 bytes",
tests/qemu-iotests/iotests.py +84 -4

486 lines not on this page · raw message

Signature · 1 line
2.29.2
author
#27 lore raw

Re: [PATCH v3 11/10] iotests: add flake8 linter

Quoted 19 lines · Vladimir Sementsov-Ogievskiy

pylint is good, but doesn't cover the PEP8. Let's add flake8, to be sure that our code sutisfy PEP8. Add new linter and fix some code style in checked files.

---

Hi!

Here is my small addition to Max's series, hope you like it!

Note, that this is not the first occurrence of 'flake8' in Qemu:

     # git grep flake8
     python/qemu/.flake8:[flake8]
     scripts/qapi/.flake8:[flake8]
     scripts/qapi/.flake8:extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
Quoted 7 lines
  tests/qemu-iotests/129        |  6 ++-
  tests/qemu-iotests/254        |  2 +-
  tests/qemu-iotests/297        | 21 ++++++---
  tests/qemu-iotests/297.out    |  1 +
  tests/qemu-iotests/300        |  4 +-
  tests/qemu-iotests/iotests.py | 88 +++++++++++++++++++++++++++++++++--
  6 files changed, 106 insertions(+), 16 deletions(-)

Looks reasonable to me, but perhaps it should just be a dedicated series. I think there’s enough in here to justify that.

Quoted 13 lines
tests/qemu-iotests/129 +1 -0
index 201d9e0a0b..28e6666c1d 100755
@@ -21,6 +21,7 @@
  import os
  import iotests
 
+
  class TestStopWithBlockJob(iotests.QMPTestCase):
      test_img = os.path.join(iotests.test_dir, 'test.img')
      target_img = os.path.join(iotests.test_dir, 'target.img')

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 44

Quoted 7 lines
tests/qemu-iotests/129 +2 -2
@@ -39,8 +40,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
          source_drive = 'driver=throttle,' \
                         'node-name=source,' \
                         'throttle-group=tg0,' \
-                      f'file.driver={iotests.imgfmt},' \
-                      f'file.file.filename={self.test_img}'
+                       f'file.driver={iotests.imgfmt},' \
+                       f'file.file.filename={self.test_img}'

Interesting, when indenting this, I was wondering whether pylint would complain. I was so glad it didn’t. I really don’t like PEP8.

(Though I understand that style guides like PEP8 are there specifically so when someone like me goes “but I like this style better :(”, everyone else can say “but you’re objectively wrong”. So me hating it kind of is its point, I guess.)

Max

#28 lore raw

Re: [PATCH v3 11/10] iotests: add flake8 linter

Quoted 28 lines · Max Reitz
Quoted 19 lines · Vladimir Sementsov-Ogievskiy

pylint is good, but doesn't cover the PEP8. Let's add flake8, to be sure that our code sutisfy PEP8. Add new linter and fix some code style in checked files.

---

Hi!

Here is my small addition to Max's series, hope you like it!

Note, that this is not the first occurrence of 'flake8' in Qemu:

     # git grep flake8
     python/qemu/.flake8:[flake8]
     scripts/qapi/.flake8:[flake8]
     scripts/qapi/.flake8:extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
Quoted 7 lines
  tests/qemu-iotests/129        |  6 ++-
  tests/qemu-iotests/254        |  2 +-
  tests/qemu-iotests/297        | 21 ++++++---
  tests/qemu-iotests/297.out    |  1 +
  tests/qemu-iotests/300        |  4 +-
  tests/qemu-iotests/iotests.py | 88 +++++++++++++++++++++++++++++++++--
  6 files changed, 106 insertions(+), 16 deletions(-)
Quoted 1 line

Looks reasonable to me, but perhaps it should just be a dedicated series.  I think there’s enough in here to justify that.

Not a problem, I can resend :)

Quoted 23 lines
Quoted 13 lines
tests/qemu-iotests/129 +0 -0
index 201d9e0a0b..28e6666c1d 100755
@@ -21,6 +21,7 @@

  import os
  import iotests
+
  class TestStopWithBlockJob(iotests.QMPTestCase):
      test_img = os.path.join(iotests.test_dir, 'test.img') target_img = os.path.join(iotests.test_dir, 'target.img')

tests/qemu-iotests/129 class TestStopWithBlockJob(iotests.QMPTestCase): · line 44

Quoted 7 lines
tests/qemu-iotests/129 +2 -2
@@ -39,8 +40,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
           source_drive = 'driver=throttle,' \
                          'node-name=source,' \
                          'throttle-group=tg0,' \
-                      f'file.driver={iotests.imgfmt},' \
-                      f'file.file.filename={self.test_img}'
+                       f'file.driver={iotests.imgfmt},' \
+                       f'file.file.filename={self.test_img}'

Interesting, when indenting this, I was wondering whether pylint would complain.  I was so glad it didn’t.  I really don’t like PEP8.

Quoted 2 lines

(Though I understand that style guides like PEP8 are there specifically so when someone like me goes “but I like this style better :(”, everyone else can say “but you’re objectively wrong”.  So me hating it kind of is its point, I guess.)

When I noted how you indent 'string' with f'string', I thought "o, interesting idea".. I just very like the idea of standardized code-style for the language, so I just used to follow PEP8 since first time I learned about it.

Signature · 2 lines
Best regards,
Vladimir