Skip to contentSkip to conversation

iotests: Run pylint and mypy in a testcase

PATCH In review Kevin Wolf 3 participants 7 messages May 11, 2020 – May 13, 2020

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

cover iotests: Run pylint and mypy in a testcase

Kevin Wolf (2):
iotests: Fix incomplete type declarations
iotests: Run pylint and mypy in a testcase

 tests/qemu-iotests/iotests.py |  8 +++----
 tests/qemu-iotests/297        | 44 +++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out    |  3 +++
 tests/qemu-iotests/group      |  1 +
 4 files changed, 52 insertions(+), 4 deletions(-)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out
Signature · 1 line
2.25.3
author
#1 lore raw

PATCH 1/2 iotests: Fix incomplete type declarations

We need to fix only a few places so that iotests.py can pass mypy --disallow-incomplete-defs, which seems to be a desirable option to have enabled in the long run.

---
 tests/qemu-iotests/iotests.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +4 -4
index 6c0e781af7..1d7f6fd7cf 100644
@@ -1040,7 +1040,7 @@ def _verify_cache_mode(supported_cache_modes: Sequence[str] = ()) -> None:
     if supported_cache_modes and (cachemode not in supported_cache_modes):
         notrun('not suitable for this cache mode: %s' % cachemode)
 
-def _verify_aio_mode(supported_aio_modes: Sequence[str] = ()):
+def _verify_aio_mode(supported_aio_modes: Sequence[str] = ()) -> None:
     if supported_aio_modes and (aiomode not in supported_aio_modes):
         notrun('not suitable for this aio mode: %s' % aiomode)
 
@@ -1087,7 +1087,8 @@ def skip_if_unsupported(required_formats=(), read_only=False):
     '''Skip Test Decorator
        Runs the test if all the required formats are whitelisted'''
     def skip_test_decorator(func):
-        def func_wrapper(test_case: QMPTestCase, *args, **kwargs):
+        def func_wrapper(test_case: QMPTestCase, *args: List[Any],
+                         **kwargs: Dict[str, Any]) -> None:
             if callable(required_formats):
                 fmts = required_formats(test_case)
             else:
@@ -1097,9 +1098,8 @@ def skip_if_unsupported(required_formats=(), read_only=False):
             if usf_list:
                 msg = f'{test_case}: formats {usf_list} are not whitelisted'
                 test_case.case_skip(msg)
-                return None
             else:
-                return func(test_case, *args, **kwargs)
+                func(test_case, *args, **kwargs)
         return func_wrapper
     return skip_test_decorator
 
Signature · 1 line
2.25.3
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] iotests: Fix incomplete type declarations

Quoted 8 lines · Kevin Wolf

We need to fix only a few places so that iotests.py can pass mypy --disallow-incomplete-defs, which seems to be a desirable option to have enabled in the long run.

---
 tests/qemu-iotests/iotests.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
author
#3 lore raw

PATCH 2/2 iotests: Run pylint and mypy in a testcase

In reply to Kevin Wolf

We made sure that iotests.py passes pylint. It would be a shame if we allowed new patches in that break this again, so let's just add a meta-test case that runs pylint on it.

While we don't pass mypy --strict yet, we can already run it with a few options that would be part of --strict to make sure that we won't regress on these aspects at least until we can enable the full thing.

---
 tests/qemu-iotests/297     | 44 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out |  3 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 48 insertions(+)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out
tests/qemu-iotests/297 new file +44 -0
new file mode 100755
index 0000000000..5c5420712b
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2020 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+# get standard environment
+. ./common.rc
+
+if ! type -p "pylint-3" > /dev/null; then
+    _notrun "pylint-3 not found"
+fi
+if ! type -p "mypy" > /dev/null; then
+    _notrun "mypy not found"
+fi
+
+pylint-3 --score=n iotests.py
+
+MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
+    --disallow-any-generics --disallow-incomplete-defs \
+    --disallow-untyped-decorators --no-implicit-optional \
+    --warn-redundant-casts --warn-unused-ignores \
+    --no-implicit-reexport iotests.py
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/297.out new file +3 -0
new file mode 100644
index 0000000000..6acc843649
@@ -0,0 +1,3 @@
+QA output created by 297
+Success: no issues found in 1 source file
+*** done
tests/qemu-iotests/group +1 -0
index fe649c5b73..c514975880 100644
@@ -299,3 +299,4 @@
 289 rw quick
 290 rw auto quick
 292 rw auto quick
+297 meta
Signature · 1 line
2.25.3
Reviewed-by on v1 2/2
#4 lore raw
Quoted 8 lines · Kevin Wolf

We made sure that iotests.py passes pylint. It would be a shame if we allowed new patches in that break this again, so let's just add a meta-test case that runs pylint on it.

While we don't pass mypy --strict yet, we can already run it with a few options that would be part of --strict to make sure that we won't regress on these aspects at least until we can enable the full thing.

Quoted 8 lines
---
 tests/qemu-iotests/297     | 44 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out |  3 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 48 insertions(+)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out

Bit of a shame that this takes 8 s (on my machine at least) and will run with every format/protocol combination unless I explictly exclude it with -x meta... But I suppose the actual problem here is the fact that tests still can’t just define a “This is the format/protocol combination I require” and then you can just let all tests run once with that default combination. (And maybe afterwards run all tests again with some custom combinations, but only when that makes sense.)

Well. Not a new problem.

author
#5 lore raw

Am 13.05.2020 um 14:14 hat Max Reitz geschrieben:

Quoted 18 lines · Kevin Wolf
Quoted 8 lines

We made sure that iotests.py passes pylint. It would be a shame if we allowed new patches in that break this again, so let's just add a meta-test case that runs pylint on it.

While we don't pass mypy --strict yet, we can already run it with a few options that would be part of --strict to make sure that we won't regress on these aspects at least until we can enable the full thing.

Quoted 8 lines
---
 tests/qemu-iotests/297     | 44 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out |  3 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 48 insertions(+)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out
Quoted 3 lines

Bit of a shame that this takes 8 s (on my machine at least) and will run with every format/protocol combination unless I explictly exclude it with -x meta...

Yes, it's surprising how slow these tools are. At least mypy caches some stuff, so the second run is considerably faster, but pylint doesn't do that.

I wonder if there is some overlap between mypy and pylint that we could configure away in pylint to speed it up.

Quoted 5 lines

But I suppose the actual problem here is the fact that
tests still can’t just define a “This is the format/protocol combination I require” and then you can just let all tests run once with that default combination. (And maybe afterwards run all tests again with some custom combinations, but only when that makes sense.)

It's probably not hard to find more "actual problems" in the test harness...

Kevin

Reviewed-by on the v1 series
#6 lore raw
In reply to Kevin Wolf
Quoted 4 lines · Kevin Wolf

Kevin Wolf (2):
iotests: Fix incomplete type declarations
iotests: Run pylint and mypy in a testcase

Quoted 8 lines
 tests/qemu-iotests/iotests.py |  8 +++----
 tests/qemu-iotests/297        | 44 +++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out    |  3 +++
 tests/qemu-iotests/group      |  1 +
 4 files changed, 52 insertions(+), 4 deletions(-)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out

Cool!

Maybe the approach of putting this in 'make check' somewhere can alleviate that expensive test being performed in iotests directly (as Max notes), but, uhm. whatever.

For now, this will help prevent the backslide.