Skip to contentSkip to conversation

qapi: static typing conversion, pt1

PATCH Superseded John Snow 5 participants 98 messages Sep 15, 2020 – Sep 25, 2020

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover qapi: static typing conversion, pt1

Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one single warning.

- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.

- After patch 7, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.

- After patch 16, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.

Preliminary refactoring and prerequisites:

001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools' 003/37: 'qapi-gen: Separate arg-parsing from generation' 004/37: 'qapi: move generator entrypoint into module' 005/37: 'qapi: Remove wildcard includes' 006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'

common.py (and params.py):

008/37: 'qapi/common.py: Remove python compatibility workaround' 009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable' 012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints' 014/37: 'qapi/common.py: Move comments into docstrings' 015/37: 'qapi/common.py: split build_params into new file' 016/37: 'qapi: establish mypy type-checking baseline'

events.py:

017/37: 'qapi/events.py: add notational type hints' 018/37: 'qapi/events.py: Move comments into docstrings'

commands.py:

019/37: 'qapi/commands.py: Don't re-bind to variable of different type' 020/37: 'qapi/commands.py: add notational type hints' 021/37: 'qapi/commands.py: enable checking with mypy'

source.py:

022/37: 'qapi/source.py: add notational type hints' 023/37: 'qapi/source.py: delint with pylint'

gen.py:

024/37: 'qapi/gen.py: Fix edge-case of _is_user_module' 025/37: 'qapi/gen.py: add notational type hints' 026/37: 'qapi/gen.py: Enable checking with mypy' 027/37: 'qapi/gen.py: Remove unused parameter' 028/37: 'qapi/gen.py: update write() to be more idiomatic' 029/37: 'qapi/gen.py: delint with pylint'

introspect.py:

030/37: 'qapi/introspect.py: Add a typed 'extra' structure' 031/37: 'qapi/introspect.py: add _gen_features helper' 032/37: 'qapi/introspect.py: create a typed 'Node' data structure' 033/37: 'qapi/introspect.py: add notational type hints'

types.py:

034/37: 'qapi/types.py: add notational type hints' 035/37: 'qapi/types.py: remove one-letter variables'

visit.py:

036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object' 037/37: 'qapi/visit.py: add notational type hints'

John Snow (37):
python: Require 3.6+
[DO-NOT-MERGE] qapi: add debugging tools
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into module
qapi: Remove wildcard includes
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable qapi/common.py: check with pylint
qapi/common.py: add notational type hints qapi/common.py: Move comments into docstrings qapi/common.py: split build_params into new file
qapi: establish mypy type-checking baseline qapi/events.py: add notational type hints qapi/events.py: Move comments into docstrings qapi/commands.py: Don't re-bind to variable of different type qapi/commands.py: add notational type hints qapi/commands.py: enable checking with mypy qapi/source.py: add notational type hints qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module qapi/gen.py: add notational type hints
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic qapi/gen.py: delint with pylint
qapi/introspect.py: Add a typed 'extra' structure qapi/introspect.py: add _gen_features helper qapi/introspect.py: create a typed 'Node' data structure qapi/introspect.py: add notational type hints qapi/types.py: add notational type hints qapi/types.py: remove one-letter variables qapi/visit.py: remove unused parameters from gen_visit_object qapi/visit.py: add notational type hints

 docs/conf.py                  |   4 +-
 configure                     |   6 +-
 .readthedocs.yml              |   2 +-
 .travis.yml                   |   8 --
 scripts/qapi-gen.py           |  59 ++--------
 scripts/qapi/.flake8          |   2 +
 scripts/qapi/commands.py      |  85 +++++++++++----
 scripts/qapi/common.py        | 161 +++++++++++++++------------
 scripts/qapi/debug.py         |  78 +++++++++++++
 scripts/qapi/doc.py           |   3 +-
 scripts/qapi/events.py        |  61 ++++++++---
 scripts/qapi/expr.py          |   4 +-
 scripts/qapi/gen.py           | 147 +++++++++++++------------
 scripts/qapi/introspect.py    | 200 +++++++++++++++++++++++-----------
 scripts/qapi/mypy.ini         |  30 +++++
 scripts/qapi/params.py        |  40 +++++++
 scripts/qapi/parser.py        |   4 +-
 scripts/qapi/pylintrc         |  71 ++++++++++++
 scripts/qapi/schema.py        |  33 +++---
 scripts/qapi/script.py        |  91 ++++++++++++++++
 scripts/qapi/source.py        |  34 +++---
 scripts/qapi/types.py         | 125 ++++++++++++++-------
 scripts/qapi/visit.py         | 104 +++++++++++++-----
 tests/qemu-iotests/iotests.py |   2 -
 24 files changed, 947 insertions(+), 407 deletions(-)
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/debug.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/params.py
 create mode 100644 scripts/qapi/pylintrc
 create mode 100644 scripts/qapi/script.py
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH 1/37 python: Require 3.6+

Python 3.5 is now EOL. Python 3.6 brings with it several nice things:

- Literal string interpolation (PEP 498)
- Syntax for variable annotations (PEP 526)
- Preserving keyword argument order (PEP 468) *

PEP 526 in particular will allow us to convert the QAPI module to the statically typed subset of the Python language. These static type hints improve the interactive editing experience in IDEs and help make refactoring and adding new features faster, easier and safer.

*Note: **kwargs is now guaranteed to preserve keyword ordering, but dict itself is still allowed to have an arbitrary order based on implementation. It is not until Python 3.7 that the dict type guarantees insertion order.

---
 docs/conf.py                  | 4 ++--
 configure                     | 6 +++---
 .readthedocs.yml              | 2 +-
 .travis.yml                   | 8 --------
 tests/qemu-iotests/iotests.py | 2 --
 5 files changed, 6 insertions(+), 16 deletions(-)
docs/conf.py +2 -2
index 0dbd90dc11..8aeac40124 100644
@@ -36,9 +36,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,5):
+if sys.version_info < (3,6):
     raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.5 or better\n")
+        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
configure +3 -3
index ce27eafb0a..33292500e7 100755
@@ -913,7 +913,7 @@ fi
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
+# We prefer python 3.6+. A bare 'python' is traditionally
 # python 2.x, but some distros have it as python 3.x, so
 # we check that too
 python=
@@ -1961,8 +1961,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
-  error_exit "Cannot use '$python', Python >= 3.5 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+  error_exit "Cannot use '$python', Python >= 3.6 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
.readthedocs.yml +1 -1
index 8355dbc634..7fb7b8dd61 100644
@@ -17,4 +17,4 @@ formats: all
 # we require for other Python in our codebase (our conf.py
 # enforces this, and some code needs it.)
 python:
-  version: 3.5
+  version: 3.6
.travis.yml +0 -8
index 65341634d0..0dca15a5e8 100644
@@ -267,14 +267,6 @@ jobs:
 
 
     # Python builds
-    - name: "GCC Python 3.5 (x86_64-softmmu)"
-      env:
-        - CONFIG="--target-list=x86_64-softmmu"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-      language: python
-      python: 3.5
-
-
     - name: "GCC Python 3.6 (x86_64-softmmu)"
       env:
         - CONFIG="--target-list=x86_64-softmmu"
tests/qemu-iotests/iotests.py +0 -2
index 91e4a57126..f48460480a 100644
@@ -40,8 +40,6 @@
 from qemu import qtest
 from qemu.qmp import QMPMessage
 
-assert sys.version_info >= (3, 6)
-
 # Use this logger for logging messages directly from the iotests module
 logger = logging.getLogger('qemu.iotests')
 logger.addHandler(logging.NullHandler())
Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH 01/37] python: Require 3.6+

Quoted 46 lines · John Snow

Python 3.5 is now EOL. Python 3.6 brings with it several nice things:

- Literal string interpolation (PEP 498)
- Syntax for variable annotations (PEP 526)
- Preserving keyword argument order (PEP 468) *

PEP 526 in particular will allow us to convert the QAPI module to the statically typed subset of the Python language. These static type hints improve the interactive editing experience in IDEs and help make refactoring and adding new features faster, easier and safer.

*Note: **kwargs is now guaranteed to preserve keyword ordering, but dict itself is still allowed to have an arbitrary order based on implementation. It is not until Python 3.7 that the dict type guarantees insertion order.

---
 docs/conf.py                  | 4 ++--
 configure                     | 6 +++---
 .readthedocs.yml              | 2 +-
 .travis.yml                   | 8 --------
 tests/qemu-iotests/iotests.py | 2 --
 5 files changed, 6 insertions(+), 16 deletions(-)
docs/conf.py +2 -2
index 0dbd90dc11..8aeac40124 100644
@@ -36,9 +36,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,5):
+if sys.version_info < (3,6):
     raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.5 or better\n")
+        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
configure +0 -0
index ce27eafb0a..33292500e7 100755

configure fi · line 916

Quoted 5 lines
configure +1 -1
@@ -913,7 +913,7 @@ fi
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
+# We prefer python 3.6+. A bare 'python' is traditionally

We require

Quoted 56 lines
configure +2 -2
 # python 2.x, but some distros have it as python 3.x, so
 # we check that too
 python=
@@ -1961,8 +1961,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
-  error_exit "Cannot use '$python', Python >= 3.5 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+  error_exit "Cannot use '$python', Python >= 3.6 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
.readthedocs.yml +1 -1
index 8355dbc634..7fb7b8dd61 100644
@@ -17,4 +17,4 @@ formats: all
 # we require for other Python in our codebase (our conf.py
 # enforces this, and some code needs it.)
 python:
-  version: 3.5
+  version: 3.6
.travis.yml +0 -8
index 65341634d0..0dca15a5e8 100644
@@ -267,14 +267,6 @@ jobs:
 
 
     # Python builds
-    - name: "GCC Python 3.5 (x86_64-softmmu)"
-      env:
-        - CONFIG="--target-list=x86_64-softmmu"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-      language: python
-      python: 3.5
-
-
     - name: "GCC Python 3.6 (x86_64-softmmu)"
       env:
         - CONFIG="--target-list=x86_64-softmmu"
tests/qemu-iotests/iotests.py +0 -2
index 91e4a57126..f48460480a 100644
@@ -40,8 +40,6 @@
 from qemu import qtest
 from qemu.qmp import QMPMessage
 
-assert sys.version_info >= (3, 6)
-
 # Use this logger for logging messages directly from the iotests module
 logger = logging.getLogger('qemu.iotests')
 logger.addHandler(logging.NullHandler())
#3 lore raw

Re: [PATCH 01/37] python: Require 3.6+

In reply to John Snow
Quoted 1 line · John Snow

Python 3.5 is now EOL. Python 3.6 brings with it several nice things:

NB Upstream EOL is not relevant. We need to base min version changes on the OS platform support matrix. So it would be better for the commit message to provide details of the version in the various distros we target, to show that the min version change is valid.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#4 lore raw

PATCH 2/37 qapi: add debugging tools

In reply to John Snow

This adds some really childishly simple debugging tools. Maybe they're interesting for someone else, too?

---
 scripts/qapi/debug.py | 78 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
 create mode 100644 scripts/qapi/debug.py
scripts/qapi/debug.py new file +78 -0
new file mode 100644
index 0000000000..bacf5ee180
@@ -0,0 +1,78 @@
+"""
+Small debugging facilities for mypy static analysis work.
+(C) 2020 John Snow, for Red Hat, Inc.
+"""
+
+import inspect
+import json
+from typing import Dict, List, Any
+from types import FrameType
+
+
+OBSERVED_TYPES: Dict[str, List[str]] = {}
+
+
+# You have no idea how long it took to find this return type...
+def caller_frame() -> FrameType:
+    """
+    Returns the stack frame of the caller's caller.
+    e.g. foo() -> caller() -> caller_frame() return's foo's stack frame.
+    """
+    stack = inspect.stack()
+    caller = stack[2].frame
+    if caller is None:
+        msg = "Python interpreter does not support stack frame inspection"
+        raise RuntimeError(msg)
+    return caller
+
+
+def _add_type_record(name: str, typestr: str) -> None:
+    seen = OBSERVED_TYPES.setdefault(name, [])
+    if typestr not in seen:
+        seen.append(typestr)
+
+
+def record_type(name: str, value: Any, dict_names: bool = False) -> None:
+    """
+    Record the type of a variable.
+
+    :param name: The name of the variable
+    :param value: The value of the variable
+    """
+    _add_type_record(name, str(type(value)))
+
+    try:
+        for key, subvalue in value.items():
+            subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]"
+            _add_type_record(subname, str(type(subvalue)))
+        return
+    except AttributeError:
+        # (Wasn't a dict or anything resembling one.)
+        pass
+
+    # str is iterable, but not in the way we want!
+    if isinstance(value, str):
+        return
+
+    try:
+        for elem in value:
+            _add_type_record(f"{name}.[list_elem]", str(type(elem)))
+    except TypeError:
+        # (Wasn't a list or anything else iterable.)
+        pass
+
+
+def show_types() -> None:
+    """
+    Print all of the currently known variable types to stdout.
+    """
+    print(json.dumps(OBSERVED_TYPES, indent=2))
+
+
+def record_locals(show: bool = False, dict_names: bool = False) -> None:
+    caller = caller_frame()
+    name = caller.f_code.co_name
+    for key, value in caller.f_locals.items():
+        record_type(f"{name}.{key}", value, dict_names=dict_names)
+    if show:
+        show_types()
Signature · 1 line
2.26.2
author
#5 lore raw

PATCH 3/37 qapi-gen: Separate arg-parsing from generation

In reply to John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
 scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++-------------
 1 file changed, 63 insertions(+), 24 deletions(-)
scripts/qapi-gen.py +63 -24
index 4b03f7d53b..59becba3e1 100644
@@ -1,9 +1,13 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
@@ -11,21 +15,65 @@
 
 from qapi.commands import gen_commands
 from qapi.doc import gen_doc
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match and match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+    gen_doc(schema, output_dir, prefix)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
     parser = argparse.ArgumentParser(
         description='Generate code from a QAPI schema')
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                         help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                         help="prefix for symbols")
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
@@ -33,26 +81,17 @@ def main(argv):
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
     try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
     except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
-    gen_doc(schema, args.output_dir, args.prefix)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
 
 if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())
Signature · 1 line
2.26.2
author
#6 lore raw

PATCH 4/37 qapi: move generator entrypoint into module

In reply to John Snow

As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.

As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.

This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."

---
 scripts/qapi-gen.py        | 94 +++-----------------------------------
 scripts/qapi/commands.py   |  4 +-
 scripts/qapi/doc.py        |  2 +-
 scripts/qapi/events.py     |  8 ++--
 scripts/qapi/expr.py       |  4 +-
 scripts/qapi/gen.py        |  4 +-
 scripts/qapi/introspect.py |  8 ++--
 scripts/qapi/parser.py     |  4 +-
 scripts/qapi/schema.py     |  8 ++--
 scripts/qapi/script.py     | 91 ++++++++++++++++++++++++++++++++++++
 scripts/qapi/types.py      |  6 +--
 scripts/qapi/visit.py      |  6 +--
 12 files changed, 124 insertions(+), 115 deletions(-)
 create mode 100644 scripts/qapi/script.py
scripts/qapi-gen.py +6 -88
index 59becba3e1..e649f8dd44 100644
@@ -1,97 +1,15 @@
 #!/usr/bin/env python3
-
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-
 """
-QAPI Generator
+QAPI code generation execution shim.
 
-This script is the main entry point for generating C code from the QAPI schema.
+This file exists primarily to facilitate the running of the QAPI code
+generator without needing to install the python module to the current
+execution environment.
 """
 
-import argparse
-import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.doc import gen_doc
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
-def generate(schema_file: str,
-             output_dir: str,
-             prefix: str,
-             unmask: bool = False,
-             builtins: bool = False) -> None:
-    """
-    generate uses a given schema to produce C code in the target directory.
-
-    :param schema_file: The primary QAPI schema file.
-    :param output_dir: The output directory to store generated code.
-    :param prefix: Optional C-code prefix for symbol names.
-    :param unmask: Expose non-ABI names through introspection?
-    :param builtins: Generate code for built-in types?
-
-    :raise QAPIError: On failures.
-    """
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
-    if match and match.end() != len(prefix):
-        msg = "funny character '{:s}' in prefix '{:s}'".format(
-            prefix[match.end()], prefix)
-        raise QAPIError('', None, msg)
-
-    schema = QAPISchema(schema_file)
-    gen_types(schema, output_dir, prefix, builtins)
-    gen_visit(schema, output_dir, prefix, builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, unmask)
-    gen_doc(schema, output_dir, prefix)
-
-
-def main() -> int:
-    """
-    gapi-gen shell script entrypoint.
-    Expects arguments via sys.argv, see --help for details.
-
-    :return: int, 0 on success, 1 on failure.
-    """
-    parser = argparse.ArgumentParser(
-        description='Generate code from a QAPI schema')
-    parser.add_argument('-b', '--builtins', action='store_true',
-                        help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store',
-                        default=DEFAULT_OUTPUT_DIR,
-                        help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store',
-                        default=DEFAULT_PREFIX,
-                        help="prefix for symbols")
-    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
-                        dest='unmask',
-                        help="expose non-ABI names in introspection")
-    parser.add_argument('schema', action='store')
-    args = parser.parse_args()
-
-    try:
-        generate(args.schema,
-                 output_dir=args.output_dir,
-                 prefix=args.prefix,
-                 unmask=args.unmask,
-                 builtins=args.builtins)
-    except QAPIError as err:
-        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
-        return 1
-    return 0
-
+from qapi import script
 
 if __name__ == '__main__':
-    sys.exit(main())
+    sys.exit(script.main())
scripts/qapi/commands.py +2 -2
index 3cf9e1110b..ce5926146a 100644
@@ -13,8 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import *
+from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/doc.py +1 -1
index 92f584edcf..cbf7076ed9 100644
@@ -5,7 +5,7 @@
 """This script produces the documentation of a qapi schema in texinfo format"""
 
 import re
-from qapi.gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc, QAPISchemaVisitor
 
 
 MSG_FMT = """
scripts/qapi/events.py +4 -4
index b544af5a1c..0467272438 100644
@@ -12,10 +12,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember
-from qapi.types import gen_enum, gen_enum_lookup
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember
+from .types import gen_enum, gen_enum_lookup
 
 
 def build_event_send_proto(name, arg_type, boxed):
scripts/qapi/expr.py +2 -2
index 2942520399..03b31ecfc1 100644
@@ -16,8 +16,8 @@
 
 import re
 from collections import OrderedDict
-from qapi.common import c_name
-from qapi.error import QAPISemError
+from .common import c_name
+from .error import QAPISemError
 
 
 # Names must be letters, numbers, -, and _.  They must start with letter,
scripts/qapi/gen.py +2 -2
index bf5552a4e7..8df19a0df0 100644
@@ -17,8 +17,8 @@
 import re
 from contextlib import contextmanager
 
-from qapi.common import *
-from qapi.schema import QAPISchemaVisitor
+from .common import *
+from .schema import QAPISchemaVisitor
 
 
 class QAPIGen:
scripts/qapi/introspect.py +4 -4
index 23652be810..2a34cd1e8e 100644
@@ -10,10 +10,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaMonolithicCVisitor
-from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                         QAPISchemaType)
+from .common import *
+from .gen import QAPISchemaMonolithicCVisitor
+from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
+                     QAPISchemaType)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/parser.py +2 -2
index 165925ca72..327cf05736 100644
@@ -18,8 +18,8 @@
 import re
 from collections import OrderedDict
 
-from qapi.error import QAPIParseError, QAPISemError
-from qapi.source import QAPISourceInfo
+from .error import QAPIParseError, QAPISemError
+from .source import QAPISourceInfo
 
 
 class QAPISchemaParser:
scripts/qapi/schema.py +4 -4
index 78309a00f0..a835ee6fde 100644
@@ -18,10 +18,10 @@
 import re
 from collections import OrderedDict
 
-from qapi.common import c_name, pointer_suffix
-from qapi.error import QAPIError, QAPISemError
-from qapi.expr import check_exprs
-from qapi.parser import QAPISchemaParser
+from .common import c_name, pointer_suffix
+from .error import QAPIError, QAPISemError
+from .expr import check_exprs
+from .parser import QAPISchemaParser
 
 
 class QAPISchemaEntity:
scripts/qapi/script.py new file +91 -0

91 lines not on this page · raw message

scripts/qapi/types.py +3 -3
index 3640f17cd6..ca9a5aacb3 100644
@@ -13,9 +13,9 @@
 # See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
 
 # variants must be emitted before their container; track what has already
scripts/qapi/visit.py +3 -3
index cdabc5fa28..7850f6e848 100644
@@ -13,9 +13,9 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
Signature · 1 line
2.26.2
#7 lore raw

Re: [PATCH 04/37] qapi: move generator entrypoint into module

Quoted 6 lines · John Snow

As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.

As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.

Should this be split into two commits, one for each of the paragraphs above?

PEP 8 recommends absolute imports, with one exception:

However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:

from . import sibling
from .sibling import example

Standard library code should avoid complex package layouts and always use absolute imports.

Do you think it covers your use of relative imports?

Quoted 3 lines

This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."

Can you give me an example of some tool having troube?

Quoted 16 lines
---
 scripts/qapi-gen.py        | 94 +++-----------------------------------
 scripts/qapi/commands.py   |  4 +-
 scripts/qapi/doc.py        |  2 +-
 scripts/qapi/events.py     |  8 ++--
 scripts/qapi/expr.py       |  4 +-
 scripts/qapi/gen.py        |  4 +-
 scripts/qapi/introspect.py |  8 ++--
 scripts/qapi/parser.py     |  4 +-
 scripts/qapi/schema.py     |  8 ++--
 scripts/qapi/script.py     | 91 ++++++++++++++++++++++++++++++++++++
 scripts/qapi/types.py      |  6 +--
 scripts/qapi/visit.py      |  6 +--
 12 files changed, 124 insertions(+), 115 deletions(-)
 create mode 100644 scripts/qapi/script.py

Naming is hard... main.py?

scripts/qapi-gen.py line 5

Quoted 11 lines
scripts/qapi-gen.py +0 -4
index 59becba3e1..e649f8dd44 100644
@@ -1,97 +1,15 @@
 #!/usr/bin/env python3
-
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-

Keep the license blurb.

[...]

author
#8 lore raw

Re: [PATCH 04/37] qapi: move generator entrypoint into module

Quoted 6 lines · Markus Armbruster
Quoted 4 lines · John Snow

As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.

Quoted 6 lines
Quoted 2 lines

As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.

Should this be split into two commits, one for each of the paragraphs above?

Hmm ... I hadn't considered it was possible, but actually ... I guess I can split those out, yeah.

Quoted 6 lines

PEP 8 recommends absolute imports, with one exception:

However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:

Quoted 8 lines

from . import sibling
from .sibling import example

Standard library code should avoid complex package layouts and always use absolute imports.

Do you think it covers your use of relative imports?

Admittedly I am going by trial and error; my experience is that the relative imports behave the nicest.

There is a historical fear of explicit relative imports because they are "new" and years of Python2 compatibility rendered many afraid of them. It is advice safely ignored in my opinion.

Using absolute imports (e.g. from qapi.sibling import foo) gets messy in virtual environments when you have *installed* the module in question: it becomes ambiguous as to *which* qapi you meant: the one in this folder, or the one installed to the environment?

Python, mypy, pylint, flake8 etc disagree sometimes, or can get confused; thinking there are two copies of each module.

Just my experience that relative imports seem to give me the least trouble.

Quoted 6 lines
Quoted 3 lines

This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."

Can you give me an example of some tool having troube?

I'd have to code up some examples. I have some hobby code unrelated to QEMU where I struggled to get flake8, mypy, and pylint all cooperating with an import regime until I gave up and used explicit relative imports.

Quoted 17 lines
Quoted 15 lines
---
  scripts/qapi-gen.py        | 94 +++-----------------------------------
  scripts/qapi/commands.py   |  4 +-
  scripts/qapi/doc.py        |  2 +-
  scripts/qapi/events.py     |  8 ++--
  scripts/qapi/expr.py       |  4 +-
  scripts/qapi/gen.py        |  4 +-
  scripts/qapi/introspect.py |  8 ++--
  scripts/qapi/parser.py     |  4 +-
  scripts/qapi/schema.py     |  8 ++--
  scripts/qapi/script.py     | 91 ++++++++++++++++++++++++++++++++++++
  scripts/qapi/types.py      |  6 +--
  scripts/qapi/visit.py      |  6 +--
  12 files changed, 124 insertions(+), 115 deletions(-)
Quoted 1 line

create mode 100644 scripts/qapi/script.py

Quoted 2 lines

Naming is hard... main.py?

I was thinking of changing this myself, so this convinced me.

Quoted 14 lines

scripts/qapi-gen.py line 5

Quoted 11 lines
scripts/qapi-gen.py +0 -4
index 59becba3e1..e649f8dd44 100644
@@ -1,97 +1,15 @@
  #!/usr/bin/env python3
-
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-

Keep the license blurb.

This is a mistake. I tried to convince git to "move" the old file and then add a "new" file to preserve history, but of course that's not how git manages file histories, so it didn't work.

TLDR: I didn't delete the license blurb, I just didn't "add" it again. I'll "fix" that.

Quoted 2 lines

[...]

#9 lore raw

Re: [PATCH 04/37] qapi: move generator entrypoint into module

Quoted 29 lines · John Snow
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · John Snow

As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.

Quoted 6 lines
Quoted 2 lines

As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.

Should this be split into two commits, one for each of the paragraphs
above?

Hmm ... I hadn't considered it was possible, but actually ... I guess I can split those out, yeah.

Quoted 11 lines

PEP 8 recommends absolute imports, with one exception:
However, explicit relative imports are an acceptable
alternative to
absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:
from . import sibling
from .sibling import example
Standard library code should avoid complex package layouts and always use absolute imports.
Do you think it covers your use of relative imports?

Quoted 5 lines

Admittedly I am going by trial and error; my experience is that the relative imports behave the nicest.

There is a historical fear of explicit relative imports because they are "new" and years of Python2 compatibility

Having to struggle with that for years was bound to damage brains.

Quoted 16 lines

rendered many afraid of
them. It is advice safely ignored in my opinion.

Using absolute imports (e.g. from qapi.sibling import foo) gets messy in virtual environments when you have *installed* the module in
question: it becomes ambiguous as to *which* qapi you meant: the one in this folder, or the one installed to the environment?

Python, mypy, pylint, flake8 etc disagree sometimes, or can get confused; thinking there are two copies of each module.

Just my experience that relative imports seem to give me the least trouble.

Quoted 3 lines
Quoted 3 lines

This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."

Quoted 6 lines
Quoted 2 lines

Can you give me an example of some tool having troube?

I'd have to code up some examples. I have some hobby code unrelated to QEMU where I struggled to get flake8, mypy, and pylint all cooperating with an import regime until I gave up and used explicit relative imports.

I'd make the switch when we actually do run into trouble.

But I'm willing to take your advice and switch now.

[...]

author
#10 lore raw

PATCH 5/37 qapi: Remove wildcard includes

In reply to John Snow

Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.

Remove them and include things explicitly by name instead.

---
 scripts/qapi/commands.py   |  6 +++++-
 scripts/qapi/events.py     |  7 ++++++-
 scripts/qapi/gen.py        | 12 +++++++++---
 scripts/qapi/introspect.py |  7 ++++++-
 scripts/qapi/types.py      |  8 +++++++-
 scripts/qapi/visit.py      | 10 +++++++++-
 6 files changed, 42 insertions(+), 8 deletions(-)
scripts/qapi/commands.py +5 -1
index ce5926146a..e1df0e341f 100644
@@ -13,7 +13,11 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    build_params,
+    c_name,
+    mcgen,
+)
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
scripts/qapi/events.py +6 -1
index 0467272438..6b3afa14d7 100644
@@ -12,7 +12,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    build_params,
+    c_enum_const,
+    c_name,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +9 -3
index 8df19a0df0..11472ba043 100644
@@ -11,13 +11,19 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-
+from contextlib import contextmanager
 import errno
 import os
 import re
-from contextlib import contextmanager
 
-from .common import *
+from .common import (
+    c_fname,
+    gen_endif,
+    gen_if,
+    guardend,
+    guardstart,
+    mcgen,
+)
 from .schema import QAPISchemaVisitor
 
 
scripts/qapi/introspect.py +6 -1
index 2a34cd1e8e..b036fcf9ce 100644
@@ -10,7 +10,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
                      QAPISchemaType)
scripts/qapi/types.py +7 -1
index ca9a5aacb3..53b47f9e58 100644
@@ -13,7 +13,13 @@
 # See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
scripts/qapi/visit.py +9 -1
index 7850f6e848..ea277e7704 100644
@@ -13,7 +13,15 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+    pop_indent,
+    push_indent,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
 
Signature · 1 line
2.26.2
author
#11 lore raw

PATCH 6/37 qapi: delint using flake8

In reply to John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

---
 scripts/qapi/.flake8     |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +++++---
 scripts/qapi/visit.py    | 15 ++++++++++-----
 4 files changed, 19 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
new file mode 100644
index 0000000000..45d8146f3f
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.
\ No newline at end of file
scripts/qapi/commands.py +2 -1
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)
 {
     Visitor *v;
 
scripts/qapi/schema.py +5 -3
index a835ee6fde..b77e0e56b2 100644
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if not self.tag_member: # flat union
+        if not self.tag_member:  # flat union
             self.tag_member = seen.get(c_name(self._tag_name))
             base = "'base'"
             # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
         self._entity_dict = {}
         self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
-        self._make_module(None) # built-ins
+        self._make_module(None)  # built-ins
         self._make_module(fname)
         self._predefining = True
         self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
             # But it's not tight: the disjunction need not imply it.  We
             # may end up compiling useless wrapper types.
             # TODO kill simple unions or implement the disjunction
-            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+            # pylint: disable=protected-access
+            assert (ifcond or []) == typ._ifcond
         else:
             self._def_entity(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +10 -5
index ea277e7704..31bf46e7f7 100644
@@ -31,7 +31,8 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj,
+                           Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +126,8 @@ def gen_visit_object_members(name, base, members, variants):
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+                           Error **errp)
 {
     bool ok = false;
     %(c_name)s *tail;
@@ -158,7 +160,8 @@ def gen_visit_list(name, element_type):
 def gen_visit_enum(name):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj,
+                           Error **errp)
 {
     int value = *obj;
     bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +175,8 @@ def gen_visit_enum(name):
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+                           Error **errp)
 {
     bool ok = false;
 
@@ -247,7 +251,8 @@ def gen_visit_alternate(name, variants):
 def gen_visit_object(name, base, members, variants):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+                           Error **errp)
 {
     bool ok = false;
 
Signature · 1 line
2.26.2
#12 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 19 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

---
 scripts/qapi/.flake8     |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +++++---
 scripts/qapi/visit.py    | 15 ++++++++++-----
 4 files changed, 19 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +0 -0
new file mode 100644
index 0000000000..45d8146f3f

scripts/qapi/.flake8 line 2

Quoted 2 lines
scripts/qapi/.flake8 new file +2 -0
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.

I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with flake8's E722 compared to pylint's W0702?

Quoted 1 line

\ No newline at end of file

So put one there :)

scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 73

Quoted 11 lines
scripts/qapi/commands.py +2 -1
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)

The continued parameter list may become misalignd in generated C. E.g.

    static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
                                              Error **errp)
    {
    ...
    }

Do we care?

More of the same below.

Quoted 3 lines
scripts/qapi/commands.py +0 -0
 {
     Visitor *v;
 

[...]

author
#13 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 15 lines · Markus Armbruster
Quoted 13 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

---
  scripts/qapi/.flake8     |  2 ++
  scripts/qapi/commands.py |  3 ++-
  scripts/qapi/schema.py   |  8 +++++---
  scripts/qapi/visit.py    | 15 ++++++++++-----
  4 files changed, 19 insertions(+), 9 deletions(-)

create mode 100644 scripts/qapi/.flake8

Quoted 12 lines

scripts/qapi/.flake8 line 2

Quoted 8 lines
scripts/qapi/.flake8 new file +2 -0
new file mode 100644
index 0000000000..45d8146f3f
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.

I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with flake8's E722 compared to pylint's W0702?

Flake8 will warn on *any* bare except, but Pylint's is context-aware and will suppress the warning if you re-raise the exception.

I don't actually think this comes up in the qapi code base, but it does come up in the ./python/qemu code base.

(One of my goals is unifying the lint checking regime for both packages.)

Quoted 4 lines
Quoted 1 line

\ No newline at end of file

So put one there :)

Whupps, okay.

Quoted 14 lines

scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 73

Quoted 11 lines
scripts/qapi/commands.py +2 -1
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
  def gen_marshal_output(ret_type):
      return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)

The continued parameter list may become misalignd in generated C. E.g.

Quoted 8 lines
     static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
                                               Error **errp)
     {
     ...
     }

Do we care?

Yeah, I don't know. Do we?

It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.

Path of least resistance led me here, but I can try and appease both systems if you'd prefer.

Quoted 7 lines

More of the same below.

Quoted 3 lines
scripts/qapi/commands.py +0 -0
  {
      Visitor *v;
 

[...]

#14 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 29 lines · John Snow
Quoted 15 lines · Markus Armbruster
Quoted 13 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

---
  scripts/qapi/.flake8     |  2 ++
  scripts/qapi/commands.py |  3 ++-
  scripts/qapi/schema.py   |  8 +++++---
  scripts/qapi/visit.py    | 15 ++++++++++-----
  4 files changed, 19 insertions(+), 9 deletions(-)

create mode 100644 scripts/qapi/.flake8

Quoted 12 lines

scripts/qapi/.flake8 line 2

Quoted 8 lines
scripts/qapi/.flake8 new file +2 -0
new file mode 100644
index 0000000000..45d8146f3f
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.

I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with
flake8's E722 compared to pylint's W0702?

Quoted 2 lines

Flake8 will warn on *any* bare except, but Pylint's is context-aware and will suppress the warning if you re-raise the exception.

Should this information be worked into the comment?

Quoted 33 lines

I don't actually think this comes up in the qapi code base, but it does come up in the ./python/qemu code base.

(One of my goals is unifying the lint checking regime for both packages.)

Quoted 3 lines
Quoted 1 line

\ No newline at end of file

So put one there :)

Whupps, okay.

Quoted 15 lines

scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74

Quoted 11 lines
scripts/qapi/commands.py +2 -0
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
  def gen_marshal_output(ret_type):
      return mcgen('''
  -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
 QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)

The continued parameter list may become misalignd in generated C. E.g.
static void qmp_marshal_output_BlockInfoList(BlockInfoList
*ret_in, QObject **ret_out,

Quoted 6 lines
                                               Error **errp)
     {
     ...
     }

Do we care?

Quoted 1 line

Yeah, I don't know. Do we?

I care, but I also care for automated style checks.

Quoted 5 lines

It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.

Path of least resistance led me here, but I can try and appease both systems if you'd prefer.

Up to now, I ran the style checkers manually, and this was just one of several complaints to ignore, so I left the code alone.

If it gets in the way of running them automatically, and messing up the generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.

Quoted 7 lines
Quoted 7 lines

More of the same below.

Quoted 3 lines
scripts/qapi/commands.py +0 -0
  {
      Visitor *v;

[...]

author
#15 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 31 lines · Markus Armbruster
Quoted 29 lines · John Snow
Quoted 27 lines · Markus Armbruster
Quoted 19 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

---
   scripts/qapi/.flake8     |  2 ++
   scripts/qapi/commands.py |  3 ++-
   scripts/qapi/schema.py   |  8 +++++---
   scripts/qapi/visit.py    | 15 ++++++++++-----
   4 files changed, 19 insertions(+), 9 deletions(-)

create mode 100644 scripts/qapi/.flake8

scripts/qapi/.flake8 new file +0 -0
new file mode 100644
index 0000000000..45d8146f3f

scripts/qapi/.flake8 line 2

Quoted 2 lines
scripts/qapi/.flake8 new file +2 -0
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.

I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with
flake8's E722 compared to pylint's W0702?

Quoted 5 lines
Quoted 2 lines

Flake8 will warn on *any* bare except, but Pylint's is context-aware and will suppress the warning if you re-raise the exception.

Should this information be worked into the comment?

I'll improve it a little, but I'll add what I wrote above to the commit message.

Quoted 43 lines
Quoted 33 lines

I don't actually think this comes up in the qapi code base, but it does come up in the ./python/qemu code base.

(One of my goals is unifying the lint checking regime for both packages.)

Quoted 3 lines
Quoted 1 line

\ No newline at end of file

So put one there :)

Whupps, okay.

Quoted 15 lines

scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74

Quoted 11 lines
scripts/qapi/commands.py +2 -0
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
   def gen_marshal_output(ret_type):
       return mcgen('''
   -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
 QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)

The continued parameter list may become misalignd in generated C. E.g.
static void qmp_marshal_output_BlockInfoList(BlockInfoList
*ret_in, QObject **ret_out,

Quoted 6 lines
                                                Error **errp)
      {
      ...
      }

Do we care?

Quoted 1 line

Yeah, I don't know. Do we?

I care, but I also care for automated style checks.

Quoted 5 lines

It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.

Path of least resistance led me here, but I can try and appease both systems if you'd prefer.

Quoted 7 lines

Up to now, I ran the style checkers manually, and this was just one of several complaints to ignore, so I left the code alone.

If it gets in the way of running them automatically, and messing up the generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.

I changed this a little to put all the args on the next line, which is slightly unusual but works okay.

I think that's a fine middle ground, because the alternative (to me) is to start using abstracted code generation tokens in a tree structure, etc etc etc.

Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

More of the same below.

Quoted 3 lines
scripts/qapi/commands.py +0 -0
   {
       Visitor *v;

[...]

#16 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 11 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

[...]

Quoted 38 lines
Quoted 29 lines
Quoted 21 lines
Quoted 13 lines

scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74

Quoted 11 lines
scripts/qapi/commands.py +2 -0
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
   def gen_marshal_output(ret_type):
       return mcgen('''
   -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
 QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)

The continued parameter list may become misalignd in generated C. E.g.

Quoted 7 lines
      static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
                                                Error **errp)
      {
      ...
      }

Do we care?

Quoted 1 line

Yeah, I don't know. Do we?

I care, but I also care for automated style checks.

Quoted 5 lines

It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.

Path of least resistance led me here, but I can try and appease both systems if you'd prefer.

Quoted 8 lines

Up to now, I ran the style checkers manually, and this was just one of
several complaints to ignore, so I left the code alone. If it gets in the way of running them automatically, and messing up the
generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.

Quoted 2 lines

I changed this a little to put all the args on the next line, which is slightly unusual but works okay.

I think it's slightly more unusual than the non-matching indentation was.

Yet another way to skin this cat:

static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
QObject **ret_out, Error **errp)

Now the second line is not aligned with the left parenthesis both in the Python source and in the generated C.

Quoted 9 lines

I think that's a fine middle ground, because the alternative (to me) is to start using abstracted code generation tokens in a tree structure, etc etc etc.

Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.

The thought "screw this, pile the output through /usr/bin/indent" has crossed my mind more than once.

author
#17 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 58 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

[...]

Quoted 38 lines
Quoted 29 lines
Quoted 21 lines
Quoted 13 lines

scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74

Quoted 11 lines
scripts/qapi/commands.py +2 -0
index e1df0e341f..2e4b4de0fa 100644
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
    def gen_marshal_output(ret_type):
        return mcgen('''
    -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
 QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)

The continued parameter list may become misalignd in generated C. E.g.

Quoted 7 lines
       static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
                                                 Error **errp)
       {
       ...
       }

Do we care?

Quoted 1 line

Yeah, I don't know. Do we?

I care, but I also care for automated style checks.

Quoted 5 lines

It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.

Path of least resistance led me here, but I can try and appease both systems if you'd prefer.

Quoted 8 lines

Up to now, I ran the style checkers manually, and this was just one of
several complaints to ignore, so I left the code alone. If it gets in the way of running them automatically, and messing up the
generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.

Quoted 2 lines

I changed this a little to put all the args on the next line, which is slightly unusual but works okay.

I think it's slightly more unusual than the non-matching indentation was.

Quoted 8 lines

Yet another way to skin this cat:

static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
QObject **ret_out, Error **errp)

Now the second line is not aligned with the left parenthesis both in the Python source and in the generated C.

Your wish is my command!
(Except in the other replies where I am arguing with you.)

Quoted 4 lines
Quoted 4 lines

I think that's a fine middle ground, because the alternative (to me) is to start using abstracted code generation tokens in a tree structure, etc etc etc.

Quoted 10 lines
Quoted 5 lines

Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.

The thought "screw this, pile the output through /usr/bin/indent" has crossed my mind more than once.

Bigger fish to fry, but with other languages like rust looming, making the core generation facilities nicer might be ...nice. Not for this series.

Two approaches in general would make sense to me:

1. Building an AST for C instead of strings, and rendering the AST.
2. Writing a templating engine that doesn't break the Python indentation flow by hoisting them into module constants and improving the rendering logic.

--js

#18 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 2 lines · John Snow
Quoted 1 line · Markus Armbruster

John Snow <[email protected]> writes:

[...]

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.

Quoted 8 lines
Quoted 4 lines

The thought "screw this, pile the output through /usr/bin/indent" has
crossed my mind more than once.

Bigger fish to fry, but with other languages like rust looming, making the core generation facilities nicer might be ...nice. Not for this series.

Definitely not.

The way we generate C now is (close to) the stupidest way that could possibly work. Stupid is *good*, until it creates actual problems.

Any improvement will have to balance readability vs. hackability.

What we have now is often hard to read, but pretty easy to hack. May sound like a contradiction, but it's been my experience.

Quoted 3 lines

Two approaches in general would make sense to me:

1. Building an AST for C instead of strings, and rendering the AST.

I fear the need to define an AST for C makes this uneconomical.

I've missed the ease of generating Lisp in Lisp many times.

Quoted 3 lines

2. Writing a templating engine that doesn't break the Python indentation flow by hoisting them into module constants and improving the rendering logic.

Too vague for me to judge, happy to look at PoC patches.

author
#19 lore raw

Re: [PATCH 06/37] qapi: delint using flake8

Quoted 12 lines · Markus Armbruster
Quoted 2 lines · John Snow
Quoted 1 line · Markus Armbruster

John Snow <[email protected]> writes:

[...]

Quoted 3 lines

2. Writing a templating engine that doesn't break the Python indentation flow by hoisting them into module constants and improving the rendering logic.

Too vague for me to judge, happy to look at PoC patches.

Okay, some other day. Not important enough presently. It's just hard to stop brainstorming ideas as I learn about and touch different areas of code.

--js

author
#20 lore raw

PATCH 7/37 qapi: add pylintrc

In reply to John Snow

Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.

Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.

Therefore, to run it, you must:

Quoted 2 lines

cd :/qemu/scripts
pylint qapi/ --rcfile=qapi/pylintrc

---
 scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +74 -0
new file mode 100644
index 0000000000..c2bbb8e8e1
@@ -0,0 +1,74 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the blacklist. The
+# regex matches against base names, not paths.
+ignore-patterns=common.py,
+                doc.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore imports when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
Signature · 1 line
2.26.2
#21 lore raw

Re: [PATCH 07/37] qapi: add pylintrc

Quoted 8 lines · John Snow

Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.

Quoted 7 lines

Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.

Therefore, to run it, you must:

Quoted 1 line

cd :/qemu/scripts

-bash: cd: :/qemu/scripts: No such file or directory

;-P

Quoted 1 line
Quoted 1 line

pylint qapi/ --rcfile=qapi/pylintrc

Why not

   $ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc
Quoted 17 lines
---
 scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +4 -0
new file mode 100644
index 0000000000..c2bbb8e8e1
@@ -0,0 +1,74 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the blacklist. The
+# regex matches against base names, not paths.

scripts/qapi/pylintrc line 14

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+ignore-patterns=common.py,
+                doc.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,

Already not ignored:

    __init__.py
    commands.py
    common.py
    debug.py
    events.py
    introspect.py
    script.py

Okay.

Quoted 9 lines
scripts/qapi/pylintrc new file +9 -0
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if

scripts/qapi/pylintrc line 33

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,

I'm fine with disabling these.

Quoted 8 lines
scripts/qapi/pylintrc new file +8 -0
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]

scripts/qapi/pylintrc line 51

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _

Isn't this the default?

scripts/qapi/pylintrc line 64

Quoted 13 lines
scripts/qapi/pylintrc new file +13 -0
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore imports when computing similarities.
+ignore-imports=yes

Why?

scripts/qapi/pylintrc line 74

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]

Looks like you started with output of --generate-rcfile,

author
#22 lore raw

Re: [PATCH 07/37] qapi: add pylintrc

Quoted 20 lines · Markus Armbruster
Quoted 8 lines · John Snow

Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.

Quoted 7 lines

Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.

Therefore, to run it, you must:

Quoted 1 line

cd :/qemu/scripts

-bash: cd: :/qemu/scripts: No such file or directory

Quoted 8 lines

;-P

Quoted 1 line
Quoted 1 line

pylint qapi/ --rcfile=qapi/pylintrc

Why not

    $ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc

No reason I'm aware of, I have just been testing with CWD at the scripts dir myself because of how python imports work.

If it works this way, enjoy!

Quoted 53 lines
Quoted 17 lines
---
  scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 74 insertions(+)

create mode 100644 scripts/qapi/pylintrc

scripts/qapi/pylintrc new file +4 -0
new file mode 100644
index 0000000000..c2bbb8e8e1
@@ -0,0 +1,74 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the blacklist. The
+# regex matches against base names, not paths.

scripts/qapi/pylintrc line 14

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+ignore-patterns=common.py,
+                doc.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,

Already not ignored:

     __init__.py
     commands.py
     common.py
     debug.py
     events.py
     introspect.py
     script.py

Okay.

Quoted 13 lines
scripts/qapi/pylintrc new file +13 -0
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
Quoted 9 lines

scripts/qapi/pylintrc line 33

Quoted 6 lines
scripts/qapi/pylintrc new file +6 -0
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,

I'm fine with disabling these.

I'd like to enable missing-docstring eventually, but that's not for today.

Quoted 12 lines
Quoted 12 lines
scripts/qapi/pylintrc new file +12 -0
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
Quoted 9 lines

scripts/qapi/pylintrc line 51

Quoted 6 lines
scripts/qapi/pylintrc new file +6 -0
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _

Isn't this the default?

Yes. I could omit it until I need to use good-names later on in the series, but I thought it would look odd to add the defaults at that point.

So it's a minor bit of prescience here.

Quoted 7 lines
Quoted 7 lines
scripts/qapi/pylintrc new file +7 -0
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
Quoted 9 lines

scripts/qapi/pylintrc line 64

Quoted 6 lines
scripts/qapi/pylintrc new file +6 -0
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore imports when computing similarities.
+ignore-imports=yes

Why?

We don't care if import statements are similar to those in other files. It's uninteresting entirely.

(It matches on from typing import ... that exceed four lines, which I do regularly by the end of the series.)

Quoted 4 lines
Quoted 4 lines
scripts/qapi/pylintrc new file +4 -0
+
+[TYPECHECK]
+
+[CLASSES]
Quoted 9 lines

scripts/qapi/pylintrc line 74

Quoted 6 lines
scripts/qapi/pylintrc new file +6 -0
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]

Looks like you started with output of --generate-rcfile,

I did,

#23 lore raw

Re: [PATCH 07/37] qapi: add pylintrc

Quoted 22 lines · John Snow
Quoted 21 lines · Markus Armbruster
Quoted 8 lines · John Snow

Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.

Quoted 7 lines

Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.

Therefore, to run it, you must:

Quoted 1 line

cd :/qemu/scripts

-bash: cd: :/qemu/scripts: No such file or directory ;-P

Quoted 1 line
Quoted 1 line

pylint qapi/ --rcfile=qapi/pylintrc

Quoted 8 lines
Quoted 3 lines

Why not

    $ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc

No reason I'm aware of, I have just been testing with CWD at the scripts dir myself because of how python imports work.

If it works this way, enjoy!

It works, and it simplifies the commmit message.

Quoted 58 lines
Quoted 48 lines
Quoted 16 lines
---
  scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 74 insertions(+)

create mode 100644 scripts/qapi/pylintrc

scripts/qapi/pylintrc new file +4 -0
new file mode 100644
index 0000000000..c2bbb8e8e1
@@ -0,0 +1,74 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the blacklist. The
+# regex matches against base names, not paths.

scripts/qapi/pylintrc line 14

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+ignore-patterns=common.py,
+                doc.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,

Already not ignored:

     __init__.py
     commands.py
     common.py
     debug.py
     events.py
     introspect.py
     script.py

Okay.

Quoted 12 lines
scripts/qapi/pylintrc new file +12 -0
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
Quoted 9 lines

scripts/qapi/pylintrc line 33

Quoted 7 lines
scripts/qapi/pylintrc new file +7 -0
+# --disable=W".
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,

I'm fine with disabling these.

Quoted 1 line

I'd like to enable missing-docstring eventually, but that's not for today.

Understood.

Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
scripts/qapi/pylintrc new file +11 -0
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
Quoted 9 lines

scripts/qapi/pylintrc line 51

Quoted 7 lines
scripts/qapi/pylintrc new file +7 -0
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _

Isn't this the default?

Quoted 5 lines

Yes. I could omit it until I need to use good-names later on in the series, but I thought it would look odd to add the defaults at that point.

So it's a minor bit of prescience here.

Matter of taste. No objection.

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
scripts/qapi/pylintrc new file +6 -0
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
Quoted 9 lines

scripts/qapi/pylintrc line 64

Quoted 7 lines
scripts/qapi/pylintrc new file +7 -0
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore imports when computing similarities.
+ignore-imports=yes

Why?

Quoted 5 lines

We don't care if import statements are similar to those in other files. It's uninteresting entirely.

(It matches on from typing import ... that exceed four lines, which I do regularly by the end of the series.)

What about something like

# Ignore imports when computing similarities, because import # statements being similar is uninteresting entirely

Quoted 13 lines
Quoted 12 lines

scripts/qapi/pylintrc line 74

Quoted 10 lines
scripts/qapi/pylintrc new file +10 -0
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]

Looks like you started with output of --generate-rcfile,

Quoted 1 line

I did,

Let's mention that in the commit message. Education opportunity :)

author
#24 lore raw

Re: [PATCH 07/37] qapi: add pylintrc

Quoted 10 lines · Markus Armbruster
Quoted 5 lines

We don't care if import statements are similar to those in other files. It's uninteresting entirely.

(It matches on from typing import ... that exceed four lines, which I do regularly by the end of the series.)

What about something like

# Ignore imports when computing similarities, because import # statements being similar is uninteresting entirely

OK. I left the "default" comments that pylint itself wrote. I'm now writing:

"Ignore import statements themselves when computing similarities."

author
#25 lore raw

PATCH 8/37 qapi/common.py: Remove python compatibility workaround

In reply to John Snow
---
 scripts/qapi/common.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +1 -4
index ba35abea47..4fb265a8bf 100644
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
     raw = code % kwds
     if indent_level:
         indent = genindent(indent_level)
-        # re.subn() lacks flags support before Python 2.7, use re.compile()
-        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-                      indent, raw)
-        raw = raw[0]
+        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
Signature · 1 line
2.26.2
#26 lore raw

Re: [PATCH 08/37] qapi/common.py: Remove python compatibility workaround

Quoted 10 lines · John Snow
---
 scripts/qapi/common.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +0 -0
index ba35abea47..4fb265a8bf 100644

scripts/qapi/common.py def cgen(code, **kwds): · line 123

Quoted 9 lines
scripts/qapi/common.py +1 -4
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
     raw = code % kwds
     if indent_level:
         indent = genindent(indent_level)
-        # re.subn() lacks flags support before Python 2.7, use re.compile()
-        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-                      indent, raw)
-        raw = raw[0]
+        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)

I missed this one in my "qapi: Bye-bye Python 2" series.

Can we use re.sub() instead?

author
#27 lore raw

Re: [PATCH 08/37] qapi/common.py: Remove python compatibility workaround

Quoted 24 lines · Markus Armbruster
Quoted 10 lines · John Snow
---
  scripts/qapi/common.py | 5 +----
  1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +0 -0
index ba35abea47..4fb265a8bf 100644

scripts/qapi/common.py def cgen(code, **kwds): · line 123

Quoted 9 lines
scripts/qapi/common.py +1 -4
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
      raw = code % kwds
      if indent_level:
          indent = genindent(indent_level)
-        # re.subn() lacks flags support before Python 2.7, use re.compile()
-        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-                      indent, raw)
-        raw = raw[0]
+        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
      return re.sub(re.escape(eatspace) + r' *', '', raw)

I missed this one in my "qapi: Bye-bye Python 2" series.

Quoted 2 lines

Can we use re.sub() instead?

Yup.

author
#28 lore raw

PATCH 9/37 qapi/common.py: Add indent manager

In reply to John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Make a little indent level manager instead.

---
 scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 38 insertions(+), 19 deletions(-)
scripts/qapi/common.py +35 -15
index 4fb265a8bf..87d87b95e5 100644
@@ -93,33 +93,53 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indent:
+    """
+    Indent-level management.
 
+    :param initial: Initial value, default 0.
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        """Return the indent as an integer."""
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the indent as a string."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indent."""
+        return bool(self._level)
 
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def push(self, amount: int = 4) -> int:
+        """Push `amount` spaces onto the indent, default four."""
+        self._level += amount
+        return self._level
+
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
+        self._level -= amount
+        return self._level
+
+
+INDENT = Indent(0)
 
 
 # Generate @code with @kwds interpolated.
 # Obey indent_level, and strip eatspace.
 def cgen(code, **kwds):
     raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if INDENT:
+        raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
scripts/qapi/visit.py +3 -4
index 31bf46e7f7..66ce3dc52e 100644
@@ -19,8 +19,7 @@
     gen_endif,
     gen_if,
     mcgen,
-    pop_indent,
-    push_indent,
+    INDENT,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
     if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
                          name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            INDENT.push()
         ret += mcgen('''
     if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
         return false;
@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
         if memb.optional:
-            pop_indent()
+            INDENT.pop()
             ret += mcgen('''
     }
 ''')
Signature · 1 line
2.26.2
#29 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 18 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Make a little indent level manager instead.

---
 scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 38 insertions(+), 19 deletions(-)
scripts/qapi/common.py +0 -0
index 4fb265a8bf..87d87b95e5 100644

scripts/qapi/common.py def c_name(name, protect=True): · line 96

Quoted 9 lines
scripts/qapi/common.py +1 -5
@@ -93,33 +93,53 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indent:

Since this class name will be used just once... Indentation or IndentationManager?

scripts/qapi/common.py def c_name(name, protect=True): · line 100

Quoted 4 lines
scripts/qapi/common.py +3 -0
+    """
+    Indent-level management.
 
+    :param initial: Initial value, default 0.

The only caller passes 0.

Let's drop the parameter, and hardcode the default initial value until we have an actual use for another initial value.

scripts/qapi/common.py def c_name(name, protect=True): · line 106

Quoted 7 lines
scripts/qapi/common.py +5 -1
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        """Return the indent as an integer."""

Isn't "as an integer" obvious?

Let's replace "the indent" by "the indentation" globally.

scripts/qapi/common.py def c_name(name, protect=True): · line 110

Quoted 4 lines
scripts/qapi/common.py +3 -0
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)

Is __repr__ needed?

scripts/qapi/common.py def c_name(name, protect=True): · line 118

Quoted 11 lines
scripts/qapi/common.py +6 -3
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the indent as a string."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indent."""
+        return bool(self._level)

This one lets us shorten

if int(INDENT):

to

if INDENT:

There's just one instance. Marginal. I'm not rejecting it.

Quoted 5 lines
scripts/qapi/common.py +2 -3
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def push(self, amount: int = 4) -> int:
+        """Push `amount` spaces onto the indent, default four."""

scripts/qapi/common.py def c_name(name, protect=True): · line 131

Quoted 10 lines
scripts/qapi/common.py +10 -0
+        self._level += amount
+        return self._level
+
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
+        self._level -= amount
+        return self._level

The push / pop names never made sense. The functions don't push onto / pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.

The @amount parameter has never been used. I don't mind keeping it.

scripts/qapi/common.py def c_name(name, protect=True): · line 134

Quoted 3 lines
scripts/qapi/common.py +3 -0
+
+
+INDENT = Indent(0)

Uh, does this really qualify as a constant? It looks quite variable to me...

Quoted 44 lines
scripts/qapi/common.py +2 -3
 # Generate @code with @kwds interpolated.
 # Obey indent_level, and strip eatspace.
 def cgen(code, **kwds):
     raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if INDENT:
+        raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
scripts/qapi/visit.py +3 -4
index 31bf46e7f7..66ce3dc52e 100644
@@ -19,8 +19,7 @@
     gen_endif,
     gen_if,
     mcgen,
-    pop_indent,
-    push_indent,
+    INDENT,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
     if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
                          name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            INDENT.push()
         ret += mcgen('''
     if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
         return false;
@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
         if memb.optional:
-            pop_indent()
+            INDENT.pop()
             ret += mcgen('''
     }
 ''')
author
#30 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 23 lines · Markus Armbruster
Quoted 21 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Make a little indent level manager instead.

---
  scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
  scripts/qapi/visit.py  |  7 +++---
  2 files changed, 38 insertions(+), 19 deletions(-)
scripts/qapi/common.py +0 -0
index 4fb265a8bf..87d87b95e5 100644
@@ -93,33 +93,53 @@ def c_name(name, protect=True):
  pointer_suffix = ' *' + eatspace
 
 
Quoted 10 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 96

Quoted 6 lines
scripts/qapi/common.py +1 -5
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indent:

Since this class name will be used just once... Indentation or IndentationManager?

Indentation is fine, if you'd like. IndentationManager makes me feel ashamed for writing this patch, like I am punishing myself with Java.

Quoted 10 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 100

Quoted 4 lines
scripts/qapi/common.py +3 -0
+    """
+    Indent-level management.
 
+    :param initial: Initial value, default 0.

The only caller passes 0.

Let's drop the parameter, and hardcode the default initial value until we have an actual use for another initial value.

The parameter is nice because it gives meaning to the output of repr(), see below.

Quoted 9 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 106

Quoted 7 lines
scripts/qapi/common.py +5 -1
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        """Return the indent as an integer."""

Isn't "as an integer" obvious?

Just a compulsion to write complete sentences that got lost in the sea of many patches. I'll nix this one, because dunder methods do not need to be documented.

Quoted 3 lines

Let's replace "the indent" by "the indentation" globally.

They're both cromulent as nouns and one is shorter.

I'll switch in good faith; do you prefer "Indentation" as a noun?

Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 110

Quoted 4 lines
scripts/qapi/common.py +3 -0
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)

Is __repr__ needed?

Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.

repr(Indent(4)) == "Indent(4)"

Quoted 14 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 118

Quoted 11 lines
scripts/qapi/common.py +6 -3
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the indent as a string."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indent."""
+        return bool(self._level)

This one lets us shorten

Quoted 8 lines

if int(INDENT):

to

if INDENT:

There's just one instance. Marginal. I'm not rejecting it.

Yep, it's trivial in either direction. Still, because it's a custom type, I thought I'd be courteous and have it support bool().

Quoted 11 lines
Quoted 11 lines
scripts/qapi/common.py +8 -3
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def push(self, amount: int = 4) -> int:
+        """Push `amount` spaces onto the indent, default four."""
+        self._level += amount
+        return self._level
+
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
Quoted 9 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 131

Quoted 4 lines
scripts/qapi/common.py +4 -0
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
+        self._level -= amount
+        return self._level

The push / pop names never made sense. The functions don't push onto / pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.

OK.

I briefly thought about using __isub__ and __iadd__ to support e.g. indent += 4, but actually it'd be annoying to have to specify 4 everywhere.

Since you didn't suggest anything, I am going to change it to 'increase' and 'decrease'.

Quoted 2 lines

The @amount parameter has never been used. I don't mind keeping it.

I'll keep it, because I like having the default amount show up in the signature instead of as a magic constant in the function body.

Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 134

Quoted 3 lines
scripts/qapi/common.py +3 -0
+
+
+INDENT = Indent(0)

Uh, does this really qualify as a constant? It looks quite variable to me...

Ever make a mistake? I thought I did once, but I was mistaken.

Quoted 44 lines
Quoted 44 lines
scripts/qapi/common.py +2 -3
  # Generate @code with @kwds interpolated.
  # Obey indent_level, and strip eatspace.
  def cgen(code, **kwds):
      raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if INDENT:
+        raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
      return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
scripts/qapi/visit.py +3 -4
index 31bf46e7f7..66ce3dc52e 100644
@@ -19,8 +19,7 @@
      gen_endif,
      gen_if,
      mcgen,
-    pop_indent,
-    push_indent,
+    INDENT,
  )
  from .gen import QAPISchemaModularCVisitor, ifcontext
  from .schema import QAPISchemaObjectType
@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
      if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
  ''',
                           name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            INDENT.push()
          ret += mcgen('''
      if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
          return false;
@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
                       c_type=memb.type.c_name(), name=memb.name,
                       c_name=c_name(memb.name))
          if memb.optional:
-            pop_indent()
+            INDENT.pop()
              ret += mcgen('''
      }
  ''')
#31 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 33 lines · John Snow
Quoted 22 lines · Markus Armbruster
Quoted 20 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Make a little indent level manager instead.

---
  scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
  scripts/qapi/visit.py  |  7 +++---
  2 files changed, 38 insertions(+), 19 deletions(-)
scripts/qapi/common.py +0 -0
index 4fb265a8bf..87d87b95e5 100644
@@ -93,33 +93,53 @@ def c_name(name, protect=True):
  pointer_suffix = ' *' + eatspace
 
Quoted 9 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 96

Quoted 6 lines
scripts/qapi/common.py +1 -5
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indent:

Since this class name will be used just once... Indentation or IndentationManager?

Quoted 2 lines

Indentation is fine, if you'd like. IndentationManager makes me feel ashamed for writing this patch, like I am punishing myself with Java.

Hah! Point taken.

Quoted 24 lines
Quoted 8 lines

scripts/qapi/common.py def c_name(name, protect=True):

Quoted 3 lines
scripts/qapi/common.py +2 -0
+    """
+    Indent-level management.
  +    :param initial: Initial value, default 0.

The only caller passes 0.
Let's drop the parameter, and hardcode the default initial value until
we have an actual use for another initial value.

The parameter is nice because it gives meaning to the output of repr(), see below.

Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 106

Quoted 6 lines
scripts/qapi/common.py +5 -0
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
  -indent_level = 0
+    def __int__(self) -> int:
+        """Return the indent as an integer."""

Isn't "as an integer" obvious?

Just a compulsion to write complete sentences that got lost in the sea of many patches. I'll nix this one, because dunder methods do not need to be documented.

Quoted 6 lines
Quoted 2 lines

Let's replace "the indent" by "the indentation" globally.

They're both cromulent as nouns and one is shorter.

I'll switch in good faith; do you prefer "Indentation" as a noun?

Use of "indent" as a noun was new to me, but what do I know; you're the native speaker. Use your judgement. Applies to the class name, too.

Quoted 10 lines
Quoted 5 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 110

Quoted 3 lines
scripts/qapi/common.py +2 -0
+        return self._level
  +    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)

Is __repr__ needed?

Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.

repr(Indent(4)) == "Indent(4)"

Meh. There's another three dozen classes for you to put lipstick on :)

Quoted 19 lines
Quoted 9 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 118

Quoted 9 lines
scripts/qapi/common.py +5 -2
  -def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the indent as a string."""
+        return ' ' * self._level
  +    def __bool__(self) -> bool:
+        """True when there is a non-zero indent."""
+        return bool(self._level)
Quoted 6 lines

This one lets us shorten
if int(INDENT):
to
if INDENT:
There's just one instance. Marginal. I'm not rejecting it.

Yep, it's trivial in either direction. Still, because it's a custom type, I thought I'd be courteous and have it support bool().

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
scripts/qapi/common.py +2 -3
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def push(self, amount: int = 4) -> int:
+        """Push `amount` spaces onto the indent, default four."""
Quoted 8 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 129

Quoted 8 lines
scripts/qapi/common.py +8 -0
+        self._level += amount
+        return self._level
+
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))

I think assert(amount <= self._level) would do just fine.

Quoted 8 lines
Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 131

Quoted 2 lines
scripts/qapi/common.py +2 -0
+        self._level -= amount
+        return self._level

The push / pop names never made sense. The functions don't push onto /
pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.

Quoted 8 lines

OK.

I briefly thought about using __isub__ and __iadd__ to support e.g. indent += 4, but actually it'd be annoying to have to specify 4 everywhere.

Since you didn't suggest anything, I am going to change it to 'increase' and 'decrease'.

Works for me. So would inc and dec.

Quoted 13 lines
Quoted 2 lines

The @amount parameter has never been used. I don't mind keeping it.

I'll keep it, because I like having the default amount show up in the signature instead of as a magic constant in the function body.

Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 134

Quoted 3 lines
scripts/qapi/common.py +3 -0
+
+
+INDENT = Indent(0)

Uh, does this really qualify as a constant? It looks quite variable to
me...

Quoted 1 line

Ever make a mistake? I thought I did once, but I was mistaken.

"If I had any humility, I'd be perfect!"

author
#32 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 6 lines · Markus Armbruster

John Snow <[email protected]> writes:

Let's replace "the indent" by "the indentation" globally.

Quoted 7 lines
Quoted 3 lines

They're both cromulent as nouns and one is shorter.

I'll switch in good faith; do you prefer "Indentation" as a noun?

Use of "indent" as a noun was new to me, but what do I know; you're the native speaker. Use your judgement. Applies to the class name, too.

I made the change; see gitlab commits or wait for v2 to see if it reads better to you.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 110

Quoted 3 lines
scripts/qapi/common.py +2 -0
+        return self._level
   +    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)

Is __repr__ needed?

Quoted 7 lines
Quoted 4 lines

Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.

repr(Indent(4)) == "Indent(4)"

Meh. There's another three dozen classes for you to put lipstick on :)

We'll get to them in due time. For now, please admire the lipstick.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 129

Quoted 5 lines
scripts/qapi/common.py +5 -0
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))

I think assert(amount <= self._level) would do just fine.

Secretly, asserts can be disabled in Python just like they can in C code.

My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.

(See the gitlab branch for the updated version of this patch, which has changed the code slightly.)

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 131

Quoted 2 lines
scripts/qapi/common.py +2 -0
+        self._level -= amount
+        return self._level

The push / pop names never made sense. The functions don't push onto /
pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.

OK.

Quoted 9 lines
Quoted 6 lines

I briefly thought about using __isub__ and __iadd__ to support e.g. indent += 4, but actually it'd be annoying to have to specify 4 everywhere.

Since you didn't suggest anything, I am going to change it to 'increase' and 'decrease'.

Works for me. So would inc and dec.

increase and decrease it is.

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines

The @amount parameter has never been used. I don't mind keeping it.

I'll keep it, because I like having the default amount show up in the signature instead of as a magic constant in the function body.

Quoted 7 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 134

Quoted 3 lines
scripts/qapi/common.py +3 -0
+
+
+INDENT = Indent(0)

Uh, does this really qualify as a constant? It looks quite variable to
me...

Quoted 4 lines
Quoted 1 line

Ever make a mistake? I thought I did once, but I was mistaken.

"If I had any humility, I'd be perfect!"

:)

#33 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 31 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 6 lines · Markus Armbruster

John Snow <[email protected]> writes:

Let's replace "the indent" by "the indentation" globally.

Quoted 7 lines
Quoted 3 lines

They're both cromulent as nouns and one is shorter.

I'll switch in good faith; do you prefer "Indentation" as a noun?

Use of "indent" as a noun was new to me, but what do I know; you're the
native speaker. Use your judgement. Applies to the class name, too.

I made the change; see gitlab commits or wait for v2 to see if it reads better to you.

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 110

Quoted 3 lines
scripts/qapi/common.py +2 -0
+        return self._level
   +    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)

Is __repr__ needed?

Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

repr(Indent(4)) == "Indent(4)"

Meh. There's another three dozen classes for you to put lipstick on :)

We'll get to them in due time. For now, please admire the lipstick.

If I take off my glasses and step six feet back, I just might be able to overlook it.

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 129

Quoted 5 lines
scripts/qapi/common.py +5 -0
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))

I think assert(amount <= self._level) would do just fine.

Secretly, asserts can be disabled in Python just like they can in C code.

There are compilers that let you switch off array bounds checking. Would you advocate manual bounds checking to protect people from their own folly?

Quoted 4 lines

My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.

I check function preconditions ruthlessly with assert. There's no sane way to recover anyway.

Without a way to recover, the only benefit is crashing more prettily. If the error is six levels deep in a some fancy-pants framework, then prettier crashes might actually help someone finding the error slightly faster. But it ain't.

My final argument is local consistency: use of assertions to check preconditions is pervasive in scripts/qapi/.

Quoted 2 lines

(See the gitlab branch for the updated version of this patch, which has changed the code slightly.)

[...]

author
#34 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 7 lines · Markus Armbruster
Quoted 1 line · John Snow

We'll get to them in due time. For now, please admire the lipstick.

If I take off my glasses and step six feet back, I just might be able to overlook it.

I consider writing a nice __repr__ good habit, I'd prefer not to delete it just because the rest of our code doesn't do so yet. (Give me time.)

I spend a lot of my time in the interactive python console: having nice __repr__ methods is a good habit, not an unsightly blemish.

Quoted 22 lines
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 129

Quoted 5 lines
scripts/qapi/common.py +5 -0
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))

I think assert(amount <= self._level) would do just fine.

Secretly, asserts can be disabled in Python just like they can in C code.

There are compilers that let you switch off array bounds checking. Would you advocate manual bounds checking to protect people from their own folly?

Quoted 4 lines

My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.

I check function preconditions ruthlessly with assert. There's no sane way to recover anyway.

Quoted 8 lines

Without a way to recover, the only benefit is crashing more prettily. If the error is six levels deep in a some fancy-pants framework, then prettier crashes might actually help someone finding the error slightly faster. But it ain't.

My final argument is local consistency: use of assertions to check preconditions is pervasive in scripts/qapi/.

You're right that there's no safe recovery from an error such as this. The only actual difference is whether you see AssertionError or ArithmeticError.

One can be disabled (But you rightly shouldn't), the other can't. One has more semantic meaning and information to it.

I prefer what I've currently written.

--js

#35 lore raw

Re: [PATCH 09/37] qapi/common.py: Add indent manager

Quoted 45 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 1 line · John Snow

We'll get to them in due time. For now, please admire the lipstick.

If I take off my glasses and step six feet back, I just might be able to
overlook it.

I consider writing a nice __repr__ good habit, I'd prefer not to delete it just because the rest of our code doesn't do so yet. (Give me time.)

I spend a lot of my time in the interactive python console: having nice __repr__ methods is a good habit, not an unsightly blemish.

Quoted 17 lines
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))

I think assert(amount <= self._level) would do just fine.

Secretly, asserts can be disabled in Python just like they can in C code.

There are compilers that let you switch off array bounds checking. Would you advocate manual bounds checking to protect people from their own folly?

Quoted 4 lines

My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.

Quoted 11 lines

I check function preconditions ruthlessly with assert. There's no sane
way to recover anyway.
Without a way to recover, the only benefit is crashing more prettily.
If the error is six levels deep in a some fancy-pants framework, then prettier crashes might actually help someone finding the error slightly faster. But it ain't.
My final argument is local consistency: use of assertions to check preconditions is pervasive in scripts/qapi/.

Quoted 6 lines

You're right that there's no safe recovery from an error such as this. The only actual difference is whether you see AssertionError or ArithmeticError.

One can be disabled (But you rightly shouldn't), the other can't. One has more semantic meaning and information to it.

YAGNI.

Quoted 1 line

I prefer what I've currently written.

Where personal preference collides with local consistency, I'm with local consistency.

You can get the two in line: change everything to your preference.

You signalled intent to do that for __repr__(): "Give me time". Alright, having such __repr__() is obviously more important / useful to you than avoiding the extra code is to me.

I received no such signal for checking preconditions. Good, because I'd go "are you serious?" :)

author
#36 lore raw

PATCH 10/37 qapi/common.py: delint with pylint

In reply to John Snow
---
 scripts/qapi/common.py | 16 +++++++---------
 scripts/qapi/schema.py | 14 +++++++-------
 2 files changed, 14 insertions(+), 16 deletions(-)
scripts/qapi/common.py +7 -9
index 87d87b95e5..c665e67495 100644
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+c_name_trans = str.maketrans('.-', '__')
+
+
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True):
     return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indent:
     """
     Indent-level management.
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey INDENT level, and strip EATSPACE.
 def cgen(code, **kwds):
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):
scripts/qapi/schema.py +7 -7
index b77e0e56b2..b4921b46c9 100644
@@ -18,7 +18,7 @@
 import re
 from collections import OrderedDict
 
-from .common import c_name, pointer_suffix
+from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
 from .expr import check_exprs
 from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self):
         return True
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'array'
@@ -430,7 +430,7 @@ def c_name(self):
 
     def c_type(self):
         assert not self.is_implicit()
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def c_unboxed_type(self):
         return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None):
             v.connect_doc(doc)
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type):
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
-        for t in [('str',    'string',  'char' + pointer_suffix),
+        for t in [('str',    'string',  'char' + POINTER_SUFFIX),
                   ('number', 'number',  'double'),
                   ('int',    'int',     'int64_t'),
                   ('int8',   'int',     'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self):
                   ('uint64', 'int',     'uint64_t'),
                   ('size',   'int',     'uint64_t'),
                   ('bool',   'boolean', 'bool'),
-                  ('any',    'value',   'QObject' + pointer_suffix),
-                  ('null',   'null',    'QNull' + pointer_suffix)]:
+                  ('any',    'value',   'QObject' + POINTER_SUFFIX),
+                  ('null',   'null',    'QNull' + POINTER_SUFFIX)]:
             self._def_builtin_type(*t)
         self.the_empty_object_type = QAPISchemaObjectType(
             'q_empty', None, None, None, None, None, [], None)
Signature · 1 line
2.26.2
#37 lore raw

Re: [PATCH 10/37] qapi/common.py: delint with pylint

Quoted 11 lines · John Snow
---
 scripts/qapi/common.py | 16 +++++++---------
 scripts/qapi/schema.py | 14 +++++++-------
 2 files changed, 14 insertions(+), 16 deletions(-)
scripts/qapi/common.py +0 -0
index 87d87b95e5..c665e67495 100644

scripts/qapi/common.py line 21

Quoted 8 lines
scripts/qapi/common.py +5 -0
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+c_name_trans = str.maketrans('.-', '__')
+
+

You rename and move. pylint gripes about the names, but it doesn't actually ask for the move, as far as I can tell. Can you explain why you move?

Quoted 25 lines
scripts/qapi/common.py +0 -7
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True):
     return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indent:
     """
     Indent-level management.

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 136

Quoted 5 lines
scripts/qapi/common.py +1 -1
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey INDENT level, and strip EATSPACE.

Is the change to INDENT intentional?

Quoted 9 lines
scripts/qapi/common.py +1 -1
 def cgen(code, **kwds):
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):

[...]

author
#38 lore raw

Re: [PATCH 10/37] qapi/common.py: delint with pylint

Quoted 16 lines · Markus Armbruster
Quoted 14 lines · John Snow
---
  scripts/qapi/common.py | 16 +++++++---------
  scripts/qapi/schema.py | 14 +++++++-------
  2 files changed, 14 insertions(+), 16 deletions(-)
scripts/qapi/common.py +0 -0
index 87d87b95e5..c665e67495 100644
@@ -14,6 +14,11 @@
  import re
 
 
Quoted 10 lines

scripts/qapi/common.py line 21

Quoted 5 lines
scripts/qapi/common.py +5 -0
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+c_name_trans = str.maketrans('.-', '__')
+
+

You rename and move. pylint gripes about the names, but it doesn't actually ask for the move, as far as I can tell. Can you explain why you move?

Preference. I like constants and globals at the top so you can audit any code that runs at import time in one place. Since they are externally visible objects, having them near other "header" style information makes sense to me.

Quoted 8 lines

scripts/qapi/common.py def c_enum_const(type_name, const_name, prefix=None): · line 45

Quoted 8 lines
scripts/qapi/common.py +0 -1
  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
  # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')

(This one winds up being a constant, so I renamed it in my v2.)

Quoted 19 lines
Quoted 19 lines
scripts/qapi/common.py +0 -6
-
-
  # Map @name to a valid C identifier.
  # If @protect, avoid returning certain ticklish identifiers (like
  # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True):
      return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
  class Indent:
      """
      Indent-level management.
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
 
 
Quoted 6 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 136

Quoted 3 lines
scripts/qapi/common.py +1 -1
  # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey INDENT level, and strip EATSPACE.

Is the change to INDENT intentional?

Kind of, but it's either late (should have been with the indent manager patch) or early (Should be with the patch that moves comments into docstrings.)

When this comment becomes a docstring, I use INDENT to indicate it as a proper object. This in and of itself is prescient, as we are not using sphinx/apidoc to generate any documentation about the QAPI package yet.

(The pending v2 uses indent after you pointed out that it was not a constant.)

Quoted 11 lines
Quoted 9 lines
scripts/qapi/common.py +1 -1
  def cgen(code, **kwds):
      raw = code % kwds
      if INDENT:
          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
  def mcgen(code, **kwds):

[...]

#39 lore raw

Re: [PATCH 10/37] qapi/common.py: delint with pylint

Quoted 26 lines · John Snow
Quoted 15 lines · Markus Armbruster
Quoted 13 lines · John Snow
---
  scripts/qapi/common.py | 16 +++++++---------
  scripts/qapi/schema.py | 14 +++++++-------
  2 files changed, 14 insertions(+), 16 deletions(-)
scripts/qapi/common.py +0 -0
index 87d87b95e5..c665e67495 100644
@@ -14,6 +14,11 @@
  import re
 
Quoted 9 lines

scripts/qapi/common.py line 21

Quoted 5 lines
scripts/qapi/common.py +5 -0
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+c_name_trans = str.maketrans('.-', '__')
+
+

You rename and move. pylint gripes about the names, but it doesn't actually ask for the move, as far as I can tell. Can you explain why you move?

Quoted 2 lines

Preference. I like constants and globals at the top so you can audit any code that runs at import time in one place.

I can buy this argument.

Quoted 3 lines

Since they are
externally visible objects, having them near other "header" style information makes sense to me.

This one I find unconvincing. Functions and classes are just as visible.

Mention the move in the commit message, along with the (convincing part of the) rationale.

Aside: EATSPACE and c_name_trans are actually local to common.py. EATSPACE sort of leaks out only via the contract of mcgen(). The contract could be rephrased in terms of POINTER_SUFFIX. Not sure it's worthwhile.

Quoted 55 lines
Quoted 7 lines

scripts/qapi/common.py def c_enum_const(type_name, const_name, prefix=None): · line 44

Quoted 7 lines
scripts/qapi/common.py +0 -1
  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
  # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
-c_name_trans = str.maketrans('.-', '__')

(This one winds up being a constant, so I renamed it in my v2.)

Quoted 17 lines
Quoted 17 lines
scripts/qapi/common.py +0 -6
-
-
  # Map @name to a valid C identifier.
  # If @protect, avoid returning certain ticklish identifiers (like
  # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True):
      return name
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
  class Indent:
      """
      Indent-level management.
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
 
Quoted 5 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 136

Quoted 3 lines
scripts/qapi/common.py +1 -1
  # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey INDENT level, and strip EATSPACE.

Is the change to INDENT intentional?

Kind of, but it's either late (should have been with the indent manager patch) or early (Should be with the patch that moves comments into docstrings.)

When this comment becomes a docstring, I use INDENT to indicate it as a proper object. This in and of itself is prescient, as we are not using sphinx/apidoc to generate any documentation about the QAPI package yet.

(The pending v2 uses indent after you pointed out that it was not a constant.)

Quoted 10 lines
Quoted 8 lines
scripts/qapi/common.py +1 -1
  def cgen(code, **kwds):
      raw = code % kwds
      if INDENT:
          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
  def mcgen(code, **kwds):

[...]

author
#40 lore raw

PATCH 11/37 qapi/common.py: Replace one-letter 'c' variable

In reply to John Snow
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
index c665e67495..4c079755d3 100644
@@ -30,14 +30,14 @@ def camel_to_upper(value):
     new_name = ''
     length = len(c_fun_str)
     for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+        char = c_fun_str[i]
+        # When char is upper and no '_' appears before, do more checks
+        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
             if i < length - 1 and c_fun_str[i + 1].islower():
                 new_name += '_'
             elif c_fun_str[i - 1].isdigit():
                 new_name += '_'
-        new_name += c
+        new_name += char
     return new_name.lstrip('_').upper()
 
 
Signature · 1 line
2.26.2
#41 lore raw

Re: [PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable

Quoted 10 lines · John Snow
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +0 -0
index c665e67495..4c079755d3 100644

scripts/qapi/common.py def camel_to_upper(value): · line 34

Quoted 8 lines
scripts/qapi/common.py +2 -3
@@ -30,14 +30,14 @@ def camel_to_upper(value):
     new_name = ''
     length = len(c_fun_str)
     for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+        char = c_fun_str[i]
+        # When char is upper and no '_' appears before, do more checks

Good opportunity to tweak the language: "When char is upper case".

Quoted 8 lines
scripts/qapi/common.py +2 -1
+        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
             if i < length - 1 and c_fun_str[i + 1].islower():
                 new_name += '_'
             elif c_fun_str[i - 1].isdigit():
                 new_name += '_'
-        new_name += c
+        new_name += char
     return new_name.lstrip('_').upper()
author
#42 lore raw

Re: [PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable

Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
---
  scripts/qapi/common.py | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
Quoted 16 lines

scripts/qapi/common.py def camel_to_upper(value): · line 34

Quoted 13 lines
scripts/qapi/common.py +2 -3
index c665e67495..4c079755d3 100644
@@ -30,14 +30,14 @@ def camel_to_upper(value):
      new_name = ''
      length = len(c_fun_str)
      for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+        char = c_fun_str[i]
+        # When char is upper and no '_' appears before, do more checks

Good opportunity to tweak the language: "When char is upper case".

ACK, change made.

author
#43 lore raw

PATCH 12/37 qapi/common.py: check with pylint

In reply to John Snow

Remove qapi/common.py from the pylintrc ignore list.

---
 scripts/qapi/pylintrc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi/pylintrc +1 -2
index c2bbb8e8e1..c800055589 100644
@@ -2,8 +2,7 @@
 
 # Add files or directories matching the regex patterns to the blacklist. The
 # regex matches against base names, not paths.
-ignore-patterns=common.py,
-                doc.py,
+ignore-patterns=doc.py,
                 error.py,
                 expr.py,
                 gen.py,
Signature · 1 line
2.26.2
author
#44 lore raw

PATCH 13/37 qapi/common.py: add notational type hints

In reply to John Snow
---
 scripts/qapi/common.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +16 -11
index 4c079755d3..af01348b35 100644
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Union, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
     return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
 
 # Generate @code with @kwds interpolated.
 # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
     return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:
     if code[0] == '\n':
         code = code[1:]
     return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
     return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -160,7 +163,7 @@ def guardstart(name):
                  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
     return mcgen('''
 
 #endif /* %(name)s */
@@ -168,7 +171,7 @@ def guardend(name):
                  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in ifcond:
         ret += mcgen('''
@@ -177,7 +180,7 @@ def gen_if(ifcond):
     return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in reversed(ifcond):
         ret += mcgen('''
@@ -186,7 +189,9 @@ def gen_endif(ifcond):
     return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
     ret = ''
     sep = ''
     if boxed:
Signature · 1 line
2.26.2
#45 lore raw

Re: [PATCH 13/37] qapi/common.py: add notational type hints

Question on the subject line: what makes a type hint notational?

Quoted 47 lines · John Snow
---
 scripts/qapi/common.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +6 -3
index 4c079755d3..af01348b35 100644
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Union, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
     return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 140

Quoted 5 lines
scripts/qapi/common.py +1 -1
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
 
 # Generate @code with @kwds interpolated.
 # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:

Hmm.

The @kwds values can be anything, provided they match the conversion specifiers in @code:

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141

Quoted 1 line
scripts/qapi/common.py +0 -0
     raw = code % kwds

Your type hint adds a restriction that wasn't there before.

Is there a better way?

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147

Quoted 7 lines
scripts/qapi/common.py +1 -1
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
     return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:

Likewise.

Quoted 53 lines
scripts/qapi/common.py +8 -6
     if code[0] == '\n':
         code = code[1:]
     return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
     return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -160,7 +163,7 @@ def guardstart(name):
                  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
     return mcgen('''
 
 #endif /* %(name)s */
@@ -168,7 +171,7 @@ def guardend(name):
                  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in ifcond:
         ret += mcgen('''
@@ -177,7 +180,7 @@ def gen_if(ifcond):
     return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in reversed(ifcond):
         ret += mcgen('''
@@ -186,7 +189,9 @@ def gen_endif(ifcond):
     return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
     ret = ''
     sep = ''
     if boxed:
author
#46 lore raw

Re: [PATCH 13/37] qapi/common.py: add notational type hints

Quoted 2 lines · Markus Armbruster

Question on the subject line: what makes a type hint notational?

My cover letter explains that every time I use this phrase, I mean to state that "This patch adds exclusively type notations and makes no functional changes to the runtime operation whatsoever."

i.e. notations-only.

Quoted 57 lines
Quoted 47 lines · John Snow
---
  scripts/qapi/common.py | 27 ++++++++++++++++-----------
  1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +6 -3
index 4c079755d3..af01348b35 100644
@@ -12,6 +12,7 @@
  # See the COPYING file in the top-level directory.
 
  import re
+from typing import Optional, Union, Sequence
 
 
  EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
  # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
      c_fun_str = c_name(value, False)
      if value.isupper():
          return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
      return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
      if prefix is not None:
          type_name = prefix
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
  # into substrings of a generated C function name.
  # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
  # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
      # ANSI X3J11/88-090, 3.1.1
      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                       'default', 'do', 'double', 'else', 'enum', 'extern',

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 140

Quoted 5 lines
scripts/qapi/common.py +1 -1
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
 
  # Generate @code with @kwds interpolated.
  # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:

Hmm.

Quoted 9 lines

The @kwds values can be anything, provided they match the conversion specifiers in @code:

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141

Quoted 1 line
scripts/qapi/common.py +0 -0
      raw = code % kwds

Your type hint adds a restriction that wasn't there before.

Is there a better way?

Maybe there are format-like type annotation tricks you can do to enforce this, but I did not research them. I tried to resist "improving" our usage of the old % formatter prematurely. I may do a wholesale f-string conversion at some point, but not now, it's not important.

In practice, we pass strings and integers. This typing *is* artificially restrictive, though. We can declare the type to be "Any" and allow the function to fail or succeed at runtime if you'd prefer.

Quoted 10 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147

Quoted 7 lines
scripts/qapi/common.py +1 -1
      if INDENT:
          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
      return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:

Likewise.

Unresearched idea: It's possible that we can subclass the string.Formatter class and extend it to perform our special variable replacements (chomping EATSPACE, etc.)

And *maybe* because it inherits from the standard formatter, we would benefit from any analysis Mypy performs on such things.

Basically, replace mcgen/cgen with class CFormatter(string.Formatter).

(maybe. assume that none of what I just said will work or is feasible.)

--js

author
#47 lore raw

Re: [PATCH 13/37] qapi/common.py: add notational type hints

Quoted 3 lines · John Snow

Your type hint adds a restriction that wasn't there before.

Is there a better way?

I've settled on using the object type for now, which is slightly more restrictive than Any.

Any and object both allow any type of argument, but Any effectively creates a gradually typed boundary in which you are implicitly casting to whatever typed boundary it enters next.

object is an explicit cast to the most abstracted type; any further usage that is not supported the base object will be rejected.

This is nice, because if someone adds a call at or below the mcgen level, mypy will certainly complain that they are assuming too much about the type -- and they would be!

The rest of my series will now take this approach: prefer object to Any.

--js

#48 lore raw

Re: [PATCH 13/37] qapi/common.py: add notational type hints

In reply to John Snow
Quoted 9 lines · John Snow
Quoted 2 lines · Markus Armbruster

Question on the subject line: what makes a type hint notational?

My cover letter explains that every time I use this phrase, I mean to state that "This patch adds exclusively type notations and makes no functional changes to the runtime operation whatsoever."

i.e. notations-only.

By the time I get to PATCH 13, details explained in the cover letter have been flushed from my memory. Moreover, the cover letter won't make it into Git. Best to repeat them right in the commit message. Perhaps:

qapi/common; Add type hints

Type hints do not change behavior.

Quoted 82 lines
Quoted 54 lines
Quoted 44 lines · John Snow
---
  scripts/qapi/common.py | 27 ++++++++++++++++-----------
  1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +6 -3
index 4c079755d3..af01348b35 100644
@@ -12,6 +12,7 @@
  # See the COPYING file in the top-level directory.
    import re
+from typing import Optional, Union, Sequence
 
  EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
  # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
      c_fun_str = c_name(value, False)
      if value.isupper():
          return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
      return new_name.lstrip('_').upper()
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
      if prefix is not None:
          type_name = prefix
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
  # into substrings of a generated C function name.
  # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
  # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
      # ANSI X3J11/88-090, 3.1.1
      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                       'default', 'do', 'double', 'else', 'enum', 'extern',

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 139

Quoted 4 lines
scripts/qapi/common.py +1 -1
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
    # Generate @code with @kwds interpolated.
  # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:

Hmm.
The @kwds values can be anything, provided they match the conversion specifiers in @code:

Quoted 4 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141

Quoted 1 line
scripts/qapi/common.py +0 -0
      raw = code % kwds

Your type hint adds a restriction that wasn't there before. Is there a better way?

Maybe there are format-like type annotation tricks you can do to enforce this, but I did not research them. I tried to resist "improving" our usage of the old % formatter prematurely. I may do a wholesale f-string conversion at some point, but not now, it's not important.

In practice, we pass strings and integers. This typing *is* artificially restrictive, though. We can declare the type to be "Any" and allow the function to fail or succeed at runtime if you'd prefer.

Quoted 8 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147

Quoted 6 lines
scripts/qapi/common.py +1 -1
      if INDENT:
          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
      return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:

Likewise.

Unresearched idea: It's possible that we can subclass the string.Formatter class and extend it to perform our special variable replacements (chomping EATSPACE, etc.)

Quoted 6 lines

And *maybe* because it inherits from the standard formatter, we would benefit from any analysis Mypy performs on such things.

Basically, replace mcgen/cgen with class CFormatter(string.Formatter).

(maybe. assume that none of what I just said will work or is feasible.)

Sounds worth exploring. No need to do it now, of course.

author
#49 lore raw

Re: [PATCH 13/37] qapi/common.py: add notational type hints

Quoted 12 lines · Markus Armbruster
Quoted 9 lines · John Snow
Quoted 2 lines · Markus Armbruster

Question on the subject line: what makes a type hint notational?

My cover letter explains that every time I use this phrase, I mean to state that "This patch adds exclusively type notations and makes no functional changes to the runtime operation whatsoever."

i.e. notations-only.

Quoted 8 lines

By the time I get to PATCH 13, details explained in the cover letter have been flushed from my memory. Moreover, the cover letter won't make it into Git. Best to repeat them right in the commit message. Perhaps:

qapi/common; Add type hints

Type hints do not change behavior.

ACK, done.

Quoted 65 lines
Quoted 65 lines
Quoted 54 lines
Quoted 44 lines · John Snow
---
   scripts/qapi/common.py | 27 ++++++++++++++++-----------
   1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +6 -3
index 4c079755d3..af01348b35 100644
@@ -12,6 +12,7 @@
   # See the COPYING file in the top-level directory.
     import re
+from typing import Optional, Union, Sequence
 
   EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
   # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
   # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
   # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
       c_fun_str = c_name(value, False)
       if value.isupper():
           return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
       return new_name.lstrip('_').upper()
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
       if prefix is not None:
           type_name = prefix
       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
   # into substrings of a generated C function name.
   # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
   # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
       # ANSI X3J11/88-090, 3.1.1
       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                        'default', 'do', 'double', 'else', 'enum', 'extern',

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 139

Quoted 4 lines
scripts/qapi/common.py +1 -1
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
     # Generate @code with @kwds interpolated.
   # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:

Hmm.
The @kwds values can be anything, provided they match the conversion specifiers in @code:

Quoted 4 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141

Quoted 1 line
scripts/qapi/common.py +0 -0
       raw = code % kwds

Your type hint adds a restriction that wasn't there before. Is there a better way?

Maybe there are format-like type annotation tricks you can do to enforce this, but I did not research them. I tried to resist "improving" our usage of the old % formatter prematurely. I may do a wholesale f-string conversion at some point, but not now, it's not important.

Quoted 4 lines
Quoted 4 lines

In practice, we pass strings and integers. This typing *is* artificially restrictive, though. We can declare the type to be "Any" and allow the function to fail or succeed at runtime if you'd prefer.

I went with the 'object' type in new revisions.

Quoted 22 lines
Quoted 13 lines
Quoted 8 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147

Quoted 6 lines
scripts/qapi/common.py +1 -1
       if INDENT:
           raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
       return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:

Likewise.

Unresearched idea: It's possible that we can subclass the string.Formatter class and extend it to perform our special variable replacements (chomping EATSPACE, etc.)

Quoted 6 lines

And *maybe* because it inherits from the standard formatter, we would benefit from any analysis Mypy performs on such things.

Basically, replace mcgen/cgen with class CFormatter(string.Formatter).

(maybe. assume that none of what I just said will work or is feasible.)

Sounds worth exploring. No need to do it now, of course.

author
#50 lore raw

PATCH 14/37 qapi/common.py: Move comments into docstrings

In reply to John Snow

As docstrings, they'll show up in documentation and IDE help.

---
 scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------
 1 file changed, 36 insertions(+), 14 deletions(-)
scripts/qapi/common.py +36 -14
index af01348b35..38d380f0a9 100644
@@ -20,10 +20,18 @@
 c_name_trans = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+    """
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
                  const_name: str,
                  prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map `name` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending 'q_'.
+    """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
 INDENT = Indent(0)
 
 
-# Generate @code with @kwds interpolated.
-# Obey INDENT level, and strip EATSPACE.
 def cgen(code: str, **kwds: Union[str, int]) -> str:
+    """
+    Generate `code` with `kwds` interpolated.
+    Obey `INDENT`, and strip `EATSPACE`.
+    """
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
Signature · 1 line
2.26.2
#51 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 34 lines · John Snow

As docstrings, they'll show up in documentation and IDE help.

---
 scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------
 1 file changed, 36 insertions(+), 14 deletions(-)
scripts/qapi/common.py +11 -3
index af01348b35..38d380f0a9 100644
@@ -20,10 +20,18 @@
 c_name_trans = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+    """
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str

scripts/qapi/common.py def camel_to_upper(value: str) -> str: · line 62

Quoted 10 lines
scripts/qapi/common.py +7 -0
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
                  const_name: str,
                  prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """

Not actually a move. Suggest to retitle

qapi/common: Turn comments in docstrings, and add more

Quoted 36 lines
scripts/qapi/common.py +14 -9
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map `name` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending 'q_'.
+    """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
 INDENT = Indent(0)
 
 

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 165

Quoted 10 lines
scripts/qapi/common.py +4 -2
-# Generate @code with @kwds interpolated.
-# Obey INDENT level, and strip EATSPACE.
 def cgen(code: str, **kwds: Union[str, int]) -> str:
+    """
+    Generate `code` with `kwds` interpolated.
+    Obey `INDENT`, and strip `EATSPACE`.
+    """
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)

Can you point to documentation on the docstring conventions and markup to use?

author
#52 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 49 lines · Markus Armbruster
Quoted 34 lines · John Snow

As docstrings, they'll show up in documentation and IDE help.

---
  scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------
  1 file changed, 36 insertions(+), 14 deletions(-)
scripts/qapi/common.py +11 -3
index af01348b35..38d380f0a9 100644
@@ -20,10 +20,18 @@
  c_name_trans = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
  def camel_to_upper(value: str) -> str:
+    """
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """
      c_fun_str = c_name(value, False)
      if value.isupper():
          return c_fun_str

scripts/qapi/common.py def camel_to_upper(value: str) -> str: · line 62

Quoted 10 lines
scripts/qapi/common.py +7 -0
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
  def c_enum_const(type_name: str,
                   const_name: str,
                   prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """

Not actually a move. Suggest to retitle

Quoted 2 lines

qapi/common: Turn comments in docstrings, and add more

OK.

Quoted 36 lines
Quoted 36 lines
scripts/qapi/common.py +14 -9
      if prefix is not None:
          type_name = prefix
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
  def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map `name` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending 'q_'.
+    """
      # ANSI X3J11/88-090, 3.1.1
      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                       'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
  INDENT = Indent(0)
 
 
Quoted 14 lines

scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 165

Quoted 10 lines
scripts/qapi/common.py +4 -2
-# Generate @code with @kwds interpolated.
-# Obey INDENT level, and strip EATSPACE.
  def cgen(code: str, **kwds: Union[str, int]) -> str:
+    """
+    Generate `code` with `kwds` interpolated.
+    Obey `INDENT`, and strip `EATSPACE`.
+    """
      raw = code % kwds
      if INDENT:
          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)

Can you point to documentation on the docstring conventions and markup to use?

Short answer: No.

Long answer:

It's actually completely arbitrary, with major competing de-facto standards. Their primary function is to be stored to the __doc__ attribute of a module/class/method and can be displayed when using the interactive function help(...).

https://www.python.org/dev/peps/pep-0257/ covers docstrings only in an extremely broad sense. In summary, it asks:

- Use full sentences that end in periods
- Use the triple-double quote style
- multi-line docstrings should have their closing quote on a line by itself
- multi-line docstrings should use a one-sentence summary, a blank line, and then a more elaborate description.

It recommends you document arguments, types, return values and types, exceptions and so on but does not dictate a format. Two popular conventions are the google-style [1] and the NumPy-style [2] docstring formats.

I write docstrings assuming we will be using *Sphinx* as our documentation tool. Sphinx does not read docstrings at all by default, but *autodoc* does. Autodoc assumes your docstrings are written in the Sphinx dialect of ReST.

What you really want to look for is the "Python domain" documentation in Sphinx: https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html

Autodoc will annotate function/method docstrings with "py:function" or "py:method" as appropriate, but the actual contents of the block are still up to you.

For those, you want to look up the Python domain info field lists that are supported by Sphinx, which are here: https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Taken together with PEP 257, you generally want something like this:

"""
Function f converts uncertainty into happiness.

Function f only works on days that do not end in 'y'. Caution should be used when integrating this function into threaded code.

:param uncertainty: All of your doubts. :raise RuntimeError: When it is a day ending in 'y'. :return: Your newfound happiness.
"""

I use the single-backtick as the Sphinx-ese "default role" resolver, which generally should resolve to a reference to some Python entity. The double-backtick is used to do formatted text for things like string literals and so on.

Coffee break.

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

--js

[1] https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

[2] https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy

#53 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]

Quoted 8 lines

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote.

With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html.

I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.

---
 docs/conf.py           |  5 +++-
 docs/devel/index.rst   |  1 +
 docs/devel/qapi.rst    | 67 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/common.py |  2 +-
 scripts/qapi/gen.py    |  2 +-
 5 files changed, 74 insertions(+), 3 deletions(-)
 create mode 100644 docs/devel/qapi.rst
docs/conf.py +4 -1
index 8aeac40124..85be0e1860 100644
@@ -54,6 +54,9 @@ except NameError:
 #
 sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
 
+# Make scripts/qapi module available for autodoc
+sys.path.insert(0, os.path.join(qemu_docdir, "../scripts"))
+
 
 # -- General configuration ------------------------------------------------
 
@@ -67,7 +70,7 @@ needs_sphinx = '1.6'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile']
+extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'sphinx.ext.autodoc']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
docs/devel/index.rst +1 -0
index 04773ce076..a4d2cb9893 100644
@@ -31,3 +31,4 @@ Contents:
    reset
    s390-dasd-ipl
    clocks
+   qapi
docs/devel/qapi.rst new file +67 -0

67 lines not on this page · raw message

scripts/qapi/common.py +1 -1
index 0b1af694e6..7c8c4cb846 100644
@@ -78,7 +78,7 @@ def c_name(name: str, protect: bool = True) -> str:
 
     :param name: The name to map.
     :param protect: If true, avoid returning certain ticklish identifiers
-                    (like C keywords) by prepending 'q_'.
+                    (like C keywords) by prepending ``q_``.
     """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
scripts/qapi/gen.py +1 -1
index 0a72aecdd0..cf33732256 100644
@@ -161,7 +161,7 @@ def ifcontext(ifcond: List[str],
               *args: QAPIGenCCode) -> Generator[None, None, None]:
     """A 'with' statement context manager to wrap with start_if()/end_if()
 
-    *args: any number of QAPIGenCCode
+    :param args: any number of QAPIGenCCode
 
     Example::
 
Signature · 4 lines
2.26.2

-- 
Eduardo
author
#54 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 11 lines · Eduardo Habkost

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]

Quoted 8 lines

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

Quoted 9 lines

After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote.

With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html.

I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.

Awesome!

I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.

When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.

Quoted 142 lines
---
  docs/conf.py           |  5 +++-
  docs/devel/index.rst   |  1 +
  docs/devel/qapi.rst    | 67 ++++++++++++++++++++++++++++++++++++++++++
  scripts/qapi/common.py |  2 +-
  scripts/qapi/gen.py    |  2 +-
  5 files changed, 74 insertions(+), 3 deletions(-)

create mode 100644 docs/devel/qapi.rst

docs/conf.py +4 -1
index 8aeac40124..85be0e1860 100644
@@ -54,6 +54,9 @@ except NameError:
  #
  sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
 
+# Make scripts/qapi module available for autodoc
+sys.path.insert(0, os.path.join(qemu_docdir, "../scripts"))
+
 
  # -- General configuration ------------------------------------------------
 
@@ -67,7 +70,7 @@ needs_sphinx = '1.6'
  # Add any Sphinx extension module names here, as strings. They can be
  # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
  # ones.
-extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile']
+extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'sphinx.ext.autodoc']
 
  # Add any paths that contain templates here, relative to this directory.
  templates_path = ['_templates']
docs/devel/index.rst +1 -0
index 04773ce076..a4d2cb9893 100644
@@ -31,3 +31,4 @@ Contents:
     reset
     s390-dasd-ipl
     clocks
+   qapi
docs/devel/qapi.rst new file +67 -0
new file mode 100644
index 0000000000..9130ef84c6
@@ -0,0 +1,67 @@
+QAPI Python module reference
+============================
+
+.. automodule:: qapi
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.commands
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.common
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.debug
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.error
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.events
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.expr
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.gen
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.introspect
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.params
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.parser
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.schema
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.script
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.source
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.types
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.visit
+   :members:
+   :undoc-members:
+
scripts/qapi/common.py +1 -1
index 0b1af694e6..7c8c4cb846 100644
@@ -78,7 +78,7 @@ def c_name(name: str, protect: bool = True) -> str:
 
      :param name: The name to map.
      :param protect: If true, avoid returning certain ticklish identifiers
-                    (like C keywords) by prepending 'q_'.
+                    (like C keywords) by prepending ``q_``.
      """
      # ANSI X3J11/88-090, 3.1.1
      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
scripts/qapi/gen.py +1 -1
index 0a72aecdd0..cf33732256 100644
@@ -161,7 +161,7 @@ def ifcontext(ifcond: List[str],
                *args: QAPIGenCCode) -> Generator[None, None, None]:
      """A 'with' statement context manager to wrap with start_if()/end_if()
 
-    *args: any number of QAPIGenCCode
+    :param args: any number of QAPIGenCCode
 
      Example::
 
#55 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]

Quoted 8 lines

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

You are deliberately trying to "approximate" Sphinx style, and ...

Quoted 9 lines
Quoted 7 lines

After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.

Awesome!

... actually almost nail it.

Please mention your choice of style in the commit message.

Quoted 7 lines

I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.

When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.

Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...

author
#56 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 24 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]

Quoted 8 lines

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

You are deliberately trying to "approximate" Sphinx style, and ...

Quoted 8 lines
Quoted 7 lines

After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.

Quoted 6 lines
Quoted 1 line

Awesome!

... actually almost nail it.

Please mention your choice of style in the commit message.

OK, I'll try to summarize it. I guess I'll do it in this commit message, but it's not likely to be too visible in the future that way, depending on how you run git blame and what you're looking at.

I want to resurface my other python style patches soon; perhaps a python coding style document should go in alongside those patches.

Quoted 11 lines
Quoted 7 lines

I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.

When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.

Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...

Yes, I just mean in terms of rebasing, docstrings and signatures fight with each other for context and make re-spinning 125 patches a major chore. I wasn't prepared to have the debate on if we wanted to add Python code into the Sphinx generator and have that entire discussion yet.

So, I figured it would be a separate series afterwards. I mentioned somewhere else that I anticipated doing it when I removed the "missing-docstring" warning.

I will investigate doing it (using Eduardo's patches) as a starting point while reviews continue to filter in. If I figure it out in time, I might squash the formatting changes in, but leave the sphinx enablement patches themselves out.

--js

#57 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 24 lines · John Snow
Quoted 23 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]

Quoted 8 lines

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

You are deliberately trying to "approximate" Sphinx style, and ...

Quoted 8 lines
Quoted 7 lines

After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Awesome!

... actually almost nail it.
Please mention your choice of style in the commit message.

OK, I'll try to summarize it. I guess I'll do it in this commit message, but it's not likely to be too visible in the future that way, depending on how you run git blame and what you're looking at.

Thanks!

Quoted 20 lines

I want to resurface my other python style patches soon; perhaps a python coding style document should go in alongside those patches.

Quoted 10 lines
Quoted 7 lines

I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.

When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.

Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...

Yes, I just mean in terms of rebasing, docstrings and signatures fight with each other for context and make re-spinning 125 patches a major chore. I wasn't prepared to have the debate on if we wanted to add Python code into the Sphinx generator and have that entire discussion yet.

Quoted 8 lines

So, I figured it would be a separate series afterwards. I mentioned somewhere else that I anticipated doing it when I removed the "missing-docstring" warning.

I will investigate doing it (using Eduardo's patches) as a starting point while reviews continue to filter in. If I figure it out in time, I might squash the formatting changes in, but leave the sphinx enablement patches themselves out.

Use your judgement.

author
#58 lore raw

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

Quoted 31 lines · Markus Armbruster
Quoted 29 lines · John Snow
Quoted 27 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]

Quoted 8 lines

Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.

So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.

You are deliberately trying to "approximate" Sphinx style, and ...

Quoted 9 lines
Quoted 7 lines

After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.

Awesome!

... actually almost nail it.
Please mention your choice of style in the commit message.

Quoted 35 lines
Quoted 3 lines

OK, I'll try to summarize it. I guess I'll do it in this commit message, but it's not likely to be too visible in the future that way, depending on how you run git blame and what you're looking at.

Thanks!

Quoted 28 lines

I want to resurface my other python style patches soon; perhaps a python coding style document should go in alongside those patches.

Quoted 10 lines
Quoted 7 lines

I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.

When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.

Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...

Yes, I just mean in terms of rebasing, docstrings and signatures fight with each other for context and make re-spinning 125 patches a major chore. I wasn't prepared to have the debate on if we wanted to add Python code into the Sphinx generator and have that entire discussion yet.

So, I figured it would be a separate series afterwards. I mentioned somewhere else that I anticipated doing it when I removed the "missing-docstring" warning.

I will investigate doing it (using Eduardo's patches) as a starting point while reviews continue to filter in. If I figure it out in time, I might squash the formatting changes in, but leave the sphinx enablement patches themselves out.

I wound up figuring it out before I sent V3 out :)

Quoted 2 lines

Use your judgement.

Yep. You can see how this played out in V3: I ensure that nothing is "wrong" but I haven't converted the entire submodule yet.

A thing to clarify for the list as well: Nothing validates these docstrings in the content-aware sense; nothing makes sure you documented the right parameter names, return values, exceptions, etc.

The only validation we can perform automatically right now that I am aware of is:

1. Did we have a docstring, yes/no?
2. Is anything referred to in backticks a valid reference?
3. Did that docstring parse as Sphinx-dialect ReST?

I would like to improve this situation someday, but it's not a task for the near future.

It would be very, very cool if autodoc itself was able to parse the integrity of the docstring and perform some simple validity checks, namely:

1. Every parameter has a :param FOO: in the docstring somewhere
2. Every :param FOO: matches a real parameter.
3. If the function has a return value, it is described with :returns:
4. If the function raises an exception, it must be mentioned with :raises:.

With perhaps some kind of exception for "simple" functions (How do I measure that?) where a one-line description is likely efficient enough.

--js

author
#59 lore raw

PATCH 15/37 qapi/common.py: split build_params into new file

In reply to John Snow

Including it in common.py creates a circular import dependency, because schema relies on common.py. To type build_params properly, it needs to be moved outside of the chain.

---
 scripts/qapi/commands.py |  2 +-
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  2 +-
 scripts/qapi/params.py   | 40 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 42 insertions(+), 25 deletions(-)
 create mode 100644 scripts/qapi/params.py
scripts/qapi/commands.py +1 -1
index 2e4b4de0fa..0c0fe854fe 100644
@@ -14,10 +14,10 @@
 """
 
 from .common import (
-    build_params,
     c_name,
     mcgen,
 )
+from .params import build_params
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
scripts/qapi/common.py +0 -23
index 38d380f0a9..0b1af694e6 100644
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
     return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
scripts/qapi/events.py +1 -1
index 6b3afa14d7..75eda72534 100644
@@ -13,11 +13,11 @@
 """
 
 from .common import (
-    build_params,
     c_enum_const,
     c_name,
     mcgen,
 )
+from .params import build_params
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
scripts/qapi/params.py new file +40 -0
new file mode 100644
index 0000000000..4d4b02f60d
@@ -0,0 +1,40 @@
+#
+# QAPI helper library
+#
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2018 Red Hat Inc.
+#
+# Authors:
+#  Anthony Liguori <[email protected]>
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from typing import Optional
+
+from .common import c_name
+from .schema import QAPISchemaObjectType
+
+
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
Signature · 1 line
2.26.2
#60 lore raw

Re: [PATCH 15/37] qapi/common.py: split build_params into new file

Quoted 4 lines · John Snow

Including it in common.py creates a circular import dependency, because schema relies on common.py. To type build_params properly, it needs to be moved outside of the chain.

Quoted 8 lines
---
 scripts/qapi/commands.py |  2 +-
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  2 +-
 scripts/qapi/params.py   | 40 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 42 insertions(+), 25 deletions(-)
 create mode 100644 scripts/qapi/params.py

Ugh.

Would moving it gen.py work?

author
#61 lore raw

Re: [PATCH 15/37] qapi/common.py: split build_params into new file

Quoted 15 lines · Markus Armbruster
Quoted 4 lines · John Snow

Including it in common.py creates a circular import dependency, because schema relies on common.py. To type build_params properly, it needs to be moved outside of the chain.

Quoted 8 lines
---
  scripts/qapi/commands.py |  2 +-
  scripts/qapi/common.py   | 23 -----------------------
  scripts/qapi/events.py   |  2 +-
  scripts/qapi/params.py   | 40 ++++++++++++++++++++++++++++++++++++++++
  4 files changed, 42 insertions(+), 25 deletions(-)

create mode 100644 scripts/qapi/params.py

Quoted 4 lines

Ugh.

Would moving it gen.py work?

Actually, yes.

I have an experimental patch way, way later in the series that does this:

1. Leaves common.py with *just* functions and constants used by schema.py: c_name, POINTER_SUFFIX, and transitively EATSPACE.

2. Splits gen_c.py out of gen.py, putting all of the C-specific generator classes in there.

3. Adds params.py and the C-specific bits of common.py into gen_c.py.

In effect, you get gen_c.py with all of the C-specific bits in it, all of the other code-generation modules import from gen_c (marking them obviously as C code generators), and schema.py and other parsing friends import only the tiny common.py for c_name().

I'll adjust this patch to stash this in gen.py for now. It's too disruptive to shift the other refactor around in my stack.

--js

author
#63 lore raw

PATCH 16/37 qapi: establish mypy type-checking baseline

In reply to John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

---
 scripts/qapi/doc.py    |  3 +-
 scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py |  3 +-
 3 files changed, 69 insertions(+), 2 deletions(-)
 create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/doc.py +2 -1
index cbf7076ed9..70f7cdfaa6 100644
@@ -5,7 +5,8 @@
 """This script produces the documentation of a qapi schema in texinfo format"""
 
 import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor
 
 
 MSG_FMT = """
scripts/qapi/mypy.ini new file +65 -0

65 lines not on this page · raw message

scripts/qapi/schema.py +2 -1
index b4921b46c9..bb0cd717f1 100644
@@ -17,6 +17,7 @@
 import os
 import re
 from collections import OrderedDict
+from typing import Optional
 
 from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
Signature · 1 line
2.26.2
#64 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 6 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

I get:

    $ mypy --config-file qapi/mypy.ini qapi
    qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
    qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
    Found 1 error in 1 file (checked 18 source files)

Is this expected?

In case it matters:

    $ mypy --version
    mypy 0.761
Quoted 13 lines
---
 scripts/qapi/doc.py    |  3 +-
 scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py |  3 +-
 3 files changed, 69 insertions(+), 2 deletions(-)
 create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/doc.py +0 -0
index cbf7076ed9..70f7cdfaa6 100644

scripts/qapi/doc.py line 9

Quoted 6 lines
scripts/qapi/doc.py +2 -1
@@ -5,7 +5,8 @@
 """This script produces the documentation of a qapi schema in texinfo format"""
 
 import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor

Your mypy doesn't like such lazy imports? Mine seems not to care.

Quoted 64 lines
scripts/qapi/doc.py +0 -0
 MSG_FMT = """
scripts/qapi/mypy.ini new file +55 -0
new file mode 100644
index 0000000000..a0f2365a53
@@ -0,0 +1,65 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.doc]
+disallow_subclassing_any = False
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False

scripts/qapi/mypy.ini line 65

Quoted 10 lines
scripts/qapi/mypy.ini new file +10 -0
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False

Greek to me. I'll learn in due time.

Quoted 21 lines
scripts/qapi/schema.py +2 -1
index b4921b46c9..bb0cd717f1 100644
@@ -17,6 +17,7 @@
 import os
 import re
 from collections import OrderedDict
+from typing import Optional
 
 from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
author
#65 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

Quoted 13 lines · Markus Armbruster

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 6 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

Quoted 9 lines

I get:

     $ mypy --config-file qapi/mypy.ini qapi
     qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
     qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
     Found 1 error in 1 file (checked 18 source files)

Is this expected?

Nope.

Quoted 5 lines

In case it matters:

     $ mypy --version
     mypy 0.761

I am using mypy 0.782.

I will investigate to see if there is an *easy* win to allow older versions to work.

In the meantime, please consider trying this:

pip install --user mypy==0.782 ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/

Quoted 15 lines
Quoted 15 lines
---
  scripts/qapi/doc.py    |  3 +-
  scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
  scripts/qapi/schema.py |  3 +-
  3 files changed, 69 insertions(+), 2 deletions(-)

create mode 100644 scripts/qapi/mypy.ini

scripts/qapi/doc.py +0 -0
index cbf7076ed9..70f7cdfaa6 100644
@@ -5,7 +5,8 @@
  """This script produces the documentation of a qapi schema in texinfo format"""
 
Quoted 7 lines

scripts/qapi/doc.py line 9

Quoted 4 lines
scripts/qapi/doc.py +2 -1
  import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor

Your mypy doesn't like such lazy imports? Mine seems not to care.

Yeah, it specifically complained that no such definition existed in that file.

Quoted 68 lines
Quoted 68 lines
scripts/qapi/doc.py +0 -0
  MSG_FMT = """
scripts/qapi/mypy.ini new file +59 -0
new file mode 100644
index 0000000000..a0f2365a53
@@ -0,0 +1,65 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.doc]
+disallow_subclassing_any = False
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
Quoted 9 lines

scripts/qapi/mypy.ini line 65

Quoted 6 lines
scripts/qapi/mypy.ini new file +6 -0
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False

Greek to me. I'll learn in due time.

I am using these options:

--strict, which is effectively -Wall.

--no-strict-optional, which disables type checking errors on conflict between Optional[T] and T. Namely, when you initialize a class field to None and set that variable after initialization, callers must be prepared to see if that field was None or not. We do that effectively nowhere.

As Python will surely explode in a noticeable way if we got an unexpected 'None', I am just suppressing these warnings "for now".

--allow-untyped-calls silences errors in files that have calls to functions in files I still have not typed. By the end of the series, this option goes away, because there's nothing untyped left.

For each untyped file, we are actually starting with all of the above options and then layering these options on top. Any egregious typing errors present in these "ignored" files will be spotted.

To get the bad files to pass, we only need three options:

allow untyped defs -- Simply permits us to have functions without annotations.

allow incomplete defs -- allows functions that are only partially typed.

check untyped defs = False -- Don't try to type check untyped definitions.

Quoted 21 lines
Quoted 21 lines
scripts/qapi/schema.py +2 -1
index b4921b46c9..bb0cd717f1 100644
@@ -17,6 +17,7 @@
  import os
  import re
  from collections import OrderedDict
+from typing import Optional
 
  from .common import c_name, POINTER_SUFFIX
  from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
  class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
      def __init__(self, name, info, doc, ifcond=None, features=None):
          assert name is None or isinstance(name, str)
#66 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

Quoted 29 lines · John Snow
Quoted 10 lines · Markus Armbruster

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 5 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 8 lines
Quoted 1 line
Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

I get:

     $ mypy --config-file qapi/mypy.ini qapi
     qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
     qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
     Found 1 error in 1 file (checked 18 source files)
Is this expected?

Nope.

Quoted 4 lines

In case it matters:

     $ mypy --version
     mypy 0.761

I am using mypy 0.782.

Quoted 7 lines

I will investigate to see if there is an *easy* win to allow older versions to work.

In the meantime, please consider trying this:

pip install --user mypy==0.782 ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/

I'll consider dragging my feet until upgrading Fedora gives it to me for free.

The less I interact with package managers, the happier I am.

Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
---
  scripts/qapi/doc.py    |  3 +-
  scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
  scripts/qapi/schema.py |  3 +-
  3 files changed, 69 insertions(+), 2 deletions(-)

create mode 100644 scripts/qapi/mypy.ini

Quoted 12 lines

scripts/qapi/doc.py line 8

Quoted 10 lines
scripts/qapi/doc.py +2 -1
index cbf7076ed9..70f7cdfaa6 100644
@@ -5,7 +5,8 @@
  """This script produces the documentation of a qapi schema in texinfo format"""
    import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor

Your mypy doesn't like such lazy imports? Mine seems not to care.

Quoted 2 lines

Yeah, it specifically complained that no such definition existed in that file.

I sense a certain wobbliness in mypy. Perhaps to be expected from a tool with major version zero. There's a risk that developers' local "make check" and our gating CI differ too much. We'll see.

Quoted 80 lines
Quoted 66 lines
Quoted 66 lines
scripts/qapi/doc.py +0 -0
  MSG_FMT = """
scripts/qapi/mypy.ini new file +58 -0
new file mode 100644
index 0000000000..a0f2365a53
@@ -0,0 +1,65 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.doc]
+disallow_subclassing_any = False
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
Quoted 9 lines

scripts/qapi/mypy.ini line 65

Quoted 7 lines
scripts/qapi/mypy.ini new file +7 -0
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False

Greek to me. I'll learn in due time.

I am using these options:

--strict, which is effectively -Wall.

Quoted 8 lines

--no-strict-optional, which disables type checking errors on conflict
between Optional[T] and T. Namely, when you initialize a class field
to None and set that variable after initialization, callers must be
prepared to see if that field was None or not. We do that effectively nowhere.

As Python will surely explode in a noticeable way if we got an unexpected 'None', I am just suppressing these warnings "for now".

Okay.

We may want to rethink how we define and initialize these variables. We initialize mostly to keep pylint happy. We initialize to None precisely to make use before the real initialization explode.

Quoted 9 lines

--allow-untyped-calls silences errors in files that have calls to
functions in files I still have not typed. By the end of the series,
this option goes away, because there's nothing untyped left.

For each untyped file, we are actually starting with all of the above options and then layering these options on top. Any egregious typing errors present in these "ignored" files will be spotted.

Quoted 8 lines

To get the bad files to pass, we only need three options:

allow untyped defs -- Simply permits us to have functions without annotations.

allow incomplete defs -- allows functions that are only partially typed.

check untyped defs = False -- Don't try to type check untyped definitions.

Thanks!

[...]

author
#67 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

Quoted 36 lines · Markus Armbruster
Quoted 34 lines · John Snow
Quoted 10 lines · Markus Armbruster

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 5 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 8 lines
Quoted 1 line
Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

I get:

      $ mypy --config-file qapi/mypy.ini qapi
      qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
      qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
      Found 1 error in 1 file (checked 18 source files)
Is this expected?

Nope.

Quoted 4 lines

In case it matters:

      $ mypy --version
      mypy 0.761

I am using mypy 0.782.

I will investigate to see if there is an *easy* win to allow older versions to work.

In the meantime, please consider trying this:

Quoted 8 lines
Quoted 2 lines

pip install --user mypy==0.782 ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/

I'll consider dragging my feet until upgrading Fedora gives it to me for free.

The less I interact with package managers, the happier I am.

I'll probably be working on making sure CI runs in a virtual environment with specifically pinned versions.

We don't need mypy to run or build, so I have been less worried about requiring bleeding edge versions for development and regression testing.

Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 13 lines
---
   scripts/qapi/doc.py    |  3 +-
   scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
   scripts/qapi/schema.py |  3 +-
   3 files changed, 69 insertions(+), 2 deletions(-)

create mode 100644 scripts/qapi/mypy.ini

scripts/qapi/doc.py +0 -0
index cbf7076ed9..70f7cdfaa6 100644

scripts/qapi/doc.py line 8

Quoted 5 lines
scripts/qapi/doc.py +2 -1
@@ -5,7 +5,8 @@
   """This script produces the documentation of a qapi schema in texinfo format"""
     import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor

Your mypy doesn't like such lazy imports? Mine seems not to care.

Quoted 7 lines
Quoted 2 lines

Yeah, it specifically complained that no such definition existed in that file.

I sense a certain wobbliness in mypy. Perhaps to be expected from a tool with major version zero. There's a risk that developers' local "make check" and our gating CI differ too much. We'll see.

It is indeed very cutting edge. It wasn't really viable to use until Python 3.6.

Quoted 86 lines
Quoted 86 lines
Quoted 66 lines
Quoted 66 lines
scripts/qapi/doc.py +0 -0
   MSG_FMT = """
scripts/qapi/mypy.ini new file +58 -0
new file mode 100644
index 0000000000..a0f2365a53
@@ -0,0 +1,65 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.doc]
+disallow_subclassing_any = False
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
Quoted 9 lines

scripts/qapi/mypy.ini line 65

Quoted 7 lines
scripts/qapi/mypy.ini new file +7 -0
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False

Greek to me. I'll learn in due time.

I am using these options:

--strict, which is effectively -Wall.

--no-strict-optional, which disables type checking errors on conflict
between Optional[T] and T. Namely, when you initialize a class field
to None and set that variable after initialization, callers must be
prepared to see if that field was None or not. We do that effectively nowhere.

Quoted 9 lines
Quoted 2 lines

As Python will surely explode in a noticeable way if we got an unexpected 'None', I am just suppressing these warnings "for now".

Okay.

We may want to rethink how we define and initialize these variables. We initialize mostly to keep pylint happy. We initialize to None precisely to make use before the real initialization explode.

The pattern we have is roughly correct; it's just that we rely on that implicit explosion, which mypy will warn about if I don't disable this option.

It's not worth addressing in our first pass, but if you look at schema.py's @property ifcond code, that's the type of thing that would "fix" this warning -- basically access shims that add a "real" error message when it's None, but convert the return type from Optional[T] to T.

It's later stuff.

Quoted 22 lines
Quoted 9 lines

--allow-untyped-calls silences errors in files that have calls to
functions in files I still have not typed. By the end of the series,
this option goes away, because there's nothing untyped left.

For each untyped file, we are actually starting with all of the above options and then layering these options on top. Any egregious typing errors present in these "ignored" files will be spotted.

Quoted 8 lines

To get the bad files to pass, we only need three options:

allow untyped defs -- Simply permits us to have functions without annotations.

allow incomplete defs -- allows functions that are only partially typed.

check untyped defs = False -- Don't try to type check untyped definitions.

Thanks!

[...]

#68 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

In reply to Markus Armbruster

On Mon, Sep 21, 2020 at 10:05:24AM +0200, Markus Armbruster wrote:
[...]

Quoted 3 lines

I sense a certain wobbliness in mypy. Perhaps to be expected from a tool with major version zero. There's a risk that developers' local "make check" and our gating CI differ too much. We'll see.

This possibility shouldn't exist. If you don't have the required version in your system, the validation script must know how to create a virtualenv and install the right version inside it.

Signature · 1 line
Eduardo
author
#69 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

In reply to Markus Armbruster
Quoted 20 lines · Markus Armbruster

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 6 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

I get:

     $ mypy --config-file qapi/mypy.ini qapi
     qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
     qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
     Found 1 error in 1 file (checked 18 source files)
Quoted 7 lines

Is this expected?

In case it matters:

     $ mypy --version
     mypy 0.761

(Warning; FiSH and stgit ahead)

cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8

### Testing mypy 0.770

pip install mypy==0.770
stg goto qapi-establish-mypy-type

while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end

pip uninstall mypy

###

0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option) 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8) 0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.

0.760 doesn't support strict in the config file (It needs component options), and it wants a few extra annotations where its inference power is weaker. Well, easy enough to fix up.

0.630 fails again and insists that __init__ should have a return type annotation of None. Modern mypy is smart enough to know that's what that type is supposed to be. Arbitrarily, this is my cutoff for what seems reasonable to even want to support.

I still find the lack of "strict=true" in the config file irritating and might ask to target 0.770 or newer. There should be no reason we can't install that in a venv for CI to chew on.

Humbly ask I take the lazy way out and document that we support mypy >= 0.770.

--js

#70 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

Quoted 70 lines · John Snow
Quoted 12 lines · Markus Armbruster

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 6 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

I get:

Quoted 9 lines
     $ mypy --config-file qapi/mypy.ini qapi
     qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
     qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
     Found 1 error in 1 file (checked 18 source files)
Is this expected?
In case it matters:
     $ mypy --version
     mypy 0.761

(Warning; FiSH and stgit ahead)

cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8

### Testing mypy 0.770

pip install mypy==0.770
stg goto qapi-establish-mypy-type

while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end

pip uninstall mypy

###

0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option) 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8) 0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.

0.760 doesn't support strict in the config file (It needs component options), and it wants a few extra annotations where its inference power is weaker. Well, easy enough to fix up.

0.630 fails again and insists that __init__ should have a return type annotation of None. Modern mypy is smart enough to know that's what that type is supposed to be. Arbitrarily, this is my cutoff for what seems reasonable to even want to support.

Quoted 6 lines

I still find the lack of "strict=true" in the config file irritating and might ask to target 0.770 or newer. There should be no reason we can't install that in a venv for CI to chew on.

Humbly ask I take the lazy way out and document that we support mypy

Quoted 1 line

= 0.770.

Our "supported build platforms" policy puts hard limits on the minimum versions for tools the build requires.

Mypy is not such a tool. I hope we get to the point where we can have "make check" run it, but skipping the test when we don't have a sufficiently modern mypy feels okay to me, as long as our gating CI still guards the master branch.

author
#71 lore raw

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

Quoted 79 lines · Markus Armbruster
Quoted 70 lines · John Snow
Quoted 12 lines · Markus Armbruster

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?

Quoted 6 lines · John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

I get:

Quoted 9 lines
      $ mypy --config-file qapi/mypy.ini qapi
      qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
      qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
      Found 1 error in 1 file (checked 18 source files)
Is this expected?
In case it matters:
      $ mypy --version
      mypy 0.761

(Warning; FiSH and stgit ahead)

cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8

### Testing mypy 0.770

pip install mypy==0.770
stg goto qapi-establish-mypy-type

while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end

pip uninstall mypy

###

0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option) 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8) 0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.

0.760 doesn't support strict in the config file (It needs component options), and it wants a few extra annotations where its inference power is weaker. Well, easy enough to fix up.

0.630 fails again and insists that __init__ should have a return type annotation of None. Modern mypy is smart enough to know that's what that type is supposed to be. Arbitrarily, this is my cutoff for what seems reasonable to even want to support.

Quoted 6 lines

I still find the lack of "strict=true" in the config file irritating and might ask to target 0.770 or newer. There should be no reason we can't install that in a venv for CI to chew on.

Humbly ask I take the lazy way out and document that we support mypy

Quoted 1 line

= 0.770.

Quoted 8 lines

Our "supported build platforms" policy puts hard limits on the minimum versions for tools the build requires.

Mypy is not such a tool. I hope we get to the point where we can have "make check" run it, but skipping the test when we don't have a sufficiently modern mypy feels okay to me, as long as our gating CI still guards the master branch.

Yes, that's been my view. These tools *are* new and they *do* change often. Chasing a wide compatibility window with them is prohibitively difficult. Luckily, they're not needed for running the packages at all.

If I do add linting to 'make check', it is going to be through a virtual environment that deploys *specific versions* of these tools, and it will happen transparently. The VM tests already do this.

--js

author
#72 lore raw

PATCH 17/37 qapi/events.py: add notational type hints

In reply to John Snow

This is enough to enable type-checking on events.py, so enable that in this patch, too.

---
 scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
 scripts/qapi/mypy.ini  |  5 -----
 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +35 -11
index 75eda72534..449a700ffe 100644
@@ -12,6 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import (
     c_enum_const,
     c_name,
@@ -19,17 +21,27 @@
 )
 from .params import build_params
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+                           arg_type: QAPISchemaObjectType,
+                           boxed: bool) -> str:
     return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
         'c_name': c_name(name.lower()),
         'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+                        arg_type: QAPISchemaObjectType,
+                        boxed: bool) -> str:
     return mcgen('''
 
 %(proto)s;
@@ -38,7 +50,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
@@ -66,7 +78,11 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+                   arg_type: QAPISchemaObjectType,
+                   boxed: bool,
+                   event_enum_name: str,
+                   event_emit: str) -> str:
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -142,15 +158,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members = []
+        self._event_enum_members: List[QAPISchemaEnumMember] = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         events = self._module_basename('qapi-events', name)
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
@@ -173,7 +189,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('emit', ' * QAPI Events emission')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
@@ -194,7 +210,13 @@ def visit_end(self):
                              event_emit=self._event_emit_name,
                              event_enum=self._event_enum_name))
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(self,
+                    name: str,
+                    info: QAPISourceInfo,
+                    ifcond: List[str],
+                    features: List[QAPISchemaFeature],
+                    arg_type: QAPISchemaObjectType,
+                    boxed: bool) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_event_send_decl(name, arg_type, boxed))
             self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -205,7 +227,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed):
         self._event_enum_members.append(QAPISchemaEnumMember(name, None))
 
 
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+               output_dir: str,
+               prefix: str) -> None:
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
index a0f2365a53..b668776d94 100644
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.events]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.expr]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
Signature · 1 line
2.26.2
author
#73 lore raw

PATCH 18/37 qapi/events.py: Move comments into docstrings

In reply to John Snow
---
 scripts/qapi/events.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/events.py +3 -1
index 449a700ffe..0b49cf5fea 100644
@@ -49,8 +49,10 @@ def gen_event_send_decl(name: str,
                  proto=build_event_send_proto(name, arg_type, boxed))
 
 
-# Declare and initialize an object 'qapi' using parameters from build_params()
 def gen_param_var(typ: QAPISchemaObjectType) -> str:
+    """
+    Declare and initialize a qapi object, using parameters from `build_params`.
+    """
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
Signature · 1 line
2.26.2
author
#74 lore raw

PATCH 19/37 qapi/commands.py: Don't re-bind to variable of different type

In reply to John Snow

Mypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.

---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +1 -3
index 0c0fe854fe..b215e58357 100644
@@ -194,14 +194,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     if not options:
         options = ['QCO_NO_OPTIONS']
 
-    options = " | ".join(options)
-
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
                          qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
-                opts=options)
+                opts=" | ".join(options))
     return ret
 
 
Signature · 1 line
2.26.2
author
#75 lore raw

PATCH 20/37 qapi/commands.py: add notational type hints

In reply to John Snow
---
 scripts/qapi/commands.py | 72 ++++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 18 deletions(-)
scripts/qapi/commands.py +54 -18
index b215e58357..3c950da10a 100644
@@ -13,15 +13,32 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import Dict, Optional, List, Set
+
 from .common import (
     c_name,
     mcgen,
 )
+from .gen import (
+    QAPIGenC,
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    ifcontext,
+)
 from .params import build_params
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .schema import (
+    QAPISchema,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+                     arg_type: Optional[QAPISchemaObjectType],
+                     boxed: bool,
+                     ret_type: Optional[QAPISchemaType]) -> str:
     return mcgen('''
 %(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
@@ -30,7 +47,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
                  params=build_params(arg_type, boxed, 'Error **errp'))
 
 
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+             arg_type: Optional[QAPISchemaObjectType],
+             boxed: bool,
+             ret_type: Optional[QAPISchemaType]) -> str:
     ret = ''
 
     argstr = ''
@@ -66,7 +86,7 @@ def gen_call(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
     return mcgen('''
 
 static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
@@ -87,19 +107,22 @@ def gen_marshal_output(ret_type):
                  c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
     return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
             % c_name(name))
 
 
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
     return mcgen('''
 %(proto)s;
 ''',
                  proto=build_marshal_proto(name))
 
 
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+                arg_type: Optional[QAPISchemaObjectType],
+                boxed: bool,
+                ret_type: Optional[QAPISchemaType]) -> str:
     have_args = boxed or (arg_type and not arg_type.is_empty())
 
     ret = mcgen('''
@@ -181,7 +204,10 @@ def gen_marshal(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_register_command(name, success_response, allow_oob, allow_preconfig):
+def gen_register_command(name: str,
+                         success_response: bool,
+                         allow_oob: bool,
+                         allow_preconfig: bool) -> str:
     options = []
 
     if not success_response:
@@ -203,7 +229,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     return ret
 
 
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -220,15 +246,14 @@ def gen_registry(registry, prefix):
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', None, __doc__)
         self._regy = QAPIGenCCode(None)
-        self._visited_ret_types = {}
+        self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         self._visited_ret_types[self._genc] = set()
         commands = self._module_basename('qapi-commands', name)
         types = self._module_basename('qapi-types', name)
@@ -252,7 +277,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('init', ' * QAPI Commands initialization')
         self._genh.add(mcgen('''
 #include "qapi/qmp/dispatch.h"
@@ -268,9 +293,18 @@ def visit_end(self):
                                       prefix=self._prefix))
         self._genc.add(gen_registry(self._regy.get_content(), self._prefix))
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig):
+    def visit_command(self,
+                      name: str,
+                      info: QAPISourceInfo,
+                      ifcond: List[str],
+                      features: List[QAPISchemaFeature],
+                      arg_type: Optional[QAPISchemaObjectType],
+                      ret_type: Optional[QAPISchemaType],
+                      gen: bool,
+                      success_response: bool,
+                      boxed: bool,
+                      allow_oob: bool,
+                      allow_preconfig: bool) -> None:
         if not gen:
             return
         # FIXME: If T is a user-defined type, the user is responsible
@@ -291,7 +325,9 @@ def visit_command(self, name, info, ifcond, features,
                                                 allow_oob, allow_preconfig))
 
 
-def gen_commands(schema, output_dir, prefix):
+def gen_commands(schema: QAPISchema,
+                 output_dir: str,
+                 prefix: str) -> None:
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
Signature · 1 line
2.26.2
author
#76 lore raw

PATCH 21/37 qapi/commands.py: enable checking with mypy

In reply to John Snow
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
index b668776d94..9da1dccef4 100644
@@ -4,11 +4,6 @@ strict_optional = False
 disallow_untyped_calls = False
 python_version = 3.6
 
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.doc]
 disallow_subclassing_any = False
 disallow_untyped_defs = False
Signature · 1 line
2.26.2
author
#77 lore raw

PATCH 22/37 qapi/source.py: add notational type hints

In reply to John Snow
---
 scripts/qapi/mypy.ini  |  5 -----
 scripts/qapi/source.py | 31 ++++++++++++++++++-------------
 2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 9da1dccef4..43c8bd1973 100644
@@ -39,11 +39,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
scripts/qapi/source.py +18 -13
index e97b9a8e15..1cc6a5b82d 100644
@@ -11,37 +11,42 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
         # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []
         # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
         self.fname = fname
         self.line = line
         self.parent = parent
-        self.pragma = parent.pragma if parent else QAPISchemaPragma()
-        self.defn_meta = None
-        self.defn_name = None
+        self.pragma: QAPISchemaPragma = (
+            parent.pragma if parent else QAPISchemaPragma()
+        )
+        self.defn_meta: Optional[str] = None
+        self.defn_name: Optional[str] = None
 
-    def set_defn(self, meta, name):
+    def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self):
+    def next_line(self: T) -> T:
         info = copy.copy(self)
         info.line += 1
         return info
 
-    def loc(self):
+    def loc(self) -> str:
         if self.fname is None:
             return sys.argv[0]
         ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
             ret += ':%d' % self.line
         return ret
 
-    def in_defn(self):
+    def in_defn(self) -> str:
         if self.defn_name:
             return "%s: In %s '%s':\n" % (self.fname,
                                           self.defn_meta, self.defn_name)
         return ''
 
-    def include_path(self):
+    def include_path(self) -> str:
         ret = ''
         parent = self.parent
         while parent:
@@ -63,5 +68,5 @@ def include_path(self):
             parent = parent.parent
         return ret
 
-    def __str__(self):
+    def __str__(self) -> str:
         return self.include_path() + self.in_defn() + self.loc()
Signature · 1 line
2.26.2
author
#78 lore raw

PATCH 23/37 qapi/source.py: delint with pylint

In reply to John Snow

Shush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.

---
 scripts/qapi/pylintrc  | 1 -
 scripts/qapi/source.py | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/pylintrc +0 -1
index c800055589..c50a6656b7 100644
@@ -8,7 +8,6 @@ ignore-patterns=doc.py,
                 gen.py,
                 parser.py,
                 schema.py,
-                source.py,
                 types.py,
                 visit.py,
 
scripts/qapi/source.py +3 -0
index 1cc6a5b82d..ba991d798f 100644
@@ -15,6 +15,9 @@
 
 
 class QAPISchemaPragma:
+    # Replace with @dataclass in Python 3.7+
+    # pylint: disable=too-few-public-methods
+
     def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
Signature · 1 line
2.26.2
author
#79 lore raw

PATCH 24/37 qapi/gen.py: Fix edge-case of _is_user_module

In reply to John Snow

The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.

---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +1 -1
index 11472ba043..fc46813d1a 100644
@@ -221,7 +221,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
 
     @staticmethod
     def _is_user_module(name):
-        return name and not name.startswith('./')
+        return name is not None and not name.startswith('./')
 
     @staticmethod
     def _is_builtin_module(name):
Signature · 1 line
2.26.2
author
#80 lore raw

PATCH 25/37 qapi/gen.py: add notational type hints

In reply to John Snow
---
 scripts/qapi/gen.py | 105 +++++++++++++++++++++++---------------------
 1 file changed, 56 insertions(+), 49 deletions(-)
scripts/qapi/gen.py +56 -49
index fc46813d1a..c87c84e7c9 100644
@@ -15,6 +15,7 @@
 import errno
 import os
 import re
+from typing import Dict, Generator, List, Optional, Tuple
 
 from .common import (
     c_fname,
@@ -24,32 +25,34 @@
     guardstart,
     mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import (
+    QAPISchemaVisitor,
+)
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         self.fname = fname
         self._preamble = ''
         self._body = ''
 
-    def preamble_add(self, text):
+    def preamble_add(self, text: str) -> None:
         self._preamble += text
 
-    def add(self, text):
+    def add(self, text: str) -> None:
         self._body += text
 
-    def get_content(self):
+    def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
-    def _top(self):
+    def _top(self) -> str:
         return ''
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return ''
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         # Include paths starting with ../ are used to reuse modules of the main
         # schema in specialised schemas. Don't overwrite the files that are
         # already generated for the main schema.
@@ -74,7 +77,7 @@ def write(self, output_dir):
         f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
     if before == after:
         return after   # suppress empty #if ... #endif
 
@@ -91,40 +94,38 @@ def _wrap_ifcond(ifcond, before, after):
 
 
 class QAPIGenCCode(QAPIGen):
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         super().__init__(fname)
-        self._start_if = None
+        self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-    def start_if(self, ifcond):
+    def start_if(self, ifcond: List[str]) -> None:
         assert self._start_if is None
         self._start_if = (ifcond, self._body, self._preamble)
 
-    def end_if(self):
+    def end_if(self) -> None:
         assert self._start_if
         self._wrap_ifcond()
         self._start_if = None
 
-    def _wrap_ifcond(self):
+    def _wrap_ifcond(self) -> None:
         self._body = _wrap_ifcond(self._start_if[0],
                                   self._start_if[1], self._body)
         self._preamble = _wrap_ifcond(self._start_if[0],
                                       self._start_if[2], self._preamble)
 
-    def get_content(self):
+    def get_content(self) -> str:
         assert self._start_if is None
         return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-    def __init__(self, fname, blurb, pydoc):
+    def __init__(self, fname: str, blurb: str, pydoc: str):
         super().__init__(fname)
         self._blurb = blurb
         self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
                                                   re.MULTILINE))
 
-    def _top(self):
+    def _top(self) -> str:
         return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -140,7 +141,7 @@ def _top(self):
 ''',
                      blurb=self._blurb, copyright=self._copyright)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -150,16 +151,16 @@ def _bottom(self):
 
 
 class QAPIGenH(QAPIGenC):
-
-    def _top(self):
+    def _top(self) -> str:
         return super()._top() + guardstart(self.fname)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str],
+              *args: QAPIGenCCode) -> Generator[None, None, None]:
     """A 'with' statement context manager to wrap with start_if()/end_if()
 
     *args: any number of QAPIGenCCode
@@ -185,15 +186,17 @@ def ifcontext(ifcond, *args):
 
 
 class QAPIGenDoc(QAPIGen):
-
-    def _top(self):
+    def _top(self) -> str:
         return (super()._top()
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 blurb: str,
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -201,38 +204,42 @@ def __init__(self, prefix, what, blurb, pydoc):
         self._genh = QAPIGenH(self._prefix + self._what + '.h',
                               blurb, pydoc)
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         self._genc.write(output_dir)
         self._genh.write(output_dir)
 
 
 class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 user_blurb: str,
+                 builtin_blurb: Optional[str],
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._user_blurb = user_blurb
         self._builtin_blurb = builtin_blurb
         self._pydoc = pydoc
-        self._genc = None
-        self._genh = None
-        self._module = {}
-        self._main_module = None
+        self._genc: Optional[QAPIGenC] = None
+        self._genh: Optional[QAPIGenH] = None
+        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+        self._main_module: Optional[str] = None
 
     @staticmethod
-    def _is_user_module(name):
+    def _is_user_module(name: Optional[str]) -> bool:
         return name is not None and not name.startswith('./')
 
     @staticmethod
-    def _is_builtin_module(name):
+    def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what, name):
+    def _module_dirname(self, what: str, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
 
-    def _module_basename(self, what, name):
+    def _module_basename(self, what: str, name: Optional[str]) -> str:
         ret = '' if self._is_builtin_module(name) else self._prefix
         if self._is_user_module(name):
             basename = os.path.basename(name)
@@ -244,27 +251,27 @@ def _module_basename(self, what, name):
             ret += re.sub(r'-', '-' + name + '-', what)
         return ret
 
-    def _module_filename(self, what, name):
+    def _module_filename(self, what: str, name: Optional[str]) -> str:
         return os.path.join(self._module_dirname(what, name),
                             self._module_basename(what, name))
 
-    def _add_module(self, name, blurb):
+    def _add_module(self, name: Optional[str], blurb: str) -> None:
         basename = self._module_filename(self._what, name)
         genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
         genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
         self._module[name] = (genc, genh)
         self._genc, self._genh = self._module[name]
 
-    def _add_user_module(self, name, blurb):
+    def _add_user_module(self, name: str, blurb: str) -> None:
         assert self._is_user_module(name)
         if self._main_module is None:
             self._main_module = name
         self._add_module(name, blurb)
 
-    def _add_system_module(self, name, blurb):
+    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
         self._add_module(name and './' + name, blurb)
 
-    def write(self, output_dir, opt_builtins=False):
+    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
         for name in self._module:
             if self._is_builtin_module(name) and not opt_builtins:
                 continue
@@ -272,13 +279,13 @@ def write(self, output_dir, opt_builtins=False):
             genc.write(output_dir)
             genh.write(output_dir)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         pass
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: Optional[str]) -> None:
         if name is None:
             if self._builtin_blurb:
                 self._add_system_module(None, self._builtin_blurb)
@@ -292,7 +299,7 @@ def visit_module(self, name):
             self._add_user_module(name, self._user_blurb)
             self._begin_user_module(name)
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
         relname = os.path.relpath(self._module_filename(self._what, name),
                                   os.path.dirname(self._genh.fname))
         self._genh.preamble_add(mcgen('''
Signature · 1 line
2.26.2
author
#81 lore raw

PATCH 26/37 qapi/gen.py: Enable checking with mypy

In reply to John Snow
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 43c8bd1973..dbfeda748c 100644
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.gen]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.introspect]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
Signature · 1 line
2.26.2
author
#82 lore raw

PATCH 27/37 qapi/gen.py: Remove unused parameter

In reply to John Snow

module_basename doesn't use the 'what' argument, so remove it.

---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
index c87c84e7c9..e873901672 100644
@@ -234,7 +234,7 @@ def _is_user_module(name: Optional[str]) -> bool:
     def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what: str, name: Optional[str]) -> str:
+    def _module_dirname(self, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
@@ -252,7 +252,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
         return ret
 
     def _module_filename(self, what: str, name: Optional[str]) -> str:
-        return os.path.join(self._module_dirname(what, name),
+        return os.path.join(self._module_dirname(name),
                             self._module_basename(what, name))
 
     def _add_module(self, name: Optional[str], blurb: str) -> None:
Signature · 1 line
2.26.2
author
#83 lore raw

PATCH 28/37 qapi/gen.py: update write() to be more idiomatic

In reply to John Snow

Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.

---
 scripts/qapi/gen.py | 24 ++++++++++--------------
 1 file changed, 10 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +10 -14
index e873901672..bd67f6b069 100644
@@ -12,7 +12,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import Dict, Generator, List, Optional, Tuple
@@ -60,21 +59,18 @@ def write(self, output_dir: str) -> None:
             return
         pathname = os.path.join(output_dir, self.fname)
         odir = os.path.dirname(pathname)
+
         if odir:
-            try:
-                os.makedirs(odir)
-            except os.error as e:
-                if e.errno != errno.EEXIST:
-                    raise
+            os.makedirs(odir, exist_ok=True)
+
         fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-        f = open(fd, 'r+', encoding='utf-8')
-        text = self.get_content()
-        oldtext = f.read(len(text) + 1)
-        if text != oldtext:
-            f.seek(0)
-            f.truncate(0)
-            f.write(text)
-        f.close()
+        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+            text = self.get_content()
+            oldtext = fp.read(len(text) + 1)
+            if text != oldtext:
+                fp.seek(0)
+                fp.truncate(0)
+                fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Signature · 1 line
2.26.2
author
#84 lore raw

PATCH 29/37 qapi/gen.py: delint with pylint

In reply to John Snow

'fp' and 'fd' are self-evident in context, add them to the list of OK names.

_top and _bottom also need to stay class methods because some users override the method and need to use self. Tell pylint to shush.

---
 scripts/qapi/gen.py   | 2 ++
 scripts/qapi/pylintrc | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -0
index bd67f6b069..0a72aecdd0 100644
@@ -46,9 +46,11 @@ def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
     def _top(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def _bottom(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def write(self, output_dir: str) -> None:
scripts/qapi/pylintrc +3 -2
index c50a6656b7..3aebb0d228 100644
@@ -5,7 +5,6 @@
 ignore-patterns=doc.py,
                 error.py,
                 expr.py,
-                gen.py,
                 parser.py,
                 schema.py,
                 types.py,
@@ -46,7 +45,9 @@ good-names=i,
            k,
            ex,
            Run,
-           _
+           _,
+           fp,  # fp = open(...)
+           fd,  # fd = os.open(...)
 
 [VARIABLES]
 
Signature · 1 line
2.26.2
author
#85 lore raw

PATCH 30/37 qapi/introspect.py: Add a typed 'extra' structure

In reply to John Snow

Typing arbitrarily shaped dicts with mypy is difficult prior to Python 3.8; using explicit structures is nicer.

Since we always define an Extra type now, the return type of _make_tree simplifies and always returns the tuple.

---
 scripts/qapi/introspect.py | 31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)
scripts/qapi/introspect.py +19 -12
index b036fcf9ce..41ca8afc67 100644
@@ -10,6 +10,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import (NamedTuple, Optional, Sequence)
+
 from .common import (
     c_name,
     gen_endif,
@@ -21,16 +23,21 @@
                      QAPISchemaType)
 
 
-def _make_tree(obj, ifcond, features, extra=None):
-    if extra is None:
-        extra = {}
-    if ifcond:
-        extra['if'] = ifcond
+class Extra(NamedTuple):
+    """
+    Extra contains data that isn't intended for output by introspection.
+    """
+    comment: Optional[str] = None
+    ifcond: Sequence[str] = tuple()
+
+
+def _make_tree(obj, ifcond, features,
+               extra: Optional[Extra] = None):
+    comment = extra.comment if extra else None
+    extra = Extra(comment, ifcond)
     if features:
-        obj['features'] = [(f.name, {'if': f.ifcond}) for f in features]
-    if extra:
-        return (obj, extra)
-    return obj
+        obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
+    return (obj, extra)
 
 
 def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -40,8 +47,8 @@ def indent(level):
 
     if isinstance(obj, tuple):
         ifobj, extra = obj
-        ifcond = extra.get('if')
-        comment = extra.get('comment')
+        ifcond = extra.ifcond
+        comment = extra.comment
         ret = ''
         if comment:
             ret += indent(level) + '/* %s */\n' % comment
@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
             if not self._unmask:
                 # Output a comment to make it easy to map masked names
                 # back to the source when reading the generated output.
-                extra = {'comment': '"%s" = %s' % (self._name(name), name)}
+                extra = Extra(comment=f'"{self._name(name)}" = {name}')
             name = self._name(name)
         obj['name'] = name
         obj['meta-type'] = mtype
Signature · 1 line
2.26.2
author
#86 lore raw

PATCH 31/37 qapi/introspect.py: add _gen_features helper

In reply to John Snow

_make_tree doesn't know if it is receiving an object or some other type; adding features information should arguably be performed by the caller.

This will help us refactor _make_tree more gracefully in the next patch.

---
 scripts/qapi/introspect.py | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
scripts/qapi/introspect.py +14 -8
index 41ca8afc67..e1edd0b179 100644
@@ -10,7 +10,7 @@
 See the COPYING file in the top-level directory.
 """
 
-from typing import (NamedTuple, Optional, Sequence)
+from typing import (NamedTuple, List, Optional, Sequence)
 
 from .common import (
     c_name,
@@ -20,7 +20,7 @@
 )
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaType)
+                     QAPISchemaFeature, QAPISchemaType)
 
 
 class Extra(NamedTuple):
@@ -31,12 +31,10 @@ class Extra(NamedTuple):
     ifcond: Sequence[str] = tuple()
 
 
-def _make_tree(obj, ifcond, features,
+def _make_tree(obj, ifcond,
                extra: Optional[Extra] = None):
     comment = extra.comment if extra else None
     extra = Extra(comment, ifcond)
-    if features:
-        obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
     return (obj, extra)
 
 
@@ -169,6 +167,10 @@ def _use_type(self, typ):
             return '[' + self._use_type(typ.element_type) + ']'
         return self._name(typ.name)
 
+    @classmethod
+    def _gen_features(cls, features: List[QAPISchemaFeature]):
+        return [_make_tree(f.name, f.ifcond) for f in features]
+
     def _gen_tree(self, name, mtype, obj, ifcond, features):
         extra = None
         if mtype not in ('command', 'event', 'builtin', 'array'):
@@ -179,13 +181,17 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
             name = self._name(name)
         obj['name'] = name
         obj['meta-type'] = mtype
-        self._trees.append(_make_tree(obj, ifcond, features, extra))
+        if features:
+            obj['features'] = self._gen_features(features)
+        self._trees.append(_make_tree(obj, ifcond, extra))
 
     def _gen_member(self, member):
         obj = {'name': member.name, 'type': self._use_type(member.type)}
         if member.optional:
             obj['default'] = None
-        return _make_tree(obj, member.ifcond, member.features)
+        if member.features:
+            obj['features'] = self._gen_features(member.features)
+        return _make_tree(obj, member.ifcond)
 
     def _gen_variants(self, tag_name, variants):
         return {'tag': tag_name,
@@ -193,7 +199,7 @@ def _gen_variants(self, tag_name, variants):
 
     def _gen_variant(self, variant):
         obj = {'case': variant.name, 'type': self._use_type(variant.type)}
-        return _make_tree(obj, variant.ifcond, None)
+        return _make_tree(obj, variant.ifcond)
 
     def visit_builtin_type(self, name, info, json_type):
         self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
Signature · 1 line
2.26.2
author
#87 lore raw

PATCH 32/37 qapi/introspect.py: create a typed 'Node' data structure

In reply to John Snow

Replacing the un-typed tuple, add a typed Node that we can add typed metadata to.

---
 scripts/qapi/introspect.py | 53 ++++++++++++++++++++------------------
 1 file changed, 28 insertions(+), 25 deletions(-)
scripts/qapi/introspect.py +28 -25
index e1edd0b179..e0f5007ab7 100644
@@ -31,11 +31,18 @@ class Extra(NamedTuple):
     ifcond: Sequence[str] = tuple()
 
 
-def _make_tree(obj, ifcond,
-               extra: Optional[Extra] = None):
-    comment = extra.comment if extra else None
-    extra = Extra(comment, ifcond)
-    return (obj, extra)
+class Node:
+    """
+    Node generally contains a SchemaInfo-like type (as a dict),
+    But it also used to wrap comments/ifconds around leaf value types.
+    """
+    # Remove after 3.7 adds @dataclass:
+    # pylint: disable=too-few-public-methods
+    def __init__(self, data, ifcond: List[str],
+                 extra: Optional[Extra] = None):
+        self.data = data
+        comment = extra.comment if extra else None
+        self.extra = Extra(comment, ifcond)
 
 
 def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -43,18 +50,15 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
     def indent(level):
         return level * 4 * ' '
 
-    if isinstance(obj, tuple):
-        ifobj, extra = obj
-        ifcond = extra.ifcond
-        comment = extra.comment
+    if isinstance(obj, Node):
         ret = ''
-        if comment:
-            ret += indent(level) + '/* %s */\n' % comment
-        if ifcond:
-            ret += gen_if(ifcond)
-        ret += _tree_to_qlit(ifobj, level)
-        if ifcond:
-            ret += '\n' + gen_endif(ifcond)
+        if obj.extra.comment:
+            ret += indent(level) + '/* %s */\n' % obj.extra.comment
+        if obj.extra.ifcond:
+            ret += gen_if(obj.extra.ifcond)
+        ret += _tree_to_qlit(obj.data, level)
+        if obj.extra.ifcond:
+            ret += '\n' + gen_endif(obj.extra.ifcond)
         return ret
 
     ret = ''
@@ -169,7 +173,7 @@ def _use_type(self, typ):
 
     @classmethod
     def _gen_features(cls, features: List[QAPISchemaFeature]):
-        return [_make_tree(f.name, f.ifcond) for f in features]
+        return [Node(f.name, f.ifcond) for f in features]
 
     def _gen_tree(self, name, mtype, obj, ifcond, features):
         extra = None
@@ -183,7 +187,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
         obj['meta-type'] = mtype
         if features:
             obj['features'] = self._gen_features(features)
-        self._trees.append(_make_tree(obj, ifcond, extra))
+        self._trees.append(Node(obj, ifcond, extra))
 
     def _gen_member(self, member):
         obj = {'name': member.name, 'type': self._use_type(member.type)}
@@ -191,7 +195,7 @@ def _gen_member(self, member):
             obj['default'] = None
         if member.features:
             obj['features'] = self._gen_features(member.features)
-        return _make_tree(obj, member.ifcond)
+        return Node(obj, member.ifcond)
 
     def _gen_variants(self, tag_name, variants):
         return {'tag': tag_name,
@@ -199,15 +203,14 @@ def _gen_variants(self, tag_name, variants):
 
     def _gen_variant(self, variant):
         obj = {'case': variant.name, 'type': self._use_type(variant.type)}
-        return _make_tree(obj, variant.ifcond)
+        return Node(obj, variant.ifcond)
 
     def visit_builtin_type(self, name, info, json_type):
         self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
 
     def visit_enum_type(self, name, info, ifcond, features, members, prefix):
         self._gen_tree(name, 'enum',
-                       {'values': [_make_tree(m.name, m.ifcond, None)
-                                   for m in members]},
+                       {'values': [Node(m.name, m.ifcond) for m in members]},
                        ifcond, features)
 
     def visit_array_type(self, name, info, ifcond, element_type):
@@ -227,9 +230,9 @@ def visit_object_type_flat(self, name, info, ifcond, features,
     def visit_alternate_type(self, name, info, ifcond, features, variants):
         self._gen_tree(name, 'alternate',
                        {'members': [
-                           _make_tree({'type': self._use_type(m.type)},
-                                      m.ifcond, None)
-                           for m in variants.variants]},
+                           Node({'type': self._use_type(m.type)}, m.ifcond)
+                           for m in variants.variants
+                       ]},
                        ifcond, features)
 
     def visit_command(self, name, info, ifcond, features,
Signature · 1 line
2.26.2
author
#88 lore raw

PATCH 33/37 qapi/introspect.py: add notational type hints

In reply to John Snow

Add a tiny hint into schema.py to allow introspect.py to type check, too.

---
 scripts/qapi/introspect.py | 123 ++++++++++++++++++++++++++-----------
 scripts/qapi/mypy.ini      |   5 --
 scripts/qapi/schema.py     |   2 +-
 3 files changed, 87 insertions(+), 43 deletions(-)
scripts/qapi/introspect.py +86 -37
index e0f5007ab7..4e1ea3a00c 100644
@@ -10,7 +10,16 @@
 See the COPYING file in the top-level directory.
 """
 
-from typing import (NamedTuple, List, Optional, Sequence)
+from typing import (
+    Any,
+    Dict,
+    Generic,
+    List,
+    NamedTuple,
+    Optional,
+    Sequence,
+    TypeVar,
+)
 
 from .common import (
     c_name,
@@ -19,8 +28,23 @@
     mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaFeature, QAPISchemaType)
+from .schema import (
+    QAPISchema,
+    QAPISchemaArrayType,
+    QAPISchemaBuiltinType,
+    QAPISchemaEntity,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariant,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+
+_NodeType = TypeVar('_NodeType')
 
 
 class Extra(NamedTuple):
@@ -31,23 +55,24 @@ class Extra(NamedTuple):
     ifcond: Sequence[str] = tuple()
 
 
-class Node:
+class Node(Generic[_NodeType]):
     """
     Node generally contains a SchemaInfo-like type (as a dict),
     But it also used to wrap comments/ifconds around leaf value types.
     """
     # Remove after 3.7 adds @dataclass:
     # pylint: disable=too-few-public-methods
-    def __init__(self, data, ifcond: List[str],
+    def __init__(self, data: _NodeType, ifcond: List[str],
                  extra: Optional[Extra] = None):
         self.data = data
         comment = extra.comment if extra else None
         self.extra = Extra(comment, ifcond)
 
 
-def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
+def _tree_to_qlit(obj: Any, level: int = 0,
+                  suppress_first_indent: bool = False) -> str:
 
-    def indent(level):
+    def indent(level: int) -> str:
         return level * 4 * ' '
 
     if isinstance(obj, Node):
@@ -94,21 +119,20 @@ def indent(level):
     return ret
 
 
-def to_c_string(string):
+def to_c_string(string: str) -> str:
     return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor):
-
-    def __init__(self, prefix, unmask):
+    def __init__(self, prefix: str, unmask: bool):
         super().__init__(
             prefix, 'qapi-introspect',
             ' * QAPI/QMP schema introspection', __doc__)
         self._unmask = unmask
-        self._schema = None
-        self._trees = []
-        self._used_types = []
-        self._name_map = {}
+        self._schema: Optional[QAPISchema] = None
+        self._trees: List[Node[Dict[str, Any]]] = []
+        self._used_types: List[QAPISchemaType] = []
+        self._name_map: Dict[str, str] = {}
         self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqapi-introspect.h"
@@ -116,10 +140,10 @@ def __init__(self, prefix, unmask):
 ''',
                              prefix=prefix))
 
-    def visit_begin(self, schema):
+    def visit_begin(self, schema: QAPISchema) -> None:
         self._schema = schema
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         # visit the types that are actually used
         for typ in self._used_types:
             typ.visit(self)
@@ -141,18 +165,18 @@ def visit_end(self):
         self._used_types = []
         self._name_map = {}
 
-    def visit_needed(self, entity):
+    def visit_needed(self, entity: QAPISchemaEntity) -> bool:
         # Ignore types on first pass; visit_end() will pick up used types
         return not isinstance(entity, QAPISchemaType)
 
-    def _name(self, name):
+    def _name(self, name: str) -> str:
         if self._unmask:
             return name
         if name not in self._name_map:
             self._name_map[name] = '%d' % len(self._name_map)
         return self._name_map[name]
 
-    def _use_type(self, typ):
+    def _use_type(self, typ: QAPISchemaType) -> str:
         # Map the various integer types to plain int
         if typ.json_type() == 'int':
             typ = self._schema.lookup_type('int')
@@ -172,10 +196,13 @@ def _use_type(self, typ):
         return self._name(typ.name)
 
     @classmethod
-    def _gen_features(cls, features: List[QAPISchemaFeature]):
+    def _gen_features(cls,
+                      features: List[QAPISchemaFeature]) -> List[Node[str]]:
         return [Node(f.name, f.ifcond) for f in features]
 
-    def _gen_tree(self, name, mtype, obj, ifcond, features):
+    def _gen_tree(self, name: str, mtype: str, obj: Dict[str, Any],
+                  ifcond: List[str],
+                  features: Optional[List[QAPISchemaFeature]]) -> None:
         extra = None
         if mtype not in ('command', 'event', 'builtin', 'array'):
             if not self._unmask:
@@ -189,7 +216,8 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
             obj['features'] = self._gen_features(features)
         self._trees.append(Node(obj, ifcond, extra))
 
-    def _gen_member(self, member):
+    def _gen_member(self, member: QAPISchemaObjectTypeMember,
+                    ) -> Node[Dict[str, Any]]:
         obj = {'name': member.name, 'type': self._use_type(member.type)}
         if member.optional:
             obj['default'] = None
@@ -197,29 +225,39 @@ def _gen_member(self, member):
             obj['features'] = self._gen_features(member.features)
         return Node(obj, member.ifcond)
 
-    def _gen_variants(self, tag_name, variants):
+    def _gen_variants(self, tag_name: str,
+                      variants: List[QAPISchemaVariant]) -> Dict[str, Any]:
         return {'tag': tag_name,
                 'variants': [self._gen_variant(v) for v in variants]}
 
-    def _gen_variant(self, variant):
+    def _gen_variant(self, variant: QAPISchemaVariant) -> Node[Dict[str, Any]]:
         obj = {'case': variant.name, 'type': self._use_type(variant.type)}
         return Node(obj, variant.ifcond)
 
-    def visit_builtin_type(self, name, info, json_type):
+    def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
+                           json_type: str) -> None:
         self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(self, name: str, info: QAPISourceInfo,
+                        ifcond: List[str], features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
         self._gen_tree(name, 'enum',
                        {'values': [Node(m.name, m.ifcond) for m in members]},
                        ifcond, features)
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
+                         ifcond: List[str],
+                         element_type: QAPISchemaType) -> None:
         element = self._use_type(element_type)
         self._gen_tree('[' + element + ']', 'array', {'element-type': element},
                        ifcond, None)
 
-    def visit_object_type_flat(self, name, info, ifcond, features,
-                               members, variants):
+    def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
+                               ifcond: List[str],
+                               features: List[QAPISchemaFeature],
+                               members: Sequence[QAPISchemaObjectTypeMember],
+                               variants: Optional[QAPISchemaVariants]) -> None:
         obj = {'members': [self._gen_member(m) for m in members]}
         if variants:
             obj.update(self._gen_variants(variants.tag_member.name,
@@ -227,7 +265,10 @@ def visit_object_type_flat(self, name, info, ifcond, features,
 
         self._gen_tree(name, 'object', obj, ifcond, features)
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name: str, info: QAPISourceInfo,
+                             ifcond: List[str],
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
         self._gen_tree(name, 'alternate',
                        {'members': [
                            Node({'type': self._use_type(m.type)}, m.ifcond)
@@ -235,24 +276,32 @@ def visit_alternate_type(self, name, info, ifcond, features, variants):
                        ]},
                        ifcond, features)
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig):
+    def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
+                      features: List[QAPISchemaFeature],
+                      arg_type: QAPISchemaObjectType,
+                      ret_type: Optional[QAPISchemaObjectType], gen: bool,
+                      success_response: bool, boxed: bool, allow_oob: bool,
+                      allow_preconfig: bool) -> None:
         arg_type = arg_type or self._schema.the_empty_object_type
         ret_type = ret_type or self._schema.the_empty_object_type
-        obj = {'arg-type': self._use_type(arg_type),
-               'ret-type': self._use_type(ret_type)}
+        obj: Dict[str, Any] = {
+            'arg-type': self._use_type(arg_type),
+            'ret-type': self._use_type(ret_type)
+        }
         if allow_oob:
             obj['allow-oob'] = allow_oob
         self._gen_tree(name, 'command', obj, ifcond, features)
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(self, name: str, info: QAPISourceInfo,
+                    ifcond: List[str], features: List[QAPISchemaFeature],
+                    arg_type: QAPISchemaObjectType, boxed: bool) -> None:
         arg_type = arg_type or self._schema.the_empty_object_type
         self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)},
                        ifcond, features)
 
 
-def gen_introspect(schema, output_dir, prefix, opt_unmask):
+def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str,
+                   opt_unmask: bool) -> None:
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
index dbfeda748c..9ce8b56f22 100644
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.introspect]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.parser]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
scripts/qapi/schema.py +1 -1
index bb0cd717f1..3023bab44b 100644
@@ -28,7 +28,7 @@
 class QAPISchemaEntity:
     meta: Optional[str] = None
 
-    def __init__(self, name, info, doc, ifcond=None, features=None):
+    def __init__(self, name: str, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
         for f in features or []:
             assert isinstance(f, QAPISchemaFeature)
Signature · 1 line
2.26.2
author
#89 lore raw

PATCH 34/37 qapi/types.py: add notational type hints

In reply to John Snow
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 64 insertions(+), 27 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 9ce8b56f22..3babc380a8 100644
@@ -29,11 +29,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.visit]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
scripts/qapi/types.py +64 -22

207 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#90 lore raw

PATCH 35/37 qapi/types.py: remove one-letter variables

In reply to John Snow

"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++++++++++++++--------------
 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
index 3aebb0d228..9ca6b60caf 100644
@@ -7,7 +7,6 @@ ignore-patterns=doc.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                types.py,
                 visit.py,
 
 
scripts/qapi/types.py +15 -14
index 274d62fac8..a0a06b68c5 100644
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
     .array = (const char *const[]) {
 ''',
                 c_name=c_name(name))
-    for m in members:
-        ret += gen_if(m.ifcond)
-        index = c_enum_const(name, m.name, prefix)
+    for member in members:
+        ret += gen_if(member.ifcond)
+        index = c_enum_const(name, member.name, prefix)
         ret += mcgen('''
         [%(index)s] = "%(name)s",
 ''',
-                     index=index, name=m.name)
-        ret += gen_endif(m.ifcond)
+                     index=index, name=member.name)
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
     },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
 ''',
                 c_name=c_name(name))
 
-    for m in enum_members:
-        ret += gen_if(m.ifcond)
+    for member in enum_members:
+        ret += gen_if(member.ifcond)
         ret += mcgen('''
     %(c_enum)s,
 ''',
-                     c_enum=c_enum_const(name, m.name, prefix))
-        ret += gen_endif(m.ifcond)
+                     c_enum=c_enum_const(name, member.name, prefix))
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
 } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
     objects_seen.add(name)
 
     ret = ''
-    if variants:
-        for v in variants.variants:
-            if isinstance(v.type, QAPISchemaObjectType):
-                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-                                  v.type.local_members, v.type.variants)
+    for variant in variants.variants if variants else ():
+        obj = variant.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret += gen_object(obj.name, obj.ifcond, obj.base,
+                          obj.local_members, obj.variants)
 
     ret += mcgen('''
 
Signature · 1 line
2.26.2
author
#91 lore raw

PATCH 36/37 qapi/visit.py: remove unused parameters from gen_visit_object

In reply to John Snow

And this fixes the pylint report for this file, so make sure we check this in the future, too.

---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/qapi/pylintrc +0 -1
index 9ca6b60caf..9d7a1d285f 100644
@@ -7,7 +7,6 @@ ignore-patterns=doc.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                visit.py,
 
 
 [MESSAGES CONTROL]
scripts/qapi/visit.py +2 -2
index 66ce3dc52e..ce71659a03 100644
@@ -247,7 +247,7 @@ def gen_visit_alternate(name, variants):
     return ret
 
 
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -340,7 +340,7 @@ def visit_object_type(self, name, info, ifcond, features,
             if not name.startswith('q_'):
                 # only explicit types need an allocating visit
                 self._genh.add(gen_visit_decl(name))
-                self._genc.add(gen_visit_object(name, base, members, variants))
+                self._genc.add(gen_visit_object(name))
 
     def visit_alternate_type(self, name, info, ifcond, features, variants):
         with ifcontext(ifcond, self._genh, self._genc):
Signature · 1 line
2.26.2
author
#92 lore raw

PATCH 37/37 qapi/visit.py: add notational type hints

In reply to John Snow
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 72 +++++++++++++++++++++++++++++++++----------
 2 files changed, 55 insertions(+), 22 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 3babc380a8..5ab3433c5f 100644
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
scripts/qapi/visit.py +55 -17

163 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#94 lore raw
In reply to John Snow
Quoted 18 lines · John Snow

Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one single warning.

Quoted 9 lines

- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.

- After patch 7, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.

- After patch 16, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.

Addressed today's feedback and pushed to gitlab.
Let's call it "Version 1.3".

https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/

Waiting on feedback for 16-37 before re-authoring to list.

--js

#95 lore raw
Quoted 26 lines · John Snow
Quoted 15 lines · John Snow

Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files. The tougher files -- schema.py, expr.py, parser.py -- will each receive
their own series as a follow-up to this one.
Notes:
- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
- Any patch named "add notational type hints" changes ONLY signatures,
which have no runtime impact whatsoever. These are big patches, but fairly straightforward.
- Most other patches are as bite-sized as possible, generally fixing one
single warning.

Quoted 9 lines

- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.
- After patch 7, pylint --rcfile=qapi/pylintrc qapi/ should pass 100%
on this and every future commit.
- After patch 16, mypy --config-file=qapi/mypy.ini qapi/ should pass
100% on this and every future commit.

Quoted 6 lines

Addressed today's feedback and pushed to gitlab.
Let's call it "Version 1.3".

https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/

Waiting on feedback for 16-37 before re-authoring to list.

If 03-12 don't actually depend on Python 3.6, you may want to respin these for me to queue right away.

#96 lore raw
In reply to John Snow

Cc'ing Ben who offered to help with having the QEMU python scripts packaged (and this series is a step toward that).

Quoted 90 lines · John Snow

Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one single warning.

- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.

- After patch 7, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.

- After patch 16, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.

Preliminary refactoring and prerequisites:

001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools' 003/37: 'qapi-gen: Separate arg-parsing from generation' 004/37: 'qapi: move generator entrypoint into module' 005/37: 'qapi: Remove wildcard includes' 006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'

common.py (and params.py):

008/37: 'qapi/common.py: Remove python compatibility workaround' 009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable' 012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints' 014/37: 'qapi/common.py: Move comments into docstrings' 015/37: 'qapi/common.py: split build_params into new file' 016/37: 'qapi: establish mypy type-checking baseline'

events.py:

017/37: 'qapi/events.py: add notational type hints' 018/37: 'qapi/events.py: Move comments into docstrings'

commands.py:

019/37: 'qapi/commands.py: Don't re-bind to variable of different type' 020/37: 'qapi/commands.py: add notational type hints' 021/37: 'qapi/commands.py: enable checking with mypy'

source.py:

022/37: 'qapi/source.py: add notational type hints' 023/37: 'qapi/source.py: delint with pylint'

gen.py:

024/37: 'qapi/gen.py: Fix edge-case of _is_user_module' 025/37: 'qapi/gen.py: add notational type hints' 026/37: 'qapi/gen.py: Enable checking with mypy' 027/37: 'qapi/gen.py: Remove unused parameter' 028/37: 'qapi/gen.py: update write() to be more idiomatic' 029/37: 'qapi/gen.py: delint with pylint'

introspect.py:

030/37: 'qapi/introspect.py: Add a typed 'extra' structure' 031/37: 'qapi/introspect.py: add _gen_features helper' 032/37: 'qapi/introspect.py: create a typed 'Node' data structure' 033/37: 'qapi/introspect.py: add notational type hints'

types.py:

034/37: 'qapi/types.py: add notational type hints' 035/37: 'qapi/types.py: remove one-letter variables'

visit.py:

036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object' 037/37: 'qapi/visit.py: add notational type hints'

author
#97 lore raw
Quoted 3 lines · Philippe Mathieu-Daudé

Cc'ing Ben who offered to help with having the QEMU python scripts packaged (and this series is a step toward that).

FWIW, I want to move ./python/qemu/ into ./python/qemu/core, and then move ./scripts/qapi to ./python/qemu/qapi.

I will leave the qapi-gen.py stub script behind for compatibility.

Once I've done that, I can add package setup scripts.

I have a long series adding mypy type hints and so forth to ./python/qemu and adding packaging glue there too, but I shelved it as 5.1 released and right now I am focusing on QAPI.

Quoted 92 lines · John Snow
Quoted 90 lines

Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one single warning.

- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.

- After patch 7, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.

- After patch 16, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.

Preliminary refactoring and prerequisites:

001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools' 003/37: 'qapi-gen: Separate arg-parsing from generation' 004/37: 'qapi: move generator entrypoint into module' 005/37: 'qapi: Remove wildcard includes' 006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'

common.py (and params.py):

008/37: 'qapi/common.py: Remove python compatibility workaround' 009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable' 012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints' 014/37: 'qapi/common.py: Move comments into docstrings' 015/37: 'qapi/common.py: split build_params into new file' 016/37: 'qapi: establish mypy type-checking baseline'

events.py:

017/37: 'qapi/events.py: add notational type hints' 018/37: 'qapi/events.py: Move comments into docstrings'

commands.py:

019/37: 'qapi/commands.py: Don't re-bind to variable of different type' 020/37: 'qapi/commands.py: add notational type hints' 021/37: 'qapi/commands.py: enable checking with mypy'

source.py:

022/37: 'qapi/source.py: add notational type hints' 023/37: 'qapi/source.py: delint with pylint'

gen.py:

024/37: 'qapi/gen.py: Fix edge-case of _is_user_module' 025/37: 'qapi/gen.py: add notational type hints' 026/37: 'qapi/gen.py: Enable checking with mypy' 027/37: 'qapi/gen.py: Remove unused parameter' 028/37: 'qapi/gen.py: update write() to be more idiomatic' 029/37: 'qapi/gen.py: delint with pylint'

introspect.py:

030/37: 'qapi/introspect.py: Add a typed 'extra' structure' 031/37: 'qapi/introspect.py: add _gen_features helper' 032/37: 'qapi/introspect.py: create a typed 'Node' data structure' 033/37: 'qapi/introspect.py: add notational type hints'

types.py:

034/37: 'qapi/types.py: add notational type hints' 035/37: 'qapi/types.py: remove one-letter variables'

visit.py:

036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object' 037/37: 'qapi/visit.py: add notational type hints'