qapi: static typing conversion, pt1
cover qapi: static typing conversion, pt1
Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.
The tougher files -- schema.py, expr.py, parser.py -- will each receive their own series as a follow-up to this one.
Notes:
- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.
- Most other patches are as bite-sized as possible, generally fixing one single warning.
- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.
- After patch 7, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.
- After patch 16, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.
Preliminary refactoring and prerequisites:
001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools' 003/37: 'qapi-gen: Separate arg-parsing from generation' 004/37: 'qapi: move generator entrypoint into module' 005/37: 'qapi: Remove wildcard includes' 006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'
common.py (and params.py):
008/37: 'qapi/common.py: Remove python compatibility workaround' 009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable' 012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints' 014/37: 'qapi/common.py: Move comments into docstrings' 015/37: 'qapi/common.py: split build_params into new file' 016/37: 'qapi: establish mypy type-checking baseline'
events.py:
017/37: 'qapi/events.py: add notational type hints' 018/37: 'qapi/events.py: Move comments into docstrings'
commands.py:
019/37: 'qapi/commands.py: Don't re-bind to variable of different type' 020/37: 'qapi/commands.py: add notational type hints' 021/37: 'qapi/commands.py: enable checking with mypy'
source.py:
022/37: 'qapi/source.py: add notational type hints' 023/37: 'qapi/source.py: delint with pylint'
gen.py:
024/37: 'qapi/gen.py: Fix edge-case of _is_user_module' 025/37: 'qapi/gen.py: add notational type hints' 026/37: 'qapi/gen.py: Enable checking with mypy' 027/37: 'qapi/gen.py: Remove unused parameter' 028/37: 'qapi/gen.py: update write() to be more idiomatic' 029/37: 'qapi/gen.py: delint with pylint'
introspect.py:
030/37: 'qapi/introspect.py: Add a typed 'extra' structure' 031/37: 'qapi/introspect.py: add _gen_features helper' 032/37: 'qapi/introspect.py: create a typed 'Node' data structure' 033/37: 'qapi/introspect.py: add notational type hints'
types.py:
034/37: 'qapi/types.py: add notational type hints' 035/37: 'qapi/types.py: remove one-letter variables'
visit.py:
036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object' 037/37: 'qapi/visit.py: add notational type hints'
John Snow (37):
python: Require 3.6+
[DO-NOT-MERGE] qapi: add debugging tools
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into module
qapi: Remove wildcard includes
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable qapi/common.py: check with pylint
qapi/common.py: add notational type hints qapi/common.py: Move comments into docstrings qapi/common.py: split build_params into new file
qapi: establish mypy type-checking baseline qapi/events.py: add notational type hints qapi/events.py: Move comments into docstrings qapi/commands.py: Don't re-bind to variable of different type qapi/commands.py: add notational type hints qapi/commands.py: enable checking with mypy qapi/source.py: add notational type hints qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module qapi/gen.py: add notational type hints
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic qapi/gen.py: delint with pylint
qapi/introspect.py: Add a typed 'extra' structure qapi/introspect.py: add _gen_features helper qapi/introspect.py: create a typed 'Node' data structure qapi/introspect.py: add notational type hints qapi/types.py: add notational type hints qapi/types.py: remove one-letter variables qapi/visit.py: remove unused parameters from gen_visit_object qapi/visit.py: add notational type hints
docs/conf.py | 4 +- configure | 6 +- .readthedocs.yml | 2 +- .travis.yml | 8 -- scripts/qapi-gen.py | 59 ++-------- scripts/qapi/.flake8 | 2 + scripts/qapi/commands.py | 85 +++++++++++---- scripts/qapi/common.py | 161 +++++++++++++++------------ scripts/qapi/debug.py | 78 +++++++++++++ scripts/qapi/doc.py | 3 +- scripts/qapi/events.py | 61 ++++++++--- scripts/qapi/expr.py | 4 +- scripts/qapi/gen.py | 147 +++++++++++++------------ scripts/qapi/introspect.py | 200 +++++++++++++++++++++++----------- scripts/qapi/mypy.ini | 30 +++++ scripts/qapi/params.py | 40 +++++++ scripts/qapi/parser.py | 4 +- scripts/qapi/pylintrc | 71 ++++++++++++ scripts/qapi/schema.py | 33 +++--- scripts/qapi/script.py | 91 ++++++++++++++++ scripts/qapi/source.py | 34 +++--- scripts/qapi/types.py | 125 ++++++++++++++------- scripts/qapi/visit.py | 104 +++++++++++++----- tests/qemu-iotests/iotests.py | 2 - 24 files changed, 947 insertions(+), 407 deletions(-) create mode 100644 scripts/qapi/.flake8 create mode 100644 scripts/qapi/debug.py create mode 100644 scripts/qapi/mypy.ini create mode 100644 scripts/qapi/params.py create mode 100644 scripts/qapi/pylintrc create mode 100644 scripts/qapi/script.py
Signature · 1 line
2.26.2
PATCH 1/37 python: Require 3.6+
Python 3.5 is now EOL. Python 3.6 brings with it several nice things:
- Literal string interpolation (PEP 498)
- Syntax for variable annotations (PEP 526)
- Preserving keyword argument order (PEP 468) *
PEP 526 in particular will allow us to convert the QAPI module to the statically typed subset of the Python language. These static type hints improve the interactive editing experience in IDEs and help make refactoring and adding new features faster, easier and safer.
*Note: **kwargs is now guaranteed to preserve keyword ordering, but dict itself is still allowed to have an arbitrary order based on implementation. It is not until Python 3.7 that the dict type guarantees insertion order.
--- docs/conf.py | 4 ++-- configure | 6 +++--- .readthedocs.yml | 2 +- .travis.yml | 8 -------- tests/qemu-iotests/iotests.py | 2 -- 5 files changed, 6 insertions(+), 16 deletions(-)
docs/conf.py +2 -2
@@ -36,9 +36,9 @@ # In newer versions of Sphinx this will display nicely; in older versions # Sphinx will also produce a Python backtrace but at least the information # gets printed... -if sys.version_info < (3,5): +if sys.version_info < (3,6): raise ConfigError( - "QEMU requires a Sphinx that uses Python 3.5 or better\n") + "QEMU requires a Sphinx that uses Python 3.6 or better\n") # The per-manual conf.py will set qemu_docdir for a single-manual build; # otherwise set it here if this is an entire-manual-set build.
configure +3 -3
@@ -913,7 +913,7 @@ fi : ${make=${MAKE-make}} -# We prefer python 3.x. A bare 'python' is traditionally +# We prefer python 3.6+. A bare 'python' is traditionally # python 2.x, but some distros have it as python 3.x, so # we check that too python=
@@ -1961,8 +1961,8 @@ fi # Note that if the Python conditional here evaluates True we will exit # with status 1 which is a shell 'false' value. -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then - error_exit "Cannot use '$python', Python >= 3.5 is required." \ +if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then + error_exit "Cannot use '$python', Python >= 3.6 is required." \ "Use --python=/path/to/python to specify a supported Python." fi
.readthedocs.yml +1 -1
@@ -17,4 +17,4 @@ formats: all # we require for other Python in our codebase (our conf.py # enforces this, and some code needs it.) python: - version: 3.5 + version: 3.6
.travis.yml +0 -8
@@ -267,14 +267,6 @@ jobs: # Python builds - - name: "GCC Python 3.5 (x86_64-softmmu)" - env: - - CONFIG="--target-list=x86_64-softmmu" - - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" - language: python - python: 3.5 - - - name: "GCC Python 3.6 (x86_64-softmmu)" env: - CONFIG="--target-list=x86_64-softmmu"
tests/qemu-iotests/iotests.py +0 -2
@@ -40,8 +40,6 @@ from qemu import qtest from qemu.qmp import QMPMessage -assert sys.version_info >= (3, 6) - # Use this logger for logging messages directly from the iotests module logger = logging.getLogger('qemu.iotests') logger.addHandler(logging.NullHandler())
Signature · 1 line
2.26.2
Re: [PATCH 01/37] python: Require 3.6+
Quoted 46 lines · John Snow
Python 3.5 is now EOL. Python 3.6 brings with it several nice things:
- Literal string interpolation (PEP 498)
- Syntax for variable annotations (PEP 526)
- Preserving keyword argument order (PEP 468) *PEP 526 in particular will allow us to convert the QAPI module to the statically typed subset of the Python language. These static type hints improve the interactive editing experience in IDEs and help make refactoring and adding new features faster, easier and safer.
*Note: **kwargs is now guaranteed to preserve keyword ordering, but dict itself is still allowed to have an arbitrary order based on implementation. It is not until Python 3.7 that the dict type guarantees insertion order.
Signed-off-by John Snow <[email protected]>--- docs/conf.py | 4 ++-- configure | 6 +++--- .readthedocs.yml | 2 +- .travis.yml | 8 -------- tests/qemu-iotests/iotests.py | 2 -- 5 files changed, 6 insertions(+), 16 deletions(-)docs/conf.py +2
-2@@ -36,9 +36,9 @@ # In newer versions of Sphinx this will display nicely; in older versions # Sphinx will also produce a Python backtrace but at least the information # gets printed... -if sys.version_info < (3,5): +if sys.version_info < (3,6): raise ConfigError( - "QEMU requires a Sphinx that uses Python 3.5 or better\n") + "QEMU requires a Sphinx that uses Python 3.6 or better\n") # The per-manual conf.py will set qemu_docdir for a single-manual build; # otherwise set it here if this is an entire-manual-set build.configure +0
-0
configure fi · line 916
Quoted 5 lines
configure +1
-1@@ -913,7 +913,7 @@ fi : ${make=${MAKE-make}} -# We prefer python 3.x. A bare 'python' is traditionally +# We prefer python 3.6+. A bare 'python' is traditionally
We require
Quoted 56 lines
configure +2
-2# python 2.x, but some distros have it as python 3.x, so # we check that too python=@@ -1961,8 +1961,8 @@ fi # Note that if the Python conditional here evaluates True we will exit # with status 1 which is a shell 'false' value. -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then - error_exit "Cannot use '$python', Python >= 3.5 is required." \ +if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then + error_exit "Cannot use '$python', Python >= 3.6 is required." \ "Use --python=/path/to/python to specify a supported Python." fi.readthedocs.yml +1
-1@@ -17,4 +17,4 @@ formats: all # we require for other Python in our codebase (our conf.py # enforces this, and some code needs it.) python: - version: 3.5 + version: 3.6.travis.yml +0
-8@@ -267,14 +267,6 @@ jobs: # Python builds - - name: "GCC Python 3.5 (x86_64-softmmu)" - env: - - CONFIG="--target-list=x86_64-softmmu" - - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default" - language: python - python: 3.5 - - - name: "GCC Python 3.6 (x86_64-softmmu)" env: - CONFIG="--target-list=x86_64-softmmu"tests/qemu-iotests/iotests.py +0
-2@@ -40,8 +40,6 @@ from qemu import qtest from qemu.qmp import QMPMessage -assert sys.version_info >= (3, 6) - # Use this logger for logging messages directly from the iotests module logger = logging.getLogger('qemu.iotests') logger.addHandler(logging.NullHandler())
Re: [PATCH 01/37] python: Require 3.6+
In reply to John SnowQuoted 1 line · John Snow
Python 3.5 is now EOL. Python 3.6 brings with it several nice things:
NB Upstream EOL is not relevant. We need to base min version changes on the OS platform support matrix. So it would be better for the commit message to provide details of the version in the various distros we target, to show that the min version change is valid.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 2/37 qapi: add debugging tools
In reply to John SnowThis adds some really childishly simple debugging tools. Maybe they're interesting for someone else, too?
--- scripts/qapi/debug.py | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 scripts/qapi/debug.py
scripts/qapi/debug.py new file +78 -0
@@ -0,0 +1,78 @@ +""" +Small debugging facilities for mypy static analysis work. +(C) 2020 John Snow, for Red Hat, Inc. +""" + +import inspect +import json +from typing import Dict, List, Any +from types import FrameType + + +OBSERVED_TYPES: Dict[str, List[str]] = {} + + +# You have no idea how long it took to find this return type... +def caller_frame() -> FrameType: + """ + Returns the stack frame of the caller's caller. + e.g. foo() -> caller() -> caller_frame() return's foo's stack frame. + """ + stack = inspect.stack() + caller = stack[2].frame + if caller is None: + msg = "Python interpreter does not support stack frame inspection" + raise RuntimeError(msg) + return caller + + +def _add_type_record(name: str, typestr: str) -> None: + seen = OBSERVED_TYPES.setdefault(name, []) + if typestr not in seen: + seen.append(typestr) + + +def record_type(name: str, value: Any, dict_names: bool = False) -> None: + """ + Record the type of a variable. + + :param name: The name of the variable + :param value: The value of the variable + """ + _add_type_record(name, str(type(value))) + + try: + for key, subvalue in value.items(): + subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]" + _add_type_record(subname, str(type(subvalue))) + return + except AttributeError: + # (Wasn't a dict or anything resembling one.) + pass + + # str is iterable, but not in the way we want! + if isinstance(value, str): + return + + try: + for elem in value: + _add_type_record(f"{name}.[list_elem]", str(type(elem))) + except TypeError: + # (Wasn't a list or anything else iterable.) + pass + + +def show_types() -> None: + """ + Print all of the currently known variable types to stdout. + """ + print(json.dumps(OBSERVED_TYPES, indent=2)) + + +def record_locals(show: bool = False, dict_names: bool = False) -> None: + caller = caller_frame() + name = caller.f_code.co_name + for key, value in caller.f_locals.items(): + record_type(f"{name}.{key}", value, dict_names=dict_names) + if show: + show_types()
Signature · 1 line
2.26.2
PATCH 3/37 qapi-gen: Separate arg-parsing from generation
In reply to John SnowThis is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.
--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)
scripts/qapi-gen.py +63 -24
@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re
@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + gen_doc(schema, output_dir, prefix) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',
@@ -33,26 +81,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) - gen_doc(schema, args.output_dir, args.prefix) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())
Signature · 1 line
2.26.2
PATCH 4/37 qapi: move generator entrypoint into module
In reply to John SnowAs part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.
As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.
This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."
--- scripts/qapi-gen.py | 94 +++----------------------------------- scripts/qapi/commands.py | 4 +- scripts/qapi/doc.py | 2 +- scripts/qapi/events.py | 8 ++-- scripts/qapi/expr.py | 4 +- scripts/qapi/gen.py | 4 +- scripts/qapi/introspect.py | 8 ++-- scripts/qapi/parser.py | 4 +- scripts/qapi/schema.py | 8 ++-- scripts/qapi/script.py | 91 ++++++++++++++++++++++++++++++++++++ scripts/qapi/types.py | 6 +-- scripts/qapi/visit.py | 6 +-- 12 files changed, 124 insertions(+), 115 deletions(-) create mode 100644 scripts/qapi/script.py
scripts/qapi-gen.py +6 -88
@@ -1,97 +1,15 @@ #!/usr/bin/env python3 - -# This work is licensed under the terms of the GNU GPL, version 2 or later. -# See the COPYING file in the top-level directory. - """ -QAPI Generator +QAPI code generation execution shim. -This script is the main entry point for generating C code from the QAPI schema. +This file exists primarily to facilitate the running of the QAPI code +generator without needing to install the python module to the current +execution environment. """ -import argparse -import re import sys -from qapi.commands import gen_commands -from qapi.doc import gen_doc -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit - - -DEFAULT_OUTPUT_DIR = '' -DEFAULT_PREFIX = '' - - -def generate(schema_file: str, - output_dir: str, - prefix: str, - unmask: bool = False, - builtins: bool = False) -> None: - """ - generate uses a given schema to produce C code in the target directory. - - :param schema_file: The primary QAPI schema file. - :param output_dir: The output directory to store generated code. - :param prefix: Optional C-code prefix for symbol names. - :param unmask: Expose non-ABI names through introspection? - :param builtins: Generate code for built-in types? - - :raise QAPIError: On failures. - """ - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) - if match and match.end() != len(prefix): - msg = "funny character '{:s}' in prefix '{:s}'".format( - prefix[match.end()], prefix) - raise QAPIError('', None, msg) - - schema = QAPISchema(schema_file) - gen_types(schema, output_dir, prefix, builtins) - gen_visit(schema, output_dir, prefix, builtins) - gen_commands(schema, output_dir, prefix) - gen_events(schema, output_dir, prefix) - gen_introspect(schema, output_dir, prefix, unmask) - gen_doc(schema, output_dir, prefix) - - -def main() -> int: - """ - gapi-gen shell script entrypoint. - Expects arguments via sys.argv, see --help for details. - - :return: int, 0 on success, 1 on failure. - """ - parser = argparse.ArgumentParser( - description='Generate code from a QAPI schema') - parser.add_argument('-b', '--builtins', action='store_true', - help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', - default=DEFAULT_OUTPUT_DIR, - help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', - default=DEFAULT_PREFIX, - help="prefix for symbols") - parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', - dest='unmask', - help="expose non-ABI names in introspection") - parser.add_argument('schema', action='store') - args = parser.parse_args() - - try: - generate(args.schema, - output_dir=args.output_dir, - prefix=args.prefix, - unmask=args.unmask, - builtins=args.builtins) - except QAPIError as err: - print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) - return 1 - return 0 - +from qapi import script if __name__ == '__main__': - sys.exit(main()) + sys.exit(script.main())
scripts/qapi/commands.py +2 -2
@@ -13,8 +13,8 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import * +from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/doc.py +1 -1
@@ -5,7 +5,7 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from qapi.gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc, QAPISchemaVisitor MSG_FMT = """
scripts/qapi/events.py +4 -4
@@ -12,10 +12,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember -from qapi.types import gen_enum, gen_enum_lookup +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember +from .types import gen_enum, gen_enum_lookup def build_event_send_proto(name, arg_type, boxed):
scripts/qapi/expr.py +2 -2
@@ -16,8 +16,8 @@ import re from collections import OrderedDict -from qapi.common import c_name -from qapi.error import QAPISemError +from .common import c_name +from .error import QAPISemError # Names must be letters, numbers, -, and _. They must start with letter,
scripts/qapi/gen.py +2 -2
@@ -17,8 +17,8 @@ import re from contextlib import contextmanager -from qapi.common import * -from qapi.schema import QAPISchemaVisitor +from .common import * +from .schema import QAPISchemaVisitor class QAPIGen:
scripts/qapi/introspect.py +4 -4
@@ -10,10 +10,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaMonolithicCVisitor -from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .common import * +from .gen import QAPISchemaMonolithicCVisitor +from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, + QAPISchemaType) def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/parser.py +2 -2
@@ -18,8 +18,8 @@ import re from collections import OrderedDict -from qapi.error import QAPIParseError, QAPISemError -from qapi.source import QAPISourceInfo +from .error import QAPIParseError, QAPISemError +from .source import QAPISourceInfo class QAPISchemaParser:
scripts/qapi/schema.py +4 -4
@@ -18,10 +18,10 @@ import re from collections import OrderedDict -from qapi.common import c_name, pointer_suffix -from qapi.error import QAPIError, QAPISemError -from qapi.expr import check_exprs -from qapi.parser import QAPISchemaParser +from .common import c_name, pointer_suffix +from .error import QAPIError, QAPISemError +from .expr import check_exprs +from .parser import QAPISchemaParser class QAPISchemaEntity:
scripts/qapi/script.py new file +91 -0
91 lines not on this page · raw message
scripts/qapi/types.py +3 -3
@@ -13,9 +13,9 @@ # See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember, QAPISchemaObjectType # variants must be emitted before their container; track what has already
scripts/qapi/visit.py +3 -3
@@ -13,9 +13,9 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
Signature · 1 line
2.26.2
Re: [PATCH 04/37] qapi: move generator entrypoint into module
Quoted 6 lines · John Snow
As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.
As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.
Should this be split into two commits, one for each of the paragraphs above?
PEP 8 recommends absolute imports, with one exception:
However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:
from . import sibling
from .sibling import example
Standard library code should avoid complex package layouts and always use absolute imports.
Do you think it covers your use of relative imports?
Quoted 3 lines
This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."
Can you give me an example of some tool having troube?
Quoted 16 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 94 +++----------------------------------- scripts/qapi/commands.py | 4 +- scripts/qapi/doc.py | 2 +- scripts/qapi/events.py | 8 ++-- scripts/qapi/expr.py | 4 +- scripts/qapi/gen.py | 4 +- scripts/qapi/introspect.py | 8 ++-- scripts/qapi/parser.py | 4 +- scripts/qapi/schema.py | 8 ++-- scripts/qapi/script.py | 91 ++++++++++++++++++++++++++++++++++++ scripts/qapi/types.py | 6 +-- scripts/qapi/visit.py | 6 +-- 12 files changed, 124 insertions(+), 115 deletions(-) create mode 100644 scripts/qapi/script.py
Naming is hard... main.py?
scripts/qapi-gen.py line 5
Quoted 11 lines
scripts/qapi-gen.py +0
-4@@ -1,97 +1,15 @@ #!/usr/bin/env python3 - -# This work is licensed under the terms of the GNU GPL, version 2 or later. -# See the COPYING file in the top-level directory. -
Keep the license blurb.
[...]
Re: [PATCH 04/37] qapi: move generator entrypoint into module
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · John Snow
As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.
Quoted 6 lines
Quoted 2 lines
As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.
Should this be split into two commits, one for each of the paragraphs above?
Hmm ... I hadn't considered it was possible, but actually ... I guess I can split those out, yeah.
Quoted 6 lines
PEP 8 recommends absolute imports, with one exception:
However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:
Quoted 8 lines
from . import sibling
from .sibling import exampleStandard library code should avoid complex package layouts and always use absolute imports.
Do you think it covers your use of relative imports?
Admittedly I am going by trial and error; my experience is that the relative imports behave the nicest.
There is a historical fear of explicit relative imports because they are "new" and years of Python2 compatibility rendered many afraid of them. It is advice safely ignored in my opinion.
Using absolute imports (e.g. from qapi.sibling import foo) gets messy in virtual environments when you have *installed* the module in question: it becomes ambiguous as to *which* qapi you meant: the one in this folder, or the one installed to the environment?
Python, mypy, pylint, flake8 etc disagree sometimes, or can get confused; thinking there are two copies of each module.
Just my experience that relative imports seem to give me the least trouble.
Quoted 6 lines
Quoted 3 lines
This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."
Can you give me an example of some tool having troube?
I'd have to code up some examples. I have some hobby code unrelated to QEMU where I struggled to get flake8, mypy, and pylint all cooperating with an import regime until I gave up and used explicit relative imports.
Quoted 17 lines
Quoted 15 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 94 +++----------------------------------- scripts/qapi/commands.py | 4 +- scripts/qapi/doc.py | 2 +- scripts/qapi/events.py | 8 ++-- scripts/qapi/expr.py | 4 +- scripts/qapi/gen.py | 4 +- scripts/qapi/introspect.py | 8 ++-- scripts/qapi/parser.py | 4 +- scripts/qapi/schema.py | 8 ++-- scripts/qapi/script.py | 91 ++++++++++++++++++++++++++++++++++++ scripts/qapi/types.py | 6 +-- scripts/qapi/visit.py | 6 +-- 12 files changed, 124 insertions(+), 115 deletions(-)Quoted 1 line
create mode 100644
scripts/qapi/script.py
Quoted 2 lines
Naming is hard...
main.py?
I was thinking of changing this myself, so this convinced me.
Quoted 14 lines
scripts/qapi-gen.py line 5
Quoted 11 lines
scripts/qapi-gen.py +0
-4@@ -1,97 +1,15 @@ #!/usr/bin/env python3 - -# This work is licensed under the terms of the GNU GPL, version 2 or later. -# See the COPYING file in the top-level directory. -Keep the license blurb.
This is a mistake. I tried to convince git to "move" the old file and then add a "new" file to preserve history, but of course that's not how git manages file histories, so it didn't work.
TLDR: I didn't delete the license blurb, I just didn't "add" it again. I'll "fix" that.
Quoted 2 lines
[...]
Re: [PATCH 04/37] qapi: move generator entrypoint into module
Quoted 29 lines · John Snow
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · John Snow
As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the module, only leaving a very tiny entrypoint shim outside of the module.
Quoted 6 lines
Quoted 2 lines
As a result, all of the include statements are reworked to be module-aware, as explicit relative imports.
Should this be split into two commits, one for each of the paragraphs
above?Hmm ... I hadn't considered it was possible, but actually ... I guess I can split those out, yeah.
Quoted 11 lines
PEP 8 recommends absolute imports, with one exception:
However, explicit relative imports are an acceptable
alternative to
absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose:
from . import sibling
from .sibling import example
Standard library code should avoid complex package layouts and always use absolute imports.
Do you think it covers your use of relative imports?
Quoted 5 lines
Admittedly I am going by trial and error; my experience is that the relative imports behave the nicest.
There is a historical fear of explicit relative imports because they are "new" and years of Python2 compatibility
Having to struggle with that for years was bound to damage brains.
Quoted 16 lines
rendered many afraid of
them. It is advice safely ignored in my opinion.Using absolute imports (e.g. from qapi.sibling import foo) gets messy in virtual environments when you have *installed* the module in
question: it becomes ambiguous as to *which* qapi you meant: the one in this folder, or the one installed to the environment?Python, mypy, pylint, flake8 etc disagree sometimes, or can get confused; thinking there are two copies of each module.
Just my experience that relative imports seem to give me the least trouble.
Quoted 3 lines
Quoted 3 lines
This is done primarily for the benefit of python tooling (pylint, mypy, flake8, et al) which otherwise has trouble consistently resolving "qapi.x" to mean "a sibling file in this folder."
Quoted 6 lines
Quoted 2 lines
Can you give me an example of some tool having troube?
I'd have to code up some examples. I have some hobby code unrelated to QEMU where I struggled to get flake8, mypy, and pylint all cooperating with an import regime until I gave up and used explicit relative imports.
I'd make the switch when we actually do run into trouble.
But I'm willing to take your advice and switch now.
[...]
PATCH 5/37 qapi: Remove wildcard includes
In reply to John SnowWildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
Remove them and include things explicitly by name instead.
--- scripts/qapi/commands.py | 6 +++++- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 42 insertions(+), 8 deletions(-)
scripts/qapi/commands.py +5 -1
@@ -13,7 +13,11 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_name, + mcgen, +) from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
scripts/qapi/events.py +6 -1
@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +9 -3
@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitor
scripts/qapi/introspect.py +6 -1
@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)
scripts/qapi/types.py +7 -1
@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
scripts/qapi/visit.py +9 -1
@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
Signature · 1 line
2.26.2
PATCH 6/37 qapi: delint using flake8
In reply to John SnowPetty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 15 ++++++++++----- 4 files changed, 19 insertions(+), 9 deletions(-) create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Pylint handles this, but smarter.
scripts/qapi/commands.py +2 -1
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp) { Visitor *v;
scripts/qapi/schema.py +5 -3
@@ -536,7 +536,7 @@ def set_defined_in(self, name): v.set_defined_in(name) def check(self, schema, seen): - if not self.tag_member: # flat union + if not self.tag_member: # flat union self.tag_member = seen.get(c_name(self._tag_name)) base = "'base'" # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname): self._entity_dict = {} self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) - self._make_module(None) # built-ins + self._make_module(None) # built-ins self._make_module(fname) self._predefining = True self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members): # But it's not tight: the disjunction need not imply it. We # may end up compiling useless wrapper types. # TODO kill simple unions or implement the disjunction - assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access + + # pylint: disable=protected-access + assert (ifcond or []) == typ._ifcond else: self._def_entity(QAPISchemaObjectType( name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +10 -5
@@ -31,7 +31,8 @@ def gen_visit_decl(name, scalar=False): if not scalar: c_type += '*' return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp); +bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, + Error **errp); ''', c_name=c_name(name), c_type=c_type)
@@ -125,7 +126,8 @@ def gen_visit_object_members(name, base, members, variants): def gen_visit_list(name, element_type): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, + Error **errp) { bool ok = false; %(c_name)s *tail;
@@ -158,7 +160,8 @@ def gen_visit_list(name, element_type): def gen_visit_enum(name): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, + Error **errp) { int value = *obj; bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +175,8 @@ def gen_visit_enum(name): def gen_visit_alternate(name, variants): ret = mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, + Error **errp) { bool ok = false;
@@ -247,7 +251,8 @@ def gen_visit_alternate(name, variants): def gen_visit_object(name, base, members, variants): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, + Error **errp) { bool ok = false;
Signature · 1 line
2.26.2
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 19 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 15 ++++++++++----- 4 files changed, 19 insertions(+), 9 deletions(-) create mode 100644 scripts/qapi/.flake8scripts/qapi/.flake8 new file +0
-0
scripts/qapi/.flake8 line 2
Quoted 2 lines
scripts/qapi/.flake8 new file +2
-0@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Pylint handles this, but smarter.
I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with flake8's E722 compared to pylint's W0702?
Quoted 1 line
\ No newline at end of file
So put one there :)
scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 73
Quoted 11 lines
scripts/qapi/commands.py +2
-1@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp)
The continued parameter list may become misalignd in generated C. E.g.
static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
Error **errp)
{
...
}Do we care?
More of the same below.
Quoted 3 lines
scripts/qapi/commands.py +0
-0{ Visitor *v;
[...]
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 15 lines · Markus Armbruster
Quoted 13 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 15 ++++++++++----- 4 files changed, 19 insertions(+), 9 deletions(-)create mode 100644
scripts/qapi/.flake8
Quoted 12 lines
scripts/qapi/.flake8 line 2
Quoted 8 lines
scripts/qapi/.flake8 new file +2
-0@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Pylint handles this, but smarter.I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with flake8's E722 compared to pylint's W0702?
Flake8 will warn on *any* bare except, but Pylint's is context-aware and will suppress the warning if you re-raise the exception.
I don't actually think this comes up in the qapi code base, but it does come up in the ./python/qemu code base.
(One of my goals is unifying the lint checking regime for both packages.)
Quoted 4 lines
Quoted 1 line
\ No newline at end of file
So put one there :)
Whupps, okay.
Quoted 14 lines
scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 73
Quoted 11 lines
scripts/qapi/commands.py +2
-1@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp)The continued parameter list may become misalignd in generated C. E.g.
Quoted 8 lines
static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out, Error **errp) { ... }Do we care?
Yeah, I don't know. Do we?
It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.
Path of least resistance led me here, but I can try and appease both systems if you'd prefer.
Quoted 7 lines
More of the same below.
Quoted 3 lines
scripts/qapi/commands.py +0
-0{ Visitor *v;[...]
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 29 lines · John Snow
Quoted 15 lines · Markus Armbruster
Quoted 13 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 15 ++++++++++----- 4 files changed, 19 insertions(+), 9 deletions(-)create mode 100644
scripts/qapi/.flake8Quoted 12 lines
scripts/qapi/.flake8 line 2
Quoted 8 lines
scripts/qapi/.flake8 new file +2
-0@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Pylint handles this, but smarter.I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with
flake8's E722 compared to pylint's W0702?
Quoted 2 lines
Flake8 will warn on *any* bare except, but Pylint's is context-aware and will suppress the warning if you re-raise the exception.
Should this information be worked into the comment?
Quoted 33 lines
I don't actually think this comes up in the qapi code base, but it does come up in the ./python/qemu code base.
(One of my goals is unifying the lint checking regime for both packages.)
Quoted 3 lines
Quoted 1 line
\ No newline at end of file
So put one there :)
Whupps, okay.
Quoted 15 lines
scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74
Quoted 11 lines
scripts/qapi/commands.py +2
-0@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp)The continued parameter list may become misalignd in generated C. E.g.
static void qmp_marshal_output_BlockInfoList(BlockInfoList*ret_in, QObject **ret_out,Quoted 6 lines
Error **errp) { ... }Do we care?
Quoted 1 line
Yeah, I don't know. Do we?
I care, but I also care for automated style checks.
Quoted 5 lines
It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.
Path of least resistance led me here, but I can try and appease both systems if you'd prefer.
Up to now, I ran the style checkers manually, and this was just one of several complaints to ignore, so I left the code alone.
If it gets in the way of running them automatically, and messing up the generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.
Quoted 7 lines
Quoted 7 lines
More of the same below.
Quoted 3 lines
scripts/qapi/commands.py +0
-0{ Visitor *v;[...]
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · John Snow
Quoted 27 lines · Markus Armbruster
Quoted 19 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 15 ++++++++++----- 4 files changed, 19 insertions(+), 9 deletions(-)create mode 100644
scripts/qapi/.flake8scripts/qapi/.flake8 new file +0
-0scripts/qapi/.flake8 line 2
Quoted 2 lines
scripts/qapi/.flake8 new file +2
-0@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Pylint handles this, but smarter.I guess you mean pylint's W0702 a.k.a. bare-except. What's wrong with
flake8's E722 compared to pylint's W0702?
Quoted 5 lines
Quoted 2 lines
Flake8 will warn on *any* bare except, but Pylint's is context-aware and will suppress the warning if you re-raise the exception.
Should this information be worked into the comment?
I'll improve it a little, but I'll add what I wrote above to the commit message.
Quoted 43 lines
Quoted 33 lines
I don't actually think this comes up in the qapi code base, but it does come up in the ./python/qemu code base.
(One of my goals is unifying the lint checking regime for both packages.)
Quoted 3 lines
Quoted 1 line
\ No newline at end of file
So put one there :)
Whupps, okay.
Quoted 15 lines
scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74
Quoted 11 lines
scripts/qapi/commands.py +2
-0@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp)The continued parameter list may become misalignd in generated C. E.g.
static void qmp_marshal_output_BlockInfoList(BlockInfoList*ret_in, QObject **ret_out,Quoted 6 lines
Error **errp) { ... }Do we care?
Quoted 1 line
Yeah, I don't know. Do we?
I care, but I also care for automated style checks.
Quoted 5 lines
It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.
Path of least resistance led me here, but I can try and appease both systems if you'd prefer.
Quoted 7 lines
Up to now, I ran the style checkers manually, and this was just one of several complaints to ignore, so I left the code alone.
If it gets in the way of running them automatically, and messing up the generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.
I changed this a little to put all the args on the next line, which is slightly unusual but works okay.
I think that's a fine middle ground, because the alternative (to me) is to start using abstracted code generation tokens in a tree structure, etc etc etc.
Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
More of the same below.
Quoted 3 lines
scripts/qapi/commands.py +0
-0{ Visitor *v;[...]
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 11 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
Signed-off-by John Snow <[email protected]>
[...]
Quoted 38 lines
Quoted 29 lines
Quoted 21 lines
Quoted 13 lines
scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74
Quoted 11 lines
scripts/qapi/commands.py +2
-0@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp)The continued parameter list may become misalignd in generated C. E.g.
Quoted 7 lines
static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out, Error **errp) { ... }Do we care?
Quoted 1 line
Yeah, I don't know. Do we?
I care, but I also care for automated style checks.
Quoted 5 lines
It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.
Path of least resistance led me here, but I can try and appease both systems if you'd prefer.
Quoted 8 lines
Up to now, I ran the style checkers manually, and this was just one of
several complaints to ignore, so I left the code alone. If it gets in the way of running them automatically, and messing up the
generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.
Quoted 2 lines
I changed this a little to put all the args on the next line, which is slightly unusual but works okay.
I think it's slightly more unusual than the non-matching indentation was.
Yet another way to skin this cat:
static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
QObject **ret_out, Error **errp)
Now the second line is not aligned with the left parenthesis both in the Python source and in the generated C.
Quoted 9 lines
I think that's a fine middle ground, because the alternative (to me) is to start using abstracted code generation tokens in a tree structure, etc etc etc.
Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.
The thought "screw this, pile the output through /usr/bin/indent" has crossed my mind more than once.
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 58 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
Signed-off-by John Snow <[email protected]>[...]
Quoted 38 lines
Quoted 29 lines
Quoted 21 lines
Quoted 13 lines
scripts/qapi/commands.py def gen_call(name, arg_type, boxed, ret_type): · line 74
Quoted 11 lines
scripts/qapi/commands.py +2
-0@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, + Error **errp)The continued parameter list may become misalignd in generated C. E.g.
Quoted 7 lines
static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out, Error **errp) { ... }Do we care?
Quoted 1 line
Yeah, I don't know. Do we?
I care, but I also care for automated style checks.
Quoted 5 lines
It actually seemed more annoying to try and get flake8 to make an exception for these handful of examples.
Path of least resistance led me here, but I can try and appease both systems if you'd prefer.
Quoted 8 lines
Up to now, I ran the style checkers manually, and this was just one of
several complaints to ignore, so I left the code alone. If it gets in the way of running them automatically, and messing up the
generated code slightly is the easiest way to get it out of the way, then I can accept the slight mess.Quoted 2 lines
I changed this a little to put all the args on the next line, which is slightly unusual but works okay.
I think it's slightly more unusual than the non-matching indentation was.
Quoted 8 lines
Yet another way to skin this cat:
static void qmp_marshal_output_%(c_name)s(%(c_type)s
ret_in,
QObject **ret_out, Error **errp)Now the second line is not aligned with the left parenthesis both in the Python source and in the generated C.
Your wish is my command!
(Except in the other replies where I am arguing with you.)
Quoted 4 lines
Quoted 4 lines
I think that's a fine middle ground, because the alternative (to me) is to start using abstracted code generation tokens in a tree structure, etc etc etc.
Quoted 10 lines
Quoted 5 lines
Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.
The thought "screw this, pile the output through
/usr/bin/indent" has crossed my mind more than once.
Bigger fish to fry, but with other languages like rust looming, making the core generation facilities nicer might be ...nice. Not for this series.
Two approaches in general would make sense to me:
1. Building an AST for C instead of strings, and rendering the AST.
2. Writing a templating engine that doesn't break the Python indentation flow by hoisting them into module constants and improving the rendering logic.
--js
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 2 lines · John Snow
Quoted 1 line · Markus Armbruster
John Snow <[email protected]> writes:
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Embedded templates are always gonna look kinda nasty, I think, because you're trying to fight style guidelines in two languages simultaneously and it's never gonna quite work out exactly how you want without some pretty complex abstraction mechanisms that are well beyond the power we need right now.
Quoted 8 lines
Quoted 4 lines
The thought "screw this, pile the output through
/usr/bin/indent" has
crossed my mind more than once.Bigger fish to fry, but with other languages like rust looming, making the core generation facilities nicer might be ...nice. Not for this series.
Definitely not.
The way we generate C now is (close to) the stupidest way that could possibly work. Stupid is *good*, until it creates actual problems.
Any improvement will have to balance readability vs. hackability.
What we have now is often hard to read, but pretty easy to hack. May sound like a contradiction, but it's been my experience.
Quoted 3 lines
Two approaches in general would make sense to me:
1. Building an AST for C instead of strings, and rendering the AST.
I fear the need to define an AST for C makes this uneconomical.
I've missed the ease of generating Lisp in Lisp many times.
Quoted 3 lines
2. Writing a templating engine that doesn't break the Python indentation flow by hoisting them into module constants and improving the rendering logic.
Too vague for me to judge, happy to look at PoC patches.
Re: [PATCH 06/37] qapi: delint using flake8
Quoted 12 lines · Markus Armbruster
Quoted 2 lines · John Snow
Quoted 1 line · Markus Armbruster
John Snow <[email protected]> writes:
[...]
Quoted 3 lines
2. Writing a templating engine that doesn't break the Python indentation flow by hoisting them into module constants and improving the rendering logic.
Too vague for me to judge, happy to look at PoC patches.
Okay, some other day. Not important enough presently. It's just hard to stop brainstorming ideas as I learn about and touch different areas of code.
--js
PATCH 7/37 qapi: add pylintrc
In reply to John SnowAdd a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.
Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.
Therefore, to run it, you must:
Quoted 2 lines
cd :/qemu/scripts
pylint qapi/ --rcfile=qapi/pylintrc
--- scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +74 -0
@@ -0,0 +1,74 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the blacklist. The +# regex matches against base names, not paths. +ignore-patterns=common.py, + doc.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py, + + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes, + +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _ + +[VARIABLES] + +[STRING] + +[SPELLING] + +[FORMAT] + +[SIMILARITIES] + +# Ignore imports when computing similarities. +ignore-imports=yes + +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]
Signature · 1 line
2.26.2
Re: [PATCH 07/37] qapi: add pylintrc
Quoted 8 lines · John Snow
Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.
Quoted 7 lines
Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.
Therefore, to run it, you must:
Quoted 1 line
cd :/qemu/scripts
-bash: cd: :/qemu/scripts: No such file or directory
;-P
Quoted 1 line
Quoted 1 line
pylint qapi/ --rcfile=qapi/pylintrc
Why not
$ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc
Quoted 17 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 scripts/qapi/pylintrcscripts/qapi/pylintrc new file +4
-0@@ -0,0 +1,74 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the blacklist. The +# regex matches against base names, not paths.
scripts/qapi/pylintrc line 14
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+ignore-patterns=common.py, + doc.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py,
Already not ignored:
__init__.py
commands.py
common.py
debug.py
events.py
introspect.py
script.pyOkay.
Quoted 9 lines
scripts/qapi/pylintrc new file +9
-0+ + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if
scripts/qapi/pylintrc line 33
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes,
I'm fine with disabling these.
Quoted 8 lines
scripts/qapi/pylintrc new file +8
-0+ +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING]
scripts/qapi/pylintrc line 51
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+ +[BASIC] + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _
Isn't this the default?
scripts/qapi/pylintrc line 64
Quoted 13 lines
scripts/qapi/pylintrc new file +13
-0+ +[VARIABLES] + +[STRING] + +[SPELLING] + +[FORMAT] + +[SIMILARITIES] + +# Ignore imports when computing similarities. +ignore-imports=yes
Why?
scripts/qapi/pylintrc line 74
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+ +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]
Looks like you started with output of --generate-rcfile,
Re: [PATCH 07/37] qapi: add pylintrc
Quoted 20 lines · Markus Armbruster
Quoted 8 lines · John Snow
Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.
Quoted 7 lines
Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.
Therefore, to run it, you must:
Quoted 1 line
cd :/qemu/scripts
-bash: cd: :/qemu/scripts: No such file or directory
Quoted 8 lines
;-P
Quoted 1 line
Quoted 1 line
pylint qapi/ --rcfile=qapi/pylintrc
Why not
$ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc
No reason I'm aware of, I have just been testing with CWD at the scripts dir myself because of how python imports work.
If it works this way, enjoy!
Quoted 53 lines
Quoted 17 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+)create mode 100644
scripts/qapi/pylintrcscripts/qapi/pylintrc new file +4
-0@@ -0,0 +1,74 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the blacklist. The +# regex matches against base names, not paths.scripts/qapi/pylintrc line 14
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+ignore-patterns=common.py, + doc.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py,Already not ignored:
__init__.py commands.py common.py debug.py events.py introspect.py script.pyOkay.
Quoted 13 lines
scripts/qapi/pylintrc new file +13
-0+ + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W".
Quoted 9 lines
scripts/qapi/pylintrc line 33
Quoted 6 lines
scripts/qapi/pylintrc new file +6
-0+disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes,I'm fine with disabling these.
I'd like to enable missing-docstring eventually, but that's not for today.
Quoted 12 lines
Quoted 12 lines
scripts/qapi/pylintrc new file +12
-0+ +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] + +# Good variable names which should always be accepted, separated by a comma.
Quoted 9 lines
scripts/qapi/pylintrc line 51
Quoted 6 lines
scripts/qapi/pylintrc new file +6
-0+good-names=i, + j, + k, + ex, + Run, + _Isn't this the default?
Yes. I could omit it until I need to use good-names later on in the series, but I thought it would look odd to add the defaults at that point.
So it's a minor bit of prescience here.
Quoted 7 lines
Quoted 7 lines
scripts/qapi/pylintrc new file +7
-0+ +[VARIABLES] + +[STRING] + +[SPELLING] +
Quoted 9 lines
scripts/qapi/pylintrc line 64
Quoted 6 lines
scripts/qapi/pylintrc new file +6
-0+[FORMAT] + +[SIMILARITIES] + +# Ignore imports when computing similarities. +ignore-imports=yesWhy?
We don't care if import statements are similar to those in other files. It's uninteresting entirely.
(It matches on from typing import ... that exceed four lines, which I do regularly by the end of the series.)
Quoted 4 lines
Quoted 4 lines
scripts/qapi/pylintrc new file +4
-0+ +[TYPECHECK] + +[CLASSES]
Quoted 9 lines
scripts/qapi/pylintrc line 74
Quoted 6 lines
scripts/qapi/pylintrc new file +6
-0+ +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]Looks like you started with output of --generate-rcfile,
I did,
Re: [PATCH 07/37] qapi: add pylintrc
Quoted 22 lines · John Snow
Quoted 21 lines · Markus Armbruster
Quoted 8 lines · John Snow
Add a skeleton pylintrc file. Right now, it ignores quite a few things. Files will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder now.
Quoted 7 lines
Due to some bugs in different versions of pylint (2.5.x), pylint does not correctly recognize when it is being run from "inside" a module, and must be run *outside* of the module.
Therefore, to run it, you must:
Quoted 1 line
cd :/qemu/scripts
-bash: cd: :/qemu/scripts: No such file or directory ;-P
Quoted 1 line
Quoted 1 line
pylint qapi/ --rcfile=qapi/pylintrc
Quoted 8 lines
Quoted 3 lines
Why not
$ pylint scripts/qapi --rcfile=scripts/qapi/pylintrcNo reason I'm aware of, I have just been testing with CWD at the scripts dir myself because of how python imports work.
If it works this way, enjoy!
It works, and it simplifies the commmit message.
Quoted 58 lines
Quoted 48 lines
Quoted 16 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+)create mode 100644
scripts/qapi/pylintrcscripts/qapi/pylintrc new file +4
-0@@ -0,0 +1,74 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the blacklist. The +# regex matches against base names, not paths.scripts/qapi/pylintrc line 14
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+ignore-patterns=common.py, + doc.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py,Already not ignored:
__init__.py commands.py common.py debug.py events.py introspect.py script.pyOkay.
Quoted 12 lines
scripts/qapi/pylintrc new file +12
-0+ + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classesQuoted 9 lines
scripts/qapi/pylintrc line 33
Quoted 7 lines
scripts/qapi/pylintrc new file +7
-0+# --disable=W". +disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes,I'm fine with disabling these.
Quoted 1 line
I'd like to enable missing-docstring eventually, but that's not for today.
Understood.
Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
scripts/qapi/pylintrc new file +11
-0+ +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] +Quoted 9 lines
scripts/qapi/pylintrc line 51
Quoted 7 lines
scripts/qapi/pylintrc new file +7
-0+# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _Isn't this the default?
Quoted 5 lines
Yes. I could omit it until I need to use good-names later on in the series, but I thought it would look odd to add the defaults at that point.
So it's a minor bit of prescience here.
Matter of taste. No objection.
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
scripts/qapi/pylintrc new file +6
-0+ +[VARIABLES] + +[STRING] + +[SPELLING]Quoted 9 lines
scripts/qapi/pylintrc line 64
Quoted 7 lines
scripts/qapi/pylintrc new file +7
-0+ +[FORMAT] + +[SIMILARITIES] + +# Ignore imports when computing similarities. +ignore-imports=yesWhy?
Quoted 5 lines
We don't care if import statements are similar to those in other files. It's uninteresting entirely.
(It matches on from typing import ... that exceed four lines, which I do regularly by the end of the series.)
What about something like
# Ignore imports when computing similarities, because import # statements being similar is uninteresting entirely
Quoted 13 lines
Quoted 12 lines
scripts/qapi/pylintrc line 74
Quoted 10 lines
scripts/qapi/pylintrc new file +10
-0+ +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]Looks like you started with output of --generate-rcfile,
Quoted 1 line
I did,
Let's mention that in the commit message. Education opportunity :)
Re: [PATCH 07/37] qapi: add pylintrc
Quoted 10 lines · Markus Armbruster
Quoted 5 lines
We don't care if import statements are similar to those in other files. It's uninteresting entirely.
(It matches on from typing import ... that exceed four lines, which I do regularly by the end of the series.)
What about something like
# Ignore imports when computing similarities, because import # statements being similar is uninteresting entirely
OK. I left the "default" comments that pylint itself wrote. I'm now writing:
"Ignore import statements themselves when computing similarities."
PATCH 8/37 qapi/common.py: Remove python compatibility workaround
In reply to John Snow--- scripts/qapi/common.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +1 -4
@@ -119,10 +119,7 @@ def cgen(code, **kwds): raw = code % kwds if indent_level: indent = genindent(indent_level) - # re.subn() lacks flags support before Python 2.7, use re.compile() - raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE), - indent, raw) - raw = raw[0] + raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
Signature · 1 line
2.26.2
Re: [PATCH 08/37] qapi/common.py: Remove python compatibility workaround
Quoted 10 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)scripts/qapi/common.py +0
-0
scripts/qapi/common.py def cgen(code, **kwds): · line 123
Quoted 9 lines
scripts/qapi/common.py +1
-4@@ -119,10 +119,7 @@ def cgen(code, **kwds): raw = code % kwds if indent_level: indent = genindent(indent_level) - # re.subn() lacks flags support before Python 2.7, use re.compile() - raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE), - indent, raw) - raw = raw[0] + raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
I missed this one in my "qapi: Bye-bye Python 2" series.
Can we use re.sub() instead?
Re: [PATCH 08/37] qapi/common.py: Remove python compatibility workaround
Quoted 24 lines · Markus Armbruster
Quoted 10 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)scripts/qapi/common.py +0
-0scripts/qapi/common.py def cgen(code, **kwds): · line 123
Quoted 9 lines
scripts/qapi/common.py +1
-4@@ -119,10 +119,7 @@ def cgen(code, **kwds): raw = code % kwds if indent_level: indent = genindent(indent_level) - # re.subn() lacks flags support before Python 2.7, use re.compile() - raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE), - indent, raw) - raw = raw[0] + raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)I missed this one in my "qapi: Bye-bye Python 2" series.
Quoted 2 lines
Can we use
re.sub()instead?
Yup.
PATCH 9/37 qapi/common.py: Add indent manager
In reply to John SnowCode style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
--- scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 19 deletions(-)
scripts/qapi/common.py +35 -15
@@ -93,33 +93,53 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indent: + """ + Indent-level management. + :param initial: Initial value, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + """Return the indent as an integer.""" + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the indent as a string.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indent.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def push(self, amount: int = 4) -> int: + """Push `amount` spaces onto the indent, default four.""" + self._level += amount + return self._level + + def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self))) + self._level -= amount + return self._level + + +INDENT = Indent(0) # Generate @code with @kwds interpolated. # Obey indent_level, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if INDENT: + raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
scripts/qapi/visit.py +3 -4
@@ -19,8 +19,7 @@ gen_endif, gen_if, mcgen, - pop_indent, - push_indent, + INDENT, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + INDENT.push() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;
@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + INDENT.pop() ret += mcgen(''' } ''')
Signature · 1 line
2.26.2
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 18 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 19 deletions(-)scripts/qapi/common.py +0
-0
scripts/qapi/common.py def c_name(name, protect=True): · line 96
Quoted 9 lines
scripts/qapi/common.py +1
-5@@ -93,33 +93,53 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indent:
Since this class name will be used just once... Indentation or IndentationManager?
scripts/qapi/common.py def c_name(name, protect=True): · line 100
Quoted 4 lines
scripts/qapi/common.py +3
-0+ """ + Indent-level management. + :param initial: Initial value, default 0.
The only caller passes 0.
Let's drop the parameter, and hardcode the default initial value until we have an actual use for another initial value.
scripts/qapi/common.py def c_name(name, protect=True): · line 106
Quoted 7 lines
scripts/qapi/common.py +5
-1+ """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + """Return the indent as an integer."""
Isn't "as an integer" obvious?
Let's replace "the indent" by "the indentation" globally.
scripts/qapi/common.py def c_name(name, protect=True): · line 110
Quoted 4 lines
scripts/qapi/common.py +3
-0+ return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level)
Is __repr__ needed?
scripts/qapi/common.py def c_name(name, protect=True): · line 118
Quoted 11 lines
scripts/qapi/common.py +6
-3-def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the indent as a string.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indent.""" + return bool(self._level)
This one lets us shorten
if int(INDENT):
to
if INDENT:
There's just one instance. Marginal. I'm not rejecting it.
Quoted 5 lines
scripts/qapi/common.py +2
-3-def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def push(self, amount: int = 4) -> int: + """Push `amount` spaces onto the indent, default four."""
scripts/qapi/common.py def c_name(name, protect=True): · line 131
Quoted 10 lines
scripts/qapi/common.py +10
-0+ self._level += amount + return self._level + + def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self))) + self._level -= amount + return self._level
The push / pop names never made sense. The functions don't push onto / pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.
The @amount parameter has never been used. I don't mind keeping it.
scripts/qapi/common.py def c_name(name, protect=True): · line 134
Quoted 3 lines
scripts/qapi/common.py +3
-0+ + +INDENT = Indent(0)
Uh, does this really qualify as a constant? It looks quite variable to me...
Quoted 44 lines
scripts/qapi/common.py +2
-3# Generate @code with @kwds interpolated. # Obey indent_level, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if INDENT: + raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)scripts/qapi/visit.py +3
-4@@ -19,8 +19,7 @@ gen_endif, gen_if, mcgen, - pop_indent, - push_indent, + INDENT, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + INDENT.push() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + INDENT.pop() ret += mcgen(''' } ''')
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 23 lines · Markus Armbruster
Quoted 21 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 19 deletions(-)scripts/qapi/common.py +0
-0@@ -93,33 +93,53 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace
Quoted 10 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 96
Quoted 6 lines
scripts/qapi/common.py +1
-5-def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indent:Since this class name will be used just once... Indentation or IndentationManager?
Indentation is fine, if you'd like. IndentationManager makes me feel ashamed for writing this patch, like I am punishing myself with Java.
Quoted 10 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 100
Quoted 4 lines
scripts/qapi/common.py +3
-0+ """ + Indent-level management. + :param initial: Initial value, default 0.The only caller passes 0.
Let's drop the parameter, and hardcode the default initial value until we have an actual use for another initial value.
The parameter is nice because it gives meaning to the output of repr(), see below.
Quoted 9 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 106
Quoted 7 lines
scripts/qapi/common.py +5
-1+ """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + """Return the indent as an integer."""Isn't "as an integer" obvious?
Just a compulsion to write complete sentences that got lost in the sea of many patches. I'll nix this one, because dunder methods do not need to be documented.
Quoted 3 lines
Let's replace "the indent" by "the indentation" globally.
They're both cromulent as nouns and one is shorter.
I'll switch in good faith; do you prefer "Indentation" as a noun?
Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 110
Quoted 4 lines
scripts/qapi/common.py +3
-0+ return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level)Is
__repr__needed?
Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.
repr(Indent(4)) == "Indent(4)"
Quoted 14 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 118
Quoted 11 lines
scripts/qapi/common.py +6
-3-def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the indent as a string.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indent.""" + return bool(self._level)This one lets us shorten
Quoted 8 lines
if int(INDENT):
to
if INDENT:
There's just one instance. Marginal. I'm not rejecting it.
Yep, it's trivial in either direction. Still, because it's a custom type, I thought I'd be courteous and have it support bool().
Quoted 11 lines
Quoted 11 lines
scripts/qapi/common.py +8
-3-def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def push(self, amount: int = 4) -> int: + """Push `amount` spaces onto the indent, default four.""" + self._level += amount + return self._level + + def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount:
Quoted 9 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 131
Quoted 4 lines
scripts/qapi/common.py +4
-0+ raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self))) + self._level -= amount + return self._levelThe push / pop names never made sense. The functions don't push onto / pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.
OK.
I briefly thought about using __isub__ and __iadd__ to support e.g. indent += 4, but actually it'd be annoying to have to specify 4 everywhere.
Since you didn't suggest anything, I am going to change it to 'increase' and 'decrease'.
Quoted 2 lines
The @amount parameter has never been used. I don't mind keeping it.
I'll keep it, because I like having the default amount show up in the signature instead of as a magic constant in the function body.
Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 134
Quoted 3 lines
scripts/qapi/common.py +3
-0+ + +INDENT = Indent(0)Uh, does this really qualify as a constant? It looks quite variable to me...
Ever make a mistake? I thought I did once, but I was mistaken.
Quoted 44 lines
Quoted 44 lines
scripts/qapi/common.py +2
-3# Generate @code with @kwds interpolated. # Obey indent_level, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if INDENT: + raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)scripts/qapi/visit.py +3
-4@@ -19,8 +19,7 @@ gen_endif, gen_if, mcgen, - pop_indent, - push_indent, + INDENT, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + INDENT.push() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + INDENT.pop() ret += mcgen(''' } ''')
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 33 lines · John Snow
Quoted 22 lines · Markus Armbruster
Quoted 20 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 19 deletions(-)scripts/qapi/common.py +0
-0@@ -93,33 +93,53 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspaceQuoted 9 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 96
Quoted 6 lines
scripts/qapi/common.py +1
-5-def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indent:Since this class name will be used just once... Indentation or IndentationManager?
Quoted 2 lines
Indentation is fine, if you'd like. IndentationManager makes me feel ashamed for writing this patch, like I am punishing myself with Java.
Hah! Point taken.
Quoted 24 lines
Quoted 8 lines
scripts/qapi/common.py def c_name(name, protect=True):
Quoted 3 lines
scripts/qapi/common.py +2
-0+ """ + Indent-level management. + :param initial: Initial value, default 0.The only caller passes 0.
Let's drop the parameter, and hardcode the default initial value until
we have an actual use for another initial value.The parameter is nice because it gives meaning to the output of
repr(), see below.Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 106
Quoted 6 lines
scripts/qapi/common.py +5
-0+ """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + """Return the indent as an integer."""Isn't "as an integer" obvious?
Just a compulsion to write complete sentences that got lost in the sea of many patches. I'll nix this one, because dunder methods do not need to be documented.
Quoted 6 lines
Quoted 2 lines
Let's replace "the indent" by "the indentation" globally.
They're both cromulent as nouns and one is shorter.
I'll switch in good faith; do you prefer "Indentation" as a noun?
Use of "indent" as a noun was new to me, but what do I know; you're the native speaker. Use your judgement. Applies to the class name, too.
Quoted 10 lines
Quoted 5 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 110
Quoted 3 lines
scripts/qapi/common.py +2
-0+ return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level)Is
__repr__needed?Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.
repr(Indent(4)) == "Indent(4)"
Meh. There's another three dozen classes for you to put lipstick on :)
Quoted 19 lines
Quoted 9 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 118
Quoted 9 lines
scripts/qapi/common.py +5
-2-def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the indent as a string.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indent.""" + return bool(self._level)Quoted 6 lines
This one lets us shorten
if int(INDENT):
to
if INDENT:
There's just one instance. Marginal. I'm not rejecting it.Yep, it's trivial in either direction. Still, because it's a custom type, I thought I'd be courteous and have it support
bool().
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
scripts/qapi/common.py +2
-3-def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def push(self, amount: int = 4) -> int: + """Push `amount` spaces onto the indent, default four."""Quoted 8 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 129
Quoted 8 lines
scripts/qapi/common.py +8
-0+ self._level += amount + return self._level + + def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
I think assert(amount <= self._level) would do just fine.
Quoted 8 lines
Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 131
Quoted 2 lines
scripts/qapi/common.py +2
-0+ self._level -= amount + return self._levelThe push / pop names never made sense. The functions don't push onto /
pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.
Quoted 8 lines
OK.
I briefly thought about using
__isub__and__iadd__to support e.g. indent += 4, but actually it'd be annoying to have to specify 4 everywhere.Since you didn't suggest anything, I am going to change it to 'increase' and 'decrease'.
Works for me. So would inc and dec.
Quoted 13 lines
Quoted 2 lines
The @amount parameter has never been used. I don't mind keeping it.
I'll keep it, because I like having the default amount show up in the signature instead of as a magic constant in the function body.
Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 134
Quoted 3 lines
scripts/qapi/common.py +3
-0+ + +INDENT = Indent(0)Uh, does this really qualify as a constant? It looks quite variable to
me...
Quoted 1 line
Ever make a mistake? I thought I did once, but I was mistaken.
"If I had any humility, I'd be perfect!"
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 6 lines · Markus Armbruster
John Snow <[email protected]> writes:
Let's replace "the indent" by "the indentation" globally.
Quoted 7 lines
Quoted 3 lines
They're both cromulent as nouns and one is shorter.
I'll switch in good faith; do you prefer "Indentation" as a noun?
Use of "indent" as a noun was new to me, but what do I know; you're the native speaker. Use your judgement. Applies to the class name, too.
I made the change; see gitlab commits or wait for v2 to see if it reads better to you.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 110
Quoted 3 lines
scripts/qapi/common.py +2
-0+ return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level)Is
__repr__needed?
Quoted 7 lines
Quoted 4 lines
Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.
repr(Indent(4)) == "Indent(4)"
Meh. There's another three dozen classes for you to put lipstick on :)
We'll get to them in due time. For now, please admire the lipstick.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 129
Quoted 5 lines
scripts/qapi/common.py +5
-0+ def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))I think
assert(amount <= self._level)would do just fine.
Secretly, asserts can be disabled in Python just like they can in C code.
My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.
(See the gitlab branch for the updated version of this patch, which has changed the code slightly.)
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 131
Quoted 2 lines
scripts/qapi/common.py +2
-0+ self._level -= amount + return self._levelThe push / pop names never made sense. The functions don't push onto /
pop from a stack, they increment / decrement a counter, and so do the methods. Good opportunity to fix the naming.OK.
Quoted 9 lines
Quoted 6 lines
I briefly thought about using
__isub__and__iadd__to support e.g. indent += 4, but actually it'd be annoying to have to specify 4 everywhere.Since you didn't suggest anything, I am going to change it to 'increase' and 'decrease'.
Works for me. So would inc and dec.
increase and decrease it is.
Quoted 13 lines
Quoted 13 lines
Quoted 2 lines
The @amount parameter has never been used. I don't mind keeping it.
I'll keep it, because I like having the default amount show up in the signature instead of as a magic constant in the function body.
Quoted 7 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 134
Quoted 3 lines
scripts/qapi/common.py +3
-0+ + +INDENT = Indent(0)Uh, does this really qualify as a constant? It looks quite variable to
me...
Quoted 4 lines
Quoted 1 line
Ever make a mistake? I thought I did once, but I was mistaken.
"If I had any humility, I'd be perfect!"
:)
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 31 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
Quoted 6 lines · Markus Armbruster
John Snow <[email protected]> writes:
Let's replace "the indent" by "the indentation" globally.
Quoted 7 lines
Quoted 3 lines
They're both cromulent as nouns and one is shorter.
I'll switch in good faith; do you prefer "Indentation" as a noun?
Use of "indent" as a noun was new to me, but what do I know; you're the
native speaker. Use your judgement. Applies to the class name, too.I made the change; see gitlab commits or wait for v2 to see if it reads better to you.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 110
Quoted 3 lines
scripts/qapi/common.py +2
-0+ return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level)Is
__repr__needed?Yes; it's used in the underflow exception , and it is nice when using the python shell interactively.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
repr(Indent(4)) == "Indent(4)"
Meh. There's another three dozen classes for you to put lipstick on :)
We'll get to them in due time. For now, please admire the lipstick.
If I take off my glasses and step six feet back, I just might be able to overlook it.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 129
Quoted 5 lines
scripts/qapi/common.py +5
-0+ def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))I think
assert(amount <= self._level)would do just fine.Secretly, asserts can be disabled in Python just like they can in C code.
There are compilers that let you switch off array bounds checking. Would you advocate manual bounds checking to protect people from their own folly?
Quoted 4 lines
My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.
I check function preconditions ruthlessly with assert. There's no sane way to recover anyway.
Without a way to recover, the only benefit is crashing more prettily. If the error is six levels deep in a some fancy-pants framework, then prettier crashes might actually help someone finding the error slightly faster. But it ain't.
My final argument is local consistency: use of assertions to check preconditions is pervasive in scripts/qapi/.
Quoted 2 lines
(See the gitlab branch for the updated version of this patch, which has changed the code slightly.)
[...]
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 7 lines · Markus Armbruster
Quoted 1 line · John Snow
We'll get to them in due time. For now, please admire the lipstick.
If I take off my glasses and step six feet back, I just might be able to overlook it.
I consider writing a nice __repr__ good habit, I'd prefer not to delete it just because the rest of our code doesn't do so yet. (Give me time.)
I spend a lot of my time in the interactive python console: having nice __repr__ methods is a good habit, not an unsightly blemish.
Quoted 22 lines
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 129
Quoted 5 lines
scripts/qapi/common.py +5
-0+ def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))I think
assert(amount <= self._level)would do just fine.Secretly, asserts can be disabled in Python just like they can in C code.
There are compilers that let you switch off array bounds checking. Would you advocate manual bounds checking to protect people from their own folly?
Quoted 4 lines
My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.
I check function preconditions ruthlessly with assert. There's no sane way to recover anyway.
Quoted 8 lines
Without a way to recover, the only benefit is crashing more prettily. If the error is six levels deep in a some fancy-pants framework, then prettier crashes might actually help someone finding the error slightly faster. But it ain't.
My final argument is local consistency: use of assertions to check preconditions is pervasive in
scripts/qapi/.
You're right that there's no safe recovery from an error such as this. The only actual difference is whether you see AssertionError or ArithmeticError.
One can be disabled (But you rightly shouldn't), the other can't. One has more semantic meaning and information to it.
I prefer what I've currently written.
--js
Re: [PATCH 09/37] qapi/common.py: Add indent manager
Quoted 45 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 1 line · John Snow
We'll get to them in due time. For now, please admire the lipstick.
If I take off my glasses and step six feet back, I just might be able to
overlook it.I consider writing a nice
__repr__good habit, I'd prefer not to delete it just because the rest of our code doesn't do so yet. (Give me time.)I spend a lot of my time in the interactive python console: having nice
__repr__methods is a good habit, not an unsightly blemish.Quoted 17 lines
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ def pop(self, amount: int = 4) -> int: + """Pop `amount` spaces off of the indent, default four.""" + if self._level < amount: + raise ArithmeticError( + "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))I think
assert(amount <= self._level)would do just fine.Secretly, asserts can be disabled in Python just like they can in C code.
There are compilers that let you switch off array bounds checking. Would you advocate manual bounds checking to protect people from their own folly?
Quoted 4 lines
My habit: if it's something that should already be true given the nature of how the code is laid out, use an assertion. If I am preventing an erroneous state (Especially from callers in other modules), explicitly raise an exception.
Quoted 11 lines
I check function preconditions ruthlessly with assert. There's no sane
way to recover anyway.
Without a way to recover, the only benefit is crashing more prettily.
If the error is six levels deep in a some fancy-pants framework, then prettier crashes might actually help someone finding the error slightly faster. But it ain't.
My final argument is local consistency: use of assertions to check preconditions is pervasive inscripts/qapi/.
Quoted 6 lines
You're right that there's no safe recovery from an error such as this. The only actual difference is whether you see AssertionError or ArithmeticError.
One can be disabled (But you rightly shouldn't), the other can't. One has more semantic meaning and information to it.
YAGNI.
Quoted 1 line
I prefer what I've currently written.
Where personal preference collides with local consistency, I'm with local consistency.
You can get the two in line: change everything to your preference.
You signalled intent to do that for __repr__(): "Give me time". Alright, having such __repr__() is obviously more important / useful to you than avoiding the extra code is to me.
I received no such signal for checking preconditions. Good, because I'd go "are you serious?" :)
PATCH 10/37 qapi/common.py: delint with pylint
In reply to John Snow--- scripts/qapi/common.py | 16 +++++++--------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 14 insertions(+), 16 deletions(-)
scripts/qapi/common.py +7 -9
@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +c_name_trans = str.maketrans('.-', '__') + + # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__') - - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True): return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indent: """ Indent-level management.
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey INDENT level, and strip EATSPACE. def cgen(code, **kwds): raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):
scripts/qapi/schema.py +7 -7
@@ -18,7 +18,7 @@ import re from collections import OrderedDict -from .common import c_name, pointer_suffix +from .common import c_name, POINTER_SUFFIX from .error import QAPIError, QAPISemError from .expr import check_exprs from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self): return True def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'array'
@@ -430,7 +430,7 @@ def c_name(self): def c_type(self): assert not self.is_implicit() - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def c_unboxed_type(self): return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None): v.connect_doc(doc) def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type): self._make_array_type(name, None) def _def_predefineds(self): - for t in [('str', 'string', 'char' + pointer_suffix), + for t in [('str', 'string', 'char' + POINTER_SUFFIX), ('number', 'number', 'double'), ('int', 'int', 'int64_t'), ('int8', 'int', 'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self): ('uint64', 'int', 'uint64_t'), ('size', 'int', 'uint64_t'), ('bool', 'boolean', 'bool'), - ('any', 'value', 'QObject' + pointer_suffix), - ('null', 'null', 'QNull' + pointer_suffix)]: + ('any', 'value', 'QObject' + POINTER_SUFFIX), + ('null', 'null', 'QNull' + POINTER_SUFFIX)]: self._def_builtin_type(*t) self.the_empty_object_type = QAPISchemaObjectType( 'q_empty', None, None, None, None, None, [], None)
Signature · 1 line
2.26.2
Re: [PATCH 10/37] qapi/common.py: delint with pylint
Quoted 11 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 16 +++++++--------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 14 insertions(+), 16 deletions(-)scripts/qapi/common.py +0
-0
scripts/qapi/common.py line 21
Quoted 8 lines
scripts/qapi/common.py +5
-0@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +c_name_trans = str.maketrans('.-', '__') + +
You rename and move. pylint gripes about the names, but it doesn't actually ask for the move, as far as I can tell. Can you explain why you move?
Quoted 25 lines
scripts/qapi/common.py +0
-7# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__') - - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.@@ -89,10 +91,6 @@ def c_name(name, protect=True): return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indent: """ Indent-level management.
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 136
Quoted 5 lines
scripts/qapi/common.py +1
-1@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey INDENT level, and strip EATSPACE.
Is the change to INDENT intentional?
Quoted 9 lines
scripts/qapi/common.py +1
-1def cgen(code, **kwds): raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):
[...]
Re: [PATCH 10/37] qapi/common.py: delint with pylint
Quoted 16 lines · Markus Armbruster
Quoted 14 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 16 +++++++--------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 14 insertions(+), 16 deletions(-)scripts/qapi/common.py +0
-0@@ -14,6 +14,11 @@ import re
Quoted 10 lines
scripts/qapi/common.py line 21
Quoted 5 lines
scripts/qapi/common.py +5
-0+EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +c_name_trans = str.maketrans('.-', '__') + +You rename and move. pylint gripes about the names, but it doesn't actually ask for the move, as far as I can tell. Can you explain why you move?
Preference. I like constants and globals at the top so you can audit any code that runs at import time in one place. Since they are externally visible objects, having them near other "header" style information makes sense to me.
Quoted 8 lines
scripts/qapi/common.py def c_enum_const(type_name, const_name, prefix=None): · line 45
Quoted 8 lines
scripts/qapi/common.py +0
-1# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__')
(This one winds up being a constant, so I renamed it in my v2.)
Quoted 19 lines
Quoted 19 lines
scripts/qapi/common.py +0
-6- - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.@@ -89,10 +91,6 @@ def c_name(name, protect=True): return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indent: """ Indent-level management.@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
Quoted 6 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 136
Quoted 3 lines
scripts/qapi/common.py +1
-1# Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey INDENT level, and strip EATSPACE.Is the change to INDENT intentional?
Kind of, but it's either late (should have been with the indent manager patch) or early (Should be with the patch that moves comments into docstrings.)
When this comment becomes a docstring, I use INDENT to indicate it as a proper object. This in and of itself is prescient, as we are not using sphinx/apidoc to generate any documentation about the QAPI package yet.
(The pending v2 uses indent after you pointed out that it was not a constant.)
Quoted 11 lines
Quoted 9 lines
scripts/qapi/common.py +1
-1def cgen(code, **kwds): raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):[...]
Re: [PATCH 10/37] qapi/common.py: delint with pylint
Quoted 26 lines · John Snow
Quoted 15 lines · Markus Armbruster
Quoted 13 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 16 +++++++--------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 14 insertions(+), 16 deletions(-)scripts/qapi/common.py +0
-0@@ -14,6 +14,11 @@ import reQuoted 9 lines
scripts/qapi/common.py line 21
Quoted 5 lines
scripts/qapi/common.py +5
-0+EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +c_name_trans = str.maketrans('.-', '__') + +You rename and move. pylint gripes about the names, but it doesn't actually ask for the move, as far as I can tell. Can you explain why you move?
Quoted 2 lines
Preference. I like constants and globals at the top so you can audit any code that runs at import time in one place.
I can buy this argument.
Quoted 3 lines
Since they are
externally visible objects, having them near other "header" style information makes sense to me.
This one I find unconvincing. Functions and classes are just as visible.
Mention the move in the commit message, along with the (convincing part of the) rationale.
Aside: EATSPACE and c_name_trans are actually local to common.py. EATSPACE sort of leaks out only via the contract of mcgen(). The contract could be rephrased in terms of POINTER_SUFFIX. Not sure it's worthwhile.
Quoted 55 lines
Quoted 7 lines
scripts/qapi/common.py def c_enum_const(type_name, const_name, prefix=None): · line 44
Quoted 7 lines
scripts/qapi/common.py +0
-1# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__')(This one winds up being a constant, so I renamed it in my v2.)
Quoted 17 lines
Quoted 17 lines
scripts/qapi/common.py +0
-6- - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.@@ -89,10 +91,6 @@ def c_name(name, protect=True): return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indent: """ Indent-level management.@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:Quoted 5 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 136
Quoted 3 lines
scripts/qapi/common.py +1
-1# Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey INDENT level, and strip EATSPACE.Is the change to INDENT intentional?
Kind of, but it's either late (should have been with the indent manager patch) or early (Should be with the patch that moves comments into docstrings.)
When this comment becomes a docstring, I use
INDENTto indicate it as a proper object. This in and of itself is prescient, as we are not using sphinx/apidoc to generate any documentation about the QAPI package yet.(The pending v2 uses
indentafter you pointed out that it was not a constant.)Quoted 10 lines
Quoted 8 lines
scripts/qapi/common.py +1
-1def cgen(code, **kwds): raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):[...]
PATCH 11/37 qapi/common.py: Replace one-letter 'c' variable
In reply to John Snow--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_': + char = c_fun_str[i] + # When char is upper and no '_' appears before, do more checks + if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_': if i < length - 1 and c_fun_str[i + 1].islower(): new_name += '_' elif c_fun_str[i - 1].isdigit(): new_name += '_' - new_name += c + new_name += char return new_name.lstrip('_').upper()
Signature · 1 line
2.26.2
Re: [PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable
Quoted 10 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)scripts/qapi/common.py +0
-0
scripts/qapi/common.py def camel_to_upper(value): · line 34
Quoted 8 lines
scripts/qapi/common.py +2
-3@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_': + char = c_fun_str[i] + # When char is upper and no '_' appears before, do more checks
Good opportunity to tweak the language: "When char is upper case".
Quoted 8 lines
scripts/qapi/common.py +2
-1+ if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_': if i < length - 1 and c_fun_str[i + 1].islower(): new_name += '_' elif c_fun_str[i - 1].isdigit(): new_name += '_' - new_name += c + new_name += char return new_name.lstrip('_').upper()
Re: [PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Quoted 16 lines
scripts/qapi/common.py def camel_to_upper(value): · line 34
Quoted 13 lines
scripts/qapi/common.py +2
-3@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_': + char = c_fun_str[i] + # When char is upper and no '_' appears before, do more checksGood opportunity to tweak the language: "When char is upper case".
ACK, change made.
PATCH 12/37 qapi/common.py: check with pylint
In reply to John SnowRemove qapi/common.py from the pylintrc ignore list.
--- scripts/qapi/pylintrc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi/pylintrc +1 -2
@@ -2,8 +2,7 @@ # Add files or directories matching the regex patterns to the blacklist. The # regex matches against base names, not paths. -ignore-patterns=common.py, - doc.py, +ignore-patterns=doc.py, error.py, expr.py, gen.py,
Signature · 1 line
2.26.2
PATCH 13/37 qapi/common.py: add notational type hints
In reply to John Snow--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +16 -11
@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Union, Sequence EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. # Obey INDENT level, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: Union[str, int]) -> str: raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: Union[str, int]) -> str: if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s
@@ -160,7 +163,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */
@@ -168,7 +171,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''
@@ -177,7 +180,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''
@@ -186,7 +189,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:
Signature · 1 line
2.26.2
Re: [PATCH 13/37] qapi/common.py: add notational type hints
Question on the subject line: what makes a type hint notational?
Quoted 47 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +6
-3@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Union, Sequence EATSPACE = '\033EATSPACE.'@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 140
Quoted 5 lines
scripts/qapi/common.py +1
-1@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. # Obey INDENT level, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: Union[str, int]) -> str:
Hmm.
The @kwds values can be anything, provided they match the conversion specifiers in @code:
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141
Quoted 1 line
scripts/qapi/common.py +0
-0raw = code % kwds
Your type hint adds a restriction that wasn't there before.
Is there a better way?
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147
Quoted 7 lines
scripts/qapi/common.py +1
-1if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: Union[str, int]) -> str:
Likewise.
Quoted 53 lines
scripts/qapi/common.py +8
-6if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s@@ -160,7 +163,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */@@ -168,7 +171,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''@@ -177,7 +180,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''@@ -186,7 +189,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:
Re: [PATCH 13/37] qapi/common.py: add notational type hints
Quoted 2 lines · Markus Armbruster
Question on the subject line: what makes a type hint notational?
My cover letter explains that every time I use this phrase, I mean to state that "This patch adds exclusively type notations and makes no functional changes to the runtime operation whatsoever."
i.e. notations-only.
Quoted 57 lines
Quoted 47 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +6
-3@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Union, Sequence EATSPACE = '\033EATSPACE.'@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 140
Quoted 5 lines
scripts/qapi/common.py +1
-1@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. # Obey INDENT level, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: Union[str, int]) -> str:Hmm.
Quoted 9 lines
The @kwds values can be anything, provided they match the conversion specifiers in @code:
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141
Quoted 1 line
scripts/qapi/common.py +0
-0raw = code % kwdsYour type hint adds a restriction that wasn't there before.
Is there a better way?
Maybe there are format-like type annotation tricks you can do to enforce this, but I did not research them. I tried to resist "improving" our usage of the old % formatter prematurely. I may do a wholesale f-string conversion at some point, but not now, it's not important.
In practice, we pass strings and integers. This typing *is* artificially restrictive, though. We can declare the type to be "Any" and allow the function to fail or succeed at runtime if you'd prefer.
Quoted 10 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147
Quoted 7 lines
scripts/qapi/common.py +1
-1if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: Union[str, int]) -> str:Likewise.
Unresearched idea: It's possible that we can subclass the string.Formatter class and extend it to perform our special variable replacements (chomping EATSPACE, etc.)
And *maybe* because it inherits from the standard formatter, we would benefit from any analysis Mypy performs on such things.
Basically, replace mcgen/cgen with class CFormatter(string.Formatter).
(maybe. assume that none of what I just said will work or is feasible.)
--js
Re: [PATCH 13/37] qapi/common.py: add notational type hints
Quoted 3 lines · John Snow
Your type hint adds a restriction that wasn't there before.
Is there a better way?
I've settled on using the object type for now, which is slightly more restrictive than Any.
Any and object both allow any type of argument, but Any effectively creates a gradually typed boundary in which you are implicitly casting to whatever typed boundary it enters next.
object is an explicit cast to the most abstracted type; any further usage that is not supported the base object will be rejected.
This is nice, because if someone adds a call at or below the mcgen level, mypy will certainly complain that they are assuming too much about the type -- and they would be!
The rest of my series will now take this approach: prefer object to Any.
--js
Re: [PATCH 13/37] qapi/common.py: add notational type hints
In reply to John SnowQuoted 9 lines · John Snow
Quoted 2 lines · Markus Armbruster
Question on the subject line: what makes a type hint notational?
My cover letter explains that every time I use this phrase, I mean to state that "This patch adds exclusively type notations and makes no functional changes to the runtime operation whatsoever."
i.e. notations-only.
By the time I get to PATCH 13, details explained in the cover letter have been flushed from my memory. Moreover, the cover letter won't make it into Git. Best to repeat them right in the commit message. Perhaps:
qapi/common; Add type hints
Type hints do not change behavior.
Quoted 82 lines
Quoted 54 lines
Quoted 44 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +6
-3@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Union, Sequence EATSPACE = '\033EATSPACE.'@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 139
Quoted 4 lines
scripts/qapi/common.py +1
-1@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. # Obey INDENT level, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: Union[str, int]) -> str:Hmm.
The @kwds values can be anything, provided they match the conversion specifiers in @code:Quoted 4 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141
Quoted 1 line
scripts/qapi/common.py +0
-0raw = code % kwdsYour type hint adds a restriction that wasn't there before. Is there a better way?
Maybe there are format-like type annotation tricks you can do to enforce this, but I did not research them. I tried to resist "improving" our usage of the old % formatter prematurely. I may do a wholesale f-string conversion at some point, but not now, it's not important.
In practice, we pass strings and integers. This typing *is* artificially restrictive, though. We can declare the type to be "Any" and allow the function to fail or succeed at runtime if you'd prefer.
Quoted 8 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147
Quoted 6 lines
scripts/qapi/common.py +1
-1if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: Union[str, int]) -> str:Likewise.
Unresearched idea: It's possible that we can subclass the string.Formatter class and extend it to perform our special variable replacements (chomping EATSPACE, etc.)
Quoted 6 lines
And *maybe* because it inherits from the standard formatter, we would benefit from any analysis Mypy performs on such things.
Basically, replace mcgen/cgen with class CFormatter(string.Formatter).
(maybe. assume that none of what I just said will work or is feasible.)
Sounds worth exploring. No need to do it now, of course.
Re: [PATCH 13/37] qapi/common.py: add notational type hints
Quoted 12 lines · Markus Armbruster
Quoted 9 lines · John Snow
Quoted 2 lines · Markus Armbruster
Question on the subject line: what makes a type hint notational?
My cover letter explains that every time I use this phrase, I mean to state that "This patch adds exclusively type notations and makes no functional changes to the runtime operation whatsoever."
i.e. notations-only.
Quoted 8 lines
By the time I get to PATCH 13, details explained in the cover letter have been flushed from my memory. Moreover, the cover letter won't make it into Git. Best to repeat them right in the commit message. Perhaps:
qapi/common; Add type hints
Type hints do not change behavior.
ACK, done.
Quoted 65 lines
Quoted 65 lines
Quoted 54 lines
Quoted 44 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +6
-3@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Union, Sequence EATSPACE = '\033EATSPACE.'@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 139
Quoted 4 lines
scripts/qapi/common.py +1
-1@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. # Obey INDENT level, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: Union[str, int]) -> str:Hmm.
The @kwds values can be anything, provided they match the conversion specifiers in @code:Quoted 4 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 141
Quoted 1 line
scripts/qapi/common.py +0
-0raw = code % kwdsYour type hint adds a restriction that wasn't there before. Is there a better way?
Maybe there are format-like type annotation tricks you can do to enforce this, but I did not research them. I tried to resist "improving" our usage of the old % formatter prematurely. I may do a wholesale f-string conversion at some point, but not now, it's not important.
Quoted 4 lines
Quoted 4 lines
In practice, we pass strings and integers. This typing *is* artificially restrictive, though. We can declare the type to be "Any" and allow the function to fail or succeed at runtime if you'd prefer.
I went with the 'object' type in new revisions.
Quoted 22 lines
Quoted 13 lines
Quoted 8 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 147
Quoted 6 lines
scripts/qapi/common.py +1
-1if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: Union[str, int]) -> str:Likewise.
Unresearched idea: It's possible that we can subclass the string.Formatter class and extend it to perform our special variable replacements (chomping EATSPACE, etc.)
Quoted 6 lines
And *maybe* because it inherits from the standard formatter, we would benefit from any analysis Mypy performs on such things.
Basically, replace mcgen/cgen with class CFormatter(string.Formatter).
(maybe. assume that none of what I just said will work or is feasible.)
Sounds worth exploring. No need to do it now, of course.
PATCH 14/37 qapi/common.py: Move comments into docstrings
In reply to John SnowAs docstrings, they'll show up in documentation and IDE help.
--- scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 14 deletions(-)
scripts/qapi/common.py +36 -14
@@ -20,10 +20,18 @@ c_name_trans = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending 'q_'. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int: INDENT = Indent(0) -# Generate @code with @kwds interpolated. -# Obey INDENT level, and strip EATSPACE. def cgen(code: str, **kwds: Union[str, int]) -> str: + """ + Generate `code` with `kwds` interpolated. + Obey `INDENT`, and strip `EATSPACE`. + """ raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
Signature · 1 line
2.26.2
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 34 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 14 deletions(-)scripts/qapi/common.py +11
-3@@ -20,10 +20,18 @@ c_name_trans = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
scripts/qapi/common.py def camel_to_upper(value: str) -> str: · line 62
Quoted 10 lines
scripts/qapi/common.py +7
-0@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """
Not actually a move. Suggest to retitle
qapi/common: Turn comments in docstrings, and add more
Quoted 36 lines
scripts/qapi/common.py +14
-9if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending 'q_'. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int: INDENT = Indent(0)
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 165
Quoted 10 lines
scripts/qapi/common.py +4
-2-# Generate @code with @kwds interpolated. -# Obey INDENT level, and strip EATSPACE. def cgen(code: str, **kwds: Union[str, int]) -> str: + """ + Generate `code` with `kwds` interpolated. + Obey `INDENT`, and strip `EATSPACE`. + """ raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
Can you point to documentation on the docstring conventions and markup to use?
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 49 lines · Markus Armbruster
Quoted 34 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 14 deletions(-)scripts/qapi/common.py +11
-3@@ -20,10 +20,18 @@ c_name_trans = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_strscripts/qapi/common.py def camel_to_upper(value: str) -> str: · line 62
Quoted 10 lines
scripts/qapi/common.py +7
-0@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """Not actually a move. Suggest to retitle
Quoted 2 lines
qapi/common: Turn comments in docstrings, and add more
OK.
Quoted 36 lines
Quoted 36 lines
scripts/qapi/common.py +14
-9if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending 'q_'. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int: INDENT = Indent(0)
Quoted 14 lines
scripts/qapi/common.py def pop(self, amount: int = 4) -> int: · line 165
Quoted 10 lines
scripts/qapi/common.py +4
-2-# Generate @code with @kwds interpolated. -# Obey INDENT level, and strip EATSPACE. def cgen(code: str, **kwds: Union[str, int]) -> str: + """ + Generate `code` with `kwds` interpolated. + Obey `INDENT`, and strip `EATSPACE`. + """ raw = code % kwds if INDENT: raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)Can you point to documentation on the docstring conventions and markup to use?
Short answer: No.
Long answer:
It's actually completely arbitrary, with major competing de-facto standards. Their primary function is to be stored to the __doc__ attribute of a module/class/method and can be displayed when using the interactive function help(...).
https://www.python.org/dev/peps/pep-0257/ covers docstrings only in an extremely broad sense. In summary, it asks:
- Use full sentences that end in periods
- Use the triple-double quote style
- multi-line docstrings should have their closing quote on a line by itself
- multi-line docstrings should use a one-sentence summary, a blank line, and then a more elaborate description.
It recommends you document arguments, types, return values and types, exceptions and so on but does not dictate a format. Two popular conventions are the google-style [1] and the NumPy-style [2] docstring formats.
I write docstrings assuming we will be using *Sphinx* as our documentation tool. Sphinx does not read docstrings at all by default, but *autodoc* does. Autodoc assumes your docstrings are written in the Sphinx dialect of ReST.
What you really want to look for is the "Python domain" documentation in Sphinx: https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html
Autodoc will annotate function/method docstrings with "py:function" or "py:method" as appropriate, but the actual contents of the block are still up to you.
For those, you want to look up the Python domain info field lists that are supported by Sphinx, which are here: https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Taken together with PEP 257, you generally want something like this:
"""
Function f converts uncertainty into happiness.
Function f only works on days that do not end in 'y'. Caution should be used when integrating this function into threaded code.
:param uncertainty: All of your doubts. :raise RuntimeError: When it is a day ending in 'y'. :return: Your newfound happiness.
"""
I use the single-backtick as the Sphinx-ese "default role" resolver, which generally should resolve to a reference to some Python entity. The double-backtick is used to do formatted text for things like string literals and so on.
Coffee break.
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
--js
[1] https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
[2] https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]
Quoted 8 lines
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote.
With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html.
I had to explicitly skip qapi/doc.py because autodoc thinks the string constants are documentation strings.
--- docs/conf.py | 5 +++- docs/devel/index.rst | 1 + docs/devel/qapi.rst | 67 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/common.py | 2 +- scripts/qapi/gen.py | 2 +- 5 files changed, 74 insertions(+), 3 deletions(-) create mode 100644 docs/devel/qapi.rst
docs/conf.py +4 -1
@@ -54,6 +54,9 @@ except NameError: # sys.path.insert(0, os.path.join(qemu_docdir, "sphinx")) +# Make scripts/qapi module available for autodoc +sys.path.insert(0, os.path.join(qemu_docdir, "../scripts")) + # -- General configuration ------------------------------------------------
@@ -67,7 +70,7 @@ needs_sphinx = '1.6' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile'] +extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'sphinx.ext.autodoc'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates']
docs/devel/index.rst +1 -0
@@ -31,3 +31,4 @@ Contents: reset s390-dasd-ipl clocks + qapi
docs/devel/qapi.rst new file +67 -0
67 lines not on this page · raw message
scripts/qapi/common.py +1 -1
@@ -78,7 +78,7 @@ def c_name(name: str, protect: bool = True) -> str: :param name: The name to map. :param protect: If true, avoid returning certain ticklish identifiers - (like C keywords) by prepending 'q_'. + (like C keywords) by prepending ``q_``. """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
scripts/qapi/gen.py +1 -1
@@ -161,7 +161,7 @@ def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Generator[None, None, None]: """A 'with' statement context manager to wrap with start_if()/end_if() - *args: any number of QAPIGenCCode + :param args: any number of QAPIGenCCode Example::
Signature · 4 lines
2.26.2 -- Eduardo
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 11 lines · Eduardo Habkost
On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]Quoted 8 lines
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
Quoted 9 lines
After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote.
With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html.
I had to explicitly skip
qapi/doc.pybecause autodoc thinks the string constants are documentation strings.
Awesome!
I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.
When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.
Quoted 142 lines
Signed-off-by Eduardo Habkost <[email protected]>--- docs/conf.py | 5 +++- docs/devel/index.rst | 1 + docs/devel/qapi.rst | 67 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/common.py | 2 +- scripts/qapi/gen.py | 2 +- 5 files changed, 74 insertions(+), 3 deletions(-)create mode 100644
docs/devel/qapi.rstdocs/conf.py +4
-1@@ -54,6 +54,9 @@ except NameError: # sys.path.insert(0, os.path.join(qemu_docdir, "sphinx")) +# Make scripts/qapi module available for autodoc +sys.path.insert(0, os.path.join(qemu_docdir, "../scripts")) + # -- General configuration ------------------------------------------------@@ -67,7 +70,7 @@ needs_sphinx = '1.6' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile'] +extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'sphinx.ext.autodoc'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates']docs/devel/index.rst +1
-0@@ -31,3 +31,4 @@ Contents: reset s390-dasd-ipl clocks + qapidocs/devel/qapi.rst new file +67
-0@@ -0,0 +1,67 @@ +QAPI Python module reference +============================ + +.. automodule:: qapi + :members: + :undoc-members: + +.. automodule:: qapi.commands + :members: + :undoc-members: + +.. automodule:: qapi.common + :members: + :undoc-members: + +.. automodule:: qapi.debug + :members: + :undoc-members: + +.. automodule:: qapi.error + :members: + :undoc-members: + +.. automodule:: qapi.events + :members: + :undoc-members: + +.. automodule:: qapi.expr + :members: + :undoc-members: + +.. automodule:: qapi.gen + :members: + :undoc-members: + +.. automodule:: qapi.introspect + :members: + :undoc-members: + +.. automodule:: qapi.params + :members: + :undoc-members: + +.. automodule:: qapi.parser + :members: + :undoc-members: + +.. automodule:: qapi.schema + :members: + :undoc-members: + +.. automodule:: qapi.script + :members: + :undoc-members: + +.. automodule:: qapi.source + :members: + :undoc-members: + +.. automodule:: qapi.types + :members: + :undoc-members: + +.. automodule:: qapi.visit + :members: + :undoc-members: +scripts/qapi/common.py +1
-1@@ -78,7 +78,7 @@ def c_name(name: str, protect: bool = True) -> str: :param name: The name to map. :param protect: If true, avoid returning certain ticklish identifiers - (like C keywords) by prepending 'q_'. + (like C keywords) by prepending ``q_``. """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',scripts/qapi/gen.py +1
-1@@ -161,7 +161,7 @@ def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Generator[None, None, None]: """A 'with' statement context manager to wrap with start_if()/end_if() - *args: any number of QAPIGenCCode + :param args: any number of QAPIGenCCode Example::
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost
On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]Quoted 8 lines
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
You are deliberately trying to "approximate" Sphinx style, and ...
Quoted 9 lines
Quoted 7 lines
After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip
qapi/doc.pybecause autodoc thinks the string constants are documentation strings.Awesome!
... actually almost nail it.
Please mention your choice of style in the commit message.
Quoted 7 lines
I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.
When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.
Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 24 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost
On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]Quoted 8 lines
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
You are deliberately trying to "approximate" Sphinx style, and ...
Quoted 8 lines
Quoted 7 lines
After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip
qapi/doc.pybecause autodoc thinks the string constants are documentation strings.
Quoted 6 lines
Quoted 1 line
Awesome!
... actually almost nail it.
Please mention your choice of style in the commit message.
OK, I'll try to summarize it. I guess I'll do it in this commit message, but it's not likely to be too visible in the future that way, depending on how you run git blame and what you're looking at.
I want to resurface my other python style patches soon; perhaps a python coding style document should go in alongside those patches.
Quoted 11 lines
Quoted 7 lines
I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.
When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.
Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...
Yes, I just mean in terms of rebasing, docstrings and signatures fight with each other for context and make re-spinning 125 patches a major chore. I wasn't prepared to have the debate on if we wanted to add Python code into the Sphinx generator and have that entire discussion yet.
So, I figured it would be a separate series afterwards. I mentioned somewhere else that I anticipated doing it when I removed the "missing-docstring" warning.
I will investigate doing it (using Eduardo's patches) as a starting point while reviews continue to filter in. If I figure it out in time, I might squash the formatting changes in, but leave the sphinx enablement patches themselves out.
--js
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 24 lines · John Snow
Quoted 23 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost
On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]Quoted 8 lines
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
You are deliberately trying to "approximate" Sphinx style, and ...
Quoted 8 lines
Quoted 7 lines
After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip
qapi/doc.pybecause autodoc thinks the string constants are documentation strings.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Awesome!
... actually almost nail it.
Please mention your choice of style in the commit message.OK, I'll try to summarize it. I guess I'll do it in this commit message, but it's not likely to be too visible in the future that way, depending on how you run git blame and what you're looking at.
Thanks!
Quoted 20 lines
I want to resurface my other python style patches soon; perhaps a python coding style document should go in alongside those patches.
Quoted 10 lines
Quoted 7 lines
I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.
When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.
Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...
Yes, I just mean in terms of rebasing, docstrings and signatures fight with each other for context and make re-spinning 125 patches a major chore. I wasn't prepared to have the debate on if we wanted to add Python code into the Sphinx generator and have that entire discussion yet.
Quoted 8 lines
So, I figured it would be a separate series afterwards. I mentioned somewhere else that I anticipated doing it when I removed the "missing-docstring" warning.
I will investigate doing it (using Eduardo's patches) as a starting point while reviews continue to filter in. If I figure it out in time, I might squash the formatting changes in, but leave the sphinx enablement patches themselves out.
Use your judgement.
Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · John Snow
Quoted 27 lines · Markus Armbruster
Quoted 11 lines · John Snow
Quoted 10 lines · Eduardo Habkost
On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]Quoted 8 lines
Having said this, I have not found any tool to date that actually *checks* these comments for consistency. The pycharm IDE interactively highlights them when it senses that you have made a mistake, but that cannot be worked into our CI process that I know of - it's a proprietary checker.
So right now, they're just plaintext that I am writing to approximate the Sphinx style until such time as I enable autodoc for the module and fine-tune the actual formatting and so on.
You are deliberately trying to "approximate" Sphinx style, and ...
Quoted 9 lines
Quoted 7 lines
After applying this series, I only had to make two small tweaks to make Sphinx + autodoc happy with the docstrings you wrote. With the following patch, "make sphinxdocs" will generate the QAPI Python module documentation at docs/devel/qapi.html. I had to explicitly skip
qapi/doc.pybecause autodoc thinks the string constants are documentation strings.Awesome!
... actually almost nail it.
Please mention your choice of style in the commit message.
Quoted 35 lines
Quoted 3 lines
OK, I'll try to summarize it. I guess I'll do it in this commit message, but it's not likely to be too visible in the future that way, depending on how you run git blame and what you're looking at.
Thanks!
Quoted 28 lines
I want to resurface my other python style patches soon; perhaps a python coding style document should go in alongside those patches.
Quoted 10 lines
Quoted 7 lines
I think I am going to delay explicitly pursuing writing a manual for the QAPI parser for now, but it's good to know I am not too far off. I'm going to target the mypy conversions first, because they can be invasive and full of churn.
When I get there, though ... I am thinking I should add this as Devel/QAPI Parser.
Doing "actually Sphinx style" instead of "an approximation of Sphinx style" would reduce churn later on. So, if it's not too distracting...
Yes, I just mean in terms of rebasing, docstrings and signatures fight with each other for context and make re-spinning 125 patches a major chore. I wasn't prepared to have the debate on if we wanted to add Python code into the Sphinx generator and have that entire discussion yet.
So, I figured it would be a separate series afterwards. I mentioned somewhere else that I anticipated doing it when I removed the "missing-docstring" warning.
I will investigate doing it (using Eduardo's patches) as a starting point while reviews continue to filter in. If I figure it out in time, I might squash the formatting changes in, but leave the sphinx enablement patches themselves out.
I wound up figuring it out before I sent V3 out :)
Quoted 2 lines
Use your judgement.
Yep. You can see how this played out in V3: I ensure that nothing is "wrong" but I haven't converted the entire submodule yet.
A thing to clarify for the list as well: Nothing validates these docstrings in the content-aware sense; nothing makes sure you documented the right parameter names, return values, exceptions, etc.
The only validation we can perform automatically right now that I am aware of is:
1. Did we have a docstring, yes/no?
2. Is anything referred to in backticks a valid reference?
3. Did that docstring parse as Sphinx-dialect ReST?
I would like to improve this situation someday, but it's not a task for the near future.
It would be very, very cool if autodoc itself was able to parse the integrity of the docstring and perform some simple validity checks, namely:
1. Every parameter has a :param FOO: in the docstring somewhere
2. Every :param FOO: matches a real parameter.
3. If the function has a return value, it is described with :returns:
4. If the function raises an exception, it must be mentioned with :raises:.
With perhaps some kind of exception for "simple" functions (How do I measure that?) where a one-line description is likely efficient enough.
--js
PATCH 15/37 qapi/common.py: split build_params into new file
In reply to John SnowIncluding it in common.py creates a circular import dependency, because schema relies on common.py. To type build_params properly, it needs to be moved outside of the chain.
--- scripts/qapi/commands.py | 2 +- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 2 +- scripts/qapi/params.py | 40 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+), 25 deletions(-) create mode 100644 scripts/qapi/params.py
scripts/qapi/commands.py +1 -1
@@ -14,10 +14,10 @@ """ from .common import ( - build_params, c_name, mcgen, ) +from .params import build_params from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
scripts/qapi/common.py +0 -23
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'
scripts/qapi/events.py +1 -1
@@ -13,11 +13,11 @@ """ from .common import ( - build_params, c_enum_const, c_name, mcgen, ) +from .params import build_params from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/params.py new file +40 -0
@@ -0,0 +1,40 @@ +# +# QAPI helper library +# +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2018 Red Hat Inc. +# +# Authors: +# Anthony Liguori <[email protected]> +# Markus Armbruster <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2. +# See the COPYING file in the top-level directory. + +from typing import Optional + +from .common import c_name +from .schema import QAPISchemaObjectType + + +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void'
Signature · 1 line
2.26.2
Re: [PATCH 15/37] qapi/common.py: split build_params into new file
Quoted 4 lines · John Snow
Including it in
common.pycreates a circular import dependency, because schema relies oncommon.py. To typebuild_paramsproperly, it needs to be moved outside of the chain.
Quoted 8 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/commands.py | 2 +- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 2 +- scripts/qapi/params.py | 40 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+), 25 deletions(-) create mode 100644 scripts/qapi/params.py
Ugh.
Would moving it gen.py work?
Re: [PATCH 15/37] qapi/common.py: split build_params into new file
Quoted 15 lines · Markus Armbruster
Quoted 4 lines · John Snow
Including it in
common.pycreates a circular import dependency, because schema relies oncommon.py. To typebuild_paramsproperly, it needs to be moved outside of the chain.Quoted 8 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/commands.py | 2 +- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 2 +- scripts/qapi/params.py | 40 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+), 25 deletions(-)create mode 100644
scripts/qapi/params.py
Quoted 4 lines
Ugh.
Would moving it
gen.pywork?
Actually, yes.
I have an experimental patch way, way later in the series that does this:
1. Leaves common.py with *just* functions and constants used by schema.py: c_name, POINTER_SUFFIX, and transitively EATSPACE.
2. Splits gen_c.py out of gen.py, putting all of the C-specific generator classes in there.
3. Adds params.py and the C-specific bits of common.py into gen_c.py.
In effect, you get gen_c.py with all of the C-specific bits in it, all of the other code-generation modules import from gen_c (marking them obviously as C code generators), and schema.py and other parsing friends import only the tiny common.py for c_name().
I'll adjust this patch to stash this in gen.py for now. It's too disruptive to shift the other refactor around in my stack.
--js
Re: [PATCH 15/37] qapi/common.py: split build_params into new file
In reply to Markus ArmbrusterQuoted 3 lines · Markus Armbruster
Ugh.
Would moving it
gen.pywork?
Done.
PATCH 16/37 qapi: establish mypy type-checking baseline
In reply to John SnowFix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
--- scripts/qapi/doc.py | 3 +- scripts/qapi/mypy.ini | 65 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 +- 3 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/doc.py +2 -1
@@ -5,7 +5,8 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from .gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc +from .schema import QAPISchemaVisitor MSG_FMT = """
scripts/qapi/mypy.ini new file +65 -0
65 lines not on this page · raw message
scripts/qapi/schema.py +2 -1
@@ -17,6 +17,7 @@ import os import re from collections import OrderedDict +from typing import Optional from .common import c_name, POINTER_SUFFIX from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Signature · 1 line
2.26.2
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type hints?
Quoted 6 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
I get:
$ mypy --config-file qapi/mypy.ini qapi
qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
Found 1 error in 1 file (checked 18 source files)Is this expected?
In case it matters:
$ mypy --version
mypy 0.761Quoted 13 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/doc.py | 3 +- scripts/qapi/mypy.ini | 65 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 +- 3 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 scripts/qapi/mypy.iniscripts/qapi/doc.py +0
-0
scripts/qapi/doc.py line 9
Quoted 6 lines
scripts/qapi/doc.py +2
-1@@ -5,7 +5,8 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from .gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc +from .schema import QAPISchemaVisitor
Your mypy doesn't like such lazy imports? Mine seems not to care.
Quoted 64 lines
scripts/qapi/doc.py +0
-0MSG_FMT = """scripts/qapi/mypy.ini new file +55
-0@@ -0,0 +1,65 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.doc] +disallow_subclassing_any = False +disallow_untyped_defs = False +disallow_incomplete_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False
scripts/qapi/mypy.ini line 65
Quoted 10 lines
scripts/qapi/mypy.ini new file +10
-0+ +[mypy-qapi.types] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False
Greek to me. I'll learn in due time.
Quoted 21 lines
scripts/qapi/schema.py +2
-1@@ -17,6 +17,7 @@ import os import re from collections import OrderedDict +from typing import Optional from .common import c_name, POINTER_SUFFIX from .error import QAPIError, QAPISemError@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
Quoted 13 lines · Markus Armbruster
Ignorant question: why does this come after PATCH 13 "
qapi/common.py:
add notational type hints", but before all the other patches adding type hints?Quoted 6 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
Quoted 9 lines
I get:
$ mypy --config-file qapi/mypy.ini qapi qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode) qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...") Found 1 error in 1 file (checked 18 source files)Is this expected?
Nope.
Quoted 5 lines
In case it matters:
$ mypy --version mypy 0.761
I am using mypy 0.782.
I will investigate to see if there is an *easy* win to allow older versions to work.
In the meantime, please consider trying this:
pip install --user mypy==0.782 ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/
Quoted 15 lines
Quoted 15 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/doc.py | 3 +- scripts/qapi/mypy.ini | 65 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 +- 3 files changed, 69 insertions(+), 2 deletions(-)create mode 100644
scripts/qapi/mypy.iniscripts/qapi/doc.py +0
-0@@ -5,7 +5,8 @@ """This script produces the documentation of a qapi schema in texinfo format"""
Quoted 7 lines
scripts/qapi/doc.py line 9
Quoted 4 lines
scripts/qapi/doc.py +2
-1import re -from .gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc +from .schema import QAPISchemaVisitorYour mypy doesn't like such lazy imports? Mine seems not to care.
Yeah, it specifically complained that no such definition existed in that file.
Quoted 68 lines
Quoted 68 lines
scripts/qapi/doc.py +0
-0MSG_FMT = """scripts/qapi/mypy.ini new file +59
-0@@ -0,0 +1,65 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.doc] +disallow_subclassing_any = False +disallow_untyped_defs = False +disallow_incomplete_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = False +disallow_incomplete_defs = False
Quoted 9 lines
scripts/qapi/mypy.ini line 65
Quoted 6 lines
scripts/qapi/mypy.ini new file +6
-0+check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = FalseGreek to me. I'll learn in due time.
I am using these options:
--strict, which is effectively -Wall.
--no-strict-optional, which disables type checking errors on conflict between Optional[T] and T. Namely, when you initialize a class field to None and set that variable after initialization, callers must be prepared to see if that field was None or not. We do that effectively nowhere.
As Python will surely explode in a noticeable way if we got an unexpected 'None', I am just suppressing these warnings "for now".
--allow-untyped-calls silences errors in files that have calls to functions in files I still have not typed. By the end of the series, this option goes away, because there's nothing untyped left.
For each untyped file, we are actually starting with all of the above options and then layering these options on top. Any egregious typing errors present in these "ignored" files will be spotted.
To get the bad files to pass, we only need three options:
allow untyped defs -- Simply permits us to have functions without annotations.
allow incomplete defs -- allows functions that are only partially typed.
check untyped defs = False -- Don't try to type check untyped definitions.
Quoted 21 lines
Quoted 21 lines
scripts/qapi/schema.py +2
-1@@ -17,6 +17,7 @@ import os import re from collections import OrderedDict +from typing import Optional from .common import c_name, POINTER_SUFFIX from .error import QAPIError, QAPISemError@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
Quoted 29 lines · John Snow
Quoted 10 lines · Markus Armbruster
Ignorant question: why does this come after PATCH 13 "
qapi/common.py:
add notational type hints", but before all the other patches adding type hints?Quoted 5 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 8 lines
Quoted 1 line
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
I get:
$ mypy --config-file qapi/mypy.ini qapi qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode) qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...") Found 1 error in 1 file (checked 18 source files) Is this expected?Nope.
Quoted 4 lines
In case it matters:
$ mypy --version mypy 0.761I am using mypy 0.782.
Quoted 7 lines
I will investigate to see if there is an *easy* win to allow older versions to work.
In the meantime, please consider trying this:
pip install --user mypy==0.782 ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/
I'll consider dragging my feet until upgrading Fedora gives it to me for free.
The less I interact with package managers, the happier I am.
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/doc.py | 3 +- scripts/qapi/mypy.ini | 65 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 +- 3 files changed, 69 insertions(+), 2 deletions(-)create mode 100644
scripts/qapi/mypy.iniQuoted 12 lines
scripts/qapi/doc.py line 8
Quoted 10 lines
scripts/qapi/doc.py +2
-1@@ -5,7 +5,8 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from .gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc +from .schema import QAPISchemaVisitorYour mypy doesn't like such lazy imports? Mine seems not to care.
Quoted 2 lines
Yeah, it specifically complained that no such definition existed in that file.
I sense a certain wobbliness in mypy. Perhaps to be expected from a tool with major version zero. There's a risk that developers' local "make check" and our gating CI differ too much. We'll see.
Quoted 80 lines
Quoted 66 lines
Quoted 66 lines
scripts/qapi/doc.py +0
-0MSG_FMT = """scripts/qapi/mypy.ini new file +58
-0@@ -0,0 +1,65 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.doc] +disallow_subclassing_any = False +disallow_untyped_defs = False +disallow_incomplete_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = FalseQuoted 9 lines
scripts/qapi/mypy.ini line 65
Quoted 7 lines
scripts/qapi/mypy.ini new file +7
-0+disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = FalseGreek to me. I'll learn in due time.
I am using these options:
--strict, which is effectively -Wall.
Quoted 8 lines
--no-strict-optional, which disables type checking errors on conflict
between Optional[T] and T. Namely, when you initialize a class field
to None and set that variable after initialization, callers must be
prepared to see if that field was None or not. We do that effectively nowhere.As Python will surely explode in a noticeable way if we got an unexpected 'None', I am just suppressing these warnings "for now".
Okay.
We may want to rethink how we define and initialize these variables. We initialize mostly to keep pylint happy. We initialize to None precisely to make use before the real initialization explode.
Quoted 9 lines
--allow-untyped-calls silences errors in files that have calls to
functions in files I still have not typed. By the end of the series,
this option goes away, because there's nothing untyped left.For each untyped file, we are actually starting with all of the above options and then layering these options on top. Any egregious typing errors present in these "ignored" files will be spotted.
Quoted 8 lines
To get the bad files to pass, we only need three options:
allow untyped defs -- Simply permits us to have functions without annotations.
allow incomplete defs -- allows functions that are only partially typed.
check untyped defs = False -- Don't try to type check untyped definitions.
Thanks!
[...]
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
Quoted 36 lines · Markus Armbruster
Quoted 34 lines · John Snow
Quoted 10 lines · Markus Armbruster
Ignorant question: why does this come after PATCH 13 "
qapi/common.py:
add notational type hints", but before all the other patches adding type hints?Quoted 5 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 8 lines
Quoted 1 line
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
I get:
$ mypy --config-file qapi/mypy.ini qapi qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode) qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...") Found 1 error in 1 file (checked 18 source files) Is this expected?Nope.
Quoted 4 lines
In case it matters:
$ mypy --version mypy 0.761I am using mypy 0.782.
I will investigate to see if there is an *easy* win to allow older versions to work.
In the meantime, please consider trying this:
Quoted 8 lines
Quoted 2 lines
pip install --user mypy==0.782 ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/
I'll consider dragging my feet until upgrading Fedora gives it to me for free.
The less I interact with package managers, the happier I am.
I'll probably be working on making sure CI runs in a virtual environment with specifically pinned versions.
We don't need mypy to run or build, so I have been less worried about requiring bleeding edge versions for development and regression testing.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 13 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/doc.py | 3 +- scripts/qapi/mypy.ini | 65 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 +- 3 files changed, 69 insertions(+), 2 deletions(-)create mode 100644
scripts/qapi/mypy.iniscripts/qapi/doc.py +0
-0scripts/qapi/doc.py line 8
Quoted 5 lines
scripts/qapi/doc.py +2
-1@@ -5,7 +5,8 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from .gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc +from .schema import QAPISchemaVisitorYour mypy doesn't like such lazy imports? Mine seems not to care.
Quoted 7 lines
Quoted 2 lines
Yeah, it specifically complained that no such definition existed in that file.
I sense a certain wobbliness in mypy. Perhaps to be expected from a tool with major version zero. There's a risk that developers' local "make check" and our gating CI differ too much. We'll see.
It is indeed very cutting edge. It wasn't really viable to use until Python 3.6.
Quoted 86 lines
Quoted 86 lines
Quoted 66 lines
Quoted 66 lines
scripts/qapi/doc.py +0
-0MSG_FMT = """scripts/qapi/mypy.ini new file +58
-0@@ -0,0 +1,65 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.doc] +disallow_subclassing_any = False +disallow_untyped_defs = False +disallow_incomplete_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = FalseQuoted 9 lines
scripts/qapi/mypy.ini line 65
Quoted 7 lines
scripts/qapi/mypy.ini new file +7
-0+disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = FalseGreek to me. I'll learn in due time.
I am using these options:
--strict, which is effectively -Wall.
--no-strict-optional, which disables type checking errors on conflict
between Optional[T] and T. Namely, when you initialize a class field
to None and set that variable after initialization, callers must be
prepared to see if that field was None or not. We do that effectively nowhere.
Quoted 9 lines
Quoted 2 lines
As Python will surely explode in a noticeable way if we got an unexpected 'None', I am just suppressing these warnings "for now".
Okay.
We may want to rethink how we define and initialize these variables. We initialize mostly to keep pylint happy. We initialize to None precisely to make use before the real initialization explode.
The pattern we have is roughly correct; it's just that we rely on that implicit explosion, which mypy will warn about if I don't disable this option.
It's not worth addressing in our first pass, but if you look at schema.py's @property ifcond code, that's the type of thing that would "fix" this warning -- basically access shims that add a "real" error message when it's None, but convert the return type from Optional[T] to T.
It's later stuff.
Quoted 22 lines
Quoted 9 lines
--allow-untyped-calls silences errors in files that have calls to
functions in files I still have not typed. By the end of the series,
this option goes away, because there's nothing untyped left.For each untyped file, we are actually starting with all of the above options and then layering these options on top. Any egregious typing errors present in these "ignored" files will be spotted.
Quoted 8 lines
To get the bad files to pass, we only need three options:
allow untyped defs -- Simply permits us to have functions without annotations.
allow incomplete defs -- allows functions that are only partially typed.
check untyped defs = False -- Don't try to type check untyped definitions.
Thanks!
[...]
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
In reply to Markus ArmbrusterOn Mon, Sep 21, 2020 at 10:05:24AM +0200, Markus Armbruster wrote:
[...]
Quoted 3 lines
I sense a certain wobbliness in mypy. Perhaps to be expected from a tool with major version zero. There's a risk that developers' local "make check" and our gating CI differ too much. We'll see.
This possibility shouldn't exist. If you don't have the required version in your system, the validation script must know how to create a virtualenv and install the right version inside it.
Signature · 1 line
Eduardo
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
In reply to Markus ArmbrusterQuoted 20 lines · Markus Armbruster
Ignorant question: why does this come after PATCH 13 "
qapi/common.py:
add notational type hints", but before all the other patches adding type hints?Quoted 6 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
I get:
$ mypy --config-file qapi/mypy.ini qapi qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode) qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...") Found 1 error in 1 file (checked 18 source files)
Quoted 7 lines
Is this expected?
In case it matters:
$ mypy --version mypy 0.761
(Warning; FiSH and stgit ahead)
cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8
### Testing mypy 0.770
pip install mypy==0.770
stg goto qapi-establish-mypy-type
while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end
pip uninstall mypy
###
0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option) 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8) 0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.
0.760 doesn't support strict in the config file (It needs component options), and it wants a few extra annotations where its inference power is weaker. Well, easy enough to fix up.
0.630 fails again and insists that __init__ should have a return type annotation of None. Modern mypy is smart enough to know that's what that type is supposed to be. Arbitrarily, this is my cutoff for what seems reasonable to even want to support.
I still find the lack of "strict=true" in the config file irritating and might ask to target 0.770 or newer. There should be no reason we can't install that in a venv for CI to chew on.
Humbly ask I take the lazy way out and document that we support mypy >= 0.770.
--js
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
Quoted 70 lines · John Snow
Quoted 12 lines · Markus Armbruster
Ignorant question: why does this come after PATCH 13 "
qapi/common.py:
add notational type hints", but before all the other patches adding type hints?Quoted 6 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
I get:
Quoted 9 lines
$ mypy --config-file qapi/mypy.ini qapi qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode) qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...") Found 1 error in 1 file (checked 18 source files) Is this expected? In case it matters: $ mypy --version mypy 0.761(Warning; FiSH and stgit ahead)
cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8### Testing mypy 0.770
pip install mypy==0.770
stg goto qapi-establish-mypy-typewhile true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end
pip uninstall mypy
###
0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option) 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8) 0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.0.760 doesn't support strict in the config file (It needs component options), and it wants a few extra annotations where its inference power is weaker. Well, easy enough to fix up.
0.630 fails again and insists that
__init__should have a return type annotation of None. Modern mypy is smart enough to know that's what that type is supposed to be. Arbitrarily, this is my cutoff for what seems reasonable to even want to support.
Quoted 6 lines
I still find the lack of "strict=true" in the config file irritating and might ask to target 0.770 or newer. There should be no reason we can't install that in a venv for CI to chew on.
Humbly ask I take the lazy way out and document that we support mypy
Quoted 1 line
= 0.770.
Our "supported build platforms" policy puts hard limits on the minimum versions for tools the build requires.
Mypy is not such a tool. I hope we get to the point where we can have "make check" run it, but skipping the test when we don't have a sufficiently modern mypy feels okay to me, as long as our gating CI still guards the master branch.
Re: [PATCH 16/37] qapi: establish mypy type-checking baseline
Quoted 79 lines · Markus Armbruster
Quoted 70 lines · John Snow
Quoted 12 lines · Markus Armbruster
Ignorant question: why does this come after PATCH 13 "
qapi/common.py:
add notational type hints", but before all the other patches adding type hints?Quoted 6 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
I get:
Quoted 9 lines
$ mypy --config-file qapi/mypy.ini qapi qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode) qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...") Found 1 error in 1 file (checked 18 source files) Is this expected? In case it matters: $ mypy --version mypy 0.761(Warning; FiSH and stgit ahead)
cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8### Testing mypy 0.770
pip install mypy==0.770
stg goto qapi-establish-mypy-typewhile true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end
pip uninstall mypy
###
0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option) 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8) 0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.0.760 doesn't support strict in the config file (It needs component options), and it wants a few extra annotations where its inference power is weaker. Well, easy enough to fix up.
0.630 fails again and insists that
__init__should have a return type annotation of None. Modern mypy is smart enough to know that's what that type is supposed to be. Arbitrarily, this is my cutoff for what seems reasonable to even want to support.Quoted 6 lines
I still find the lack of "strict=true" in the config file irritating and might ask to target 0.770 or newer. There should be no reason we can't install that in a venv for CI to chew on.
Humbly ask I take the lazy way out and document that we support mypy
Quoted 1 line
= 0.770.
Quoted 8 lines
Our "supported build platforms" policy puts hard limits on the minimum versions for tools the build requires.
Mypy is not such a tool. I hope we get to the point where we can have "make check" run it, but skipping the test when we don't have a sufficiently modern mypy feels okay to me, as long as our gating CI still guards the master branch.
Yes, that's been my view. These tools *are* new and they *do* change often. Chasing a wide compatibility window with them is prohibitively difficult. Luckily, they're not needed for running the packages at all.
If I do add linting to 'make check', it is going to be through a virtual environment that deploys *specific versions* of these tools, and it will happen transparently. The VM tests already do this.
--js
PATCH 17/37 qapi/events.py: add notational type hints
In reply to John SnowThis is enough to enable type-checking on events.py, so enable that in this patch, too.
--- scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 ----- 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +35 -11
@@ -12,6 +12,8 @@ See the COPYING file in the top-level directory. """ +from typing import List + from .common import ( c_enum_const, c_name,
@@ -19,17 +21,27 @@ ) from .params import build_params from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaEnumMember +from .schema import ( + QAPISchema, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, +) +from .source import QAPISourceInfo from .types import gen_enum, gen_enum_lookup -def build_event_send_proto(name, arg_type, boxed): +def build_event_send_proto(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return 'void qapi_event_send_%(c_name)s(%(param)s)' % { 'c_name': c_name(name.lower()), 'param': build_params(arg_type, boxed)} -def gen_event_send_decl(name, arg_type, boxed): +def gen_event_send_decl(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return mcgen(''' %(proto)s;
@@ -38,7 +50,7 @@ def gen_event_send_decl(name, arg_type, boxed): # Declare and initialize an object 'qapi' using parameters from build_params() -def gen_param_var(typ): +def gen_param_var(typ: QAPISchemaObjectType) -> str: assert not typ.variants ret = mcgen(''' %(c_name)s param = {
@@ -66,7 +78,11 @@ def gen_param_var(typ): return ret -def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): +def gen_event_send(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool, + event_enum_name: str, + event_emit: str) -> str: # FIXME: Our declaration of local variables (and of 'errp' in the # parameter list) can collide with exploded members of the event's # data type passed in as parameters. If this collision ever hits in
@@ -142,15 +158,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-events', ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) - self._event_enum_members = [] + self._event_enum_members: List[QAPISchemaEnumMember] = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit') - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: events = self._module_basename('qapi-events', name) types = self._module_basename('qapi-types', name) visit = self._module_basename('qapi-visit', name)
@@ -173,7 +189,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('emit', ' * QAPI Events emission') self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h"
@@ -194,7 +210,13 @@ def visit_end(self): event_emit=self._event_emit_name, event_enum=self._event_enum_name)) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + boxed: bool) -> None: with ifcontext(ifcond, self._genh, self._genc): self._genh.add(gen_event_send_decl(name, arg_type, boxed)) self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -205,7 +227,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed): self._event_enum_members.append(QAPISchemaEnumMember(name, None)) -def gen_events(schema, output_dir, prefix): +def gen_events(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenEventVisitor(prefix) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.events] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.expr] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
PATCH 18/37 qapi/events.py: Move comments into docstrings
In reply to John Snow--- scripts/qapi/events.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/events.py +3 -1
@@ -49,8 +49,10 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Declare and initialize a qapi object, using parameters from `build_params`. + """ assert not typ.variants ret = mcgen(''' %(c_name)s param = {
Signature · 1 line
2.26.2
PATCH 19/37 qapi/commands.py: Don't re-bind to variable of different type
In reply to John SnowMypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
--- scripts/qapi/commands.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +1 -3
@@ -194,14 +194,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): if not options: options = ['QCO_NO_OPTIONS'] - options = " | ".join(options) - ret = mcgen(''' qmp_register_command(cmds, "%(name)s", qmp_marshal_%(c_name)s, %(opts)s); ''', name=name, c_name=c_name(name), - opts=options) + opts=" | ".join(options)) return ret
Signature · 1 line
2.26.2
PATCH 20/37 qapi/commands.py: add notational type hints
In reply to John Snow--- scripts/qapi/commands.py | 72 ++++++++++++++++++++++++++++++---------- 1 file changed, 54 insertions(+), 18 deletions(-)
scripts/qapi/commands.py +54 -18
@@ -13,15 +13,32 @@ See the COPYING file in the top-level directory. """ +from typing import Dict, Optional, List, Set + from .common import ( c_name, mcgen, ) +from .gen import ( + QAPIGenC, + QAPIGenCCode, + QAPISchemaModularCVisitor, + ifcontext, +) from .params import build_params -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .schema import ( + QAPISchema, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaType, +) +from .source import QAPISourceInfo -def gen_command_decl(name, arg_type, boxed, ret_type): +def gen_command_decl(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: return mcgen(''' %(c_type)s qmp_%(c_name)s(%(params)s); ''',
@@ -30,7 +47,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type): params=build_params(arg_type, boxed, 'Error **errp')) -def gen_call(name, arg_type, boxed, ret_type): +def gen_call(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: ret = '' argstr = ''
@@ -66,7 +86,7 @@ def gen_call(name, arg_type, boxed, ret_type): return ret -def gen_marshal_output(ret_type): +def gen_marshal_output(ret_type: QAPISchemaType) -> str: return mcgen(''' static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
@@ -87,19 +107,22 @@ def gen_marshal_output(ret_type): c_type=ret_type.c_type(), c_name=ret_type.c_name()) -def build_marshal_proto(name): +def build_marshal_proto(name: str) -> str: return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)) -def gen_marshal_decl(name): +def gen_marshal_decl(name: str) -> str: return mcgen(''' %(proto)s; ''', proto=build_marshal_proto(name)) -def gen_marshal(name, arg_type, boxed, ret_type): +def gen_marshal(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: have_args = boxed or (arg_type and not arg_type.is_empty()) ret = mcgen('''
@@ -181,7 +204,10 @@ def gen_marshal(name, arg_type, boxed, ret_type): return ret -def gen_register_command(name, success_response, allow_oob, allow_preconfig): +def gen_register_command(name: str, + success_response: bool, + allow_oob: bool, + allow_preconfig: bool) -> str: options = [] if not success_response:
@@ -203,7 +229,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): return ret -def gen_registry(registry, prefix): +def gen_registry(registry: str, prefix: str) -> str: ret = mcgen(''' void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -220,15 +246,14 @@ def gen_registry(registry, prefix): class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): - - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-commands', ' * Schema-defined QAPI/QMP commands', None, __doc__) self._regy = QAPIGenCCode(None) - self._visited_ret_types = {} + self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {} - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: self._visited_ret_types[self._genc] = set() commands = self._module_basename('qapi-commands', name) types = self._module_basename('qapi-types', name)
@@ -252,7 +277,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('init', ' * QAPI Commands initialization') self._genh.add(mcgen(''' #include "qapi/qmp/dispatch.h"
@@ -268,9 +293,18 @@ def visit_end(self): prefix=self._prefix)) self._genc.add(gen_registry(self._regy.get_content(), self._prefix)) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool) -> None: if not gen: return # FIXME: If T is a user-defined type, the user is responsible
@@ -291,7 +325,9 @@ def visit_command(self, name, info, ifcond, features, allow_oob, allow_preconfig)) -def gen_commands(schema, output_dir, prefix): +def gen_commands(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenCommandVisitor(prefix) schema.visit(vis) vis.write(output_dir)
Signature · 1 line
2.26.2
PATCH 21/37 qapi/commands.py: enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.doc] disallow_subclassing_any = False disallow_untyped_defs = False
Signature · 1 line
2.26.2
PATCH 22/37 qapi/source.py: add notational type hints
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -39,11 +39,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/source.py +18 -13
@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]): self.fname = fname self.line = line self.parent = parent - self.pragma = parent.pragma if parent else QAPISchemaPragma() - self.defn_meta = None - self.defn_name = None + self.pragma: QAPISchemaPragma = ( + parent.pragma if parent else QAPISchemaPragma() + ) + self.defn_meta: Optional[str] = None + self.defn_name: Optional[str] = None - def set_defn(self, meta, name): + def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self): + def next_line(self: T) -> T: info = copy.copy(self) info.line += 1 return info - def loc(self): + def loc(self) -> str: if self.fname is None: return sys.argv[0] ret = self.fname
@@ -49,13 +54,13 @@ def loc(self): ret += ':%d' % self.line return ret - def in_defn(self): + def in_defn(self) -> str: if self.defn_name: return "%s: In %s '%s':\n" % (self.fname, self.defn_meta, self.defn_name) return '' - def include_path(self): + def include_path(self) -> str: ret = '' parent = self.parent while parent:
@@ -63,5 +68,5 @@ def include_path(self): parent = parent.parent return ret - def __str__(self): + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc()
Signature · 1 line
2.26.2
PATCH 23/37 qapi/source.py: delint with pylint
In reply to John SnowShush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/source.py | 3 +++ 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/pylintrc +0 -1
@@ -8,7 +8,6 @@ ignore-patterns=doc.py, gen.py, parser.py, schema.py, - source.py, types.py, visit.py,
scripts/qapi/source.py +3 -0
@@ -15,6 +15,9 @@ class QAPISchemaPragma: + # Replace with @dataclass in Python 3.7+ + # pylint: disable=too-few-public-methods + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False
Signature · 1 line
2.26.2
PATCH 24/37 qapi/gen.py: Fix edge-case of _is_user_module
In reply to John SnowThe edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +1 -1
@@ -221,7 +221,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./') @staticmethod def _is_builtin_module(name):
Signature · 1 line
2.26.2
PATCH 25/37 qapi/gen.py: add notational type hints
In reply to John Snow--- scripts/qapi/gen.py | 105 +++++++++++++++++++++++--------------------- 1 file changed, 56 insertions(+), 49 deletions(-)
scripts/qapi/gen.py +56 -49
@@ -15,6 +15,7 @@ import errno import os import re +from typing import Dict, Generator, List, Optional, Tuple from .common import ( c_fname,
@@ -24,32 +25,34 @@ guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import ( + QAPISchemaVisitor, +) +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.
@@ -74,7 +77,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif
@@ -91,40 +94,38 @@ def _wrap_ifcond(ifcond, before, after): class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
@@ -140,7 +141,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */
@@ -150,16 +151,16 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], + *args: QAPIGenCCode) -> Generator[None, None, None]: """A 'with' statement context manager to wrap with start_if()/end_if() *args: any number of QAPIGenCCode
@@ -185,15 +186,17 @@ def ifcontext(ifcond, *args): class QAPIGenDoc(QAPIGen): - - def _top(self): + def _top(self) -> str: return (super()._top() + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n') class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + blurb: str, + pydoc: str): self._prefix = prefix self._what = what self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -201,38 +204,42 @@ def __init__(self, prefix, what, blurb, pydoc): self._genh = QAPIGenH(self._prefix + self._what + '.h', blurb, pydoc) - def write(self, output_dir): + def write(self, output_dir: str) -> None: self._genc.write(output_dir) self._genh.write(output_dir) class QAPISchemaModularCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + user_blurb: str, + builtin_blurb: Optional[str], + pydoc: str): self._prefix = prefix self._what = what self._user_blurb = user_blurb self._builtin_blurb = builtin_blurb self._pydoc = pydoc - self._genc = None - self._genh = None - self._module = {} - self._main_module = None + self._genc: Optional[QAPIGenC] = None + self._genh: Optional[QAPIGenH] = None + self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {} + self._main_module: Optional[str] = None @staticmethod - def _is_user_module(name): + def _is_user_module(name: Optional[str]) -> bool: return name is not None and not name.startswith('./') @staticmethod - def _is_builtin_module(name): + def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what, name): + def _module_dirname(self, what: str, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return '' - def _module_basename(self, what, name): + def _module_basename(self, what: str, name: Optional[str]) -> str: ret = '' if self._is_builtin_module(name) else self._prefix if self._is_user_module(name): basename = os.path.basename(name)
@@ -244,27 +251,27 @@ def _module_basename(self, what, name): ret += re.sub(r'-', '-' + name + '-', what) return ret - def _module_filename(self, what, name): + def _module_filename(self, what: str, name: Optional[str]) -> str: return os.path.join(self._module_dirname(what, name), self._module_basename(what, name)) - def _add_module(self, name, blurb): + def _add_module(self, name: Optional[str], blurb: str) -> None: basename = self._module_filename(self._what, name) genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) self._genc, self._genh = self._module[name] - def _add_user_module(self, name, blurb): + def _add_user_module(self, name: str, blurb: str) -> None: assert self._is_user_module(name) if self._main_module is None: self._main_module = name self._add_module(name, blurb) - def _add_system_module(self, name, blurb): + def _add_system_module(self, name: Optional[str], blurb: str) -> None: self._add_module(name and './' + name, blurb) - def write(self, output_dir, opt_builtins=False): + def write(self, output_dir: str, opt_builtins: bool = False) -> None: for name in self._module: if self._is_builtin_module(name) and not opt_builtins: continue
@@ -272,13 +279,13 @@ def write(self, output_dir, opt_builtins=False): genc.write(output_dir) genh.write(output_dir) - def _begin_system_module(self, name): + def _begin_system_module(self, name: None) -> None: pass - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: pass - def visit_module(self, name): + def visit_module(self, name: Optional[str]) -> None: if name is None: if self._builtin_blurb: self._add_system_module(None, self._builtin_blurb)
@@ -292,7 +299,7 @@ def visit_module(self, name): self._add_user_module(name, self._user_blurb) self._begin_user_module(name) - def visit_include(self, name, info): + def visit_include(self, name: str, info: QAPISourceInfo) -> None: relname = os.path.relpath(self._module_filename(self._what, name), os.path.dirname(self._genh.fname)) self._genh.preamble_add(mcgen('''
Signature · 1 line
2.26.2
PATCH 26/37 qapi/gen.py: Enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.gen] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.introspect] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
PATCH 27/37 qapi/gen.py: Remove unused parameter
In reply to John Snowmodule_basename doesn't use the 'what' argument, so remove it.
--- scripts/qapi/gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
@@ -234,7 +234,7 @@ def _is_user_module(name: Optional[str]) -> bool: def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what: str, name: Optional[str]) -> str: + def _module_dirname(self, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return ''
@@ -252,7 +252,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str: return ret def _module_filename(self, what: str, name: Optional[str]) -> str: - return os.path.join(self._module_dirname(what, name), + return os.path.join(self._module_dirname(name), self._module_basename(what, name)) def _add_module(self, name: Optional[str], blurb: str) -> None:
Signature · 1 line
2.26.2
PATCH 28/37 qapi/gen.py: update write() to be more idiomatic
In reply to John SnowMake the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.
--- scripts/qapi/gen.py | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +10 -14
@@ -12,7 +12,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import Dict, Generator, List, Optional, Tuple
@@ -60,21 +59,18 @@ def write(self, output_dir: str) -> None: return pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try: - os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True) + fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close() + with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text) def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Signature · 1 line
2.26.2
PATCH 29/37 qapi/gen.py: delint with pylint
In reply to John Snow'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_top and _bottom also need to stay class methods because some users override the method and need to use self. Tell pylint to shush.
--- scripts/qapi/gen.py | 2 ++ scripts/qapi/pylintrc | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -0
@@ -46,9 +46,11 @@ def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() def _top(self) -> str: + # pylint: disable=no-self-use return '' def _bottom(self) -> str: + # pylint: disable=no-self-use return '' def write(self, output_dir: str) -> None:
scripts/qapi/pylintrc +3 -2
@@ -5,7 +5,6 @@ ignore-patterns=doc.py, error.py, expr.py, - gen.py, parser.py, schema.py, types.py,
@@ -46,7 +45,9 @@ good-names=i, k, ex, Run, - _ + _, + fp, # fp = open(...) + fd, # fd = os.open(...) [VARIABLES]
Signature · 1 line
2.26.2
PATCH 30/37 qapi/introspect.py: Add a typed 'extra' structure
In reply to John SnowTyping arbitrarily shaped dicts with mypy is difficult prior to Python 3.8; using explicit structures is nicer.
Since we always define an Extra type now, the return type of _make_tree simplifies and always returns the tuple.
--- scripts/qapi/introspect.py | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)
scripts/qapi/introspect.py +19 -12
@@ -10,6 +10,8 @@ See the COPYING file in the top-level directory. """ +from typing import (NamedTuple, Optional, Sequence) + from .common import ( c_name, gen_endif,
@@ -21,16 +23,21 @@ QAPISchemaType) -def _make_tree(obj, ifcond, features, extra=None): - if extra is None: - extra = {} - if ifcond: - extra['if'] = ifcond +class Extra(NamedTuple): + """ + Extra contains data that isn't intended for output by introspection. + """ + comment: Optional[str] = None + ifcond: Sequence[str] = tuple() + + +def _make_tree(obj, ifcond, features, + extra: Optional[Extra] = None): + comment = extra.comment if extra else None + extra = Extra(comment, ifcond) if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] - if extra: - return (obj, extra) - return obj + obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features] + return (obj, extra) def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -40,8 +47,8 @@ def indent(level): if isinstance(obj, tuple): ifobj, extra = obj - ifcond = extra.get('if') - comment = extra.get('comment') + ifcond = extra.ifcond + comment = extra.comment ret = '' if comment: ret += indent(level) + '/* %s */\n' % comment
@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): if not self._unmask: # Output a comment to make it easy to map masked names # back to the source when reading the generated output. - extra = {'comment': '"%s" = %s' % (self._name(name), name)} + extra = Extra(comment=f'"{self._name(name)}" = {name}') name = self._name(name) obj['name'] = name obj['meta-type'] = mtype
Signature · 1 line
2.26.2
PATCH 31/37 qapi/introspect.py: add _gen_features helper
In reply to John Snow_make_tree doesn't know if it is receiving an object or some other type; adding features information should arguably be performed by the caller.
This will help us refactor _make_tree more gracefully in the next patch.
--- scripts/qapi/introspect.py | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
scripts/qapi/introspect.py +14 -8
@@ -10,7 +10,7 @@ See the COPYING file in the top-level directory. """ -from typing import (NamedTuple, Optional, Sequence) +from typing import (NamedTuple, List, Optional, Sequence) from .common import ( c_name,
@@ -20,7 +20,7 @@ ) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) + QAPISchemaFeature, QAPISchemaType) class Extra(NamedTuple):
@@ -31,12 +31,10 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -def _make_tree(obj, ifcond, features, +def _make_tree(obj, ifcond, extra: Optional[Extra] = None): comment = extra.comment if extra else None extra = Extra(comment, ifcond) - if features: - obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features] return (obj, extra)
@@ -169,6 +167,10 @@ def _use_type(self, typ): return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) + @classmethod + def _gen_features(cls, features: List[QAPISchemaFeature]): + return [_make_tree(f.name, f.ifcond) for f in features] + def _gen_tree(self, name, mtype, obj, ifcond, features): extra = None if mtype not in ('command', 'event', 'builtin', 'array'):
@@ -179,13 +181,17 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): name = self._name(name) obj['name'] = name obj['meta-type'] = mtype - self._trees.append(_make_tree(obj, ifcond, features, extra)) + if features: + obj['features'] = self._gen_features(features) + self._trees.append(_make_tree(obj, ifcond, extra)) def _gen_member(self, member): obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None - return _make_tree(obj, member.ifcond, member.features) + if member.features: + obj['features'] = self._gen_features(member.features) + return _make_tree(obj, member.ifcond) def _gen_variants(self, tag_name, variants): return {'tag': tag_name,
@@ -193,7 +199,7 @@ def _gen_variants(self, tag_name, variants): def _gen_variant(self, variant): obj = {'case': variant.name, 'type': self._use_type(variant.type)} - return _make_tree(obj, variant.ifcond, None) + return _make_tree(obj, variant.ifcond) def visit_builtin_type(self, name, info, json_type): self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
Signature · 1 line
2.26.2
PATCH 32/37 qapi/introspect.py: create a typed 'Node' data structure
In reply to John SnowReplacing the un-typed tuple, add a typed Node that we can add typed metadata to.
--- scripts/qapi/introspect.py | 53 ++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 25 deletions(-)
scripts/qapi/introspect.py +28 -25
@@ -31,11 +31,18 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -def _make_tree(obj, ifcond, - extra: Optional[Extra] = None): - comment = extra.comment if extra else None - extra = Extra(comment, ifcond) - return (obj, extra) +class Node: + """ + Node generally contains a SchemaInfo-like type (as a dict), + But it also used to wrap comments/ifconds around leaf value types. + """ + # Remove after 3.7 adds @dataclass: + # pylint: disable=too-few-public-methods + def __init__(self, data, ifcond: List[str], + extra: Optional[Extra] = None): + self.data = data + comment = extra.comment if extra else None + self.extra = Extra(comment, ifcond) def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -43,18 +50,15 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False): def indent(level): return level * 4 * ' ' - if isinstance(obj, tuple): - ifobj, extra = obj - ifcond = extra.ifcond - comment = extra.comment + if isinstance(obj, Node): ret = '' - if comment: - ret += indent(level) + '/* %s */\n' % comment - if ifcond: - ret += gen_if(ifcond) - ret += _tree_to_qlit(ifobj, level) - if ifcond: - ret += '\n' + gen_endif(ifcond) + if obj.extra.comment: + ret += indent(level) + '/* %s */\n' % obj.extra.comment + if obj.extra.ifcond: + ret += gen_if(obj.extra.ifcond) + ret += _tree_to_qlit(obj.data, level) + if obj.extra.ifcond: + ret += '\n' + gen_endif(obj.extra.ifcond) return ret ret = ''
@@ -169,7 +173,7 @@ def _use_type(self, typ): @classmethod def _gen_features(cls, features: List[QAPISchemaFeature]): - return [_make_tree(f.name, f.ifcond) for f in features] + return [Node(f.name, f.ifcond) for f in features] def _gen_tree(self, name, mtype, obj, ifcond, features): extra = None
@@ -183,7 +187,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): obj['meta-type'] = mtype if features: obj['features'] = self._gen_features(features) - self._trees.append(_make_tree(obj, ifcond, extra)) + self._trees.append(Node(obj, ifcond, extra)) def _gen_member(self, member): obj = {'name': member.name, 'type': self._use_type(member.type)}
@@ -191,7 +195,7 @@ def _gen_member(self, member): obj['default'] = None if member.features: obj['features'] = self._gen_features(member.features) - return _make_tree(obj, member.ifcond) + return Node(obj, member.ifcond) def _gen_variants(self, tag_name, variants): return {'tag': tag_name,
@@ -199,15 +203,14 @@ def _gen_variants(self, tag_name, variants): def _gen_variant(self, variant): obj = {'case': variant.name, 'type': self._use_type(variant.type)} - return _make_tree(obj, variant.ifcond) + return Node(obj, variant.ifcond) def visit_builtin_type(self, name, info, json_type): self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) def visit_enum_type(self, name, info, ifcond, features, members, prefix): self._gen_tree(name, 'enum', - {'values': [_make_tree(m.name, m.ifcond, None) - for m in members]}, + {'values': [Node(m.name, m.ifcond) for m in members]}, ifcond, features) def visit_array_type(self, name, info, ifcond, element_type):
@@ -227,9 +230,9 @@ def visit_object_type_flat(self, name, info, ifcond, features, def visit_alternate_type(self, name, info, ifcond, features, variants): self._gen_tree(name, 'alternate', {'members': [ - _make_tree({'type': self._use_type(m.type)}, - m.ifcond, None) - for m in variants.variants]}, + Node({'type': self._use_type(m.type)}, m.ifcond) + for m in variants.variants + ]}, ifcond, features) def visit_command(self, name, info, ifcond, features,
Signature · 1 line
2.26.2
PATCH 33/37 qapi/introspect.py: add notational type hints
In reply to John SnowAdd a tiny hint into schema.py to allow introspect.py to type check, too.
--- scripts/qapi/introspect.py | 123 ++++++++++++++++++++++++++----------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 87 insertions(+), 43 deletions(-)
scripts/qapi/introspect.py +86 -37
@@ -10,7 +10,16 @@ See the COPYING file in the top-level directory. """ -from typing import (NamedTuple, List, Optional, Sequence) +from typing import ( + Any, + Dict, + Generic, + List, + NamedTuple, + Optional, + Sequence, + TypeVar, +) from .common import ( c_name,
@@ -19,8 +28,23 @@ mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaFeature, QAPISchemaType) +from .schema import ( + QAPISchema, + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, +) +from .source import QAPISourceInfo + + +_NodeType = TypeVar('_NodeType') class Extra(NamedTuple):
@@ -31,23 +55,24 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -class Node: +class Node(Generic[_NodeType]): """ Node generally contains a SchemaInfo-like type (as a dict), But it also used to wrap comments/ifconds around leaf value types. """ # Remove after 3.7 adds @dataclass: # pylint: disable=too-few-public-methods - def __init__(self, data, ifcond: List[str], + def __init__(self, data: _NodeType, ifcond: List[str], extra: Optional[Extra] = None): self.data = data comment = extra.comment if extra else None self.extra = Extra(comment, ifcond) -def _tree_to_qlit(obj, level=0, suppress_first_indent=False): +def _tree_to_qlit(obj: Any, level: int = 0, + suppress_first_indent: bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, Node):
@@ -94,21 +119,20 @@ def indent(level): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[Node[Dict[str, Any]]] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h"
@@ -116,10 +140,10 @@ def __init__(self, prefix, unmask): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self)
@@ -141,18 +165,18 @@ def visit_end(self): self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: # Map the various integer types to plain int if typ.json_type() == 'int': typ = self._schema.lookup_type('int')
@@ -172,10 +196,13 @@ def _use_type(self, typ): return self._name(typ.name) @classmethod - def _gen_features(cls, features: List[QAPISchemaFeature]): + def _gen_features(cls, + features: List[QAPISchemaFeature]) -> List[Node[str]]: return [Node(f.name, f.ifcond) for f in features] - def _gen_tree(self, name, mtype, obj, ifcond, features): + def _gen_tree(self, name: str, mtype: str, obj: Dict[str, Any], + ifcond: List[str], + features: Optional[List[QAPISchemaFeature]]) -> None: extra = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask:
@@ -189,7 +216,8 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): obj['features'] = self._gen_features(features) self._trees.append(Node(obj, ifcond, extra)) - def _gen_member(self, member): + def _gen_member(self, member: QAPISchemaObjectTypeMember, + ) -> Node[Dict[str, Any]]: obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None
@@ -197,29 +225,39 @@ def _gen_member(self, member): obj['features'] = self._gen_features(member.features) return Node(obj, member.ifcond) - def _gen_variants(self, tag_name, variants): + def _gen_variants(self, tag_name: str, + variants: List[QAPISchemaVariant]) -> Dict[str, Any]: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant): + def _gen_variant(self, variant: QAPISchemaVariant) -> Node[Dict[str, Any]]: obj = {'case': variant.name, 'type': self._use_type(variant.type)} return Node(obj, variant.ifcond) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo], + json_type: str) -> None: self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree(name, 'enum', {'values': [Node(m.name, m.ifcond) for m in members]}, ifcond, features) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + features: List[QAPISchemaFeature], + members: Sequence[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: obj = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name,
@@ -227,7 +265,10 @@ def visit_object_type_flat(self, name, info, ifcond, features, self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: self._gen_tree(name, 'alternate', {'members': [ Node({'type': self._use_type(m.type)}, m.ifcond)
@@ -235,24 +276,32 @@ def visit_alternate_type(self, name, info, ifcond, features, variants): ]}, ifcond, features) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + ret_type: Optional[QAPISchemaObjectType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: Dict[str, Any] = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, boxed: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/schema.py +1 -1
@@ -28,7 +28,7 @@ class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature)
Signature · 1 line
2.26.2
PATCH 34/37 qapi/types.py: add notational type hints
In reply to John Snow--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++----------- 2 files changed, 64 insertions(+), 27 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -29,11 +29,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.types] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.visit] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/types.py +64 -22
207 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 35/37 qapi/types.py: remove one-letter variables
In reply to John Snow"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/types.py | 29 +++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -7,7 +7,6 @@ ignore-patterns=doc.py, expr.py, parser.py, schema.py, - types.py, visit.py,
scripts/qapi/types.py +15 -14
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str, .array = (const char *const[]) { ''', c_name=c_name(name)) - for m in members: - ret += gen_if(m.ifcond) - index = c_enum_const(name, m.name, prefix) + for member in members: + ret += gen_if(member.ifcond) + index = c_enum_const(name, member.name, prefix) ret += mcgen(''' [%(index)s] = "%(name)s", ''', - index=index, name=m.name) - ret += gen_endif(m.ifcond) + index=index, name=member.name) + ret += gen_endif(member.ifcond) ret += mcgen(''' },
@@ -79,13 +79,13 @@ def gen_enum(name: str, ''', c_name=c_name(name)) - for m in enum_members: - ret += gen_if(m.ifcond) + for member in enum_members: + ret += gen_if(member.ifcond) ret += mcgen(''' %(c_enum)s, ''', - c_enum=c_enum_const(name, m.name, prefix)) - ret += gen_endif(m.ifcond) + c_enum=c_enum_const(name, member.name, prefix)) + ret += gen_endif(member.ifcond) ret += mcgen(''' } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name) ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for variant in variants.variants if variants else (): + obj = variant.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''
Signature · 1 line
2.26.2
PATCH 36/37 qapi/visit.py: remove unused parameters from gen_visit_object
In reply to John SnowAnd this fixes the pylint report for this file, so make sure we check this in the future, too.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/visit.py | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -7,7 +7,6 @@ ignore-patterns=doc.py, expr.py, parser.py, schema.py, - visit.py, [MESSAGES CONTROL]
scripts/qapi/visit.py +2 -2
@@ -247,7 +247,7 @@ def gen_visit_alternate(name, variants): return ret -def gen_visit_object(name, base, members, variants): +def gen_visit_object(name): return mcgen(''' bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -340,7 +340,7 @@ def visit_object_type(self, name, info, ifcond, features, if not name.startswith('q_'): # only explicit types need an allocating visit self._genh.add(gen_visit_decl(name)) - self._genc.add(gen_visit_object(name, base, members, variants)) + self._genc.add(gen_visit_object(name)) def visit_alternate_type(self, name, info, ifcond, features, variants): with ifcontext(ifcond, self._genh, self._genc):
Signature · 1 line
2.26.2
PATCH 37/37 qapi/visit.py: add notational type hints
In reply to John Snow--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/visit.py | 72 +++++++++++++++++++++++++++++++++---------- 2 files changed, 55 insertions(+), 22 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -28,8 +28,3 @@ check_untyped_defs = False disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False - -[mypy-qapi.visit] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False
scripts/qapi/visit.py +55 -17
163 lines not on this page · raw message
Signature · 1 line
2.26.2
Quoted 4 lines · John Snow
Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.
Updated series with current feedback and pushed to gitlab. Awaiting feedback on patches 10-37 before re-spinning.
https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/
--js
Quoted 18 lines · John Snow
Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.
The tougher files --
schema.py,expr.py,parser.py-- will each receive their own series as a follow-up to this one.Notes:
- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.
- Most other patches are as bite-sized as possible, generally fixing one single warning.
Quoted 9 lines
- After patch 6,
flake8 qapi/should pass 100% on this and every future commit.- After patch 7,
pylint --rcfile=qapi/pylintrc qapi/should pass 100% on this and every future commit.- After patch 16,
mypy --config-file=qapi/mypy.ini qapi/should pass 100% on this and every future commit.
Addressed today's feedback and pushed to gitlab.
Let's call it "Version 1.3".
https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/
Waiting on feedback for 16-37 before re-authoring to list.
--js
Quoted 26 lines · John Snow
Quoted 15 lines · John Snow
Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files. The tougher files --
schema.py,expr.py,parser.py-- will each receive
their own series as a follow-up to this one.
Notes:
- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
- Any patch named "add notational type hints" changes ONLY signatures,
which have no runtime impact whatsoever. These are big patches, but fairly straightforward.
- Most other patches are as bite-sized as possible, generally fixing one
single warning.Quoted 9 lines
- After patch 6,
flake8 qapi/should pass 100% on this and every future commit.
- After patch 7,pylint --rcfile=qapi/pylintrc qapi/should pass 100%
on this and every future commit.
- After patch 16,mypy --config-file=qapi/mypy.ini qapi/should pass
100% on this and every future commit.
Quoted 6 lines
Addressed today's feedback and pushed to gitlab.
Let's call it "Version 1.3".https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/
Waiting on feedback for 16-37 before re-authoring to list.
If 03-12 don't actually depend on Python 3.6, you may want to respin these for me to queue right away.
Cc'ing Ben who offered to help with having the QEMU python scripts packaged (and this series is a step toward that).
Quoted 90 lines · John Snow
Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.
The tougher files --
schema.py,expr.py,parser.py-- will each receive their own series as a follow-up to this one.Notes:
- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.
- Most other patches are as bite-sized as possible, generally fixing one single warning.
- After patch 6,
flake8 qapi/should pass 100% on this and every future commit.- After patch 7,
pylint --rcfile=qapi/pylintrc qapi/should pass 100% on this and every future commit.- After patch 16,
mypy --config-file=qapi/mypy.ini qapi/should pass 100% on this and every future commit.Preliminary refactoring and prerequisites:
001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools' 003/37: 'qapi-gen: Separate arg-parsing from generation' 004/37: 'qapi: move generator entrypoint into module' 005/37: 'qapi: Remove wildcard includes' 006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'
common.py(andparams.py):008/37: '
qapi/common.py: Remove python compatibility workaround' 009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable' 012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints' 014/37: 'qapi/common.py: Move comments into docstrings' 015/37: 'qapi/common.py: splitbuild_paramsinto new file' 016/37: 'qapi: establish mypy type-checking baseline'
events.py:017/37: '
qapi/events.py: add notational type hints' 018/37: 'qapi/events.py: Move comments into docstrings'
commands.py:019/37: '
qapi/commands.py: Don't re-bind to variable of different type' 020/37: 'qapi/commands.py: add notational type hints' 021/37: 'qapi/commands.py: enable checking with mypy'
source.py:022/37: '
qapi/source.py: add notational type hints' 023/37: 'qapi/source.py: delint with pylint'
gen.py:024/37: '
qapi/gen.py: Fix edge-case of_is_user_module' 025/37: 'qapi/gen.py: add notational type hints' 026/37: 'qapi/gen.py: Enable checking with mypy' 027/37: 'qapi/gen.py: Remove unused parameter' 028/37: 'qapi/gen.py: updatewrite()to be more idiomatic' 029/37: 'qapi/gen.py: delint with pylint'
introspect.py:030/37: '
qapi/introspect.py: Add a typed 'extra' structure' 031/37: 'qapi/introspect.py: add_gen_featureshelper' 032/37: 'qapi/introspect.py: create a typed 'Node' data structure' 033/37: 'qapi/introspect.py: add notational type hints'
types.py:034/37: '
qapi/types.py: add notational type hints' 035/37: 'qapi/types.py: remove one-letter variables'
visit.py:036/37: '
qapi/visit.py: remove unused parameters fromgen_visit_object' 037/37: 'qapi/visit.py: add notational type hints'
Quoted 3 lines · Philippe Mathieu-Daudé
Cc'ing Ben who offered to help with having the QEMU python scripts packaged (and this series is a step toward that).
FWIW, I want to move ./python/qemu/ into ./python/qemu/core, and then move ./scripts/qapi to ./python/qemu/qapi.
I will leave the qapi-gen.py stub script behind for compatibility.
Once I've done that, I can add package setup scripts.
I have a long series adding mypy type hints and so forth to ./python/qemu and adding packaging glue there too, but I shelved it as 5.1 released and right now I am focusing on QAPI.
Quoted 92 lines · John Snow
Quoted 90 lines
Hi, this series starts adding static type hints to the QAPI module. As you can see, the series started getting quite a bit long, so this is only a partial conversion that focuses on a handful of the easier files.
The tougher files --
schema.py,expr.py,parser.py-- will each receive their own series as a follow-up to this one.Notes:
- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
- Any patch named "add notational type hints" changes ONLY signatures, which have no runtime impact whatsoever. These are big patches, but fairly straightforward.
- Most other patches are as bite-sized as possible, generally fixing one single warning.
- After patch 6,
flake8 qapi/should pass 100% on this and every future commit.- After patch 7,
pylint --rcfile=qapi/pylintrc qapi/should pass 100% on this and every future commit.- After patch 16,
mypy --config-file=qapi/mypy.ini qapi/should pass 100% on this and every future commit.Preliminary refactoring and prerequisites:
001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools' 003/37: 'qapi-gen: Separate arg-parsing from generation' 004/37: 'qapi: move generator entrypoint into module' 005/37: 'qapi: Remove wildcard includes' 006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'
common.py(andparams.py):008/37: '
qapi/common.py: Remove python compatibility workaround' 009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable' 012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints' 014/37: 'qapi/common.py: Move comments into docstrings' 015/37: 'qapi/common.py: splitbuild_paramsinto new file' 016/37: 'qapi: establish mypy type-checking baseline'
events.py:017/37: '
qapi/events.py: add notational type hints' 018/37: 'qapi/events.py: Move comments into docstrings'
commands.py:019/37: '
qapi/commands.py: Don't re-bind to variable of different type' 020/37: 'qapi/commands.py: add notational type hints' 021/37: 'qapi/commands.py: enable checking with mypy'
source.py:022/37: '
qapi/source.py: add notational type hints' 023/37: 'qapi/source.py: delint with pylint'
gen.py:024/37: '
qapi/gen.py: Fix edge-case of_is_user_module' 025/37: 'qapi/gen.py: add notational type hints' 026/37: 'qapi/gen.py: Enable checking with mypy' 027/37: 'qapi/gen.py: Remove unused parameter' 028/37: 'qapi/gen.py: updatewrite()to be more idiomatic' 029/37: 'qapi/gen.py: delint with pylint'
introspect.py:030/37: '
qapi/introspect.py: Add a typed 'extra' structure' 031/37: 'qapi/introspect.py: add_gen_featureshelper' 032/37: 'qapi/introspect.py: create a typed 'Node' data structure' 033/37: 'qapi/introspect.py: add notational type hints'
types.py:034/37: '
qapi/types.py: add notational type hints' 035/37: 'qapi/types.py: remove one-letter variables'
visit.py:036/37: '
qapi/visit.py: remove unused parameters fromgen_visit_object' 037/37: 'qapi/visit.py: add notational type hints'