qapi: static typing conversion, pt1
v6 cover qapi: static typing conversion, pt1
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)
In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
Notes:
- After patch 07, isort -c should pass 100% on this and every future commit.
- After patch 08, flake8 qapi/ should pass 100% on this and every future commit.
- After patch 09, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.
- After patch 18, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.
Review Status:
[01] docs-repair-broken-references # [02] qapi-modify-docstrings-to-be # [03] qapi-gen-separate-arg-parsing # [RB] CR,EH [TB] CR [SOB] JS [04] qapi-move-generator-entrypoint # [RB] CR,EH [TB] CR [SOB] JS [05] qapi-prefer-explicit-relative # [RB] CR,EH,PM [SOB] JS [06] qapi-remove-wildcard-includes # [RB] CR,EH,PM [SOB] JS [07] qapi-enforce-import-order # [RB] CR,MA [TB] CR [SOB] JS [08] qapi-delint-using-flake8 # [RB] CR,EH [SOB] JS [09] qapi-add-pylintrc # [RB] CR [TB] CR,EH [SOB] JS [10] qapi-common-py-remove-python # [RB] CR,EH [SOB] JS [11] qapi-common-add-indent-manager # [RB] CR,EH [SOB] JS [12] qapi-common-py-delint-with # [RB] CR,EH [SOB] JS [13] replace-c-by-char # [RB] CR,EH,PM [SOB] JS [14] qapi-common-py-check-with # [RB] CR [TB] CR,EH [SOB] JS [15] qapi-common-py-add-notational # [RB] CR,EH [SOB] JS [16] qapi-common-move-comments-into # [RB] CR,EH [SOB] JS [17] qapi-split-build_params-into # [RB] CR,EH [SOB] JS [18] qapi-establish-mypy-type # [RB] CR [TB] CR,EH [SOB] JS [19] qapi-events-py-add-notational # [RB] CR,EH [SOB] JS [20] qapi-events-move-comments-into # [RB] CR,EH [SOB] JS [21] qapi-commands-py-don-t-re-bind # [RB] CR,EH,MA [SOB] JS [22] qapi-commands-py-add # [RB] CR,EH [SOB] JS [23] qapi-commands-py-enable # [RB] CR,EH [SOB] JS [24] qapi-source-py-add-notational # [RB] CR,EH [TB] CR [SOB] JS [25] qapi-source-py-delint-with # [RB] CR,EH [TB] CR [SOB] JS [26] qapi-gen-py-fix-edge-case-of # [RB] CR,EH,PM [SOB] JS [27] qapi-gen-py-add-notational # [RB] CR,EH [SOB] JS [28] qapi-gen-py-enable-checking # [RB] CR,EH [TB] CR [SOB] JS [29] qapi-gen-py-remove-unused # [RB] CR,EH [SOB] JS [30] qapi-gen-py-update-write-to-be # [RB] CR,EH,MA [SOB] JS [31] qapi-gen-py-delint-with-pylint # [RB] CR,EH [SOB] JS [32] qapi-types-py-add-type-hint # [RB] CR,EH [SOB] JS [33] qapi-types-py-remove-one # [RB] CR,EH [SOB] JS [34] qapi-visit-py-assert # [RB] CR,EH [SOB] JS [35] qapi-visit-py-remove-unused # [RB] CR,EH,PM [TB] CR [SOB] JS [36] qapi-visit-py-add-notational # [RB] CR,EH [TB] CR [SOB] JS
Changelog:
002/36:[0001] [FC] 'qapi: modify docstrings to be sphinx-compatible' 003/36:[0030] [FC] 'qapi-gen: Separate arg-parsing from generation' 004/36:[down] 'qapi: move generator entrypoint into package' 008/36:[0008] [FC] 'qapi: delint using flake8'
016/36:[0017] [FC] 'qapi/common.py: Convert comments into docstrings, and elaborate' 017/36:[0004] [FC] 'qapi/common.py: move build_params into gen.py' 022/36:[0011] [FC] 'qapi/commands.py: add type hint annotations' 024/36:[0003] [FC] 'qapi/source.py: add type hint annotations' 033/36:[0022] [FC] 'qapi/types.py: remove one-letter variables'
V6:
- Rebase
- 02: Dropped changes not *strictly* required for sphinx building.
- 03: Split prefix check into helper function,
Removed default constants,
Moved error message back to main() function, changed wording
- 04: Changes from 003; fixed commit message ("package", not "module")
- 08: Line wrapping changes.
- 16: Sphinx formatting changes, docstring change.
- 17: Copyright string changes
- 22: Reflow and add type for new 'coroutine' parameter
- 24: Simpler QAPISourceInfo() typing
- 33: "memb" and "var" instead of "member" and "variant".
V5:
- Remove DO-NOT-MERGE patches (Now in Part2)
- Remove introspect.py patches (Now in Part2)
- 02: Docstring formatting, more commit message (Markus)
V4:
- Rebase on Peter Maydell's work;
- 05: Context differences from Peter Maydell's work
- 06: Remove qapi.doc
- 07: Remove qapi.doc, remove superfluous "if match"
- 09: Remove qapi.doc
- 13: remove qapi.doc
- 18: remove qapi.doc
- 22: remove qapi.doc
- 31: remove QAPIGenDoc changes
- Minor adjustments from list feedback;
- 01: More backticks for QAPI json files, now that they are in Sphinx-exe
- 02: Use :ref: role for the docker-ref cross-reference
- 04: Remove doc.py work changes; add references for start_if/end_if
- 10: Changes to accommodate isort
- 11: Added lines_after_imports=2
- 16: isort whitespace changes
- 24: Take Markus's docstring phrasing
- 34: add comment explaining os.open()
- 37: isort differences
V3:
- Use isort to enforce import consistency
- Use sphinx apidoc to check docstring format
V2:
- Removed Python3.6 patch in favor of Thomas Huth's
- Addressed (most) feedback from Markus
- Renamed type hint annotation commits
John Snow (36):
docs: repair broken references
qapi: modify docstrings to be sphinx-compatible
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into package
qapi: Prefer explicit relative imports
qapi: Remove wildcard includes
qapi: enforce import order/styling with isort
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable qapi/common.py: check with pylint
qapi/common.py: add type hint annotations qapi/common.py: Convert comments into docstrings, and elaborate qapi/common.py: move build_params into gen.py
qapi: establish mypy type-checking baseline qapi/events.py: add type hint annotations qapi/events.py: Move comments into docstrings qapi/commands.py: Don't re-bind to variable of different type qapi/commands.py: add type hint annotations qapi/commands.py: enable checking with mypy qapi/source.py: add type hint annotations qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module qapi/gen.py: add type hint annotations
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic qapi/gen.py: delint with pylint
qapi/types.py: add type hint annotations qapi/types.py: remove one-letter variables qapi/visit.py: assert tag_member contains a QAPISchemaEnumType qapi/visit.py: remove unused parameters from gen_visit_object qapi/visit.py: add type hint annotations
docs/devel/multi-thread-tcg.rst | 2 +- docs/devel/testing.rst | 2 +- scripts/qapi-gen.py | 57 ++-------- scripts/qapi/.flake8 | 2 + scripts/qapi/.isort.cfg | 7 ++ scripts/qapi/commands.py | 90 +++++++++++----- scripts/qapi/common.py | 164 ++++++++++++++++------------- scripts/qapi/events.py | 58 +++++++--- scripts/qapi/expr.py | 7 +- scripts/qapi/gen.py | 180 ++++++++++++++++++++------------ scripts/qapi/introspect.py | 16 ++- scripts/qapi/main.py | 95 +++++++++++++++++ scripts/qapi/mypy.ini | 30 ++++++ scripts/qapi/parser.py | 6 +- scripts/qapi/pylintrc | 70 +++++++++++++ scripts/qapi/schema.py | 33 +++--- scripts/qapi/source.py | 35 ++++--- scripts/qapi/types.py | 125 +++++++++++++++------- scripts/qapi/visit.py | 116 ++++++++++++++------ 19 files changed, 759 insertions(+), 336 deletions(-) create mode 100644 scripts/qapi/.flake8 create mode 100644 scripts/qapi/.isort.cfg create mode 100644 scripts/qapi/main.py create mode 100644 scripts/qapi/mypy.ini create mode 100644 scripts/qapi/pylintrc
Signature · 1 line
2.26.2
PATCH v6 1/36 docs: repair broken references
In two different places, we are not making a cross-reference to some resource correctly.
--- docs/devel/multi-thread-tcg.rst | 2 +- docs/devel/testing.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
docs/devel/multi-thread-tcg.rst +1 -1
@@ -267,7 +267,7 @@ of view of external observers (e.g. another processor core). They can apply to any memory operations as well as just loads or stores. The Linux kernel has an excellent `write-up -<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>` +<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`_ on the various forms of memory barrier and the guarantees they can provide.
docs/devel/testing.rst +1 -1
@@ -953,7 +953,7 @@ compiler flags are needed to build for a given target. If you have the ability to run containers as the user you can also take advantage of the build systems "Docker" support. It will then use containers to build any test case for an enabled guest where there is -no system compiler available. See :ref: `_docker-ref` for details. +no system compiler available. See :ref:`docker-ref` for details. Running subset of tests -----------------------
Signature · 1 line
2.26.2
PATCH v6 2/36 qapi: modify docstrings to be sphinx-compatible
In reply to John SnowA precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. For now, merely avoid obvious errors that cause Sphinx documentation build problems, using a style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
See also:
https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
--- scripts/qapi/gen.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -2
@@ -154,9 +154,11 @@ def _bottom(self): @contextmanager def ifcontext(ifcond, *args): - """A 'with' statement context manager to wrap with start_if()/end_if() + """ + A with-statement context manager that wraps with `start_if()` / `end_if()`. - *args: any number of QAPIGenCCode + :param ifcond: A list of conditionals, passed to `start_if()`. + :param args: any number of `QAPIGenCCode`. Example::
Signature · 1 line
2.26.2
PATCH v6 3/36 qapi-gen: Separate arg-parsing from generation
In reply to John SnowThis is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.
--- scripts/qapi-gen.py | 89 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 22 deletions(-)
scripts/qapi-gen.py +67 -22
@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys +from typing import Optional from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +def invalid_char(prefix: str) -> Optional[str]: + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + return prefix[match.end()] + return None + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + Generate C code for the given schema into the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + assert invalid_char(prefix) is None + + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen executable entry point. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default='', help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default='', help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',
@@ -32,25 +79,23 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) + funny_char = invalid_char(args.prefix) + if funny_char: + msg = f"funny character '{funny_char}' in argument of --prefix" + print(f"{sys.argv[0]}: {msg}", file=sys.stderr) + return 1 try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())
Signature · 1 line
2.26.2
Re: [PATCH v6 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 1 line · John Snow
This is a minor re-work of the entrypoint script. It isolates a
<pedantic>entrypoint is not a word</> ;-P
Quoted 35 lines
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 89 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 22 deletions(-)scripts/qapi-gen.py +8
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys +from typing import Optional from qapi.commands import gen_commands +from qapi.error import QAPIError
scripts/qapi-gen.py line 26
Quoted 10 lines
scripts/qapi-gen.py +2
-2from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +def invalid_char(prefix: str) -> Optional[str]:
Naming is hard... invalid_char() makes sense because it returns the invalid character. The name's a tad generic, though. Would first_invalid_prefix_char() be easier to understand?
Quoted 4 lines
scripts/qapi-gen.py +4
-0+ match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + return prefix[match.end()] + return None
[...]
Re: [PATCH v6 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow
This is a minor re-work of the entrypoint script. It isolates a
<pedantic>entrypoint is not a word</> ;-P
I'm not entirely sure why a German is complaining about the birth of a new and beautiful compound noun.
(I started to make sure I wrote it out as two words, but missed a spot. Too late now, unless you fixed it already.)
Quoted 43 lines
Quoted 43 lines
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 89 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 22 deletions(-)scripts/qapi-gen.py +9
-3@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys +from typing import Optional from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit
Quoted 7 lines
scripts/qapi-gen.py line 26
Quoted 2 lines
scripts/qapi-gen.py +1
-1-def main(argv): +def invalid_char(prefix: str) -> Optional[str]:Naming is hard...
invalid_char()makes sense because it returns the invalid character. The name's a tad generic, though. Wouldfirst_invalid_prefix_char()be easier to understand?
Sounds like an improvement to me. I guess that should be in a follow-up now.
Quoted 6 lines
Quoted 4 lines
scripts/qapi-gen.py +4
-0+ match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + return prefix[match.end()] + return None[...]
Re: [PATCH v6 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 9 lines · John Snow
Quoted 5 lines · Markus Armbruster
Quoted 1 line · John Snow
This is a minor re-work of the entrypoint script. It isolates a
<pedantic>entrypoint is not a word</> ;-P
I'm not entirely sure why a German is complaining about the birth of a new and beautiful compound noun.
Ha!
Quoted 2 lines
(I started to make sure I wrote it out as two words, but missed a spot. Too late now, unless you fixed it already.)
I didn't.
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 89 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 67 insertions(+), 22 deletions(-)scripts/qapi-gen.py +8
-3@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys +from typing import Optional from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit
Quoted 8 lines
Quoted 6 lines
scripts/qapi-gen.py line 26
Quoted 2 lines
scripts/qapi-gen.py +1
-1-def main(argv): +def invalid_char(prefix: str) -> Optional[str]:Naming is hard...
invalid_char()makes sense because it returns the invalid character. The name's a tad generic, though. Wouldfirst_invalid_prefix_char()be easier to understand?Sounds like an improvement to me. I guess that should be in a follow-up now.
I took the liberty to rename it to invalid_prefix_char() in my tree. Has since been merged into master as commit 52a474180ae.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
scripts/qapi-gen.py +4
-0+ match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + return prefix[match.end()] + return None[...]
PATCH v6 4/36 qapi: move generator entrypoint into package
In reply to John SnowAs part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the package, only leaving a very tiny entrypoint shim outside of the package.
--- scripts/qapi-gen.py | 94 +++---------------------------------------- scripts/qapi/main.py | 95 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+), 88 deletions(-) create mode 100644 scripts/qapi/main.py
scripts/qapi-gen.py +6 -88
@@ -4,98 +4,16 @@ # See the COPYING file in the top-level directory. """ -QAPI Generator +QAPI code generation execution shim. -This is the main entry point for generating C code from the QAPI schema. +This standalone script exists primarily to facilitate the running of the QAPI +code generator without needing to install the python module to the current +execution environment. """ -import argparse -import re import sys -from typing import Optional - -from qapi.commands import gen_commands -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit - - -def invalid_char(prefix: str) -> Optional[str]: - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) - if match.end() != len(prefix): - return prefix[match.end()] - return None - - -def generate(schema_file: str, - output_dir: str, - prefix: str, - unmask: bool = False, - builtins: bool = False) -> None: - """ - Generate C code for the given schema into the target directory. - - :param schema_file: The primary QAPI schema file. - :param output_dir: The output directory to store generated code. - :param prefix: Optional C-code prefix for symbol names. - :param unmask: Expose non-ABI names through introspection? - :param builtins: Generate code for built-in types? - - :raise QAPIError: On failures. - """ - assert invalid_char(prefix) is None - - schema = QAPISchema(schema_file) - gen_types(schema, output_dir, prefix, builtins) - gen_visit(schema, output_dir, prefix, builtins) - gen_commands(schema, output_dir, prefix) - gen_events(schema, output_dir, prefix) - gen_introspect(schema, output_dir, prefix, unmask) - - -def main() -> int: - """ - gapi-gen executable entry point. - Expects arguments via sys.argv, see --help for details. - - :return: int, 0 on success, 1 on failure. - """ - parser = argparse.ArgumentParser( - description='Generate code from a QAPI schema') - parser.add_argument('-b', '--builtins', action='store_true', - help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', - default='', - help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', - default='', - help="prefix for symbols") - parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', - dest='unmask', - help="expose non-ABI names in introspection") - parser.add_argument('schema', action='store') - args = parser.parse_args() - - funny_char = invalid_char(args.prefix) - if funny_char: - msg = f"funny character '{funny_char}' in argument of --prefix" - print(f"{sys.argv[0]}: {msg}", file=sys.stderr) - return 1 - - try: - generate(args.schema, - output_dir=args.output_dir, - prefix=args.prefix, - unmask=args.unmask, - builtins=args.builtins) - except QAPIError as err: - print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) - return 1 - return 0 +from qapi import main if __name__ == '__main__': - sys.exit(main()) + sys.exit(main.main())
scripts/qapi/main.py new file +95 -0
95 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v6 5/36 qapi: Prefer explicit relative imports
In reply to John SnowAll of the QAPI include statements are changed to be package-aware, as explicit relative imports.
A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.
When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.
This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.
--- scripts/qapi/commands.py | 4 ++-- scripts/qapi/events.py | 8 ++++---- scripts/qapi/expr.py | 4 ++-- scripts/qapi/gen.py | 4 ++-- scripts/qapi/introspect.py | 8 ++++---- scripts/qapi/main.py | 14 +++++++------- scripts/qapi/parser.py | 4 ++-- scripts/qapi/schema.py | 8 ++++---- scripts/qapi/types.py | 6 +++--- scripts/qapi/visit.py | 6 +++--- 10 files changed, 33 insertions(+), 33 deletions(-)
scripts/qapi/commands.py +2 -2
@@ -13,8 +13,8 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import * +from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/events.py +4 -4
@@ -12,10 +12,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember -from qapi.types import gen_enum, gen_enum_lookup +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember +from .types import gen_enum, gen_enum_lookup def build_event_send_proto(name, arg_type, boxed):
scripts/qapi/expr.py +2 -2
@@ -16,8 +16,8 @@ import re from collections import OrderedDict -from qapi.common import c_name -from qapi.error import QAPISemError +from .common import c_name +from .error import QAPISemError # Names must be letters, numbers, -, and _. They must start with letter,
scripts/qapi/gen.py +2 -2
@@ -17,8 +17,8 @@ import re from contextlib import contextmanager -from qapi.common import * -from qapi.schema import QAPISchemaVisitor +from .common import * +from .schema import QAPISchemaVisitor class QAPIGen:
scripts/qapi/introspect.py +4 -4
@@ -10,10 +10,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaMonolithicCVisitor -from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .common import * +from .gen import QAPISchemaMonolithicCVisitor +from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, + QAPISchemaType) def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/main.py +7 -7
@@ -12,13 +12,13 @@ import sys from typing import Optional -from qapi.commands import gen_commands -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit +from .commands import gen_commands +from .error import QAPIError +from .events import gen_events +from .introspect import gen_introspect +from .schema import QAPISchema +from .types import gen_types +from .visit import gen_visit def invalid_char(prefix: str) -> Optional[str]:
scripts/qapi/parser.py +2 -2
@@ -18,8 +18,8 @@ import re from collections import OrderedDict -from qapi.error import QAPIParseError, QAPISemError -from qapi.source import QAPISourceInfo +from .error import QAPIParseError, QAPISemError +from .source import QAPISourceInfo class QAPISchemaParser:
scripts/qapi/schema.py +4 -4
@@ -18,10 +18,10 @@ import re from collections import OrderedDict -from qapi.common import c_name, pointer_suffix -from qapi.error import QAPIError, QAPISemError -from qapi.expr import check_exprs -from qapi.parser import QAPISchemaParser +from .common import c_name, pointer_suffix +from .error import QAPIError, QAPISemError +from .expr import check_exprs +from .parser import QAPISchemaParser class QAPISchemaEntity:
scripts/qapi/types.py +3 -3
@@ -13,9 +13,9 @@ # See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember, QAPISchemaObjectType # variants must be emitted before their container; track what has already
scripts/qapi/visit.py +3 -3
@@ -13,9 +13,9 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
Signature · 1 line
2.26.2
PATCH v6 6/36 qapi: Remove wildcard includes
In reply to John SnowWildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
--- scripts/qapi/commands.py | 2 +- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 38 insertions(+), 8 deletions(-)
scripts/qapi/commands.py +1 -1
@@ -13,7 +13,7 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import build_params, c_name, mcgen from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
scripts/qapi/events.py +6 -1
@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +9 -3
@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitor
scripts/qapi/introspect.py +6 -1
@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)
scripts/qapi/types.py +7 -1
@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
scripts/qapi/visit.py +9 -1
@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
Signature · 1 line
2.26.2
PATCH v6 7/36 qapi: enforce import order/styling with isort
In reply to John SnowWhile we're mucking around with imports, we might as well formalize the style we use. Let's use isort to do it for us.
lines_after_imports=2: Use two lines after imports, to match PEP8's desire to have "two lines before and after" class definitions, which are likely to start immediately after imports.
force_sort_within_sections: Intermingles "from x" and "import x" style statements, such that sorting is always performed strictly on the module name itself.
force_grid_wrap=4: Four or more imports from a single module will force the one-per-line style that's more git-friendly. This will generally happen for 'typing' imports.
multi_line_output=3: Uses the one-per-line indented style for long imports.
include_trailing_comma: Adds a comma to the last import in a group, which makes git conflicts nicer to deal with, generally.
line_length: 72 is chosen to match PEP8's "docstrings and comments" line length limit. If you have a single line import that exceeds 72 characters, your names are too long!
--- scripts/qapi/.isort.cfg | 7 +++++++ scripts/qapi/expr.py | 3 ++- scripts/qapi/introspect.py | 7 +++++-- scripts/qapi/parser.py | 2 +- scripts/qapi/schema.py | 2 +- 5 files changed, 16 insertions(+), 5 deletions(-) create mode 100644 scripts/qapi/.isort.cfg
scripts/qapi/.isort.cfg new file +7 -0
@@ -0,0 +1,7 @@ +[settings] +force_grid_wrap=4 +force_sort_within_sections=True +include_trailing_comma=True +line_length=72 +lines_after_imports=2 +multi_line_output=3
scripts/qapi/expr.py +2 -1
@@ -14,8 +14,9 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -import re from collections import OrderedDict +import re + from .common import c_name from .error import QAPISemError
scripts/qapi/introspect.py +5 -2
@@ -17,8 +17,11 @@ mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .schema import ( + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaType, +) def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/parser.py +1 -1
@@ -14,9 +14,9 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. +from collections import OrderedDict import os import re -from collections import OrderedDict from .error import QAPIParseError, QAPISemError from .source import QAPISourceInfo
scripts/qapi/schema.py +1 -1
@@ -14,9 +14,9 @@ # TODO catching name collisions in generated code would be nice +from collections import OrderedDict import os import re -from collections import OrderedDict from .common import c_name, pointer_suffix from .error import QAPIError, QAPISemError
Signature · 1 line
2.26.2
PATCH v6 8/36 qapi: delint using flake8
In reply to John SnowPetty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 16 +++++++++++----- 4 files changed, 20 insertions(+), 9 deletions(-) create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Prefer pylint's bare-except checks to flake8's
scripts/qapi/commands.py +2 -1
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, + QObject **ret_out, Error **errp) { Visitor *v;
scripts/qapi/schema.py +5 -3
@@ -536,7 +536,7 @@ def set_defined_in(self, name): v.set_defined_in(name) def check(self, schema, seen): - if not self.tag_member: # flat union + if not self.tag_member: # flat union self.tag_member = seen.get(c_name(self._tag_name)) base = "'base'" # Pointing to the base type when not implicit would be
@@ -824,7 +824,7 @@ def __init__(self, fname): self._entity_dict = {} self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) - self._make_module(None) # built-ins + self._make_module(None) # built-ins self._make_module(fname) self._predefining = True self._def_predefineds()
@@ -968,7 +968,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members): # But it's not tight: the disjunction need not imply it. We # may end up compiling useless wrapper types. # TODO kill simple unions or implement the disjunction - assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access + + # pylint: disable=protected-access + assert (ifcond or []) == typ._ifcond else: self._def_entity(QAPISchemaObjectType( name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +11 -5
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False): if not scalar: c_type += '*' return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp); + +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_type)sobj, Error **errp); ''', c_name=c_name(name), c_type=c_type)
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants): def gen_visit_list(name, element_type): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false; %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type): def gen_visit_enum(name): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s *obj, Error **errp) { int value = *obj; bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ def gen_visit_enum(name): def gen_visit_alternate(name, variants): ret = mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false;
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants): def gen_visit_object(name, base, members, variants): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false;
Signature · 1 line
2.26.2
PATCH v6 9/36 qapi: add pylintrc
In reply to John SnowUsing pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.
I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is released), I aim to have gitlab CI running the specific targeted versions of pylint, mypy, flake8, etc in a job.
2.5.x will work if you additionally pass --disable=bad-whitespace. This warning was removed from 2.6.x, for lack of consistent support.
Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5+, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrc
--- scripts/qapi/pylintrc | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +73 -0
@@ -0,0 +1,73 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the ignore list. +# The regex matches against base names, not paths. +ignore-patterns=common.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py, + + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes, + +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _ + +[VARIABLES] + +[STRING] + +[SPELLING] + +[FORMAT] + +[SIMILARITIES] + +# Ignore import statements themselves when computing similarities. +ignore-imports=yes + +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]
Signature · 1 line
2.26.2
PATCH v6 10/36 qapi/common.py: Remove python compatibility workaround
In reply to John Snow--- scripts/qapi/common.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +1 -4
@@ -119,10 +119,7 @@ def cgen(code, **kwds): raw = code % kwds if indent_level: indent = genindent(indent_level) - # re.subn() lacks flags support before Python 2.7, use re.compile() - raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE), - indent, raw) - raw = raw[0] + raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
Signature · 1 line
2.26.2
PATCH v6 11/36 qapi/common.py: Add indent manager
In reply to John SnowCode style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
--- scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++-------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 36 insertions(+), 20 deletions(-)
scripts/qapi/common.py +33 -16
@@ -93,33 +93,50 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> None: + """Increase the indentation level by ``amount``, default 4.""" + self._level += amount + + def decrease(self, amount: int = 4) -> None: + """Decrease the indentation level by ``amount``, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + + +indent = Indentation() # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey indent, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if indent: + raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
scripts/qapi/visit.py +3 -4
@@ -18,9 +18,8 @@ c_name, gen_endif, gen_if, + indent, mcgen, - pop_indent, - push_indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + indent.increase() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + indent.decrease() ret += mcgen(''' } ''')
Signature · 1 line
2.26.2
PATCH v6 12/36 qapi/common.py: delint with pylint
In reply to John SnowAt this point, that just means using a consistent strategy for constant names. constants get UPPER_CASE and names not used externally get a leading underscore.
As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
--- scripts/qapi/common.py | 18 ++++++++---------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 15 insertions(+), 17 deletions(-)
scripts/qapi/common.py +8 -10
@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +_C_NAME_TRANS = str.maketrans('.-', '__') + + # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__') - - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True): 'not_eq', 'or', 'or_eq', 'xor', 'xor_eq']) # namespace pollution: polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386']) - name = name.translate(c_name_trans) + name = name.translate(_C_NAME_TRANS) if protect and (name in c89_words | c99_words | c11_words | gcc_words | cpp_words | polluted_words): return 'q_' + name return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indentation: """ Indentation level management.
@@ -132,12 +130,12 @@ def decrease(self, amount: int = 4) -> None: # Generate @code with @kwds interpolated. -# Obey indent, and strip eatspace. +# Obey indent, and strip EATSPACE. def cgen(code, **kwds): raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):
scripts/qapi/schema.py +7 -7
@@ -18,7 +18,7 @@ import os import re -from .common import c_name, pointer_suffix +from .common import POINTER_SUFFIX, c_name from .error import QAPIError, QAPISemError from .expr import check_exprs from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self): return True def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'array'
@@ -430,7 +430,7 @@ def c_name(self): def c_type(self): assert not self.is_implicit() - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def c_unboxed_type(self): return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None): v.connect_doc(doc) def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'value'
@@ -899,7 +899,7 @@ def _def_builtin_type(self, name, json_type, c_type): self._make_array_type(name, None) def _def_predefineds(self): - for t in [('str', 'string', 'char' + pointer_suffix), + for t in [('str', 'string', 'char' + POINTER_SUFFIX), ('number', 'number', 'double'), ('int', 'int', 'int64_t'), ('int8', 'int', 'int8_t'),
@@ -912,8 +912,8 @@ def _def_predefineds(self): ('uint64', 'int', 'uint64_t'), ('size', 'int', 'uint64_t'), ('bool', 'boolean', 'bool'), - ('any', 'value', 'QObject' + pointer_suffix), - ('null', 'null', 'QNull' + pointer_suffix)]: + ('any', 'value', 'QObject' + POINTER_SUFFIX), + ('null', 'null', 'QNull' + POINTER_SUFFIX)]: self._def_builtin_type(*t) self.the_empty_object_type = QAPISchemaObjectType( 'q_empty', None, None, None, None, None, [], None)
Signature · 1 line
2.26.2
PATCH v6 13/36 qapi/common.py: Replace one-letter 'c' variable
In reply to John Snow--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_': + char = c_fun_str[i] + # When char is upper case and no '_' appears before, do more checks + if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_': if i < length - 1 and c_fun_str[i + 1].islower(): new_name += '_' elif c_fun_str[i - 1].isdigit(): new_name += '_' - new_name += c + new_name += char return new_name.lstrip('_').upper()
Signature · 1 line
2.26.2
PATCH v6 14/36 qapi/common.py: check with pylint
In reply to John SnowRemove qapi/common.py from the pylintrc ignore list.
--- scripts/qapi/pylintrc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi/pylintrc +1 -2
@@ -2,8 +2,7 @@ # Add files or directories matching the regex patterns to the ignore list. # The regex matches against base names, not paths. -ignore-patterns=common.py, - error.py, +ignore-patterns=error.py, expr.py, gen.py, parser.py,
Signature · 1 line
2.26.2
PATCH v6 15/36 qapi/common.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
Note that build_params() cannot be fully annotated due to import dependency issues. The commit after next will take care of it.
--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +16 -11
@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Sequence EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None: # Generate @code with @kwds interpolated. # Obey indent, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: object) -> str: raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: object) -> str: if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''
@@ -183,7 +186,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:
Signature · 1 line
2.26.2
PATCH v6 16/36 qapi/common.py: Convert comments into docstrings, and elaborate
In reply to John SnowAs docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Primarily, we use :param X: descr, :return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.
(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)
--- scripts/qapi/common.py | 54 +++++++++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 14 deletions(-)
scripts/qapi/common.py +40 -14
@@ -15,15 +15,25 @@ from typing import Optional, Sequence +#: Magic string that gets removed along with all space to its right. EATSPACE = '\033EATSPACE.' POINTER_SUFFIX = ' *' + EATSPACE _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples:: + + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -45,21 +55,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map ``name`` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +151,16 @@ def decrease(self, amount: int = 4) -> None: self._level -= amount +#: Global, current indent level for code generation. indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate ``code`` with ``kwds`` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """ raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
Signature · 1 line
2.26.2
PATCH v6 17/36 qapi/common.py: move build_params into gen.py
In reply to John SnowIncluding it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.
--- scripts/qapi/commands.py | 9 +++++++-- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 9 ++------- scripts/qapi/gen.py | 29 +++++++++++++++++++++++++++-- 4 files changed, 36 insertions(+), 34 deletions(-)
scripts/qapi/commands.py +7 -2
@@ -13,8 +13,13 @@ See the COPYING file in the top-level directory. """ -from .common import build_params, c_name, mcgen -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import c_name, mcgen +from .gen import ( + QAPIGenCCode, + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
@@ -210,26 +210,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'
scripts/qapi/events.py +2 -7
@@ -12,13 +12,8 @@ See the COPYING file in the top-level directory. """ -from .common import ( - build_params, - c_enum_const, - c_name, - mcgen, -) -from .gen import QAPISchemaModularCVisitor, ifcontext +from .common import c_enum_const, c_name, mcgen +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +27 -2
@@ -2,7 +2,7 @@ # # QAPI code generation # -# Copyright (c) 2018-2019 Red Hat Inc. +# Copyright (c) 2015-2019 Red Hat Inc. # # Authors: # Markus Armbruster <[email protected]>
@@ -15,16 +15,18 @@ import errno import os import re +from typing import Optional from .common import ( c_fname, + c_name, gen_endif, gen_if, guardend, guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import QAPISchemaObjectType, QAPISchemaVisitor class QAPIGen:
@@ -90,6 +92,29 @@ def _wrap_ifcond(ifcond, before, after): return out +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void' + + class QAPIGenCCode(QAPIGen): def __init__(self, fname):
Signature · 1 line
2.26.2
PATCH v6 18/36 qapi: establish mypy type-checking baseline
In reply to John SnowFix a minor typing issue, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
This is designed and tested for mypy 0.770 or greater.
--- scripts/qapi/mypy.ini | 60 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 ++- 2 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/mypy.ini new file +60 -0
@@ -0,0 +1,60 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False
scripts/qapi/schema.py +2 -1
@@ -17,6 +17,7 @@ from collections import OrderedDict import os import re +from typing import Optional from .common import POINTER_SUFFIX, c_name from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Signature · 1 line
2.26.2
PATCH v6 19/36 qapi/events.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
Note: __init__ does not need its return type annotated, as it is special.
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
--- scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 ----- 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +35 -11
@@ -12,19 +12,31 @@ See the COPYING file in the top-level directory. """ +from typing import List + from .common import c_enum_const, c_name, mcgen from .gen import QAPISchemaModularCVisitor, build_params, ifcontext -from .schema import QAPISchemaEnumMember +from .schema import ( + QAPISchema, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, +) +from .source import QAPISourceInfo from .types import gen_enum, gen_enum_lookup -def build_event_send_proto(name, arg_type, boxed): +def build_event_send_proto(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return 'void qapi_event_send_%(c_name)s(%(param)s)' % { 'c_name': c_name(name.lower()), 'param': build_params(arg_type, boxed)} -def gen_event_send_decl(name, arg_type, boxed): +def gen_event_send_decl(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return mcgen(''' %(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed): # Declare and initialize an object 'qapi' using parameters from build_params() -def gen_param_var(typ): +def gen_param_var(typ: QAPISchemaObjectType) -> str: assert not typ.variants ret = mcgen(''' %(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ): return ret -def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): +def gen_event_send(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool, + event_enum_name: str, + event_emit: str) -> str: # FIXME: Our declaration of local variables (and of 'errp' in the # parameter list) can collide with exploded members of the event's # data type passed in as parameters. If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-events', ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) - self._event_enum_members = [] + self._event_enum_members: List[QAPISchemaEnumMember] = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit') - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: events = self._module_basename('qapi-events', name) types = self._module_basename('qapi-types', name) visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('emit', ' * QAPI Events emission') self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h"
@@ -189,7 +205,13 @@ def visit_end(self): event_emit=self._event_emit_name, event_enum=self._event_enum_name)) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + boxed: bool) -> None: with ifcontext(ifcond, self._genh, self._genc): self._genh.add(gen_event_send_decl(name, arg_type, boxed)) self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -200,7 +222,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed): self._event_enum_members.append(QAPISchemaEnumMember(name, None)) -def gen_events(schema, output_dir, prefix): +def gen_events(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenEventVisitor(prefix) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -14,11 +14,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.events] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.expr] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
PATCH v6 20/36 qapi/events.py: Move comments into docstrings
In reply to John SnowClarify them while we're here.
--- scripts/qapi/events.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/qapi/events.py +5 -1
@@ -44,8 +44,12 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Generate a struct variable holding the event parameters. + + Initialize it with the function arguments defined in `gen_event_send`. + """ assert not typ.variants ret = mcgen(''' %(c_name)s param = {
Signature · 1 line
2.26.2
PATCH v6 21/36 qapi/commands.py: Don't re-bind to variable of different type
In reply to John SnowMypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
--- scripts/qapi/commands.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +1 -3
@@ -198,14 +198,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig, if not options: options = ['QCO_NO_OPTIONS'] - options = " | ".join(options) - ret = mcgen(''' qmp_register_command(cmds, "%(name)s", qmp_marshal_%(c_name)s, %(opts)s); ''', name=name, c_name=c_name(name), - opts=options) + opts=" | ".join(options)) return ret
Signature · 1 line
2.26.2
PATCH v6 22/36 qapi/commands.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/commands.py | 74 ++++++++++++++++++++++++++++++---------- 1 file changed, 56 insertions(+), 18 deletions(-)
scripts/qapi/commands.py +56 -18
@@ -13,16 +13,34 @@ See the COPYING file in the top-level directory. """ +from typing import ( + Dict, + List, + Optional, + Set, +) + from .common import c_name, mcgen from .gen import ( + QAPIGenC, QAPIGenCCode, QAPISchemaModularCVisitor, build_params, ifcontext, ) +from .schema import ( + QAPISchema, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaType, +) +from .source import QAPISourceInfo -def gen_command_decl(name, arg_type, boxed, ret_type): +def gen_command_decl(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: return mcgen(''' %(c_type)s qmp_%(c_name)s(%(params)s); ''',
@@ -31,7 +49,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type): params=build_params(arg_type, boxed, 'Error **errp')) -def gen_call(name, arg_type, boxed, ret_type): +def gen_call(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: ret = '' argstr = ''
@@ -67,7 +88,7 @@ def gen_call(name, arg_type, boxed, ret_type): return ret -def gen_marshal_output(ret_type): +def gen_marshal_output(ret_type: QAPISchemaType) -> str: return mcgen(''' static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -88,19 +109,22 @@ def gen_marshal_output(ret_type): c_type=ret_type.c_type(), c_name=ret_type.c_name()) -def build_marshal_proto(name): +def build_marshal_proto(name: str) -> str: return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)) -def gen_marshal_decl(name): +def gen_marshal_decl(name: str) -> str: return mcgen(''' %(proto)s; ''', proto=build_marshal_proto(name)) -def gen_marshal(name, arg_type, boxed, ret_type): +def gen_marshal(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: have_args = boxed or (arg_type and not arg_type.is_empty()) ret = mcgen('''
@@ -182,8 +206,11 @@ def gen_marshal(name, arg_type, boxed, ret_type): return ret -def gen_register_command(name, success_response, allow_oob, allow_preconfig, - coroutine): +def gen_register_command(name: str, + success_response: bool, + allow_oob: bool, + allow_preconfig: bool, + coroutine: bool) -> str: options = [] if not success_response:
@@ -207,7 +234,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig, return ret -def gen_registry(registry, prefix): +def gen_registry(registry: str, prefix: str) -> str: ret = mcgen(''' void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,15 +251,14 @@ def gen_registry(registry, prefix): class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): - - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-commands', ' * Schema-defined QAPI/QMP commands', None, __doc__) self._regy = QAPIGenCCode(None) - self._visited_ret_types = {} + self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {} - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: self._visited_ret_types[self._genc] = set() commands = self._module_basename('qapi-commands', name) types = self._module_basename('qapi-types', name)
@@ -256,7 +282,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('init', ' * QAPI Commands initialization') self._genh.add(mcgen(''' #include "qapi/qmp/dispatch.h"
@@ -272,9 +298,19 @@ def visit_end(self): prefix=self._prefix)) self._genc.add(gen_registry(self._regy.get_content(), self._prefix)) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig, coroutine): + def visit_command(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool, + coroutine: bool) -> None: if not gen: return # FIXME: If T is a user-defined type, the user is responsible
@@ -296,7 +332,9 @@ def visit_command(self, name, info, ifcond, features, coroutine)) -def gen_commands(schema, output_dir, prefix): +def gen_commands(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenCommandVisitor(prefix) schema.visit(vis) vis.write(output_dir)
Signature · 1 line
2.26.2
PATCH v6 23/36 qapi/commands.py: enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.error] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
PATCH v6 24/36 qapi/source.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.
Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.
--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 32 +++++++++++++++++++------------- 2 files changed, 19 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -34,11 +34,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/source.py +19 -13
@@ -11,37 +11,43 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self, fname: str, line: int, + parent: Optional['QAPISourceInfo']): self.fname = fname self.line = line self.parent = parent - self.pragma = parent.pragma if parent else QAPISchemaPragma() - self.defn_meta = None - self.defn_name = None + self.pragma: QAPISchemaPragma = ( + parent.pragma if parent else QAPISchemaPragma() + ) + self.defn_meta: Optional[str] = None + self.defn_name: Optional[str] = None - def set_defn(self, meta, name): + def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self): + def next_line(self: T) -> T: info = copy.copy(self) info.line += 1 return info - def loc(self): + def loc(self) -> str: if self.fname is None: return sys.argv[0] ret = self.fname
@@ -49,13 +55,13 @@ def loc(self): ret += ':%d' % self.line return ret - def in_defn(self): + def in_defn(self) -> str: if self.defn_name: return "%s: In %s '%s':\n" % (self.fname, self.defn_meta, self.defn_name) return '' - def include_path(self): + def include_path(self) -> str: ret = '' parent = self.parent while parent:
@@ -63,5 +69,5 @@ def include_path(self): parent = parent.parent return ret - def __str__(self): + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc()
Signature · 1 line
2.26.2
PATCH v6 25/36 qapi/source.py: delint with pylint
In reply to John SnowShush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/source.py | 3 +++ 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/pylintrc +0 -1
@@ -7,7 +7,6 @@ ignore-patterns=error.py, gen.py, parser.py, schema.py, - source.py, types.py, visit.py,
scripts/qapi/source.py +3 -0
@@ -15,6 +15,9 @@ class QAPISchemaPragma: + # Replace with @dataclass in Python 3.7+ + # pylint: disable=too-few-public-methods + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False
Signature · 1 line
2.26.2
PATCH v6 26/36 qapi/gen.py: Fix edge-case of _is_user_module
In reply to John SnowThe edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
In practice, module names are not allowed to be the empty string, but this constraint is not modeled for the type system.
--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +1 -1
@@ -241,7 +241,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./')) @staticmethod def _is_builtin_module(name):
Signature · 1 line
2.26.2
Re: [PATCH v6 26/36] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 19 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
In practice, module names are not allowed to be the empty string, but this constraint is not modeled for the type system.
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Philippe Mathieu-Daudé <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 244
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -241,7 +241,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./'))
return not (name is None or name.startswith('./')
Looks slightly clearer to me.
Quoted 3 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):
Re: [PATCH v6 26/36] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 22 lines · Markus Armbruster
Quoted 20 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
In practice, module names are not allowed to be the empty string, but this constraint is not modeled for the type system.
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Philippe Mathieu-Daudé <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0@@ -241,7 +241,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
Quoted 8 lines
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 244
Quoted 4 lines
scripts/qapi/gen.py +1
-1@staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./'))return not (name is None or
name.startswith('./')Looks slightly clearer to me.
That would have exactly the same behavior as the
name is not None and name.startswith('./')
expression we had in v1.
Signature · 1 line
Eduardo
Re: [PATCH v6 26/36] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type. In practice, module names are not allowed to be the empty string, but this constraint is not modeled for the type system.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Philippe Mathieu-Daudé <[email protected]>
Actually:
1. We also map None to None, which is also not bool.
2. There is no declared type to violate until the next patch.
3. The subject's claim 'Fix edge-case' is misleading: this is a cleanup, not a bug fix.
Easy enough to fix:
qapi/gen: Make _is_user_module() return bool
_is_user_module() returns thruth values. The next commit wants it to return bool. Make it so.
Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 244
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -241,7 +241,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./'))return not (name is None or
name.startswith('./')
Quoted 5 lines
Quoted 1 line
Looks slightly clearer to me.
That would have exactly the same behavior as the
name is not None andname.startswith('./')
expression we had in v1.
True.
Let's review what this function should do, and what it does.
The function should return whether @name is a user module name. Returning truthy and falsy is fine; the callers expect no more.
system module names are either pathnames starting with './' (see _add_system_module(), or None.
user module names are pathnames not starting with './' (see _module_name()).
Before the patch:
if @name is falsy, i.e. None or '', return @name else return name.startswith('./')
Thus, the function maps
None -> None (1)
'' -> '' (2)
'./' + any string -> False (3)
any other string -> True (4)This is correct. Case (2) can't actually happen ('' is not a pathname).
John's version of the patch normalizes case (1) and (2) to
None -> False (1)
'' -> False (2)so the next patch can declare the function returns bool. Safe, because it doesn't change "thruthiness".
My version of the patch
if @name is None, return False,
else return not name.startswith('./')
Now the function maps
None -> False (1)
'' -> True (2)
'./' + any string -> False (3)
any other string -> True (4)The only difference to John's patch is case (2). That case is impossible, so no difference in actual behavior. Nevertheless, mapping '' to True is unclean: it claims '' is a user module name, which it isn't.
This would be clean:
@staticmethod
def _is_system_module(name):
return name is None or name.startswith('./')
Adjusting callers would be straightforward.
Not worth it right now. Taking John's patch with the rewritten commit message.
Eduardo, thanks for making me think!
PATCH v6 27/36 qapi/gen.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +57 -47
@@ -15,7 +15,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,
@@ -27,31 +33,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.
@@ -76,7 +82,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif
@@ -116,40 +122,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
@@ -165,7 +169,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */
@@ -175,16 +179,15 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]: """ A with-statement context manager that wraps with `start_if()` / `end_if()`.
@@ -212,8 +215,11 @@ def ifcontext(ifcond, *args): class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + blurb: str, + pydoc: str): self._prefix = prefix self._what = what self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -221,38 +227,42 @@ def __init__(self, prefix, what, blurb, pydoc): self._genh = QAPIGenH(self._prefix + self._what + '.h', blurb, pydoc) - def write(self, output_dir): + def write(self, output_dir: str) -> None: self._genc.write(output_dir) self._genh.write(output_dir) class QAPISchemaModularCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + user_blurb: str, + builtin_blurb: Optional[str], + pydoc: str): self._prefix = prefix self._what = what self._user_blurb = user_blurb self._builtin_blurb = builtin_blurb self._pydoc = pydoc - self._genc = None - self._genh = None - self._module = {} - self._main_module = None + self._genc: Optional[QAPIGenC] = None + self._genh: Optional[QAPIGenH] = None + self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {} + self._main_module: Optional[str] = None @staticmethod - def _is_user_module(name): + def _is_user_module(name: Optional[str]) -> bool: return bool(name and not name.startswith('./')) @staticmethod - def _is_builtin_module(name): + def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what, name): + def _module_dirname(self, what: str, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return '' - def _module_basename(self, what, name): + def _module_basename(self, what: str, name: Optional[str]) -> str: ret = '' if self._is_builtin_module(name) else self._prefix if self._is_user_module(name): basename = os.path.basename(name)
@@ -264,27 +274,27 @@ def _module_basename(self, what, name): ret += re.sub(r'-', '-' + name + '-', what) return ret - def _module_filename(self, what, name): + def _module_filename(self, what: str, name: Optional[str]) -> str: return os.path.join(self._module_dirname(what, name), self._module_basename(what, name)) - def _add_module(self, name, blurb): + def _add_module(self, name: Optional[str], blurb: str) -> None: basename = self._module_filename(self._what, name) genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) self._genc, self._genh = self._module[name] - def _add_user_module(self, name, blurb): + def _add_user_module(self, name: str, blurb: str) -> None: assert self._is_user_module(name) if self._main_module is None: self._main_module = name self._add_module(name, blurb) - def _add_system_module(self, name, blurb): + def _add_system_module(self, name: Optional[str], blurb: str) -> None: self._add_module(name and './' + name, blurb) - def write(self, output_dir, opt_builtins=False): + def write(self, output_dir: str, opt_builtins: bool = False) -> None: for name in self._module: if self._is_builtin_module(name) and not opt_builtins: continue
@@ -292,13 +302,13 @@ def write(self, output_dir, opt_builtins=False): genc.write(output_dir) genh.write(output_dir) - def _begin_system_module(self, name): + def _begin_system_module(self, name: None) -> None: pass - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: pass - def visit_module(self, name): + def visit_module(self, name: Optional[str]) -> None: if name is None: if self._builtin_blurb: self._add_system_module(None, self._builtin_blurb)
@@ -312,7 +322,7 @@ def visit_module(self, name): self._add_user_module(name, self._user_blurb) self._begin_user_module(name) - def visit_include(self, name, info): + def visit_include(self, name: str, info: QAPISourceInfo) -> None: relname = os.path.relpath(self._module_filename(self._what, name), os.path.dirname(self._genh.fname)) self._genh.preamble_add(mcgen('''
Signature · 1 line
2.26.2
PATCH v6 28/36 qapi/gen.py: Enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -14,11 +14,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.gen] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.introspect] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
PATCH v6 29/36 qapi/gen.py: Remove unused parameter
In reply to John Snow_module_dirname doesn't use the 'what' argument, so remove it.
--- scripts/qapi/gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
@@ -257,7 +257,7 @@ def _is_user_module(name: Optional[str]) -> bool: def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what: str, name: Optional[str]) -> str: + def _module_dirname(self, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return ''
@@ -275,7 +275,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str: return ret def _module_filename(self, what: str, name: Optional[str]) -> str: - return os.path.join(self._module_dirname(what, name), + return os.path.join(self._module_dirname(name), self._module_basename(what, name)) def _add_module(self, name: Optional[str], blurb: str) -> None:
Signature · 1 line
2.26.2
PATCH v6 30/36 qapi/gen.py: update write() to be more idiomatic
In reply to John SnowMake the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.
--- scripts/qapi/gen.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +11 -14
@@ -12,7 +12,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import (
@@ -65,21 +64,19 @@ def write(self, output_dir: str) -> None: return pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try: - os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True) + + # use os.open for O_CREAT to create and read a non-existant file fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close() + with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text) def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Signature · 1 line
2.26.2
PATCH v6 31/36 qapi/gen.py: delint with pylint
In reply to John Snow'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_top and _bottom also need to stay standard methods because some users override the method and need to use self. Tell pylint to shush.
--- scripts/qapi/gen.py | 2 ++ scripts/qapi/pylintrc | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -0
@@ -51,9 +51,11 @@ def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() def _top(self) -> str: + # pylint: disable=no-self-use return '' def _bottom(self) -> str: + # pylint: disable=no-self-use return '' def write(self, output_dir: str) -> None:
scripts/qapi/pylintrc +3 -2
@@ -4,7 +4,6 @@ # The regex matches against base names, not paths. ignore-patterns=error.py, expr.py, - gen.py, parser.py, schema.py, types.py,
@@ -45,7 +44,9 @@ good-names=i, k, ex, Run, - _ + _, + fp, # fp = open(...) + fd, # fd = os.open(...) [VARIABLES]
Signature · 1 line
2.26.2
PATCH v6 32/36 qapi/types.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++----------- 2 files changed, 64 insertions(+), 27 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -29,11 +29,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.types] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.visit] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/types.py +64 -22
207 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v6 33/36 qapi/types.py: remove one-letter variables
In reply to John Snow"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/types.py | 29 +++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -6,7 +6,6 @@ ignore-patterns=error.py, expr.py, parser.py, schema.py, - types.py, visit.py,
scripts/qapi/types.py +15 -14
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str, .array = (const char *const[]) { ''', c_name=c_name(name)) - for m in members: - ret += gen_if(m.ifcond) - index = c_enum_const(name, m.name, prefix) + for memb in members: + ret += gen_if(memb.ifcond) + index = c_enum_const(name, memb.name, prefix) ret += mcgen(''' [%(index)s] = "%(name)s", ''', - index=index, name=m.name) - ret += gen_endif(m.ifcond) + index=index, name=memb.name) + ret += gen_endif(memb.ifcond) ret += mcgen(''' },
@@ -79,13 +79,13 @@ def gen_enum(name: str, ''', c_name=c_name(name)) - for m in enum_members: - ret += gen_if(m.ifcond) + for memb in enum_members: + ret += gen_if(memb.ifcond) ret += mcgen(''' %(c_enum)s, ''', - c_enum=c_enum_const(name, m.name, prefix)) - ret += gen_endif(m.ifcond) + c_enum=c_enum_const(name, memb.name, prefix)) + ret += gen_endif(memb.ifcond) ret += mcgen(''' } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name) ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for var in variants.variants if variants else (): + obj = var.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''
Signature · 1 line
2.26.2
PATCH v6 34/36 qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
In reply to John SnowThis is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.
--- scripts/qapi/visit.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +7 -5
@@ -22,7 +22,7 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import QAPISchemaEnumType, QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType) + ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Signature · 1 line
2.26.2
PATCH v6 35/36 qapi/visit.py: remove unused parameters from gen_visit_object
In reply to John SnowAnd this fixes the pylint report for this file, so make sure we check this in the future, too.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/visit.py | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -6,7 +6,6 @@ ignore-patterns=error.py, expr.py, parser.py, schema.py, - visit.py, [MESSAGES CONTROL]
scripts/qapi/visit.py +2 -2
@@ -250,7 +250,7 @@ def gen_visit_alternate(name, variants): return ret -def gen_visit_object(name, base, members, variants): +def gen_visit_object(name): return mcgen(''' bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -343,7 +343,7 @@ def visit_object_type(self, name, info, ifcond, features, if not name.startswith('q_'): # only explicit types need an allocating visit self._genh.add(gen_visit_decl(name)) - self._genc.add(gen_visit_object(name, base, members, variants)) + self._genc.add(gen_visit_object(name)) def visit_alternate_type(self, name, info, ifcond, features, variants): with ifcontext(ifcond, self._genh, self._genc):
Signature · 1 line
2.26.2
PATCH v6 36/36 qapi/visit.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++---------- 2 files changed, 56 insertions(+), 22 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -28,8 +28,3 @@ check_untyped_defs = False disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False - -[mypy-qapi.visit] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False
scripts/qapi/visit.py +56 -17
164 lines not on this page · raw message
Signature · 1 line
2.26.2
Quoted 23 lines · John Snow
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
Notes:
- After patch 07,
isort -cshould pass 100% on this and every future commit.
Quoted 8 lines
- After patch 08,
flake8 qapi/should pass 100% on this and every future commit.- After patch 09,
pylint --rcfile=qapi/pylintrc qapi/should pass 100% on this and every future commit.- After patch 18,
mypy --config-file=qapi/mypy.ini qapi/should pass 100% on this and every future commit.
Series
Queued, thanks!
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · John Snow
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
Notes:
- After patch 07,
isort -cshould pass 100% on this and every future commit.- After patch 08,
flake8 qapi/should pass 100% on this and every future commit.- After patch 09,
pylint --rcfile=qapi/pylintrc qapi/should pass 100% on this and every future commit.
Quoted 8 lines
Quoted 2 lines
- After patch 18,
mypy --config-file=qapi/mypy.ini qapi/should pass 100% on this and every future commit.Series
Reviewed-by Markus Armbruster <[email protected]>Queued, thanks!
Thanks for putting up with me!
Only five left to go! Enjoy your PTO.
--js