qapi: static typing conversion, pt1
v2 cover qapi: static typing conversion, pt1
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)
In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
- Patches 1-7: generic, qapi-wide cleanups, flake8 and pylint support
- Patches 8-16: common.py cleanup (with mypy support added at the end)
- Patches 17-18: events.py cleanup
- Patches 19-21: commands.py cleanup
- Patches 22-23: source.py cleanup
- Patches 24-29: gen.py cleanup
- Patches 30-33: introspect.py cleanup
- Patches 34-35: types.py cleanup
- Patches 36-37: visit.py cleanup
Notes:
- After patch 6, flake8 qapi/ should pass 100% on this and every future commit.
- After patch 7, pylint --rcfile=3Dqapi/pylintrc qapi/ should pass 100% on this and every future commit.
- After patch 16, mypy --config-file=3Dqapi/mypy.ini qapi/ should pass 100% on this and every future commit.
V2:
- Removed Python3.6 patch in favor of Thomas Huth's
- Addressed (most) feedback from Markus
- Renamed type hint annotation commits
Changelog:
003/38:[0082] [FC] 'qapi: move generator entrypoint into module' 004/38:[down] 'qapi: Prefer explicit relative imports' 006/38:[0027] [FC] 'qapi: delint using flake8'
007/38:[0006] [FC] 'qapi: add pylintrc'
008/38:[0002] [FC] 'qapi/common.py: Remove python compatibility workaround' 009/38:[0037] [FC] 'qapi/common.py: Add indent manager' 010/38:[0008] [FC] 'qapi/common.py: delint with pylint' 011/38:[0002] [FC] 'qapi/common.py: Replace one-letter 'c' variable' 012/38:[----] [-C] 'qapi/common.py: check with pylint' 013/38:[down] 'qapi/common.py: add type hint annotations' 014/38:[down] 'qapi/common.py: Convert comments into docstrings, and elaborat= e'
015/38:[down] 'qapi/common.py: move build_params into gen.py' 017/38:[down] 'qapi/events.py: add type hint annotations' 020/38:[0006] [FC] 'qapi/commands.py: add notational type hints' 022/38:[down] 'qapi/source.py: add type hint annotations' 025/38:[down] 'qapi/gen.py: add type hint annotations' 033/38:[down] 'qapi/introspect.py: add type hint annotations' 034/38:[down] 'qapi/types.py: add type hint annotations' 036/38:[down] 'qapi/visit.py: assert tag_member contains a QAPISchemaEnumType' 037/38:[----] [-C] 'qapi/visit.py: remove unused parameters from gen_visit_ob= ject'
038/38:[down] 'qapi/visit.py: add type hint annotations'
John Snow (38):
[DO-NOT-MERGE] qapi: add debugging tools
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into module
qapi: Prefer explicit relative imports
qapi: Remove wildcard includes
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable qapi/common.py: check with pylint
qapi/common.py: add type hint annotations qapi/common.py: Convert comments into docstrings, and elaborate qapi/common.py: move build_params into gen.py
qapi: establish mypy type-checking baseline qapi/events.py: add type hint annotations qapi/events.py: Move comments into docstrings qapi/commands.py: Don't re-bind to variable of different type qapi/commands.py: add notational type hints qapi/commands.py: enable checking with mypy qapi/source.py: add type hint annotations qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module qapi/gen.py: add type hint annotations
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic qapi/gen.py: delint with pylint
qapi/introspect.py: Add a typed 'extra' structure qapi/introspect.py: add _gen_features helper qapi/introspect.py: create a typed 'Node' data structure qapi/introspect.py: add type hint annotations qapi/types.py: add type hint annotations qapi/types.py: remove one-letter variables qapi/visit.py: assert tag_member contains a QAPISchemaEnumType qapi/visit.py: remove unused parameters from gen_visit_object qapi/visit.py: add type hint annotations
scripts/qapi-gen.py | 59 ++--------- scripts/qapi/.flake8 | 2 + scripts/qapi/commands.py | 85 +++++++++++---- scripts/qapi/common.py | 163 +++++++++++++++------------- scripts/qapi/debug.py | 78 ++++++++++++++ scripts/qapi/doc.py | 3 +- scripts/qapi/events.py | 64 ++++++++--- scripts/qapi/expr.py | 4 +- scripts/qapi/gen.py | 176 ++++++++++++++++++------------ scripts/qapi/introspect.py | 212 +++++++++++++++++++++++++------------ scripts/qapi/main.py | 91 ++++++++++++++++ scripts/qapi/mypy.ini | 30 ++++++ scripts/qapi/parser.py | 4 +- scripts/qapi/pylintrc | 71 +++++++++++++ scripts/qapi/schema.py | 33 +++--- scripts/qapi/source.py | 34 +++--- scripts/qapi/types.py | 125 +++++++++++++++------- scripts/qapi/visit.py | 116 ++++++++++++++------ 18 files changed, 952 insertions(+), 398 deletions(-) create mode 100644 scripts/qapi/.flake8 create mode 100644 scripts/qapi/debug.py create mode 100644 scripts/qapi/main.py create mode 100644 scripts/qapi/mypy.ini create mode 100644 scripts/qapi/pylintrc
--=20
2.26.2
PATCH v2 1/38 qapi: add debugging tools
This adds some really childishly simple debugging tools. Maybe they're interesting for someone else, too?
--- scripts/qapi/debug.py | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 scripts/qapi/debug.py
scripts/qapi/debug.py new file +78 -0
@@ -0,0 +1,78 @@ +""" +Small debugging facilities for mypy static analysis work. +(C) 2020 John Snow, for Red Hat, Inc. +""" + +import inspect +import json +from typing import Dict, List, Any +from types import FrameType + + +OBSERVED_TYPES: Dict[str, List[str]] = {} + + +# You have no idea how long it took to find this return type... +def caller_frame() -> FrameType: + """ + Returns the stack frame of the caller's caller. + e.g. foo() -> caller() -> caller_frame() return's foo's stack frame. + """ + stack = inspect.stack() + caller = stack[2].frame + if caller is None: + msg = "Python interpreter does not support stack frame inspection" + raise RuntimeError(msg) + return caller + + +def _add_type_record(name: str, typestr: str) -> None: + seen = OBSERVED_TYPES.setdefault(name, []) + if typestr not in seen: + seen.append(typestr) + + +def record_type(name: str, value: Any, dict_names: bool = False) -> None: + """ + Record the type of a variable. + + :param name: The name of the variable + :param value: The value of the variable + """ + _add_type_record(name, str(type(value))) + + try: + for key, subvalue in value.items(): + subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]" + _add_type_record(subname, str(type(subvalue))) + return + except AttributeError: + # (Wasn't a dict or anything resembling one.) + pass + + # str is iterable, but not in the way we want! + if isinstance(value, str): + return + + try: + for elem in value: + _add_type_record(f"{name}.[list_elem]", str(type(elem))) + except TypeError: + # (Wasn't a list or anything else iterable.) + pass + + +def show_types() -> None: + """ + Print all of the currently known variable types to stdout. + """ + print(json.dumps(OBSERVED_TYPES, indent=2)) + + +def record_locals(show: bool = False, dict_names: bool = False) -> None: + caller = caller_frame() + name = caller.f_code.co_name + for key, value in caller.f_locals.items(): + record_type(f"{name}.{key}", value, dict_names=dict_names) + if show: + show_types()
Signature · 1 line
2.26.2
Re: [PATCH v2 01/38] [DO-NOT-MERGE] qapi: add debugging tools
Quoted 75 lines · John Snow
This adds some really childishly simple debugging tools. Maybe they're interesting for someone else, too?
Signed-off-by John Snow <[email protected]>--- scripts/qapi/debug.py | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 scripts/qapi/debug.pyscripts/qapi/debug.py new file +60
-0@@ -0,0 +1,78 @@ +""" +Small debugging facilities for mypy static analysis work. +(C) 2020 John Snow, for Red Hat, Inc. +""" + +import inspect +import json +from typing import Dict, List, Any +from types import FrameType + + +OBSERVED_TYPES: Dict[str, List[str]] = {} + + +# You have no idea how long it took to find this return type... +def caller_frame() -> FrameType: + """ + Returns the stack frame of the caller's caller. + e.g. foo() -> caller() -> caller_frame() return's foo's stack frame. + """ + stack = inspect.stack() + caller = stack[2].frame + if caller is None: + msg = "Python interpreter does not support stack frame inspection" + raise RuntimeError(msg) + return caller + + +def _add_type_record(name: str, typestr: str) -> None: + seen = OBSERVED_TYPES.setdefault(name, []) + if typestr not in seen: + seen.append(typestr) + + +def record_type(name: str, value: Any, dict_names: bool = False) -> None: + """ + Record the type of a variable. + + :param name: The name of the variable + :param value: The value of the variable + """ + _add_type_record(name, str(type(value))) + + try: + for key, subvalue in value.items(): + subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]" + _add_type_record(subname, str(type(subvalue))) + return + except AttributeError: + # (Wasn't a dict or anything resembling one.) + pass + + # str is iterable, but not in the way we want! + if isinstance(value, str): + return + + try: + for elem in value: + _add_type_record(f"{name}.[list_elem]", str(type(elem))) + except TypeError:
scripts/qapi/debug.py line 70
Quoted 10 lines
scripts/qapi/debug.py new file +10
-0+ # (Wasn't a list or anything else iterable.) + pass + + +def show_types() -> None: + """ + Print all of the currently known variable types to stdout. + """ + print(json.dumps(OBSERVED_TYPES, indent=2)) +
Maybe the following will be cheaper (no json conversion):
pprint.pprint(OBSERVED_TYPES, indent=2)
Other than that, I'd vote for including this if there's a bit more documentation on how to use it, or an example script. Maybe there already is, and I did not get to it yet.
- Cleber.
Quoted 11 lines
scripts/qapi/debug.py new file +8
-0+ +def record_locals(show: bool = False, dict_names: bool = False) -> None: + caller = caller_frame() + name = caller.f_code.co_name + for key, value in caller.f_locals.items(): + record_type(f"{name}.{key}", value, dict_names=dict_names) + if show: + show_types()Signature · 1 line
2.26.2
Re: [PATCH v2 01/38] [DO-NOT-MERGE] qapi: add debugging tools
Quoted 89 lines · Cleber Rosa
Quoted 75 lines · John Snow
This adds some really childishly simple debugging tools. Maybe they're interesting for someone else, too?
Signed-off-by John Snow <[email protected]>--- scripts/qapi/debug.py | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+)create mode 100644
scripts/qapi/debug.pyscripts/qapi/debug.py new file +60
-0@@ -0,0 +1,78 @@ +""" +Small debugging facilities for mypy static analysis work. +(C) 2020 John Snow, for Red Hat, Inc. +""" + +import inspect +import json +from typing import Dict, List, Any +from types import FrameType + + +OBSERVED_TYPES: Dict[str, List[str]] = {} + + +# You have no idea how long it took to find this return type... +def caller_frame() -> FrameType: + """ + Returns the stack frame of the caller's caller. + e.g. foo() -> caller() -> caller_frame() return's foo's stack frame. + """ + stack = inspect.stack() + caller = stack[2].frame + if caller is None: + msg = "Python interpreter does not support stack frame inspection" + raise RuntimeError(msg) + return caller + + +def _add_type_record(name: str, typestr: str) -> None: + seen = OBSERVED_TYPES.setdefault(name, []) + if typestr not in seen: + seen.append(typestr) + + +def record_type(name: str, value: Any, dict_names: bool = False) -> None: + """ + Record the type of a variable. + + :param name: The name of the variable + :param value: The value of the variable + """ + _add_type_record(name, str(type(value))) + + try: + for key, subvalue in value.items(): + subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]" + _add_type_record(subname, str(type(subvalue))) + return + except AttributeError: + # (Wasn't a dict or anything resembling one.) + pass + + # str is iterable, but not in the way we want! + if isinstance(value, str): + return + + try: + for elem in value: + _add_type_record(f"{name}.[list_elem]", str(type(elem))) + except TypeError:scripts/qapi/debug.py line 70
Quoted 10 lines
scripts/qapi/debug.py new file +10
-0+ # (Wasn't a list or anything else iterable.) + pass + + +def show_types() -> None: + """ + Print all of the currently known variable types to stdout. + """ + print(json.dumps(OBSERVED_TYPES, indent=2)) +Maybe the following will be cheaper (no json conversion):
Quoted 8 lines
pprint.pprint(OBSERVED_TYPES, indent=2)Other than that, I'd vote for including this if there's a bit more documentation on how to use it, or an example script. Maybe there already is, and I did not get to it yet.
- Cleber.
Nope, this is just a dumb script I did to observe types in flight.
There are apparently bigger, beefier tools that I don't know how to use yet: https://github.com/dropbox/pyannotate
I just included my own little tool as a reference thing to be archived on list, I have no desire to spruce it up. I'd rather spend my time learning pyannotate.
--js
PATCH v2 2/38 qapi-gen: Separate arg-parsing from generation
In reply to John SnowThis is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.
--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)
scripts/qapi-gen.py +63 -24
@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re
@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + gen_doc(schema, output_dir, prefix) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',
@@ -33,26 +81,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) - gen_doc(schema, args.output_dir, args.prefix) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())
Signature · 1 line
2.26.2
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
Quoted 4 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
In reply to John SnowQuoted 34 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)scripts/qapi-gen.py +7
-2@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_events
scripts/qapi-gen.py line 27
Quoted 10 lines
scripts/qapi-gen.py +3
-2from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = ''
I did not understand the purpose of these. If they're used only as the default value for the command line option parsing, I'd suggest dropping them.
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. +
scripts/qapi-gen.py line 47
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix):
Nice catch with the extra check here. Maybe worth mentioning and/or splitting the change?
Quoted 4 lines
scripts/qapi-gen.py +4
-0+ msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) +
scripts/qapi-gen.py line 61
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + gen_doc(schema, output_dir, prefix) + + +def main() -> int:
One extra Pythonic touch would be to use a bool here, and then:
sys.exit(0 if main() else 1)
But that's probably overkill.
Quoted 32 lines
scripts/qapi-gen.py +10
-9+ """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -33,26 +81,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try:
scripts/qapi-gen.py def main(argv): · line 48
Quoted 10 lines
scripts/qapi-gen.py +5
-4- schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) -
Glad to see that this "quitter" is gone in favor of one and only sys.exit().
- Cleber.
Quoted 17 lines
scripts/qapi-gen.py +4
-7- gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) - gen_doc(schema, args.output_dir, args.prefix) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())Signature · 1 line
2.26.2
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
Quoted 42 lines · Cleber Rosa
Quoted 41 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)scripts/qapi-gen.py +8
-3@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit
Quoted 8 lines
scripts/qapi-gen.py line 27
Quoted 3 lines
scripts/qapi-gen.py +2
-1-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = ''I did not understand the purpose of these. If they're used only as the default value for the command line option parsing, I'd suggest dropping them.
The alternative is setting default='' inline below, which is fine, but found them kind of buried; and looking a bit too much like a weird magic constant. Aesthetically, I liked making them obvious.
You found them! My plan worked.
Quoted 15 lines
Quoted 15 lines
scripts/qapi-gen.py +15
-0+ + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types?
Quoted 9 lines
scripts/qapi-gen.py line 47
Quoted 5 lines
scripts/qapi-gen.py +5
-0+ + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix):Nice catch with the extra check here. Maybe worth mentioning and/or splitting the change?
If you review all 125 patches, I will do this for you ;)
Quoted 17 lines
Quoted 4 lines
scripts/qapi-gen.py +4
-0+ msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) +scripts/qapi-gen.py line 61
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + gen_doc(schema, output_dir, prefix) + + +def main() -> int:One extra Pythonic touch would be to use a bool here, and then:
Quoted 4 lines
sys.exit(0 if main() else 1)But that's probably overkill.
I think a function named main() is fair enough to return int -- we are declaring that this is a shell script pretty explicitly, and it is allowed to return a status code.
Shifting the knowledge of how shell codes work up one more layer is probably not ... urgent.
Quoted 65 lines
Quoted 32 lines
scripts/qapi-gen.py +10
-9+ """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -33,26 +81,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try:scripts/qapi-gen.py def main(argv): · line 48
Quoted 10 lines
scripts/qapi-gen.py +5
-4- schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) -Glad to see that this "quitter" is gone in favor of one and only
sys.exit().- Cleber.
Quoted 17 lines
scripts/qapi-gen.py +4
-7- gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) - gen_doc(schema, args.output_dir, args.prefix) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())Signature · 1 line
2.26.2
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
Quoted 47 lines · John Snow
Quoted 36 lines · Cleber Rosa
Quoted 35 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)scripts/qapi-gen.py +8
-3@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_typesQuoted 9 lines
scripts/qapi-gen.py line 27
Quoted 4 lines
scripts/qapi-gen.py +2
-1from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = ''I did not understand the purpose of these. If they're used only as the default value for the command line option parsing, I'd suggest dropping them.
Quoted 4 lines
The alternative is setting default='' inline below, which is fine, but found them kind of buried; and looking a bit too much like a weird magic constant. Aesthetically, I liked making them obvious.
I differ in opinion and style, and I don't think you should adopt mine. Just for the record, though, consider if would do the same for dozens of command line options... IMO it'd be fine to declare them for extra explicitness, but I see no reason for them to be module globals.
Quoted 2 lines
You found them! My plan worked.
:)
- Cleber.
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
In reply to Cleber RosaQuoted 65 lines · Cleber Rosa
Quoted 34 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)scripts/qapi-gen.py +7
-2@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_eventsscripts/qapi-gen.py line 27
Quoted 10 lines
scripts/qapi-gen.py +3
-2from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = ''I did not understand the purpose of these. If they're used only as the default value for the command line option parsing, I'd suggest dropping them.
Quoted 15 lines
scripts/qapi-gen.py +15
-0+ + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types?
Quoted 8 lines
scripts/qapi-gen.py line 47
Quoted 5 lines
scripts/qapi-gen.py +5
-0+ + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix):Nice catch with the extra check here. Maybe worth mentioning and/or splitting the change?
Please do not sneak additional checking into patches advertized as pure refactoring. It makes me look for more sneakery with a microscope.
This re.match() cannot possibly fail. Three cases:
* First character is funny
The regexp matches the empty string. There's a reason the regexp ends with '?'.
* Non-first character is funny
The regexp matches the non-funny prefix.
* No character is funny
The regexp matches the complete string.
Checking impossible conditions as if they were possible is confusing. Please drop the additional check.
We can talk about checking this impossible condition with
assert(match)
if you believe it makes the code easier to understand (it does not improve its behavior).
Quoted 17 lines
Quoted 4 lines
scripts/qapi-gen.py +4
-0+ msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) +scripts/qapi-gen.py line 61
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + gen_doc(schema, output_dir, prefix) + + +def main() -> int:One extra Pythonic touch would be to use a bool here, and then:
Quoted 3 lines
sys.exit(0 if main() else 1)But that's probably overkill.
Yuck :)
Quoted 37 lines
Quoted 37 lines
scripts/qapi-gen.py +14
-10+ """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -33,26 +81,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask,
Quoted 8 lines
scripts/qapi-gen.py def main(argv): · line 48
Quoted 5 lines
scripts/qapi-gen.py +1
-3+ builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) -Glad to see that this "quitter" is gone in favor of one and only
sys.exit().
I'm totally fine with having multiple sys.exit() in main(). They become icky only when buried too deeply.
I'm fine with John's patch regardless.
[...]
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
Quoted 97 lines · Markus Armbruster
Quoted 65 lines · Cleber Rosa
Quoted 34 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 24 deletions(-)scripts/qapi-gen.py +7
-2@@ -1,9 +1,13 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re@@ -11,21 +15,65 @@ from qapi.commands import gen_commands from qapi.doc import gen_doc +from qapi.error import QAPIError from qapi.events import gen_eventsscripts/qapi-gen.py line 27
Quoted 10 lines
scripts/qapi-gen.py +3
-2from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = ''I did not understand the purpose of these. If they're used only as the default value for the command line option parsing, I'd suggest dropping them.
Quoted 15 lines
scripts/qapi-gen.py +15
-0+ + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types?Quoted 8 lines
scripts/qapi-gen.py line 47
Quoted 5 lines
scripts/qapi-gen.py +5
-0+ + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix):Nice catch with the extra check here. Maybe worth mentioning and/or splitting the change?
Please do not sneak additional checking into patches advertized as pure refactoring. It makes me look for more sneakery with a microscope.
This
re.match()cannot possibly fail. Three cases:* First character is funny
The regexp matches the empty string. There's a reason the regexp ends with '?'.
* Non-first character is funny
The regexp matches the non-funny prefix.
* No character is funny
The regexp matches the complete string.
Checking impossible conditions as if they were possible is confusing. Please drop the additional check.
Quoted 7 lines
We can talk about checking this impossible condition with
assert(match)
if you believe it makes the code easier to understand (it does not improve its behavior).
My use of strict_optional=False is what prevents this from exhibiting as an error in mypy. An assert will help convince mypy that 'match' cannot possibly be 'None'.
eh, well. I will fix this when I remove strict_optional, so I will just remove this additional check for now to avoid adding another patch to this series.
--js
Re: [PATCH v2 02/38] qapi-gen: Separate arg-parsing from generation
Quoted 8 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · Cleber Rosa
Quoted 4 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>
[...]
Quoted 45 lines
Quoted 45 lines
Quoted 13 lines
Quoted 13 lines
scripts/qapi-gen.py +13
-0+def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types?Quoted 8 lines
scripts/qapi-gen.py line 47
Quoted 5 lines
scripts/qapi-gen.py +5
-0+ + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match and match.end() != len(prefix):Nice catch with the extra check here. Maybe worth mentioning and/or splitting the change?
Please do not sneak additional checking into patches advertized as pure refactoring. It makes me look for more sneakery with a microscope.
This
re.match()cannot possibly fail. Three cases:* First character is funny
The regexp matches the empty string. There's a reason the regexp ends with '?'.
* Non-first character is funny
The regexp matches the non-funny prefix.
* No character is funny
The regexp matches the complete string.
Checking impossible conditions as if they were possible is confusing. Please drop the additional check.
We can talk about checking this impossible condition with
Quoted 8 lines
Quoted 4 lines
assert(match)
if you believe it makes the code easier to understand (it does not improve its behavior).
My use of strict_optional=False is what prevents this from exhibiting as an error in mypy. An assert will help convince mypy that 'match' cannot possibly be 'None'.
Adding assertions to help mypy along is okay.
Quoted 3 lines
eh, well. I will fix this when I remove
strict_optional, so I will just remove this additional check for now to avoid adding another patch to this series.
Makes sense to me.
PATCH v2 3/38 qapi: move generator entrypoint into module
In reply to John SnowAs part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the package, only leaving a very tiny entrypoint shim outside of the module.
--- scripts/qapi-gen.py | 90 +++---------------------------------------- scripts/qapi/main.py | 91 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 84 deletions(-) create mode 100644 scripts/qapi/main.py
scripts/qapi-gen.py +6 -84
@@ -4,94 +4,16 @@ # See the COPYING file in the top-level directory. """ -QAPI Generator +QAPI code generation execution shim. -This script is the main entry point for generating C code from the QAPI schema. +This standalone script exists primarily to facilitate the running of the QAPI +code generator without needing to install the python module to the current +execution environment. """ -import argparse -import re import sys -from qapi.commands import gen_commands -from qapi.doc import gen_doc -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit - - -DEFAULT_OUTPUT_DIR = '' -DEFAULT_PREFIX = '' - - -def generate(schema_file: str, - output_dir: str, - prefix: str, - unmask: bool = False, - builtins: bool = False) -> None: - """ - generate uses a given schema to produce C code in the target directory. - - :param schema_file: The primary QAPI schema file. - :param output_dir: The output directory to store generated code. - :param prefix: Optional C-code prefix for symbol names. - :param unmask: Expose non-ABI names through introspection? - :param builtins: Generate code for built-in types? - - :raise QAPIError: On failures. - """ - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) - if match and match.end() != len(prefix): - msg = "funny character '{:s}' in prefix '{:s}'".format( - prefix[match.end()], prefix) - raise QAPIError('', None, msg) - - schema = QAPISchema(schema_file) - gen_types(schema, output_dir, prefix, builtins) - gen_visit(schema, output_dir, prefix, builtins) - gen_commands(schema, output_dir, prefix) - gen_events(schema, output_dir, prefix) - gen_introspect(schema, output_dir, prefix, unmask) - gen_doc(schema, output_dir, prefix) - - -def main() -> int: - """ - gapi-gen shell script entrypoint. - Expects arguments via sys.argv, see --help for details. - - :return: int, 0 on success, 1 on failure. - """ - parser = argparse.ArgumentParser( - description='Generate code from a QAPI schema') - parser.add_argument('-b', '--builtins', action='store_true', - help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', - default=DEFAULT_OUTPUT_DIR, - help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', - default=DEFAULT_PREFIX, - help="prefix for symbols") - parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', - dest='unmask', - help="expose non-ABI names in introspection") - parser.add_argument('schema', action='store') - args = parser.parse_args() - - try: - generate(args.schema, - output_dir=args.output_dir, - prefix=args.prefix, - unmask=args.unmask, - builtins=args.builtins) - except QAPIError as err: - print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) - return 1 - return 0 - +from qapi import main if __name__ == '__main__': - sys.exit(main()) + sys.exit(main.main())
scripts/qapi/main.py new file +91 -0
91 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v2 03/38] qapi: move generator entrypoint into module
Quoted 11 lines · John Snow
As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the package, only leaving a very tiny entrypoint shim outside of the module.
Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 90 +++---------------------------------------- scripts/qapi/main.py | 91 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 84 deletions(-) create mode 100644 scripts/qapi/main.py
[...]
Quoted 17 lines
scripts/qapi/main.py new file +11
-0@@ -0,0 +1,91 @@ +# This work is licensed under the terms of the GNU GPL, version 2 or later. +# See the COPYING file in the top-level directory. + +""" +QAPI Generator + +This is the main entry point for generating C code from the QAPI schema. +""" + +import argparse +import re
scripts/qapi/main.py line 21
Quoted 10 lines
scripts/qapi/main.py new file +10
-0+import sys + +from qapi.commands import gen_commands +from qapi.doc import gen_doc +from qapi.error import QAPIError +from qapi.events import gen_events +from qapi.introspect import gen_introspect +from qapi.schema import QAPISchema +from qapi.types import gen_types +from qapi.visit import gen_visit
Do you plan to change this to use relative imports eventually?
In either case:
Signature · 1 line
Eduardo
Re: [PATCH v2 03/38] qapi: move generator entrypoint into module
Quoted 9 lines · Eduardo Habkost
scripts/qapi/main.py line 21
Quoted 8 lines
scripts/qapi/main.py new file +8
-0+from qapi.commands import gen_commands +from qapi.doc import gen_doc +from qapi.error import QAPIError +from qapi.events import gen_events +from qapi.introspect import gen_introspect +from qapi.schema import QAPISchema +from qapi.types import gen_types +from qapi.visit import gen_visitDo you plan to change this to use relative imports eventually?
Happens in 04/38.
--js
Re: [PATCH v2 03/38] qapi: move generator entrypoint into module
In reply to John SnowQuoted 11 lines · John Snow
As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the package, only leaving a very tiny entrypoint shim outside of the module.
Signed-off-by John Snow <[email protected]>--- scripts/qapi-gen.py | 90 +++---------------------------------------- scripts/qapi/main.py | 91 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 84 deletions(-) create mode 100644 scripts/qapi/main.py
PATCH v2 4/38 qapi: Prefer explicit relative imports
In reply to John SnowAll of the QAPI include statements are changed to be package-aware, as explicit relative imports.
A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.
When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.
This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.
--- scripts/qapi/commands.py | 4 ++-- scripts/qapi/doc.py | 2 +- scripts/qapi/events.py | 8 ++++---- scripts/qapi/expr.py | 4 ++-- scripts/qapi/gen.py | 4 ++-- scripts/qapi/introspect.py | 8 ++++---- scripts/qapi/main.py | 16 ++++++++-------- scripts/qapi/parser.py | 4 ++-- scripts/qapi/schema.py | 8 ++++---- scripts/qapi/types.py | 6 +++--- scripts/qapi/visit.py | 6 +++--- 11 files changed, 35 insertions(+), 35 deletions(-)
scripts/qapi/commands.py +2 -2
@@ -13,8 +13,8 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import * +from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/doc.py +1 -1
@@ -5,7 +5,7 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from qapi.gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc, QAPISchemaVisitor MSG_FMT = """
scripts/qapi/events.py +4 -4
@@ -12,10 +12,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember -from qapi.types import gen_enum, gen_enum_lookup +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember +from .types import gen_enum, gen_enum_lookup def build_event_send_proto(name, arg_type, boxed):
scripts/qapi/expr.py +2 -2
@@ -16,8 +16,8 @@ import re from collections import OrderedDict -from qapi.common import c_name -from qapi.error import QAPISemError +from .common import c_name +from .error import QAPISemError # Names must be letters, numbers, -, and _. They must start with letter,
scripts/qapi/gen.py +2 -2
@@ -17,8 +17,8 @@ import re from contextlib import contextmanager -from qapi.common import * -from qapi.schema import QAPISchemaVisitor +from .common import * +from .schema import QAPISchemaVisitor class QAPIGen:
scripts/qapi/introspect.py +4 -4
@@ -10,10 +10,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaMonolithicCVisitor -from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .common import * +from .gen import QAPISchemaMonolithicCVisitor +from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, + QAPISchemaType) def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/main.py +8 -8
@@ -11,14 +11,14 @@ import re import sys -from qapi.commands import gen_commands -from qapi.doc import gen_doc -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit +from .commands import gen_commands +from .doc import gen_doc +from .error import QAPIError +from .events import gen_events +from .introspect import gen_introspect +from .schema import QAPISchema +from .types import gen_types +from .visit import gen_visit DEFAULT_OUTPUT_DIR = ''
scripts/qapi/parser.py +2 -2
@@ -18,8 +18,8 @@ import re from collections import OrderedDict -from qapi.error import QAPIParseError, QAPISemError -from qapi.source import QAPISourceInfo +from .error import QAPIParseError, QAPISemError +from .source import QAPISourceInfo class QAPISchemaParser:
scripts/qapi/schema.py +4 -4
@@ -18,10 +18,10 @@ import re from collections import OrderedDict -from qapi.common import c_name, pointer_suffix -from qapi.error import QAPIError, QAPISemError -from qapi.expr import check_exprs -from qapi.parser import QAPISchemaParser +from .common import c_name, pointer_suffix +from .error import QAPIError, QAPISemError +from .expr import check_exprs +from .parser import QAPISchemaParser class QAPISchemaEntity:
scripts/qapi/types.py +3 -3
@@ -13,9 +13,9 @@ # See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember, QAPISchemaObjectType # variants must be emitted before their container; track what has already
scripts/qapi/visit.py +3 -3
@@ -13,9 +13,9 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
Signature · 1 line
2.26.2
Re: [PATCH v2 04/38] qapi: Prefer explicit relative imports
Quoted 27 lines · John Snow
All of the QAPI include statements are changed to be package-aware, as explicit relative imports.
A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.
When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.
This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.
Quoted 4 lines
It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.
Signed-off-by John Snow <[email protected]>
Here's the answer to the question I sent on patch 03/38. :)
Signature · 1 line
Eduardo
Re: [PATCH v2 04/38] qapi: Prefer explicit relative imports
In reply to John SnowQuoted 31 lines · John Snow
All of the QAPI include statements are changed to be package-aware, as explicit relative imports.
A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.
When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.
This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.
Signed-off-by John Snow <[email protected]>
Quoted 14 lines
--- scripts/qapi/commands.py | 4 ++-- scripts/qapi/doc.py | 2 +- scripts/qapi/events.py | 8 ++++---- scripts/qapi/expr.py | 4 ++-- scripts/qapi/gen.py | 4 ++-- scripts/qapi/introspect.py | 8 ++++---- scripts/qapi/main.py | 16 ++++++++-------- scripts/qapi/parser.py | 4 ++-- scripts/qapi/schema.py | 8 ++++---- scripts/qapi/types.py | 6 +++--- scripts/qapi/visit.py | 6 +++--- 11 files changed, 35 insertions(+), 35 deletions(-)
Relative imports are a source of heated debates, but when properly used in a self contained module like here, they are very posititive IMO.
Re: [PATCH v2 04/38] qapi: Prefer explicit relative imports
Quoted 3 lines · Cleber Rosa
Relative imports are a source of heated debates, but when properly used in a self contained module like here, they are very posititive IMO.
Still? I know they were loathed pre-3.5, but in my subjective experience they behave the nicest overall in the modern python dialect.
What are the downsides?
--js
Re: [PATCH v2 04/38] qapi: Prefer explicit relative imports
Quoted 11 lines · John Snow
Quoted 3 lines · Cleber Rosa
Relative imports are a source of heated debates, but when properly used in a self contained module like here, they are very posititive IMO.
Still? I know they were loathed pre-3.5, but in my subjective experience they behave the nicest overall in the modern python dialect.
What are the downsides?
--js
I'll just invite Beraldo to the discussion and let the fun begin :).
- Cleber.
Re: [PATCH v2 04/38] qapi: Prefer explicit relative imports
Quoted 6 lines · Cleber Rosa
Quoted 5 lines · John Snow
Quoted 3 lines · Cleber Rosa
Relative imports are a source of heated debates, but when properly used in a self contained module like here, they are very posititive IMO.
Quoted 8 lines
Quoted 6 lines
Still? I know they were loathed pre-3.5, but in my subjective experience they behave the nicest overall in the modern python dialect.
What are the downsides?
--js
I'll just invite Beraldo to the discussion and let the fun begin :).
Nice try, Cleber! ;)
Well, relative imports are supported by Guido, so I'm not here to say different. There are some use-cases.
I'm not fully aware of the qapi context and big picture here, but I guess that depends on how you would like to use your package/scripts.
Some may say that one "downside" is that relative imports are not as readable as absolute ones. But reading the 04/38 PATH description by jsnow, yes, looks like using relative imports is one valid option here.
I prefer to use my scripts as packages inside venvs, and I use to have a setup.py, with absolute imports whenever possible, and when in development mode, make use of python3 setup.py develop which will create the "links" for me.
--
Beraldo
Re: [PATCH v2 04/38] qapi: Prefer explicit relative imports
Quoted 10 lines · Beraldo Leal
Quoted 9 lines · Cleber Rosa
Quoted 8 lines · John Snow
Quoted 3 lines · Cleber Rosa
Relative imports are a source of heated debates, but when properly used in a self contained module like here, they are very posititive IMO.
Still? I know they were loathed pre-3.5, but in my subjective experience they behave the nicest overall in the modern python dialect.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What are the downsides?
--js
I'll just invite Beraldo to the discussion and let the fun begin :).
Nice try, Cleber! ;)
C'mon... I was hoping for nothing less than an emacs .vs. vi kind of discussion.
Quoted 6 lines
Well, relative imports are supported by Guido, so I'm not here to say different. There are some use-cases.
I'm not fully aware of the qapi context and big picture here, but I guess that depends on how you would like to use your package/scripts.
Quoted 9 lines
Some may say that one "downside" is that relative imports are not as readable as absolute ones. But reading the 04/38 PATH description by jsnow, yes, looks like using relative imports is one valid option here.
I prefer to use my scripts as packages inside venvs, and I use to have a
setup.py, with absolute imports whenever possible, and when in development mode, make use ofpython3 setup.py developwhich will create the "links" for me.
Now seriously, these are good point, thanks. John, I invited Beraldo to give his take on the subject here because he started this issue on Avocado land:
https://github.com/avocado-framework/avocado/issues/3525
So far on Avocado we've kept the relative imports on most places. We do have some occurences of "triple upper level" imports that don't look very nice IMO though:
Quoted 3 lines
--
Beraldo
Anyway, I think we're all in agreement with the approach taken here.
- Cleber.
PATCH v2 5/38 qapi: Remove wildcard includes
In reply to John SnowWildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
--- scripts/qapi/commands.py | 6 +++++- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 42 insertions(+), 8 deletions(-)
scripts/qapi/commands.py +5 -1
@@ -13,7 +13,11 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_name, + mcgen, +) from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
scripts/qapi/events.py +6 -1
@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +9 -3
@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitor
scripts/qapi/introspect.py +6 -1
@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)
scripts/qapi/types.py +7 -1
@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
scripts/qapi/visit.py +9 -1
@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
Signature · 1 line
2.26.2
Re: [PATCH v2 05/38] qapi: Remove wildcard includes
Quoted 9 lines · John Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 05/38] qapi: Remove wildcard includes
In reply to John SnowQuoted 25 lines · John Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/commands.py | 6 +++++- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 42 insertions(+), 8 deletions(-)scripts/qapi/commands.py +0
-0@@ -13,7 +13,11 @@ See the COPYING file in the top-level directory. """
scripts/qapi/commands.py line 23
Quoted 10 lines
scripts/qapi/commands.py +5
-1-from .common import * +from .common import ( + build_params, + c_name, + mcgen, +) from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
Is this import style being suggested or enforced by any tool? I've been using isort with very good results (both as a check tool, and as an emacs extension). For instance, the block about would look like:
from .common import build_params, c_name, mcgen
from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
Quoted 90 lines
scripts/qapi/events.py +6
-1@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookupscripts/qapi/gen.py +9
-3@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitorscripts/qapi/introspect.py +6
-1@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)scripts/qapi/types.py +7
-1@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectTypescripts/qapi/visit.py +0
-0@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """
scripts/qapi/visit.py line 24
Quoted 10 lines
scripts/qapi/visit.py +9
-1-from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +)
And here, isort will add the paranthesis (it does so based on space demands):
from .common import (c_enum_const, c_name, gen_endif, gen_if, mcgen,
pop_indent, push_indent)
from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
Other than those suggestions, it LGTM.
Re: [PATCH v2 05/38] qapi: Remove wildcard includes
Quoted 41 lines · Cleber Rosa
Quoted 25 lines · John Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/commands.py | 6 +++++- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 42 insertions(+), 8 deletions(-)scripts/qapi/commands.py +0
-0@@ -13,7 +13,11 @@ See the COPYING file in the top-level directory. """scripts/qapi/commands.py line 23
Quoted 10 lines
scripts/qapi/commands.py +5
-1-from .common import * +from .common import ( + build_params, + c_name, + mcgen, +) from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontextIs this import style being suggested or enforced by any tool? I've been using isort with very good results (both as a check tool, and as an emacs extension). For instance, the block about would look like:
Quoted 3 lines
from .common import
build_params,c_name, mcgen
from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
Not enforced by any tool, no. Just subjective preference for git-friendly import lines. They conflict on rebase a lot less.
I have been using emacs sort-lines to order the names in a group.
Quoted 103 lines
Quoted 90 lines
scripts/qapi/events.py +6
-1@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookupscripts/qapi/gen.py +9
-3@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitorscripts/qapi/introspect.py +6
-1@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)scripts/qapi/types.py +7
-1@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectTypescripts/qapi/visit.py +0
-0@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """scripts/qapi/visit.py line 24
Quoted 10 lines
scripts/qapi/visit.py +9
-1-from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +)And here, isort will add the paranthesis (it does so based on space demands):
Quoted 7 lines
from .common import (
c_enum_const,c_name,gen_endif,gen_if, mcgen,
pop_indent,push_indent)
from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectTypeOther than those suggestions, it LGTM.
OK. We can add a check that asserts that isort(file) == file to keep our include regimes consistent. I'll look into the tool, but it will be after this marathon of a series.
Here's a gitlab issue I made on my QEMU fork to help me keep track of Python-related issues that I intend to use: https://gitlab.com/jsnow/qemu/-/issues/6
Quoted 2 lines
Reviewed-by Cleber Rosa <[email protected]>
Thanks!
--js
Re: [PATCH v2 05/38] qapi: Remove wildcard includes
Quoted 147 lines · John Snow
Quoted 38 lines · Cleber Rosa
Quoted 23 lines · John Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/commands.py | 6 +++++- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 42 insertions(+), 8 deletions(-)scripts/qapi/commands.py +0
-0scripts/qapi/commands.py line 20
Quoted 9 lines
scripts/qapi/commands.py +5
-1@@ -13,7 +13,11 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_name, + mcgen, +) from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontextIs this import style being suggested or enforced by any tool? I've been using isort with very good results (both as a check tool, and as an emacs extension). For instance, the block about would look like:
Quoted 3 lines
from .common import
build_params,c_name, mcgen
from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontextNot enforced by any tool, no. Just subjective preference for git-friendly import lines. They conflict on rebase a lot less.
I have been using emacs sort-lines to order the names in a group.
Quoted 99 lines
Quoted 81 lines
scripts/qapi/events.py +6
-1@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookupscripts/qapi/gen.py +9
-3@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitorscripts/qapi/introspect.py +6
-1@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)scripts/qapi/types.py +7
-1@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectTypescripts/qapi/visit.py +0
-0@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """scripts/qapi/visit.py line 23
Quoted 10 lines
scripts/qapi/visit.py +9
-1-from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +)And here, isort will add the paranthesis (it does so based on space demands):
from .common import (
c_enum_const,c_name,gen_endif,gen_if, mcgen,
pop_indent,push_indent)
from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
Quoted 7 lines
Quoted 2 lines
Other than those suggestions, it LGTM.
OK. We can add a check that asserts that isort(file) == file to keep our include regimes consistent. I'll look into the tool, but it will be after this marathon of a series.
That goes without saying!
Quoted 4 lines
Here's a gitlab issue I made on my QEMU fork to help me keep track of Python-related issues that I intend to use:
https://gitlab.com/jsnow/qemu/-/issues/6
Nice!
- Cleber.
Quoted 6 lines
Quoted 2 lines
Reviewed-by Cleber Rosa <[email protected]>Thanks!
--js
Re: [PATCH v2 05/38] qapi: Remove wildcard includes
In reply to John SnowQuoted 146 lines · John Snow
Quoted 38 lines · Cleber Rosa
Quoted 31 lines · John Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by John Snow <[email protected]>---
 scripts/qapi/commands.py  | 6 +++++-  scripts/qapi/events.py    | 7 ++++++-  scripts/qapi/gen.py       | 12 +++++++++--- Âscripts/qapi/introspect.py| 7 ++++++-
 scripts/qapi/types.py     | 8 +++++++-  scripts/qapi/visit.py     | 10 +++++++++-  6 files changed, 42 insertions(+), 8 deletions(-)scripts/qapi/commands.py +0
-0@@ -13,7 +13,11 @@Â See the COPYING file in the top-level directory. Â """
scripts/qapi/commands.py +5
-1-from .common import * +from .common import ( +Â Â Â build_params, +Â Â Â c_name, +Â Â Â mcgen, +)Quoted 1 line
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
Is this import style being suggested or enforced by any tool? I've been using isort with very good results (both as a check tool, and as an emacs extension). For instance, the block about would look like:
Quoted 3 lines
   from .common import
build_params,c_name, mcgen
   from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontextNot enforced by any tool, no. Just subjective preference for git-friendly import lines. They conflict on rebase a lot less.
I have been using emacs sort-lines to order the names in a group.
Quoted 99 lines
Quoted 81 lines
scripts/qapi/events.py +0
-0@@ -12,7 +12,12 @@Â See the COPYING file in the top-level directory. Â """
scripts/qapi/events.py +6
-1-from .common import * +from .common import ( +   build_params, +   c_enum_const, +   c_name, +   mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext  from .schema import QAPISchemaEnumMember
 from .types importgen_enum,gen_enum_lookupscripts/qapi/gen.py +0
-0@@ -11,13 +11,19 @@Â # This work is licensed under the terms of the GNU GPL, version 2. Â # See the COPYING file in the top-level directory.
scripts/qapi/gen.py +1
-1- +from contextlib import contextmanager import errno
 import os
 import rescripts/qapi/gen.py +8
-2-from contextlib import contextmanager -from .common import * +from .common import ( +Â Â Â c_fname, +Â Â Â gen_endif, +Â Â Â gen_if, +Â Â Â guardend, +Â Â Â guardstart, +Â Â Â mcgen, +)Â from .schema import QAPISchemaVisitor
scripts/qapi/introspect.py +0
-0@@ -10,7 +10,12 @@Â See the COPYING file in the top-level directory. Â """
scripts/qapi/introspect.py +6
-1-from .common import * +from .common import ( +Â Â Â c_name, +Â Â Â gen_endif, +Â Â Â gen_if, +Â Â Â mcgen, +)Â from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,                       QAPISchemaType)scripts/qapi/types.py +0
-0@@ -13,7 +13,13 @@Â # See the COPYING file in the top-level directory. Â """
scripts/qapi/types.py +7
-1-from .common import * +from .common import ( +   c_enum_const, +   c_name, +   gen_endif, +   gen_if, +   mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext  from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
scripts/qapi/visit.py +0
-0@@ -13,7 +13,15 @@Â See the COPYING file in the top-level directory. Â """
scripts/qapi/visit.py
Quoted 10 lines
scripts/qapi/visit.py +9
-1-from .common import * +from .common import ( +Â Â Â c_enum_const, +Â Â Â c_name, +Â Â Â gen_endif, +Â Â Â gen_if, +Â Â Â mcgen, +Â Â Â pop_indent, +Â Â Â push_indent, +)And here, isort will add the paranthesis (it does so based on space demands):
   from .common import (
c_enum_const,c_name,gen_endif,gen_if, mcgen, Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âpop_indent,push_indent) Â Â Â from .gen import QAPISchemaModularCVisitor, ifcontext
   from .schema import QAPISchemaObjectType
Quoted 7 lines
Quoted 2 lines
Other than those suggestions, it LGTM.
OK. We can add a check that asserts that isort(file) == file to keep our include regimes consistent. I'll look into the tool, but it will be after this marathon of a series.
Here's a gitlab issue I made on my QEMU fork to help me keep track of Python-related issues that I intend to use: https://gitlab.com/jsnow/qemu/-/issues/6
I've found thatisort --force-sort-within-sections --force-grid-wrap 4 --multi-line 3 --trailing-comma
is pretty close to what I was already doing, so I'll adopt this for the respin on good faith that nobody will retract an R-B for new import orderings.
force sort: I prefer to sort by module, so I intersperse "from x" and "import x" style in one section. This keeps the module reference absolutely consistent regardless of HOW we import from it.
force grid: 4 or more imports from a module will get wrapped using the one-per-line style.
multi-line: '3' just refers to the one-per-line style of imports that I already use.
trailing comma: A little buddy that hangs out with you.
Quoted 6 lines
Quoted 2 lines
Reviewed-by Cleber Rosa <[email protected]>Thanks!
--js
PATCH v2 6/38 qapi: delint using flake8
In reply to John SnowPetty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 16 +++++++++++----- 4 files changed, 20 insertions(+), 9 deletions(-) create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Prefer pylint's bare-except checks to flake8's
scripts/qapi/commands.py +2 -1
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, + QObject **ret_out, Error **errp) { Visitor *v;
scripts/qapi/schema.py +5 -3
@@ -536,7 +536,7 @@ def set_defined_in(self, name): v.set_defined_in(name) def check(self, schema, seen): - if not self.tag_member: # flat union + if not self.tag_member: # flat union self.tag_member = seen.get(c_name(self._tag_name)) base = "'base'" # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname): self._entity_dict = {} self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) - self._make_module(None) # built-ins + self._make_module(None) # built-ins self._make_module(fname) self._predefining = True self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members): # But it's not tight: the disjunction need not imply it. We # may end up compiling useless wrapper types. # TODO kill simple unions or implement the disjunction - assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access + + # pylint: disable=protected-access + assert (ifcond or []) == typ._ifcond else: self._def_entity(QAPISchemaObjectType( name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +11 -5
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False): if not scalar: c_type += '*' return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp); + +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_type)sobj, Error **errp); ''', c_name=c_name(name), c_type=c_type)
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants): def gen_visit_list(name, element_type): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false; %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type): def gen_visit_enum(name): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s *obj, Error **errp) { int value = *obj; bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ def gen_visit_enum(name): def gen_visit_alternate(name, variants): ret = mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp) { bool ok = false;
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants): def gen_visit_object(name, base, members, variants): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp) { bool ok = false;
Signature · 1 line
2.26.2
Re: [PATCH v2 06/38] qapi: delint using flake8
Quoted 9 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 06/38] qapi: delint using flake8
In reply to John SnowQuoted 9 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
Signed-off-by John Snow <[email protected]>
PATCH v2 7/38 qapi: add pylintrc
In reply to John SnowUsing pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.
Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5.x, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrc
--- scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +74 -0
@@ -0,0 +1,74 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the ignore list. +# The regex matches against base names, not paths. +ignore-patterns=common.py, + doc.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py, + + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes, + +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _ + +[VARIABLES] + +[STRING] + +[SPELLING] + +[FORMAT] + +[SIMILARITIES] + +# Ignore import statements themselves when computing similarities. +ignore-imports=yes + +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]
Signature · 1 line
2.26.2
Re: [PATCH v2 07/38] qapi: add pylintrc
Quoted 15 lines · John Snow
Using
pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5.x, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Quoted 5 lines
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrcSigned-off-by John Snow <[email protected]>
This doesn't generate any warnings after this patch, but at the end of your -pt6 branch I got some pylint warnings. I am bisecting it to try to identify the patch where the warnings are introduced.
Signature · 1 line
Eduardo
Re: [PATCH v2 07/38] qapi: add pylintrc
In reply to John SnowQuoted 15 lines · John Snow
Using
pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5.x, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Quoted 5 lines
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrcSigned-off-by John Snow <[email protected]>
One concern I have here is that the pylint version is not defined. Based on experience, different pylint will behave differently, because among other things, it may introduce new checks.
I'd at the very least document the pylint version used in the commit message, until a "requirements.txt"-like solution pinning a version is given.
Other than that,
Re: [PATCH v2 07/38] qapi: add pylintrc
Quoted 26 lines · Cleber Rosa
Quoted 15 lines · John Snow
Using
pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5.x, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Quoted 5 lines
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrcSigned-off-by John Snow <[email protected]>One concern I have here is that the pylint version is not defined. Based on experience, different pylint will behave differently, because among other things, it may introduce new checks.
Quoted 9 lines
I'd at the very least document the pylint version used in the commit message, until a "requirements.txt"-like solution pinning a version is given.
Other than that,
Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>
Alright, I'll put it in the commit message itself instead of in the cover letter.
The next step is to re-engage on that Makefile patch that I was working on for ./python/qemu and introduce it here too, which will document the pinned versions correctly.
--js
Re: [PATCH v2 07/38] qapi: add pylintrc
Quoted 31 lines · John Snow
Quoted 30 lines · Cleber Rosa
Quoted 15 lines · John Snow
Using
pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5.x, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Quoted 5 lines
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrcSigned-off-by John Snow <[email protected]>One concern I have here is that the pylint version is not defined. Based on experience, different pylint will behave differently, because among other things, it may introduce new checks.
I'd at the very least document the pylint version used in the commit message, until a "requirements.txt"-like solution pinning a version is given.
Quoted 9 lines
Quoted 5 lines
Other than that,
Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>Alright, I'll put it in the commit message itself instead of in the cover letter.
I missed that info on the cover letter, so my apologies. But still, I think it's a good idea to have that preserved in the repo history indeed.
Quoted 5 lines
The next step is to re-engage on that
Makefilepatch that I was working on for ./python/qemu and introduce it here too, which will document the pinned versions correctly.--js
Agreed!
- Cleber.
PATCH v2 8/38 qapi/common.py: Remove python compatibility workaround
In reply to John Snow--- scripts/qapi/common.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +1 -4
@@ -119,10 +119,7 @@ def cgen(code, **kwds): raw = code % kwds if indent_level: indent = genindent(indent_level) - # re.subn() lacks flags support before Python 2.7, use re.compile() - raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE), - indent, raw) - raw = raw[0] + raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
Signature · 1 line
2.26.2
Re: [PATCH v2 08/38] qapi/common.py: Remove python compatibility workaround
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 08/38] qapi/common.py: Remove python compatibility workaround
In reply to John SnowQuoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
PATCH v2 9/38 qapi/common.py: Add indent manager
In reply to John SnowCode style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
--- scripts/qapi/common.py | 51 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 20 deletions(-)
scripts/qapi/common.py +35 -16
@@ -93,33 +93,52 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> int: + """Increase the indentation level by `amount`, default 4.""" + self._level += amount + return self._level + + def decrease(self, amount: int = 4) -> int: + """Decrease the indentation level by `amount`, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + return self._level + + +indent = Indentation() # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey indent, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if indent: + raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
scripts/qapi/visit.py +3 -4
@@ -19,8 +19,7 @@ gen_endif, gen_if, mcgen, - pop_indent, - push_indent, + indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + indent.increase() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + indent.decrease() ret += mcgen(''' } ''')
Signature · 1 line
2.26.2
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 61 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 20 deletions(-)scripts/qapi/common.py +22
-12@@ -93,33 +93,52 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> int: + """Increase the indentation level by `amount`, default 4.""" + self._level += amount + return self._level +
scripts/qapi/common.py def c_name(name, protect=True): · line 133
Quoted 10 lines
scripts/qapi/common.py +10
-0+ def decrease(self, amount: int = 4) -> int: + """Decrease the indentation level by `amount`, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + return self._level + + +indent = Indentation()
Personally, I would keep the push_indent(), pop_indent() API, and introduce an indent() function, to follow the existing API style of plain functions.
Something like:
indent_prefixes = []
def push_indent(amount=4):
"""Add amount spaces to indentation prefix""" indent_prefixes.push(' '*amount)
def pop_indent():
"""Revert the most recent push_indent() call""" indent_prefixes.pop()
def indent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
What you did is still an improvement, though, so:
Signature · 1 line
Eduardo
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 90 lines · Eduardo Habkost
Quoted 61 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 20 deletions(-)scripts/qapi/common.py +22
-12@@ -93,33 +93,52 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> int: + """Increase the indentation level by `amount`, default 4.""" + self._level += amount + return self._level +scripts/qapi/common.py def c_name(name, protect=True): · line 133
Quoted 10 lines
scripts/qapi/common.py +10
-0+ def decrease(self, amount: int = 4) -> int: + """Decrease the indentation level by `amount`, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + return self._level + + +indent = Indentation()Personally, I would keep the
push_indent(),pop_indent()API, and introduce anindent()function, to follow the existing API style of plain functions.Something like:
indent_prefixes= []
defpush_indent(amount=4):
"""Addamountspaces to indentation prefix"""indent_prefixes.push(' '*amount)
defpop_indent():
"""Revert the most recentpush_indent()call"""indent_prefixes.pop()
defindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
Quoted 4 lines
What you did is still an improvement, though, so:
Reviewed-by Eduardo Habkost <[email protected]>
Yeah, there's only one user right now, so ... I just kinda wanted to get rid of the global usage. Maybe if we make the code generator fancier we'll find out what form is best.
--js
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 84 lines · John Snow
Quoted 80 lines · Eduardo Habkost
Quoted 54 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 20 deletions(-)scripts/qapi/common.py +19
-9@@ -93,33 +93,52 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> int: + """Increase the indentation level by `amount`, default 4.""" + self._level += amount + return self._level +scripts/qapi/common.py def c_name(name, protect=True): · line 129
Quoted 10 lines
scripts/qapi/common.py +10
-0+ def decrease(self, amount: int = 4) -> int: + """Decrease the indentation level by `amount`, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + return self._level + + +indent = Indentation()Personally, I would keep the
push_indent(),pop_indent()API, and introduce anindent()function, to follow the existing API style of plain functions.
Something like:
indent_prefixes= []
defpush_indent(amount=4):
"""Addamountspaces to indentation prefix"""indent_prefixes.push(' '*amount)
defpop_indent():
"""Revert the most recentpush_indent()call"""indent_prefixes.pop()
defindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
What you did is still an improvement, though, so:Quoted 2 lines
Reviewed-by Eduardo Habkost <[email protected]>
Quoted 3 lines
Yeah, there's only one user right now, so ... I just kinda wanted to get rid of the global usage. Maybe if we make the code generator fancier we'll find out what form is best.
You don't get rid of the global variable, you just change it from integer to a class. A class can be handier when generating multiple things interleaved, because you can have one class instance per thing.
Note that we already have a class instance per thing we generate:
instances of subtypes of QAPIGen. The thought of moving the indentation machinery into QAPIGen or or maybe QAPIGenCCode crossed my mind many moons ago, but I had bigger fish to fry, and then I forgot :)
John, I suggest you don't try to make this pretty just yet. Do what needs to be done for the type hint job. We can make it pretty later.
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 2 lines · Markus Armbruster
Quoted 1 line · John Snow
On 9/22/20 6:22 PM, Eduardo Habkost wrote:
[...]
Quoted 7 lines
Quoted 3 lines
Yeah, there's only one user right now, so ... I just kinda wanted to get rid of the global usage. Maybe if we make the code generator fancier we'll find out what form is best.
You don't get rid of the global variable, you just change it from integer to a class. A class can be handier when generating multiple things interleaved, because you can have one class instance per thing.
To be fair, John doesn't claim to be getting rid of a global variable. He's getting rid of usage of the 'global' keyword to make linters happier. There are ways to do that without changing the code too much, though.
Quoted 8 lines
Note that we already have a class instance per thing we generate:
instances of subtypes of QAPIGen. The thought of moving the indentation machinery into QAPIGen or or maybe QAPIGenCCode crossed my mind many moons ago, but I had bigger fish to fry, and then I forgot :)John, I suggest you don't try to make this pretty just yet. Do what needs to be done for the type hint job. We can make it pretty later.
Signature · 1 line
Eduardo
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 8 lines · Eduardo Habkost
Quoted 2 lines · Markus Armbruster
Quoted 1 line · John Snow
On 9/22/20 6:22 PM, Eduardo Habkost wrote:
[...]
Quoted 4 lines
Quoted 3 lines
Yeah, there's only one user right now, so ... I just kinda wanted to get rid of the global usage. Maybe if we make the code generator fancier we'll find out what form is best.
Quoted 7 lines
Quoted 3 lines
You don't get rid of the global variable, you just change it from integer to a class. A class can be handier when generating multiple things interleaved, because you can have one class instance per thing.
To be fair, John doesn't claim to be getting rid of a global variable. He's getting rid of usage of the 'global' keyword to make linters happier.
True.
Quoted 2 lines
There are ways to do that without changing
the code too much, though.
Let's do something easy and cheap, because ...
Quoted 4 lines
Quoted 4 lines
Note that we already have a class instance per thing we generate:
instances of subtypes of QAPIGen. The thought of moving the indentation machinery into QAPIGen or or maybe QAPIGenCCode crossed my mind many moons ago, but I had bigger fish to fry, and then I forgot :)
... we'll probably want to move this stuff into QAPIGen later, and that's when we should make it pretty.
Quoted 2 lines
Quoted 2 lines
John, I suggest you don't try to make this pretty just yet. Do what needs to be done for the type hint job. We can make it pretty later.
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
In reply to Markus ArmbrusterQuoted 86 lines · Markus Armbruster
Quoted 84 lines · John Snow
Quoted 82 lines · Eduardo Habkost
Quoted 54 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 20 deletions(-)scripts/qapi/common.py +19
-9@@ -93,33 +93,52 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> int: + """Increase the indentation level by `amount`, default 4.""" + self._level += amount + return self._level +scripts/qapi/common.py def c_name(name, protect=True): · line 129
Quoted 10 lines
scripts/qapi/common.py +10
-0+ def decrease(self, amount: int = 4) -> int: + """Decrease the indentation level by `amount`, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + return self._level + + +indent = Indentation()Personally, I would keep the
push_indent(),pop_indent()API, and introduce anindent()function, to follow the existing API style of plain functions.
Something like:
indent_prefixes= []
defpush_indent(amount=4):
"""Addamountspaces to indentation prefix"""indent_prefixes.push(' '*amount)
defpop_indent():
"""Revert the most recentpush_indent()call"""indent_prefixes.pop()
defindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
What you did is still an improvement, though, so:Reviewed-by Eduardo Habkost <[email protected]>
Quoted 8 lines
Quoted 3 lines
Yeah, there's only one user right now, so ... I just kinda wanted to get rid of the global usage. Maybe if we make the code generator fancier we'll find out what form is best.
You don't get rid of the global variable, you just change it from integer to a class. A class can be handier when generating multiple things interleaved, because you can have one class instance per thing.
The 'global' keyword, not 'global scoped'. The key thing was to remove re-binding of the global name, which is now true. We bind to this object once and then modify object state.
The global keyword allows us to re-bind the variable at the global scope which is the pattern to avoid. Having globally scoped things you don't rebind is perfectly fine.
Quoted 8 lines
Note that we already have a class instance per thing we generate:
instances of subtypes of QAPIGen. The thought of moving the indentation machinery into QAPIGen or or maybe QAPIGenCCode crossed my mind many moons ago, but I had bigger fish to fry, and then I forgot :)John, I suggest you don't try to make this pretty just yet. Do what needs to be done for the type hint job. We can make it pretty later.
Mmhmm. I'm fine with with we have here for now. We can try to make things pretty later. We have the rust support series to consider, too, so I am not going to put the cart before the horse.
--js
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
In reply to John SnowQuoted 51 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 +++++++++++++++++++++++++++++------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 38 insertions(+), 20 deletions(-)scripts/qapi/common.py +16
-9@@ -93,33 +93,52 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation."""
scripts/qapi/common.py def c_name(name, protect=True): · line 123
Quoted 10 lines
scripts/qapi/common.py +6
-3+ return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> int: + """Increase the indentation level by `amount`, default 4.""" + self._level += amount + return self._level +
Do you have a use case for returning the level? If not, I'd go without it, and add a "level" property instead, as it'd serve more cases.
Other than that,
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 3 lines · Cleber Rosa
Do you have a use case for returning the level? If not, I'd go without it, and add a "level" property instead, as it'd serve more cases.
__int__ is doing that lifting. I can remove the return.
--js
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 6 lines · John Snow
Quoted 3 lines · Cleber Rosa
Do you have a use case for returning the level? If not, I'd go without it, and add a "level" property instead, as it'd serve more cases.
__int__is doing that lifting. I can remove the return.
I like my functions to return something useful.
Use your judgement.
Re: [PATCH v2 09/38] qapi/common.py: Add indent manager
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Quoted 3 lines · Cleber Rosa
Do you have a use case for returning the level? If not, I'd go without it, and add a "level" property instead, as it'd serve more cases.
Quoted 7 lines
Quoted 1 line
__int__is doing that lifting. I can remove the return.I like my functions to return something useful.
Use your judgement.
Eh, Cleber had a point. Nothing uses it.
(AKA: I already made the edit ...)
Like you say, we'll figure out the truly beautiful way to do code generation when we tackle this all together, holistically. What I've got works for now (and isn't terribly complex), let's just roll with it while we fry the bigger fish.
--js
PATCH v2 10/38 qapi/common.py: delint with pylint
In reply to John SnowAt this point, that just means using a consistent strategy for constant names. constants get UPPER_CASE and names not used externally get a leading underscore.
As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
--- scripts/qapi/common.py | 18 ++++++++---------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 15 insertions(+), 17 deletions(-)
scripts/qapi/common.py +8 -10
@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +_C_NAME_TRANS = str.maketrans('.-', '__') + + # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__') - - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True): 'not_eq', 'or', 'or_eq', 'xor', 'xor_eq']) # namespace pollution: polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386']) - name = name.translate(c_name_trans) + name = name.translate(_C_NAME_TRANS) if protect and (name in c89_words | c99_words | c11_words | gcc_words | cpp_words | polluted_words): return 'q_' + name return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indentation: """ Indentation level management.
@@ -134,12 +132,12 @@ def decrease(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. -# Obey indent, and strip eatspace. +# Obey indent, and strip EATSPACE. def cgen(code, **kwds): raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):
scripts/qapi/schema.py +7 -7
@@ -18,7 +18,7 @@ import re from collections import OrderedDict -from .common import c_name, pointer_suffix +from .common import c_name, POINTER_SUFFIX from .error import QAPIError, QAPISemError from .expr import check_exprs from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self): return True def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'array'
@@ -430,7 +430,7 @@ def c_name(self): def c_type(self): assert not self.is_implicit() - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def c_unboxed_type(self): return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None): v.connect_doc(doc) def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type): self._make_array_type(name, None) def _def_predefineds(self): - for t in [('str', 'string', 'char' + pointer_suffix), + for t in [('str', 'string', 'char' + POINTER_SUFFIX), ('number', 'number', 'double'), ('int', 'int', 'int64_t'), ('int8', 'int', 'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self): ('uint64', 'int', 'uint64_t'), ('size', 'int', 'uint64_t'), ('bool', 'boolean', 'bool'), - ('any', 'value', 'QObject' + pointer_suffix), - ('null', 'null', 'QNull' + pointer_suffix)]: + ('any', 'value', 'QObject' + POINTER_SUFFIX), + ('null', 'null', 'QNull' + POINTER_SUFFIX)]: self._def_builtin_type(*t) self.the_empty_object_type = QAPISchemaObjectType( 'q_empty', None, None, None, None, None, [], None)
Signature · 1 line
2.26.2
Re: [PATCH v2 10/38] qapi/common.py: delint with pylint
Quoted 8 lines · John Snow
At this point, that just means using a consistent strategy for constant names. constants get
UPPER_CASEand names not used externally get a leading underscore.As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 10/38] qapi/common.py: delint with pylint
In reply to John SnowQuoted 18 lines · John Snow
At this point, that just means using a consistent strategy for constant names. constants get
UPPER_CASEand names not used externally get a leading underscore.As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 18 ++++++++---------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 15 insertions(+), 17 deletions(-)scripts/qapi/common.py +0
-0
scripts/qapi/common.py line 19
Quoted 6 lines
scripts/qapi/common.py +3
-0@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +_C_NAME_TRANS = str.maketrans('.-', '__')
IMO _C_NAME_TRANS is solely the concern of the c_name() function, and should not be a global. If you're concerned with speed (which I don't think you should) you could still do:
def c_name(name, protect=True,
name_translation=str.maketrans('.-', '__')):
...
name = name.translate(name_translation)Keeping in mind that you're adding a mutable type to a function argument *on purpose*. I'd really favor having that statement within the only function that uses it, though.
- Cleber.
Re: [PATCH v2 10/38] qapi/common.py: delint with pylint
Quoted 22 lines · Cleber Rosa
Quoted 21 lines · John Snow
At this point, that just means using a consistent strategy for constant names. constants get
UPPER_CASEand names not used externally get a leading underscore.As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 18 ++++++++---------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 15 insertions(+), 17 deletions(-)scripts/qapi/common.py +0
-0@@ -14,6 +14,11 @@ import re
Quoted 8 lines
scripts/qapi/common.py line 19
Quoted 3 lines
scripts/qapi/common.py +3
-0+EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +_C_NAME_TRANS = str.maketrans('.-', '__')IMO
_C_NAME_TRANSis solely the concern of thec_name()function, and should not be a global. If you're concerned with speed (which I don't think you should) you could still do:
It's true, but that's why it's marked internal here with the leading underscore -- it will not be exported. It was also *already* defined at the module level, I didn't hoist it.
I think what is written here is already the simplest thing that works.
Quoted 9 lines
def c_name(name, protect=True,
name_translation=str.maketrans('.-', '__')):
...
name =name.translate(name_translation)Keeping in mind that you're adding a mutable type to a function argument *on purpose*. I'd really favor having that statement within the only function that uses it, though.
That complicates the signature, so I think we shouldn't.
--js
Cleber Rosa
Reviewed-byRe: [PATCH v2 10/38] qapi/common.py: delint with pylint
Quoted 41 lines · John Snow
Quoted 14 lines · Cleber Rosa
Quoted 13 lines · John Snow
At this point, that just means using a consistent strategy for constant names. constants get
UPPER_CASEand names not used externally get a leading underscore.As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 18 ++++++++---------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 15 insertions(+), 17 deletions(-)Quoted 14 lines
scripts/qapi/common.py line 17
Quoted 9 lines
scripts/qapi/common.py +3
-0@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +_C_NAME_TRANS = str.maketrans('.-', '__')IMO
_C_NAME_TRANSis solely the concern of thec_name()function, and should not be a global. If you're concerned with speed (which I don't think you should) you could still do:It's true, but that's why it's marked internal here with the leading underscore -- it will not be exported. It was also *already* defined at the module level, I didn't hoist it.
I think what is written here is already the simplest thing that works.
Quoted 5 lines
def c_name(name, protect=True,
name_translation=str.maketrans('.-', '__')):
...
name =name.translate(name_translation)
Quoted 10 lines
Quoted 4 lines
Keeping in mind that you're adding a mutable type to a function argument *on purpose*. I'd really favor having that statement within the only function that uses it, though.
That complicates the signature, so I think we shouldn't.
--js
Not a too strong opinion about this to block it. So if I hadn't already:
PATCH v2 11/38 qapi/common.py: Replace one-letter 'c' variable
In reply to John Snow--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_': + char = c_fun_str[i] + # When char is upper case and no '_' appears before, do more checks + if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_': if i < length - 1 and c_fun_str[i + 1].islower(): new_name += '_' elif c_fun_str[i - 1].isdigit(): new_name += '_' - new_name += c + new_name += char return new_name.lstrip('_').upper()
Signature · 1 line
2.26.2
Re: [PATCH v2 11/38] qapi/common.py: Replace one-letter 'c' variable
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 11/38] qapi/common.py: Replace one-letter 'c' variable
In reply to John SnowQuoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
PATCH v2 12/38 qapi/common.py: check with pylint
In reply to John SnowRemove qapi/common.py from the pylintrc ignore list.
--- scripts/qapi/pylintrc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi/pylintrc +1 -2
@@ -2,8 +2,7 @@ # Add files or directories matching the regex patterns to the ignore list. # The regex matches against base names, not paths. -ignore-patterns=common.py, - doc.py, +ignore-patterns=doc.py, error.py, expr.py, gen.py,
Signature · 1 line
2.26.2
Re: [PATCH v2 12/38] qapi/common.py: check with pylint
Quoted 3 lines · John Snow
Remove
qapi/common.pyfrom the pylintrc ignore list.Signed-off-by John Snow <[email protected]>
No pylint warnings detected.
Signature · 1 line
Eduardo
Re: [PATCH v2 12/38] qapi/common.py: check with pylint
In reply to John SnowQuoted 3 lines · John Snow
Remove
qapi/common.pyfrom the pylintrc ignore list.Signed-off-by John Snow <[email protected]>
$ pylint scripts/qapi/ --rcfile=scripts/qapi/pylintrc
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
I still think we should record (and maybe I'm missing it) the pylint version used though. Anyway:
Re: [PATCH v2 12/38] qapi/common.py: check with pylint
Quoted 2 lines · Cleber Rosa
I still think we should record (and maybe I'm missing it) the pylint version used though. Anyway:
Cover letter for all six parts mentions pylint 2.6.0!
In the future this will be pinned in the CI mechanisms.
--js
PATCH v2 13/38 qapi/common.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +16 -11
@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Sequence EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -133,24 +136,24 @@ def decrease(self, amount: int = 4) -> int: # Generate @code with @kwds interpolated. # Obey indent, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: object) -> str: raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: object) -> str: if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s
@@ -159,7 +162,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */
@@ -167,7 +170,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''
@@ -176,7 +179,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''
@@ -185,7 +188,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:
Signature · 1 line
2.26.2
Re: [PATCH v2 13/38] qapi/common.py: add type hint annotations
Quoted 13 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +0
-0@@ -12,6 +12,7 @@
[...]
scripts/qapi/common.py def gen_if(ifcond): · line 182
Quoted 6 lines
scripts/qapi/common.py +1
-1@@ -176,7 +179,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str:
Does this need to require a Sequence? It looks like it could be Iterable.
I don't think this should block the patch, though, so:
Quoted 17 lines
scripts/qapi/common.py +3
-1ret = '' for ifc in reversed(ifcond): ret += mcgen('''@@ -185,7 +188,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 13/38] qapi/common.py: add type hint annotations
Quoted 6 lines · Eduardo Habkost
Does this need to require a Sequence? It looks like it could be Iterable.
I don't think this should block the patch, though, so:
Reviewed-by Eduardo Habkost<[email protected]>
gen_if can take an Iterator, gen_endif needs a Sequence because it uses reversed().
I have not been very fastidious about choosing the MOST possibly abstracted type, but you are right that we SHOULD. I was hoping to find all occurrences of List and try slackening them to Collection/Sequence/Iterable etc, but as a follow-up.
In this case, since gen_endif actually does want the stricter type, I'm deciding to leave them as Sequence for signature parity.
--js
(P.S. in case it comes up in future review, there is a bug that prevents us from using Collection in 3.6: https://github.com/PyCQA/pylint/issues/2377 )
Re: [PATCH v2 13/38] qapi/common.py: add type hint annotations
Quoted 11 lines · John Snow
Quoted 6 lines · Eduardo Habkost
Does this need to require a Sequence? It looks like it could be Iterable.
I don't think this should block the patch, though, so:
Reviewed-by Eduardo Habkost<[email protected]>
gen_ifcan take an Iterator,gen_endifneeds a Sequence because it usesreversed().
Quoted 7 lines
I have not been very fastidious about choosing the MOST possibly abstracted type, but you are right that we SHOULD. I was hoping to find all occurrences of List and try slackening them to Collection/Sequence/Iterable etc, but as a follow-up.
In this case, since
gen_endifactually does want the stricter type, I'm deciding to leave them as Sequence for signature parity.
I hadn't noticed reversed() requires a Sequence. I agree using the same type on both is better. Thanks!
Signature · 1 line
Eduardo
Re: [PATCH v2 13/38] qapi/common.py: add type hint annotations
In reply to John SnowQuoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
PATCH v2 14/38 qapi/common.py: Convert comments into docstrings, and elaborate
In reply to John SnowAs docstrings, they'll show up in documentation and IDE help.
--- scripts/qapi/common.py | 51 ++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 14 deletions(-)
scripts/qapi/common.py +37 -14
@@ -20,10 +20,18 @@ _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -134,9 +154,12 @@ def decrease(self, amount: int = 4) -> int: indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate `code` with `kwds` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """ raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
Signature · 1 line
2.26.2
Re: [PATCH v2 14/38] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 3 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 14/38] qapi/common.py: Convert comments into docstrings, and elaborate
In reply to John SnowQuoted 78 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 ++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 14 deletions(-)scripts/qapi/common.py +32
-12@@ -20,10 +20,18 @@ _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -134,9 +154,12 @@ def decrease(self, amount: int = 4) -> int: indent = Indentation()
scripts/qapi/common.py def decrease(self, amount: int = 4) -> int: · line 162
Quoted 10 lines
scripts/qapi/common.py +5
-2-# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate `code` with `kwds` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """
This probably won't help on IDEs (never checked any), but sphinx will let you do:
"""
Generate code with kwds interpolated.
Obey indent, and strip :data:EATSPACE.
"""
I'm not sure that a maximum level of docstring "sphinxzation" is the goal here, though.
Re: [PATCH v2 14/38] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 96 lines · Cleber Rosa
Quoted 78 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 ++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 14 deletions(-)scripts/qapi/common.py +32
-12@@ -20,10 +20,18 @@ _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -134,9 +154,12 @@ def decrease(self, amount: int = 4) -> int: indent = Indentation()scripts/qapi/common.py def decrease(self, amount: int = 4) -> int: · line 162
Quoted 10 lines
scripts/qapi/common.py +5
-2-# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate `code` with `kwds` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """This probably won't help on IDEs (never checked any), but sphinx will let you do:
"""
Generatecodewithkwdsinterpolated.
Quoted 8 lines
Obey
indent, and strip :data:EATSPACE.
"""I'm not sure that a maximum level of docstring "sphinxzation" is the goal here, though.
Reviewed-by Cleber Rosa <[email protected]>
It isn't yet, but I intend to address that when I remove missing-docstring from pylint exemptions. Do I need :data: if I set the default role to 'any'?
I'll probably try to enable sphinx at that time (and put the docs in a devel/python manual?) and worry about the formatting at that point.
--js
Re: [PATCH v2 14/38] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 94 lines · John Snow
Quoted 93 lines · Cleber Rosa
Quoted 73 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 ++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 14 deletions(-)scripts/qapi/common.py +32
-12@@ -20,10 +20,18 @@ _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',scripts/qapi/common.py def decrease(self, amount: int = 4) -> int: · line 160
Quoted 9 lines
scripts/qapi/common.py +5
-2@@ -134,9 +154,12 @@ def decrease(self, amount: int = 4) -> int: indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate `code` with `kwds` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """This probably won't help on IDEs (never checked any), but sphinx will let you do:
"""
Generatecodewithkwdsinterpolated.Obey
indent, and strip :data:EATSPACE.
"""
Quoted 9 lines
Quoted 5 lines
I'm not sure that a maximum level of docstring "sphinxzation" is the goal here, though.
Reviewed-by Cleber Rosa <[email protected]>It isn't yet, but I intend to address that when I remove missing-docstring from pylint exemptions. Do I need :data: if I set the default role to 'any'?
That's a good question. According to the docs "any" will do its best, so it's probably a good fallback. I do still favor using the correct role from the start if I can help it.
Quoted 4 lines
I'll probably try to enable sphinx at that time (and put the docs in a devel/python manual?) and worry about the formatting at that point.
--js
Nice!
- Cleber.
Re: [PATCH v2 14/38] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 109 lines · Cleber Rosa
Quoted 94 lines · John Snow
Quoted 93 lines · Cleber Rosa
Quoted 73 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/common.py | 51 ++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 14 deletions(-)scripts/qapi/common.py +32
-12@@ -20,10 +20,18 @@ _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map `name` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',scripts/qapi/common.py def decrease(self, amount: int = 4) -> int: · line 160
Quoted 9 lines
scripts/qapi/common.py +5
-2@@ -134,9 +154,12 @@ def decrease(self, amount: int = 4) -> int: indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate `code` with `kwds` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """This probably won't help on IDEs (never checked any), but sphinx will let you do:
"""
Generatecodewithkwdsinterpolated.Obey
indent, and strip :data:EATSPACE.
"""Quoted 9 lines
Quoted 5 lines
I'm not sure that a maximum level of docstring "sphinxzation" is the goal here, though.
Reviewed-by Cleber Rosa <[email protected]>It isn't yet, but I intend to address that when I remove missing-docstring from pylint exemptions. Do I need :data: if I set the default role to 'any'?
That's a good question. According to the docs "any" will do its best, so it's probably a good fallback. I do still favor using the correct role from the start if I can help it.
Quoted 9 lines
Quoted 4 lines
I'll probably try to enable sphinx at that time (and put the docs in a devel/python manual?) and worry about the formatting at that point.
--js
Nice!
- Cleber.
As of v3, I started toying with this, as you can see. It is a goal of mine to hit full doc coverage in this package, eventually.
What I learned: you can reference data members, but only if they have a comment. Otherwise, they are skipped. Sphinx does not appear to offer an "undocumented data member" option the same way it does for "undocumented member".
Using the "Any" role is nice, and I prefer it. If it finds that a target is ambiguous (2+ references), it will throw an error and the sphinx build will fail. This is good enough for me: there's no reason to clutter the docstrings with Sphinxese if we don't have to.
I would like to try and keep these readable to humans who are just in emacs/vim editing code, too.
--js
PATCH v2 15/38 qapi/common.py: move build_params into gen.py
In reply to John SnowIncluding it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.
--- scripts/qapi/commands.py | 8 ++++++-- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 7 +++++-- scripts/qapi/gen.py | 34 ++++++++++++++++++++++++++++++++-- 4 files changed, 43 insertions(+), 29 deletions(-)
scripts/qapi/commands.py +6 -2
@@ -14,11 +14,15 @@ """ from .common import ( - build_params, c_name, mcgen, ) -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .gen import ( + QAPIGenCCode, + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'
scripts/qapi/events.py +5 -2
@@ -13,12 +13,15 @@ """ from .common import ( - build_params, c_enum_const, c_name, mcgen, ) -from .gen import QAPISchemaModularCVisitor, ifcontext +from .gen import ( + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +32 -2
@@ -2,9 +2,11 @@ # # QAPI code generation # -# Copyright (c) 2018-2019 Red Hat Inc. +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2019 Red Hat Inc. # # Authors: +# Anthony Liguori <[email protected]> # Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #
@@ -15,16 +17,21 @@ import errno import os import re +from typing import Optional from .common import ( c_fname, + c_name, gen_endif, gen_if, guardend, guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import ( + QAPISchemaObjectType, + QAPISchemaVisitor, +) class QAPIGen:
@@ -90,6 +97,29 @@ def _wrap_ifcond(ifcond, before, after): return out +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void' + + class QAPIGenCCode(QAPIGen): def __init__(self, fname):
Signature · 1 line
2.26.2
Re: [PATCH v2 15/38] qapi/common.py: move build_params into gen.py
Quoted 6 lines · John Snow
Including it in
common.pycreates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 15/38] qapi/common.py: move build_params into gen.py
In reply to John SnowQuoted 6 lines · John Snow
Including it in
common.pycreates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.Signed-off-by John Snow <[email protected]>
PATCH v2 16/38 qapi: establish mypy type-checking baseline
In reply to John SnowFix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
--- scripts/qapi/doc.py | 3 +- scripts/qapi/mypy.ini | 65 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 +- 3 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/doc.py +2 -1
@@ -5,7 +5,8 @@ """This script produces the documentation of a qapi schema in texinfo format""" import re -from .gen import QAPIGenDoc, QAPISchemaVisitor +from .gen import QAPIGenDoc +from .schema import QAPISchemaVisitor MSG_FMT = """
scripts/qapi/mypy.ini new file +65 -0
65 lines not on this page · raw message
scripts/qapi/schema.py +2 -1
@@ -17,6 +17,7 @@ import os import re from collections import OrderedDict +from typing import Optional from .common import c_name, POINTER_SUFFIX from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Signature · 1 line
2.26.2
Re: [PATCH v2 16/38] qapi: establish mypy type-checking baseline
Quoted 8 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
Signed-off-by John Snow <[email protected]>
Works with mypy==0.770.
Signature · 1 line
Eduardo
Re: [PATCH v2 16/38] qapi: establish mypy type-checking baseline
In reply to John SnowQuoted 6 lines · John Snow
Fix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
Like with pylint, let's at least document the version of mypy used. For example, with 0.740, this fails. With 0.782, it passes.
- Cleber.
PATCH v2 17/38 qapi/events.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 ----- 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +35 -11
@@ -12,6 +12,8 @@ See the COPYING file in the top-level directory. """ +from typing import List + from .common import ( c_enum_const, c_name,
@@ -22,17 +24,27 @@ build_params, ifcontext, ) -from .schema import QAPISchemaEnumMember +from .schema import ( + QAPISchema, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, +) +from .source import QAPISourceInfo from .types import gen_enum, gen_enum_lookup -def build_event_send_proto(name, arg_type, boxed): +def build_event_send_proto(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return 'void qapi_event_send_%(c_name)s(%(param)s)' % { 'c_name': c_name(name.lower()), 'param': build_params(arg_type, boxed)} -def gen_event_send_decl(name, arg_type, boxed): +def gen_event_send_decl(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return mcgen(''' %(proto)s;
@@ -41,7 +53,7 @@ def gen_event_send_decl(name, arg_type, boxed): # Declare and initialize an object 'qapi' using parameters from build_params() -def gen_param_var(typ): +def gen_param_var(typ: QAPISchemaObjectType) -> str: assert not typ.variants ret = mcgen(''' %(c_name)s param = {
@@ -69,7 +81,11 @@ def gen_param_var(typ): return ret -def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): +def gen_event_send(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool, + event_enum_name: str, + event_emit: str) -> str: # FIXME: Our declaration of local variables (and of 'errp' in the # parameter list) can collide with exploded members of the event's # data type passed in as parameters. If this collision ever hits in
@@ -145,15 +161,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-events', ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) - self._event_enum_members = [] + self._event_enum_members: List[QAPISchemaEnumMember] = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit') - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: events = self._module_basename('qapi-events', name) types = self._module_basename('qapi-types', name) visit = self._module_basename('qapi-visit', name)
@@ -176,7 +192,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('emit', ' * QAPI Events emission') self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h"
@@ -197,7 +213,13 @@ def visit_end(self): event_emit=self._event_emit_name, event_enum=self._event_enum_name)) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + boxed: bool) -> None: with ifcontext(ifcond, self._genh, self._genc): self._genh.add(gen_event_send_decl(name, arg_type, boxed)) self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -208,7 +230,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed): self._event_enum_members.append(QAPISchemaEnumMember(name, None)) -def gen_events(schema, output_dir, prefix): +def gen_events(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenEventVisitor(prefix) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.events] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.expr] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
Re: [PATCH v2 17/38] qapi/events.py: add type hint annotations
Quoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 17/38] qapi/events.py: add type hint annotations
In reply to John SnowQuoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
PATCH v2 18/38 qapi/events.py: Move comments into docstrings
In reply to John Snow--- scripts/qapi/events.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/events.py +3 -1
@@ -52,8 +52,10 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Declare and initialize a qapi object, using parameters from `build_params`. + """ assert not typ.variants ret = mcgen(''' %(c_name)s param = {
Signature · 1 line
2.26.2
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
Quoted 10 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/events.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)scripts/qapi/events.py +0
-0
scripts/qapi/events.py def gen_event_send_decl(name: str, · line 57
Quoted 7 lines
scripts/qapi/events.py +2
-1@@ -52,8 +52,10 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Declare and initialize a qapi object, using parameters from `build_params`.
The mention of "object 'qapi'" is gone, and this seems correct because there's no object called 'qapi' anywhere in this function. So:
Comments/questions for follow up patches, because it's beyond the scope of this series:
- Was the doc string supposed to say "an object 'param'" instead of "an object 'qapi'", as that's the name of the variable it declares?
- The "using parameters from build_params()" part was confusing to me. I thought it meant gen_param_var() would call build_params(). I took a while to notice it actually meant "using the C function parameters that were previously declared using build_params() at build_event_send_proto()". I don't know how we could make it clearer.
Quoted 7 lines
scripts/qapi/events.py +1
-0+ """ assert not typ.variants ret = mcgen(''' %(c_name)s param = {Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
Quoted 27 lines · Eduardo Habkost
Quoted 10 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/events.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)scripts/qapi/events.py +0
-0scripts/qapi/events.py def gen_event_send_decl(name: str, · line 57
Quoted 7 lines
scripts/qapi/events.py +2
-1@@ -52,8 +52,10 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Declare and initialize a qapi object, using parameters from `build_params`.The mention of "object 'qapi'" is gone, and this seems correct because there's no object called 'qapi' anywhere in this function. So:
Quoted 1 line
Reviewed-by Eduardo Habkost <[email protected]>Comments/questions for follow up patches, because it's beyond the scope of this series:
Quoted 10 lines
- Was the doc string supposed to say "an object 'param'" instead of "an object 'qapi'", as that's the name of the variable it declares?
- The "using parameters frombuild_params()" part was confusing to me. I thought it meantgen_param_var()would callbuild_params(). I took a while to notice it actually meant "using the C function parameters that were previously declared usingbuild_params()atbuild_event_send_proto()". I don't know how we could make it clearer.
Good questions for Markus.
(By the way, Markus: I do intend to remove the "missing-docstring" warning from the exceptions, and at such time we can have a party writing docstrings for everything.
I intend to devote an entire series to doing this during the release window.)
--js
Quoted 8 lines
Quoted 7 lines
+ """
assert not typ.variants ret = mcgen(''' %(c_name)s param = {Signature · 1 line
2.26.2
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
In reply to Eduardo HabkostQuoted 23 lines · Eduardo Habkost
Quoted 10 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/events.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)scripts/qapi/events.py +0
-0scripts/qapi/events.py def gen_event_send_decl(name: str, · line 57
Quoted 7 lines
scripts/qapi/events.py +2
-1@@ -52,8 +52,10 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Declare and initialize a qapi object, using parameters from `build_params`.The mention of "object 'qapi'" is gone, and this seems correct because there's no object called 'qapi' anywhere in this function. So:
Quoted 8 lines
Reviewed-by Eduardo Habkost <[email protected]>Comments/questions for follow up patches, because it's beyond the scope of this series:
- Was the doc string supposed to say "an object 'param'" instead of "an object 'qapi'", as that's the name of the variable it declares?
Checking history... yes. The variable was renamed from @qapi to @param during review.
Quoted 6 lines
- The "using parameters from
build_params()" part was confusing to me. I thought it meantgen_param_var()would callbuild_params(). I took a while to notice it actually meant "using the C function parameters that were previously declared usingbuild_params()atbuild_event_send_proto()". I don't know how we could make it clearer.
What about:
Generate a QAPI struct variable holding the event parameters, initialized with the function's arguments.
Quoted 7 lines
Quoted 7 lines
scripts/qapi/events.py +1
-0+ """ assert not typ.variants ret = mcgen(''' %(c_name)s param = {Signature · 1 line
2.26.2
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
Quoted 4 lines · Markus Armbruster
What about:
Generate a QAPI struct variable holding the event parameters, initialized with the function's arguments.
Line length and style-guide limitations; docstrings need a one-liner summary.
(Consistency is the hobgoblin, blah blah blah.)
I am writing:
"""
Generate a QAPI struct variable with an initializer.
The QAPI struct describes the event parameters, and the initializer references the function arguments defined in gen_event_send. """
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
Quoted 7 lines · John Snow
Quoted 3 lines · Markus Armbruster
What about:
Generate a QAPI struct variable holding the event parameters, initialized with the function's arguments.Line length and style-guide limitations; docstrings need a one-liner summary.
They do!
Quoted 10 lines
(Consistency is the hobgoblin, blah blah blah.)
I am writing:
"""
Generate a QAPI struct variable with an initializer.The QAPI struct describes the event parameters, and the initializer references the function arguments defined in
gen_event_send. """
Better.
My second try:
Generate a struct variable holding the event parameters.
Initialize it with the function arguments defined in in gen_event_send.
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
Quoted 23 lines · Markus Armbruster
Quoted 7 lines · John Snow
Quoted 3 lines · Markus Armbruster
What about:
Generate a QAPI struct variable holding the event parameters, initialized with the function's arguments.Line length and style-guide limitations; docstrings need a one-liner summary.
They do!
Quoted 10 lines
(Consistency is the hobgoblin, blah blah blah.)
I am writing:
"""
Generate a QAPI struct variable with an initializer.The QAPI struct describes the event parameters, and the initializer references the function arguments defined in
gen_event_send. """
Quoted 9 lines
Better.
My second try:
Generate a struct variable holding the event parameters.
Initialize it with the function arguments defined in in
gen_event_send.
You're the boss!
--js
Re: [PATCH v2 18/38] qapi/events.py: Move comments into docstrings
In reply to John SnowQuoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
PATCH v2 19/38 qapi/commands.py: Don't re-bind to variable of different type
In reply to John SnowMypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
--- scripts/qapi/commands.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +1 -3
@@ -198,14 +198,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): if not options: options = ['QCO_NO_OPTIONS'] - options = " | ".join(options) - ret = mcgen(''' qmp_register_command(cmds, "%(name)s", qmp_marshal_%(c_name)s, %(opts)s); ''', name=name, c_name=c_name(name), - opts=options) + opts=" | ".join(options)) return ret
Signature · 1 line
2.26.2
Re: [PATCH v2 19/38] qapi/commands.py: Don't re-bind to variable of different type
Quoted 4 lines · John Snow
Mypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 19/38] qapi/commands.py: Don't re-bind to variable of different type
In reply to John SnowQuoted 4 lines · John Snow
Mypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
Signed-off-by John Snow <[email protected]>
PATCH v2 20/38 qapi/commands.py: add notational type hints
In reply to John Snow--- scripts/qapi/commands.py | 66 +++++++++++++++++++++++++++++----------- 1 file changed, 49 insertions(+), 17 deletions(-)
scripts/qapi/commands.py +49 -17
@@ -13,19 +13,32 @@ See the COPYING file in the top-level directory. """ +from typing import Dict, Optional, List, Set + from .common import ( c_name, mcgen, ) from .gen import ( + QAPIGenC, QAPIGenCCode, QAPISchemaModularCVisitor, build_params, ifcontext, ) +from .schema import ( + QAPISchema, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaType, +) +from .source import QAPISourceInfo -def gen_command_decl(name, arg_type, boxed, ret_type): +def gen_command_decl(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: return mcgen(''' %(c_type)s qmp_%(c_name)s(%(params)s); ''',
@@ -34,7 +47,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type): params=build_params(arg_type, boxed, 'Error **errp')) -def gen_call(name, arg_type, boxed, ret_type): +def gen_call(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: ret = '' argstr = ''
@@ -70,7 +86,7 @@ def gen_call(name, arg_type, boxed, ret_type): return ret -def gen_marshal_output(ret_type): +def gen_marshal_output(ret_type: QAPISchemaType) -> str: return mcgen(''' static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -91,19 +107,22 @@ def gen_marshal_output(ret_type): c_type=ret_type.c_type(), c_name=ret_type.c_name()) -def build_marshal_proto(name): +def build_marshal_proto(name: str) -> str: return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)) -def gen_marshal_decl(name): +def gen_marshal_decl(name: str) -> str: return mcgen(''' %(proto)s; ''', proto=build_marshal_proto(name)) -def gen_marshal(name, arg_type, boxed, ret_type): +def gen_marshal(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: have_args = boxed or (arg_type and not arg_type.is_empty()) ret = mcgen('''
@@ -185,7 +204,10 @@ def gen_marshal(name, arg_type, boxed, ret_type): return ret -def gen_register_command(name, success_response, allow_oob, allow_preconfig): +def gen_register_command(name: str, + success_response: bool, + allow_oob: bool, + allow_preconfig: bool) -> str: options = [] if not success_response:
@@ -207,7 +229,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): return ret -def gen_registry(registry, prefix): +def gen_registry(registry: str, prefix: str) -> str: ret = mcgen(''' void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,15 +246,14 @@ def gen_registry(registry, prefix): class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): - - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-commands', ' * Schema-defined QAPI/QMP commands', None, __doc__) self._regy = QAPIGenCCode(None) - self._visited_ret_types = {} + self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {} - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: self._visited_ret_types[self._genc] = set() commands = self._module_basename('qapi-commands', name) types = self._module_basename('qapi-types', name)
@@ -256,7 +277,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('init', ' * QAPI Commands initialization') self._genh.add(mcgen(''' #include "qapi/qmp/dispatch.h"
@@ -272,9 +293,18 @@ def visit_end(self): prefix=self._prefix)) self._genc.add(gen_registry(self._regy.get_content(), self._prefix)) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool) -> None: if not gen: return # FIXME: If T is a user-defined type, the user is responsible
@@ -295,7 +325,9 @@ def visit_command(self, name, info, ifcond, features, allow_oob, allow_preconfig)) -def gen_commands(schema, output_dir, prefix): +def gen_commands(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenCommandVisitor(prefix) schema.visit(vis) vis.write(output_dir)
Signature · 1 line
2.26.2
Re: [PATCH v2 20/38] qapi/commands.py: add notational type hints
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 20/38] qapi/commands.py: add notational type hints
Quoted 5 lines · Eduardo Habkost
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>
Oops, I didn't use my consistent type hint commit message for this one. Changing that to avoid my invented word "notational".
--js
Re: [PATCH v2 20/38] qapi/commands.py: add notational type hints
In reply to John SnowQuoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
PATCH v2 21/38 qapi/commands.py: enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.doc] disallow_subclassing_any = False disallow_untyped_defs = False
Signature · 1 line
2.26.2
Re: [PATCH v2 21/38] qapi/commands.py: enable checking with mypy
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 21/38] qapi/commands.py: enable checking with mypy
In reply to John SnowQuoted 24 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)scripts/qapi/mypy.ini +0
-5@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.doc] disallow_subclassing_any = False disallow_untyped_defs = FalseSignature · 1 line
2.26.2
IMO, this increase of strictness for "commands" would make more sense to be squashed together with the previous changes on "commands.py". Not only here, but for the other patches for the other modules too.
Anyway,
Re: [PATCH v2 21/38] qapi/commands.py: enable checking with mypy
Quoted 30 lines · Cleber Rosa
Quoted 24 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)scripts/qapi/mypy.ini +0
-5@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.doc] disallow_subclassing_any = False disallow_untyped_defs = FalseSignature · 1 line
2.26.2IMO, this increase of strictness for "commands" would make more sense to be squashed together with the previous changes on "
commands.py". Not only here, but for the other patches for the other modules too.
Quoted 4 lines
Anyway,
Reviewed-by Cleber Rosa <[email protected]>
Admittedly, the only reason I *didn't* is because these patches have been reordered a *lot* and in some cases, it helped me to have distinct "This patch is last and enables the checks!" commits.
(I am hedging my bets that more re-ordering is in my future.)
I will squash the "Enable such-and-such" commits with whatever fixed the last error for final inclusion, but I might keep them separate for now just for my own convenience.
Sorry for the volume.
--js
PATCH v2 22/38 qapi/source.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -39,11 +39,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/source.py +18 -13
@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]): self.fname = fname self.line = line self.parent = parent - self.pragma = parent.pragma if parent else QAPISchemaPragma() - self.defn_meta = None - self.defn_name = None + self.pragma: QAPISchemaPragma = ( + parent.pragma if parent else QAPISchemaPragma() + ) + self.defn_meta: Optional[str] = None + self.defn_name: Optional[str] = None - def set_defn(self, meta, name): + def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self): + def next_line(self: T) -> T: info = copy.copy(self) info.line += 1 return info - def loc(self): + def loc(self) -> str: if self.fname is None: return sys.argv[0] ret = self.fname
@@ -49,13 +54,13 @@ def loc(self): ret += ':%d' % self.line return ret - def in_defn(self): + def in_defn(self) -> str: if self.defn_name: return "%s: In %s '%s':\n" % (self.fname, self.defn_meta, self.defn_name) return '' - def include_path(self): + def include_path(self) -> str: ret = '' parent = self.parent while parent:
@@ -63,5 +68,5 @@ def include_path(self): parent = parent.parent return ret - def __str__(self): + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc()
Signature · 1 line
2.26.2
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
Quoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
In reply to John SnowQuoted 14 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-0
scripts/qapi/mypy.ini disallow_untyped_defs = False · line 46
Quoted 8 lines
scripts/qapi/mypy.ini +0
-5@@ -39,11 +39,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False -
This is what I meant in my comment in the previous patch. It looks like a mix of commit grannurality styles. Not a blocker though.
Quoted 8 lines
scripts/qapi/mypy.ini +0
-0[mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +0
-0
scripts/qapi/source.py line 18
Quoted 9 lines
scripts/qapi/source.py +2
-1@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None:
I don't follow the reason for typing this...
Quoted 5 lines
scripts/qapi/source.py +1
-1# Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = []
scripts/qapi/source.py line 30
Quoted 10 lines
scripts/qapi/source.py +4
-2# Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):
And not this... to tune my review approach, should I assume that this series intends to add complete type hints or not?
- Cleber.
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
Quoted 18 lines · Cleber Rosa
Quoted 17 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-0@@ -39,11 +39,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False
Quoted 9 lines
scripts/qapi/mypy.ini disallow_untyped_defs = False · line 46
Quoted 5 lines
scripts/qapi/mypy.ini +0
-5-[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False -This is what I meant in my comment in the previous patch. It looks like a mix of commit grannurality styles. Not a blocker though.
Yep. Just how the chips fell. Some files were just very quick to cleanup and I didn't have to refactor them much when I split things out, so the enablements got rolled in.
I will, once reviews are in (and there is a commitment to merge), try to squash things where it seems appropriate.
Quoted 30 lines
Quoted 8 lines
scripts/qapi/mypy.ini +0
-0[mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +0
-0scripts/qapi/source.py line 18
Quoted 9 lines
scripts/qapi/source.py +2
-1@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None:I don't follow the reason for typing this...
Quoted 10 lines
scripts/qapi/source.py +2
-2# Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = []
Quoted 9 lines
scripts/qapi/source.py line 30
Quoted 5 lines
scripts/qapi/source.py +3
-1class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):And not this... to tune my review approach, should I assume that this series intends to add complete type hints or not?
This is a mypy quirk you've discovered that I've simply forgotten about.
When __init__ has *no* arguments, you need to annotate its return to explain to mypy that you have fully typed that method. It's a sentinel that says "Please type check this class!"
When __init__ has some arguments, you only need to type those arguments and not the return type. The sentinel is not needed.
__init__ *never* returns anything, so I opted to omit this useless annotation whenever it was possible to do so.
Quoted 2 lines
- Cleber.
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
Quoted 6 lines · John Snow
Quoted 5 lines · Cleber Rosa
Quoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
[...]
Quoted 39 lines
Quoted 23 lines
scripts/qapi/source.py line 16
Quoted 12 lines
scripts/qapi/source.py +2
-1@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None:I don't follow the reason for typing this...
Quoted 9 lines
scripts/qapi/source.py +2
-2# Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = []Quoted 9 lines
scripts/qapi/source.py line 30
Quoted 5 lines
scripts/qapi/source.py +3
-1class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):And not this... to tune my review approach, should I assume that this
series intends to add complete type hints or not?This is a mypy quirk you've discovered that I've simply forgotten about.
When
__init__has *no* arguments, you need to annotate its return to explain to mypy that you have fully typed that method. It's a sentinel that says "Please type check this class!"
Quoted 5 lines
When
__init__has some arguments, you only need to type those arguments and not the return type. The sentinel is not needed.
__init__*never* returns anything, so I opted to omit this useless annotation whenever it was possible to do so.
Worth capturing in a comment and/or commit message?
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
Quoted 1 line · Markus Armbruster
Worth capturing in a comment and/or commit message?
Doesn't hurt me any to do so.
It's also good fodder for a style guide document, which would centralize such things.
Here is a formal IOU: https://gitlab.com/jsnow/qemu/-/issues/7
--js
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
In reply to John SnowQuoted 70 lines · John Snow
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)Quoted 16 lines
scripts/qapi/mypy.ini disallow_untyped_defs = False · line 45
Quoted 12 lines
scripts/qapi/mypy.ini +0
-5@@ -39,11 +39,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False -This is what I meant in my comment in the previous patch. It looks like a mix of commit grannurality styles. Not a blocker though.
Yep. Just how the chips fell. Some files were just very quick to cleanup and I didn't have to refactor them much when I split things out, so the enablements got rolled in.
I will, once reviews are in (and there is a commitment to merge), try to squash things where it seems appropriate.
Quoted 31 lines
Quoted 8 lines
scripts/qapi/mypy.ini +0
-0[mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +0
-0scripts/qapi/source.py line 15
Quoted 6 lines
scripts/qapi/source.py +2
-1@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None:I don't follow the reason for typing this...
scripts/qapi/source.py line 30
Quoted 13 lines
scripts/qapi/source.py +5
-3# Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):Quoted 3 lines
And not this... to tune my review approach, should I assume that this series intends to add complete type hints or not?
Quoted 6 lines
This is a mypy quirk you've discovered that I've simply forgotten about.
When
__init__has *no* arguments, you need to annotate its return to explain to mypy that you have fully typed that method. It's a sentinel that says "Please type check this class!"
Ouch. Is this a permanent quirk or a known bug that will eventually be addressed?
- Cleber.
Re: [PATCH v2 22/38] qapi/source.py: add type hint annotations
Quoted 73 lines · Cleber Rosa
Quoted 72 lines · John Snow
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)Quoted 16 lines
scripts/qapi/mypy.ini disallow_untyped_defs = False · line 45
Quoted 12 lines
scripts/qapi/mypy.ini +0
-5@@ -39,11 +39,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False -This is what I meant in my comment in the previous patch. It looks like a mix of commit grannurality styles. Not a blocker though.
Yep. Just how the chips fell. Some files were just very quick to cleanup and I didn't have to refactor them much when I split things out, so the enablements got rolled in.
I will, once reviews are in (and there is a commitment to merge), try to squash things where it seems appropriate.
Quoted 31 lines
Quoted 8 lines
scripts/qapi/mypy.ini +0
-0[mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +0
-0scripts/qapi/source.py line 15
Quoted 6 lines
scripts/qapi/source.py +2
-1@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None:I don't follow the reason for typing this...
scripts/qapi/source.py line 30
Quoted 13 lines
scripts/qapi/source.py +5
-3# Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):Quoted 3 lines
And not this... to tune my review approach, should I assume that this series intends to add complete type hints or not?
This is a mypy quirk you've discovered that I've simply forgotten about.
Quoted 7 lines
Quoted 4 lines
When
__init__has *no* arguments, you need to annotate its return to explain to mypy that you have fully typed that method. It's a sentinel that says "Please type check this class!"Ouch. Is this a permanent quirk or a known bug that will eventually be addressed?
Permanent, it is a feature.
mypy intentionally supports gradual typing as a paradigm: it allows you to intermix "typed" and "untyped" functions.
```
def __init__(self):
pass
```Happens to pass as both untyped and fully typed. In order to distinguish it in this one case, you must add the return annotation as a declaration of intent.
However, when using '--strict' mode, you are declaring your intent to mypy that everything MUST be strictly typed, so perhaps in this case it would be possible to omit the annotation for __init__.
So maybe someday this will change; but given how uncommon it is to write no-argument init methods, I am hardly bothered by it. Mypy will remind you if you forget.
Quoted 3 lines
- Cleber.
PATCH v2 23/38 qapi/source.py: delint with pylint
In reply to John SnowShush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/source.py | 3 +++ 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/pylintrc +0 -1
@@ -8,7 +8,6 @@ ignore-patterns=doc.py, gen.py, parser.py, schema.py, - source.py, types.py, visit.py,
scripts/qapi/source.py +3 -0
@@ -15,6 +15,9 @@ class QAPISchemaPragma: + # Replace with @dataclass in Python 3.7+ + # pylint: disable=too-few-public-methods + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False
Signature · 1 line
2.26.2
Re: [PATCH v2 23/38] qapi/source.py: delint with pylint
Quoted 4 lines · John Snow
Shush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 23/38] qapi/source.py: delint with pylint
In reply to John SnowQuoted 4 lines · John Snow
Shush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.
Signed-off-by John Snow <[email protected]>
PATCH v2 24/38 qapi/gen.py: Fix edge-case of _is_user_module
In reply to John SnowThe edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +1 -1
@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./') @staticmethod def _is_builtin_module(name):
Signature · 1 line
2.26.2
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 13 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')
This changes behavior if name=='', and I guess this is OK, but I'm not sure. I miss documentation on visit_module(), visit_include(), and _is_user_module(). I don't know what name means here, and what is a "user module".
Quoted 6 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 5 lines · Eduardo Habkost
This changes behavior if name=='', and I guess this is OK, but I'm not sure. I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".
Good spot, I missed that.
I can probably do: bool(name and not name.startswith('./'))
to convert explicitly the empty string to false. I will allow Markus the chance to explain the module stuff.
--js
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 7 lines · John Snow
Quoted 5 lines · Eduardo Habkost
This changes behavior if name=='', and I guess this is OK, but I'm not sure. I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".
Quoted 6 lines
Good spot, I missed that.
I can probably do: bool(name and not
name.startswith('./'))to convert explicitly the empty string to false. I will allow Markus the chance to explain the module stuff.
Sound good to me. If the current behavior needs to be changed, it can be fixed later.
Signature · 1 line
Eduardo
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 10 lines · Eduardo Habkost
Quoted 9 lines · John Snow
Quoted 5 lines · Eduardo Habkost
This changes behavior if name=='', and I guess this is OK, but I'm not sure. I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".Good spot, I missed that.
Nice catch indeed.
Quoted 2 lines
Quoted 2 lines
I can probably do: bool(name and not
name.startswith('./'))
In this case I like my previous suggestion even better. :)
- Cleber.
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 12 lines · Cleber Rosa
Quoted 10 lines · Eduardo Habkost
Quoted 9 lines · John Snow
Quoted 5 lines · Eduardo Habkost
This changes behavior if name=='', and I guess this is OK, but I'm not sure. I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".Good spot, I missed that.
Quoted 8 lines
Nice catch indeed.
Quoted 2 lines
Quoted 2 lines
I can probably do: bool(name and not
name.startswith('./'))In this case I like my previous suggestion even better. :)
- Cleber.
Never mind me... I noticed that this actually gets called with None.
- Cleber.
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
In reply to Eduardo HabkostQuoted 15 lines · Eduardo Habkost
Quoted 14 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
Quoted 7 lines
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 4 lines
scripts/qapi/gen.py +1
-1@staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')This changes behavior if name=='', and I guess this is OK, but I'm not sure.
@name is either
(1) A module pathname relative to the main module
This is a module defined by the user.
(2) system module name, starting with './'
This is a named system module. We currently have two: './init' in commands.py, and and './emit' in events.py.
(3) None
This is the (nameless) system module for built-in stuff. It predates (2). Using './builtin' would probably be better now.
Note that (1) and (2) are disjoint: relative pathnames do not begin with './'.
name='' is not possible, because '' is not a valid pathname.
Quoted 3 lines
I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".
Valid complaints! The code is subtle in places, without helping its readers along with comments or doc strings.
Quoted 7 lines
Quoted 7 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):Signature · 1 line
2.26.2
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 38 lines · Markus Armbruster
Quoted 15 lines · Eduardo Habkost
Quoted 14 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):Quoted 7 lines
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 4 lines
scripts/qapi/gen.py +1
-1@staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')This changes behavior if name=='', and I guess this is OK, but I'm not sure.
@name is either
(1) A module pathname relative to the main module
This is a module defined by the user.
(2) system module name, starting with './'
This is a named system module. We currently have two: './init' in
commands.py, and and './emit' inevents.py.(3) None
Quoted 7 lines
This is the (nameless) system module for built-in stuff. It predates (2). Using './builtin' would probably be better now.
Note that (1) and (2) are disjoint: relative pathnames do not begin with './'.
name='' is not possible, because '' is not a valid pathname.
Thanks! So, the './' prefix is just internal state and never visible to the outside, correct? I would use a separate bool instead of trying to encode additional state inside the string.
Quoted 15 lines
Quoted 4 lines
I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".Valid complaints! The code is subtle in places, without helping its readers along with comments or doc strings.
Quoted 7 lines
Quoted 7 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 42 lines · Eduardo Habkost
Quoted 41 lines · Markus Armbruster
Quoted 15 lines · Eduardo Habkost
Quoted 14 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):Quoted 7 lines
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 4 lines
scripts/qapi/gen.py +1
-1@staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')This changes behavior if name=='', and I guess this is OK, but I'm not sure.
@name is either
(1) A module pathname relative to the main module
This is a module defined by the user.
(2) system module name, starting with './'
This is a named system module. We currently have two: './init' in
commands.py, and and './emit' inevents.py.(3) None
This is the (nameless) system module for built-in stuff. It predates (2). Using './builtin' would probably be better now.
Quoted 8 lines
Quoted 4 lines
Note that (1) and (2) are disjoint: relative pathnames do not begin with './'.
name='' is not possible, because '' is not a valid pathname.
Thanks! So, the './' prefix is just internal state and never visible to the outside, correct? I would use a separate bool instead of trying to encode additional state inside the string.
I've found only one place where the './' prefix might be leaking, and I don't know if it's intentional or not:
Is the name argument to visit_include() supposed to be always (1), or are './' pathnames allowed too?
Signature · 1 line
Eduardo
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 43 lines · Eduardo Habkost
Quoted 42 lines · Eduardo Habkost
Quoted 41 lines · Markus Armbruster
Quoted 15 lines · Eduardo Habkost
Quoted 14 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):Quoted 7 lines
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 4 lines
scripts/qapi/gen.py +1
-1@staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')This changes behavior if name=='', and I guess this is OK, but I'm not sure.
@name is either
(1) A module pathname relative to the main module
This is a module defined by the user.
(2) system module name, starting with './'
This is a named system module. We currently have two: './init' in
commands.py, and and './emit' inevents.py.(3) None
This is the (nameless) system module for built-in stuff. It predates (2). Using './builtin' would probably be better now.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Note that (1) and (2) are disjoint: relative pathnames do not begin with './'.
name='' is not possible, because '' is not a valid pathname.
Thanks! So, the './' prefix is just internal state and never visible to the outside, correct?
Yes.
Quoted 8 lines
Quoted 2 lines
I would use a separate bool
instead of trying to encode additional state inside the string.I've found only one place where the './' prefix might be leaking, and I don't know if it's intentional or not:
Is the name argument to
visit_include()supposed to be always (1), or are './' pathnames allowed too?
Always (1).
visit_include() gets passed a module name:
class QAPISchemaInclude(QAPISchemaEntity):
[...]
def visit(self, visitor):
super().visit(visitor) visitor.visit_include(self._sub_module.name, self.info)
Module names are relative to the main module's directory:
def _module_name(self, fname):
if fname is None:
return None
return os.path.relpath(fname, self._schema_dir)
os,path.relpath() normalizes away './':
$ python
Python 3.8.5 (default, Aug 12 2020, 00:00:00)
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.Quoted 1 line
Quoted 1 line
Quoted 1 line
os.path.relpath('./sub.json', '')
'sub.json'
QAPISchema._make_module() uses ._module_name() as it should:
def _make_module(self, fname):
name = self._module_name(fname)
if name not in self._module_dict:
self._module_dict[name] = QAPISchemaModule(name)
return self._module_dict[name]
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
In reply to Markus ArmbrusterQuoted 33 lines · Markus Armbruster
Quoted 15 lines · Eduardo Habkost
Quoted 14 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):Quoted 7 lines
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 4 lines
scripts/qapi/gen.py +1
-1@staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')This changes behavior if name=='', and I guess this is OK, but I'm not sure.
@name is either
(1) A module pathname relative to the main module
This is a module defined by the user.
(2) system module name, starting with './'
Quoted 8 lines
This is a named system module. We currently have two: './init' in
commands.py, and and './emit' inevents.py.(3) None
This is the (nameless) system module for built-in stuff. It predates (2). Using './builtin' would probably be better now.
Yes please! This would help simplify Optional[str] to str in many places, and removes doubt as to what "None" might imply.
Let's queue that idea up as a cleanup for after this typing series.
Quoted 19 lines
Note that (1) and (2) are disjoint: relative pathnames do not begin with './'.
name='' is not possible, because '' is not a valid pathname.
Quoted 3 lines
I miss documentation on
visit_module(),visit_include(), and_is_user_module(). I don't know whatnamemeans here, and what is a "user module".Valid complaints! The code is subtle in places, without helping its readers along with comments or doc strings.
Quoted 7 lines
Quoted 7 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):Signature · 1 line
2.26.2
For now, I've done the simpler thing and wrapped the return in bool(...), but we will be able to do much more cleanups if we eliminate the possibility of "None" module names later. I'll get it all at once.
I'm adding it to my python TODO: https://gitlab.com/jsnow/qemu/-/issues/8
--js
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
In reply to John SnowQuoted 13 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')
Another possibility here that handles the empty string situation and will never return anything but a bool:
return all([name, not name.startswith('./')])
Either way,
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 23 lines · Cleber Rosa
Quoted 13 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')Another possibility here that handles the empty string situation and will never return anything but a bool:
Quoted 2 lines
return
all([name, not name.startswith('./')])
Never mind me... I noticed that this actually gets called with None.
- Cleber.
Re: [PATCH v2 24/38] qapi/gen.py: Fix edge-case of _is_user_module
In reply to Cleber RosaQuoted 23 lines · Cleber Rosa
Quoted 13 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 254
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -251,7 +251,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return name is not None and not name.startswith('./')Another possibility here that handles the empty string situation and will never return anything but a bool:
Quoted 6 lines
return all([name, not name.startswith('./')])Either way,
Reviewed-by Cleber Rosa <[email protected]>
I wound up changing this per ehabkost's review.
--js
PATCH v2 25/38 qapi/gen.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/gen.py | 102 +++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 49 deletions(-)
scripts/qapi/gen.py +53 -49
@@ -17,7 +17,7 @@ import errno import os import re -from typing import Optional +from typing import Dict, Generator, List, Optional, Tuple from .common import ( c_fname,
@@ -32,31 +32,31 @@ QAPISchemaObjectType, QAPISchemaVisitor, ) +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.
@@ -81,7 +81,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif
@@ -121,40 +121,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
@@ -170,7 +168,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */
@@ -180,16 +178,16 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], + *args: QAPIGenCCode) -> Generator[None, None, None]: """A 'with' statement context manager to wrap with start_if()/end_if() *args: any number of QAPIGenCCode
@@ -215,15 +213,17 @@ def ifcontext(ifcond, *args): class QAPIGenDoc(QAPIGen): - - def _top(self): + def _top(self) -> str: return (super()._top() + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n') class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + blurb: str, + pydoc: str): self._prefix = prefix self._what = what self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -231,38 +231,42 @@ def __init__(self, prefix, what, blurb, pydoc): self._genh = QAPIGenH(self._prefix + self._what + '.h', blurb, pydoc) - def write(self, output_dir): + def write(self, output_dir: str) -> None: self._genc.write(output_dir) self._genh.write(output_dir) class QAPISchemaModularCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + user_blurb: str, + builtin_blurb: Optional[str], + pydoc: str): self._prefix = prefix self._what = what self._user_blurb = user_blurb self._builtin_blurb = builtin_blurb self._pydoc = pydoc - self._genc = None - self._genh = None - self._module = {} - self._main_module = None + self._genc: Optional[QAPIGenC] = None + self._genh: Optional[QAPIGenH] = None + self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {} + self._main_module: Optional[str] = None @staticmethod - def _is_user_module(name): + def _is_user_module(name: Optional[str]) -> bool: return name is not None and not name.startswith('./') @staticmethod - def _is_builtin_module(name): + def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what, name): + def _module_dirname(self, what: str, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return '' - def _module_basename(self, what, name): + def _module_basename(self, what: str, name: Optional[str]) -> str: ret = '' if self._is_builtin_module(name) else self._prefix if self._is_user_module(name): basename = os.path.basename(name)
@@ -274,27 +278,27 @@ def _module_basename(self, what, name): ret += re.sub(r'-', '-' + name + '-', what) return ret - def _module_filename(self, what, name): + def _module_filename(self, what: str, name: Optional[str]) -> str: return os.path.join(self._module_dirname(what, name), self._module_basename(what, name)) - def _add_module(self, name, blurb): + def _add_module(self, name: Optional[str], blurb: str) -> None: basename = self._module_filename(self._what, name) genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) self._genc, self._genh = self._module[name] - def _add_user_module(self, name, blurb): + def _add_user_module(self, name: str, blurb: str) -> None: assert self._is_user_module(name) if self._main_module is None: self._main_module = name self._add_module(name, blurb) - def _add_system_module(self, name, blurb): + def _add_system_module(self, name: Optional[str], blurb: str) -> None: self._add_module(name and './' + name, blurb) - def write(self, output_dir, opt_builtins=False): + def write(self, output_dir: str, opt_builtins: bool = False) -> None: for name in self._module: if self._is_builtin_module(name) and not opt_builtins: continue
@@ -302,13 +306,13 @@ def write(self, output_dir, opt_builtins=False): genc.write(output_dir) genh.write(output_dir) - def _begin_system_module(self, name): + def _begin_system_module(self, name: None) -> None: pass - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: pass - def visit_module(self, name): + def visit_module(self, name: Optional[str]) -> None: if name is None: if self._builtin_blurb: self._add_system_module(None, self._builtin_blurb)
@@ -322,7 +326,7 @@ def visit_module(self, name): self._add_user_module(name, self._user_blurb) self._begin_user_module(name) - def visit_include(self, name, info): + def visit_include(self, name: str, info: QAPISourceInfo) -> None: relname = os.path.relpath(self._module_filename(self._what, name), os.path.dirname(self._genh.fname)) self._genh.preamble_add(mcgen('''
Signature · 1 line
2.26.2
Re: [PATCH v2 25/38] qapi/gen.py: add type hint annotations
Quoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 25/38] qapi/gen.py: add type hint annotations
In reply to John SnowQuoted 136 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 102 +++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 49 deletions(-)scripts/qapi/gen.py +20
-23@@ -17,7 +17,7 @@ import errno import os import re -from typing import Optional +from typing import Dict, Generator, List, Optional, Tuple from .common import ( c_fname,@@ -32,31 +32,31 @@ QAPISchemaObjectType, QAPISchemaVisitor, ) +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.@@ -81,7 +81,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif@@ -121,40 +121,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */@@ -170,7 +168,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */@@ -180,16 +178,16 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname)
scripts/qapi/gen.py def _bottom(self): · line 190
Quoted 10 lines
scripts/qapi/gen.py +3
-2- def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], + *args: QAPIGenCCode) -> Generator[None, None, None]:
IIUC, this could simply be "Iterator[None]" instead of "Generator[None, None, None]".
Anyway,
Re: [PATCH v2 25/38] qapi/gen.py: add type hint annotations
Quoted 151 lines · Cleber Rosa
Quoted 136 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 102 +++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 49 deletions(-)scripts/qapi/gen.py +20
-23@@ -17,7 +17,7 @@ import errno import os import re -from typing import Optional +from typing import Dict, Generator, List, Optional, Tuple from .common import ( c_fname,@@ -32,31 +32,31 @@ QAPISchemaObjectType, QAPISchemaVisitor, ) +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.@@ -81,7 +81,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif@@ -121,40 +121,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */@@ -170,7 +168,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */@@ -180,16 +178,16 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname)scripts/qapi/gen.py def _bottom(self): · line 190
Quoted 10 lines
scripts/qapi/gen.py +3
-2- def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], + *args: QAPIGenCCode) -> Generator[None, None, None]:IIUC, this could simply be "Iterator[None]" instead of "Generator[None, None, None]".
Quoted 4 lines
Anyway,
Reviewed-by Cleber Rosa <[email protected]>
Oh, you're right! Let's do that instead.
Reference: https://mypy.readthedocs.io/en/stable/kinds_of_types.html#generators
Eduardo, I am making this change and keeping your R-B.
--js
Re: [PATCH v2 25/38] qapi/gen.py: add type hint annotations
Quoted 128 lines · John Snow
Quoted 122 lines · Cleber Rosa
Quoted 107 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 102 +++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 49 deletions(-)scripts/qapi/gen.py +19
-22@@ -17,7 +17,7 @@ import errno import os import re -from typing import Optional +from typing import Dict, Generator, List, Optional, Tuple from .common import ( c_fname,@@ -32,31 +32,31 @@ QAPISchemaObjectType, QAPISchemaVisitor, ) +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.@@ -81,7 +81,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif@@ -121,40 +121,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */@@ -170,7 +168,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */@@ -180,16 +178,16 @@ def _bottom(self): class QAPIGenH(QAPIGenC): -scripts/qapi/gen.py def _bottom(self): · line 185
Quoted 10 lines
scripts/qapi/gen.py +4
-3- def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], + *args: QAPIGenCCode) -> Generator[None, None, None]:IIUC, this could simply be "Iterator[None]" instead of "Generator[None, None, None]".
Quoted 4 lines
Anyway,
Reviewed-by Cleber Rosa <[email protected]>
Quoted 6 lines
Oh, you're right! Let's do that instead.
Reference:
https://mypy.readthedocs.io/en/stable/kinds_of_types.html#generatorsEduardo, I am making this change and keeping your R-B.
Agreed.
Signature · 1 line
Eduardo
PATCH v2 26/38 qapi/gen.py: Enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.gen] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.introspect] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
Re: [PATCH v2 26/38] qapi/gen.py: Enable checking with mypy
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 26/38] qapi/gen.py: Enable checking with mypy
In reply to John SnowQuoted 24 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)scripts/qapi/mypy.ini +0
-5@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.gen] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.introspect] disallow_untyped_defs = False disallow_incomplete_defs = FalseSignature · 1 line
2.26.2
Like I mentioned before, I'd squash this together with the actual changes that enable this increased coverage.
Anyway,
PATCH v2 27/38 qapi/gen.py: Remove unused parameter
In reply to John Snowmodule_basename doesn't use the 'what' argument, so remove it.
--- scripts/qapi/gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
@@ -261,7 +261,7 @@ def _is_user_module(name: Optional[str]) -> bool: def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what: str, name: Optional[str]) -> str: + def _module_dirname(self, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return ''
@@ -279,7 +279,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str: return ret def _module_filename(self, what: str, name: Optional[str]) -> str: - return os.path.join(self._module_dirname(what, name), + return os.path.join(self._module_dirname(name), self._module_basename(what, name)) def _add_module(self, name: Optional[str], blurb: str) -> None:
Signature · 1 line
2.26.2
Re: [PATCH v2 27/38] qapi/gen.py: Remove unused parameter
Quoted 3 lines · John Snow
module_basenamedoesn't use the 'what' argument, so remove it.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 27/38] qapi/gen.py: Remove unused parameter
In reply to John SnowQuoted 3 lines · John Snow
module_basenamedoesn't use the 'what' argument, so remove it.Signed-off-by John Snow <[email protected]>
PATCH v2 28/38 qapi/gen.py: update write() to be more idiomatic
In reply to John SnowMake the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.
--- scripts/qapi/gen.py | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +10 -14
@@ -14,7 +14,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import Dict, Generator, List, Optional, Tuple
@@ -64,21 +63,18 @@ def write(self, output_dir: str) -> None: return pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try: - os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True) + fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close() + with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text) def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Signature · 1 line
2.26.2
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 8 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Signed-off-by John Snow <[email protected]>
I really miss a comment below explaining why we use open(os.open(pathname, ...), ...) instead of open(pathname, ...).
Quoted 51 lines
--- scripts/qapi/gen.py | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)scripts/qapi/gen.py +10
-14@@ -14,7 +14,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import Dict, Generator, List, Optional, Tuple@@ -64,21 +63,18 @@ def write(self, output_dir: str) -> None: return pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try: - os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True) + fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close() + with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text) def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.
Quoted 7 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).
Not known to me. It was introduced in 907b846653 as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?
Markus?
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 17 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Quoted 7 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).
Quoted 5 lines
Not known to me. It was introduced in
907b846653as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?Markus?
AFACIT the change on 907b846653 is effective because of the "is new text different from old text?" conditional. I can not see how the separate/duplicate open/fdopen would contribute to that.
But, let's hear from Markus.
- Cleber.
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).
This code:
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
f = open(fd, 'r+', encoding='utf-8')Quoted 11 lines
Quoted 5 lines
Not known to me. It was introduced in
907b846653as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?Markus?
AFACIT the change on
907b846653is effective because of the "is new text different from old text?" conditional. I can not see how the separate/duplicate open/fdopen would contribute to that.But, let's hear from Markus.
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')
fails instead of creates, and
f = open(pathname, "w+", encoding='utf-8')
truncates.
If you know a better way, tell me!
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 43 lines · Markus Armbruster
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).This code:
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) f = open(fd, 'r+', encoding='utf-8')Quoted 11 lines
Quoted 5 lines
Not known to me. It was introduced in
907b846653as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?Markus?
AFACIT the change on
907b846653is effective because of the "is new text different from old text?" conditional. I can not see how the separate/duplicate open/fdopen would contribute to that.But, let's hear from Markus.
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')
Quoted 7 lines
fails instead of creates, and
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
IIUC, you need "a+" as the mode, rather than "w+"
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 6 lines · Daniel P. Berrangé
Quoted 5 lines
This code:
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) f = open(fd, 'r+', encoding='utf-8')
Quoted 8 lines
Quoted 8 lines
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')fails instead of creates, and
Checking what POSIX says for fopen():
https://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html
Yep, "r+" does not use O_CREAT.
Quoted 4 lines
Quoted 4 lines
f = open(pathname, "w+", encoding='utf-8')truncates.
Yep, "w+" uses O_TRUNC.
Quoted 4 lines
Quoted 2 lines
If you know a better way, tell me!
IIUC, you need "a+" as the mode, rather than "w+"
That uses O_APPEND. Which is fine if you are only appending and not touching existing contents, but not what you want if you are doing random access.
Yeah, the fopen() interface is rather puny, in that it does not express as many modes as open() supports.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
In reply to Daniel P. BerrangéQuoted 46 lines · Daniel P. Berrangé
Quoted 45 lines · Markus Armbruster
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).This code:
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) f = open(fd, 'r+', encoding='utf-8')Quoted 11 lines
Quoted 5 lines
Not known to me. It was introduced in
907b846653as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?Markus?
AFACIT the change on
907b846653is effective because of the "is new text different from old text?" conditional. I can not see how the separate/duplicate open/fdopen would contribute to that.But, let's hear from Markus.
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')fails instead of creates, and
Quoted 7 lines
Quoted 5 lines
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
IIUC, you need "a+" as the mode, rather than "w+"
Sure this lets me do
f.seek(0)
f.truncate(0)
f.write(text)to overwrite the old contents on all systems?
Documentation cautions:
[...] 'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position).
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 17 lines · Markus Armbruster
Quoted 9 lines
Quoted 9 lines
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')fails instead of creates, and
Quoted 7 lines
Quoted 5 lines
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
IIUC, you need "a+" as the mode, rather than "w+"
Quoted 7 lines
Sure this lets me do
f.seek(0) f.truncate(0) f.write(text)to overwrite the old contents on all systems?
As long as you do a single pass over the output (you issue a stream of f.write() after the truncate, but never a seek), then this will work.
Quoted 6 lines
Documentation cautions:
[...] 'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position).
Yes, that means that random access is impossible on such a stream. But not all file creation patterns require random access.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 18 lines · Eric Blake
Quoted 16 lines · Markus Armbruster
Quoted 16 lines
Quoted 7 lines
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')Quoted 7 lines
fails instead of creates, and
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
IIUC, you need "a+" as the mode, rather than "w+"
Quoted 8 lines
Quoted 5 lines
Sure this lets me do
f.seek(0) f.truncate(0) f.write(text)to overwrite the old contents on all systems?
As long as you do a single pass over the output (you issue a stream of
f.write()after the truncate, but never a seek), then this will work.
Well, I do seek(), right before the truncate.
Quoted 8 lines
Quoted 5 lines
Documentation cautions:
[...] 'a' for appending (which on some Unix systems, means that
all
writes append to the end of the file regardless of the current seek position).Yes, that means that random access is impossible on such a stream. But not all file creation patterns require random access.
To be honest, I still prefer the code I wrote, because there the reader only wonders why I didn't just open(), while here we get to argue about subtleties of mode "a+".
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 37 lines · Markus Armbruster
Quoted 18 lines · Eric Blake
Quoted 16 lines · Markus Armbruster
Quoted 16 lines
Quoted 7 lines
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')Quoted 7 lines
fails instead of creates, and
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
IIUC, you need "a+" as the mode, rather than "w+"
Quoted 8 lines
Quoted 5 lines
Sure this lets me do
f.seek(0) f.truncate(0) f.write(text)to overwrite the old contents on all systems?
As long as you do a single pass over the output (you issue a stream of
f.write()after the truncate, but never a seek), then this will work.Well, I do
seek(), right before the truncate.Quoted 6 lines
Quoted 5 lines
Documentation cautions:
[...] 'a' for appending (which on some Unix systems, means that
all
writes append to the end of the file regardless of the current seek position).
Quoted 7 lines
Quoted 2 lines
Yes, that means that random access is impossible on such a stream. But not all file creation patterns require random access.
To be honest, I still prefer the code I wrote, because there the reader only wonders why I didn't just
open(), while here we get to argue about subtleties of mode "a+".
I kept your os.open, I agree with you here.
(I still rewrote to use the context managers, though.)
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
In reply to Markus ArmbrusterQuoted 43 lines · Markus Armbruster
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).This code:
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) f = open(fd, 'r+', encoding='utf-8')Quoted 11 lines
Quoted 5 lines
Not known to me. It was introduced in
907b846653as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?Markus?
AFACIT the change on
907b846653is effective because of the "is new text different from old text?" conditional. I can not see how the separate/duplicate open/fdopen would contribute to that.But, let's hear from Markus.
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')
Quoted 7 lines
fails instead of creates, and
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
Thanks for the explanation!
Yeah, it looks like there's no combination of open() flags that would get translated to O_RDWR|O_CREAT.
Using os.open() like you did seems more straightforward than catching FileNotFoundError.
Signature · 1 line
Eduardo
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
Quoted 52 lines · Eduardo Habkost
Quoted 43 lines · Markus Armbruster
Quoted 10 lines · Cleber Rosa
Quoted 9 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 7 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>I really miss a comment below explaining why we use
open(os.open(pathname, ...), ...)instead ofopen(pathname, ...).This code:
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) f = open(fd, 'r+', encoding='utf-8')Quoted 11 lines
Quoted 5 lines
Not known to me. It was introduced in
907b846653as part of an effort to reduce rebuild times. Maybe this avoids a modification time change if the file already exists?Markus?
AFACIT the change on
907b846653is effective because of the "is new text different from old text?" conditional. I can not see how the separate/duplicate open/fdopen would contribute to that.But, let's hear from Markus.
This was my best attempt to open the file read/write, creating it if it doesn't exist.
Plain
f = open(pathname, "r+", encoding='utf-8')Quoted 7 lines
fails instead of creates, and
f = open(pathname, "w+", encoding='utf-8')truncates.
If you know a better way, tell me!
Quoted 8 lines
Thanks for the explanation!
Yeah, it looks like there's no combination of
open()flags that would get translated to O_RDWR|O_CREAT.Using
os.open()like you did seems more straightforward than catching FileNotFoundError.
OK. Using fdopen works for us, so let's stick with it. I will add a comment explaining our reasoning.
Thanks!
--js
Re: [PATCH v2 28/38] qapi/gen.py: update write() to be more idiomatic
In reply to John SnowQuoted 8 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Signed-off-by John Snow <[email protected]>
PATCH v2 29/38 qapi/gen.py: delint with pylint
In reply to John Snow'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_top and _bottom also need to stay class methods because some users override the method and need to use self. Tell pylint to shush.
--- scripts/qapi/gen.py | 2 ++ scripts/qapi/pylintrc | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -0
@@ -50,9 +50,11 @@ def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() def _top(self) -> str: + # pylint: disable=no-self-use return '' def _bottom(self) -> str: + # pylint: disable=no-self-use return '' def write(self, output_dir: str) -> None:
scripts/qapi/pylintrc +3 -2
@@ -5,7 +5,6 @@ ignore-patterns=doc.py, error.py, expr.py, - gen.py, parser.py, schema.py, types.py,
@@ -46,7 +45,9 @@ good-names=i, k, ex, Run, - _ + _, + fp, # fp = open(...) + fd, # fd = os.open(...) [VARIABLES]
Signature · 1 line
2.26.2
Re: [PATCH v2 29/38] qapi/gen.py: delint with pylint
Quoted 5 lines · John Snow
'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_topand_bottomalso need to stay class methods because some users override the method and need to useself. Tell pylint to shush.
Do you mean "stay instance methods"?
Quoted 49 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 ++ scripts/qapi/pylintrc | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-)scripts/qapi/gen.py +2
-0@@ -50,9 +50,11 @@ def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() def _top(self) -> str: + # pylint: disable=no-self-use return '' def _bottom(self) -> str: + # pylint: disable=no-self-use return '' def write(self, output_dir: str) -> None:scripts/qapi/pylintrc +3
-2@@ -5,7 +5,6 @@ ignore-patterns=doc.py, error.py, expr.py, - gen.py, parser.py, schema.py, types.py,@@ -46,7 +45,9 @@ good-names=i, k, ex, Run, - _ + _, + fp, # fp = open(...) + fd, # fd = os.open(...) [VARIABLES]Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 29/38] qapi/gen.py: delint with pylint
Quoted 9 lines · Eduardo Habkost
Quoted 5 lines · John Snow
'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_topand_bottomalso need to stay class methods because some users override the method and need to useself. Tell pylint to shush.Do you mean "stay instance methods"?
Yep. Fixed, thank you.
Quoted 2 lines
Reviewed-by Eduardo Habkost <[email protected]>
Thanks!
Quoted 50 lines
Quoted 49 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 2 ++ scripts/qapi/pylintrc | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-)scripts/qapi/gen.py +2
-0@@ -50,9 +50,11 @@ def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() def _top(self) -> str: + # pylint: disable=no-self-use return '' def _bottom(self) -> str: + # pylint: disable=no-self-use return '' def write(self, output_dir: str) -> None:scripts/qapi/pylintrc +3
-2@@ -5,7 +5,6 @@ ignore-patterns=doc.py, error.py, expr.py, - gen.py, parser.py, schema.py, types.py,@@ -46,7 +45,9 @@ good-names=i, k, ex, Run, - _ + _, + fp, # fp = open(...) + fd, # fd = os.open(...) [VARIABLES]Signature · 1 line
2.26.2
Re: [PATCH v2 29/38] qapi/gen.py: delint with pylint
In reply to John SnowQuoted 7 lines · John Snow
'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_topand_bottomalso need to stay class methods because some users override the method and need to useself. Tell pylint to shush.Signed-off-by John Snow <[email protected]>
With the commit message change caught by Eduardo:
PATCH v2 30/38 qapi/introspect.py: Add a typed 'extra' structure
In reply to John SnowTyping arbitrarily shaped dicts with mypy is difficult prior to Python 3.8; using explicit structures is nicer.
Since we always define an Extra type now, the return type of _make_tree simplifies and always returns the tuple.
--- scripts/qapi/introspect.py | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)
scripts/qapi/introspect.py +19 -12
@@ -10,6 +10,8 @@ See the COPYING file in the top-level directory. """ +from typing import (NamedTuple, Optional, Sequence) + from .common import ( c_name, gen_endif,
@@ -21,16 +23,21 @@ QAPISchemaType) -def _make_tree(obj, ifcond, features, extra=None): - if extra is None: - extra = {} - if ifcond: - extra['if'] = ifcond +class Extra(NamedTuple): + """ + Extra contains data that isn't intended for output by introspection. + """ + comment: Optional[str] = None + ifcond: Sequence[str] = tuple() + + +def _make_tree(obj, ifcond, features, + extra: Optional[Extra] = None): + comment = extra.comment if extra else None + extra = Extra(comment, ifcond) if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] - if extra: - return (obj, extra) - return obj + obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features] + return (obj, extra) def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -40,8 +47,8 @@ def indent(level): if isinstance(obj, tuple): ifobj, extra = obj - ifcond = extra.get('if') - comment = extra.get('comment') + ifcond = extra.ifcond + comment = extra.comment ret = '' if comment: ret += indent(level) + '/* %s */\n' % comment
@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): if not self._unmask: # Output a comment to make it easy to map masked names # back to the source when reading the generated output. - extra = {'comment': '"%s" = %s' % (self._name(name), name)} + extra = Extra(comment=f'"{self._name(name)}" = {name}') name = self._name(name) obj['name'] = name obj['meta-type'] = mtype
Signature · 1 line
2.26.2
Re: [PATCH v2 30/38] qapi/introspect.py: Add a typed 'extra' structure
Quoted 11 lines · John Snow
Typing arbitrarily shaped dicts with mypy is difficult prior to Python 3.8; using explicit structures is nicer.
Since we always define an Extra type now, the return type of
_make_treesimplifies and always returns the tuple.Signed-off-by John Snow <[email protected]>--- scripts/qapi/introspect.py | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)
Here I'm confused by both the original code and the new code.
I will try to review as a refactoring of existing code, but I'll have suggestions for follow ups:
Quoted 24 lines
scripts/qapi/introspect.py +4
-5@@ -10,6 +10,8 @@ See the COPYING file in the top-level directory. """ +from typing import (NamedTuple, Optional, Sequence) + from .common import ( c_name, gen_endif,@@ -21,16 +23,21 @@ QAPISchemaType) -def _make_tree(obj, ifcond, features, extra=None): - if extra is None: - extra = {} - if ifcond: - extra['if'] = ifcond +class Extra(NamedTuple): + """
scripts/qapi/introspect.py line 37
Quoted 10 lines
scripts/qapi/introspect.py +10
-0+ Extra contains data that isn't intended for output by introspection. + """ + comment: Optional[str] = None + ifcond: Sequence[str] = tuple() + + +def _make_tree(obj, ifcond, features, + extra: Optional[Extra] = None): + comment = extra.comment if extra else None + extra = Extra(comment, ifcond)
Here we have one big difference: now extra is being recreated, and all fields except extra.comment are being ignored. On the original version, all fields in extra were being kept. This makes the existence of the extra argument pointless.
If you are going through the trouble of changing the type of the 4rd argument to _make_tree(), this seems more obvious:
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index 41ca8afc672..c62af94c9ad 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -32,8 +32,7 @@ class Extra(NamedTuple):
def _make_tree(obj, ifcond, features,
- extra: Optional[Extra] = None):
- comment = extra.comment if extra else None
+ comment: Optional[str] = None):
extra = Extra(comment, ifcond)
if features:
obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
@@ -170,16 +169,16 @@ const QLitObject %(c_name)s = %(c_string)s;
return self._name(typ.name)
def _gen_tree(self, name, mtype, obj, ifcond, features):
- extra = None
+ comment = None
if mtype not in ('command', 'event', 'builtin', 'array'):
if not self._unmask:
# Output a comment to make it easy to map masked names # back to the source when reading the generated output.
- extra = Extra(comment=f'"{self._name(name)}" = {name}')
+ comment = f'"{self._name(name)}" = {name}'
name = self._name(name)
obj['name'] = name
obj['meta-type'] = mtype
- self._trees.append(_make_tree(obj, ifcond, features, extra))
+ self._trees.append(_make_tree(obj, ifcond, features, comment))
def _gen_member(self, member):
obj = {'name': member.name, 'type': self._use_type(member.type)}
I understand you're trying to just make minimal refactoring, and I don't think this should block your cleanup series. So:
Quoted 33 lines
scripts/qapi/introspect.py +5
-7if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] - if extra: - return (obj, extra) - return obj + obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features] + return (obj, extra) def _tree_to_qlit(obj, level=0, suppress_first_indent=False):@@ -40,8 +47,8 @@ def indent(level): if isinstance(obj, tuple): ifobj, extra = obj - ifcond = extra.get('if') - comment = extra.get('comment') + ifcond = extra.ifcond + comment = extra.comment ret = '' if comment: ret += indent(level) + '/* %s */\n' % comment@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): if not self._unmask: # Output a comment to make it easy to map masked names # back to the source when reading the generated output. - extra = {'comment': '"%s" = %s' % (self._name(name), name)} + extra = Extra(comment=f'"{self._name(name)}" = {name}') name = self._name(name) obj['name'] = name obj['meta-type'] = mtypeSignature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 30/38] qapi/introspect.py: Add a typed 'extra' structure
Quoted 49 lines · Eduardo Habkost
Quoted 11 lines · John Snow
Typing arbitrarily shaped dicts with mypy is difficult prior to Python 3.8; using explicit structures is nicer.
Since we always define an Extra type now, the return type of
_make_treesimplifies and always returns the tuple.Signed-off-by John Snow <[email protected]>--- scripts/qapi/introspect.py | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)Here I'm confused by both the original code and the new code.
I will try to review as a refactoring of existing code, but I'll have suggestions for follow ups:
Quoted 31 lines
scripts/qapi/introspect.py +11
-5@@ -10,6 +10,8 @@ See the COPYING file in the top-level directory. """ +from typing import (NamedTuple, Optional, Sequence) + from .common import ( c_name, gen_endif,@@ -21,16 +23,21 @@ QAPISchemaType) -def _make_tree(obj, ifcond, features, extra=None): - if extra is None: - extra = {} - if ifcond: - extra['if'] = ifcond +class Extra(NamedTuple): + """ + Extra contains data that isn't intended for output by introspection. + """ + comment: Optional[str] = None + ifcond: Sequence[str] = tuple() + + +def _make_tree(obj, ifcond, features,
Quoted 9 lines
scripts/qapi/introspect.py line 37
Quoted 3 lines
scripts/qapi/introspect.py +3
-0+ extra: Optional[Extra] = None): + comment = extra.comment if extra else None + extra = Extra(comment, ifcond)Here we have one big difference: now
extrais being recreated, and all fields exceptextra.commentare being ignored. On the original version, all fields inextrawere being kept. This makes the existence of theextraargument pointless.
Yup, oops.
Quoted 3 lines
If you are going through the trouble of changing the type of the 4rd argument to
_make_tree(), this seems more obvious:
Yes, agree. I came up with something similar after talking to you this morning.
Quoted 32 lines
diff --git
a/scripts/qapi/introspect.pyb/scripts/qapi/introspect.pyindex41ca8afc672..c62af94c9ad100644
---a/scripts/qapi/introspect.py
+++b/scripts/qapi/introspect.py
@@ -32,8 +32,7 @@ class Extra(NamedTuple):def _make_tree(obj, ifcond, features,
- extra: Optional[Extra] = None):
- comment = extra.comment if extra else None
+ comment: Optional[str] = None):
extra =Extra(comment, ifcond)
if features:
obj['features'] = [(f.name,Extra(None, f.ifcond)) for f in features]
@@ -170,16 +169,16 @@ const QLitObject %(c_name)s = %(c_string)s;
returnself._name(typ.name)def
_gen_tree(self, name, mtype, obj, ifcond, features):
- extra = None
+ comment = None
if mtype not in ('command', 'event', 'builtin', 'array'):
if not self._unmask:
# Output a comment to make it easy to map masked names # back to the source when reading the generated output.
- extra =Extra(comment=f'"{self._name(name)}" = {name}')
+ comment = f'"{self._name(name)}" = {name}'
name =self._name(name)
obj['name'] = name
obj['meta-type'] = mtype
-self._trees.append(_make_tree(obj, ifcond, features, extra))
+self._trees.append(_make_tree(obj, ifcond, features, comment))
Quoted 8 lines
def
_gen_member(self, member):
obj = {'name': member.name, 'type':self._use_type(member.type)}I understand you're trying to just make minimal refactoring, and I don't think this should block your cleanup series. So:
Reviewed-by Eduardo Habkost <[email protected]>
I appreciate the benefit-of-the-doubt, but I think this change is worth making while we're here.
Quoted 35 lines
Quoted 34 lines
scripts/qapi/introspect.py +5
-7if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] - if extra: - return (obj, extra) - return obj + obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features] + return (obj, extra) def _tree_to_qlit(obj, level=0, suppress_first_indent=False):@@ -40,8 +47,8 @@ def indent(level): if isinstance(obj, tuple): ifobj, extra = obj - ifcond = extra.get('if') - comment = extra.get('comment') + ifcond = extra.ifcond + comment = extra.comment ret = '' if comment: ret += indent(level) + '/* %s */\n' % comment@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): if not self._unmask: # Output a comment to make it easy to map masked names # back to the source when reading the generated output. - extra = {'comment': '"%s" = %s' % (self._name(name), name)} + extra = Extra(comment=f'"{self._name(name)}" = {name}') name = self._name(name) obj['name'] = name obj['meta-type'] = mtypeSignature · 1 line
2.26.2
PATCH v2 31/38 qapi/introspect.py: add _gen_features helper
In reply to John Snow_make_tree doesn't know if it is receiving an object or some other type; adding features information should arguably be performed by the caller.
This will help us refactor _make_tree more gracefully in the next patch.
--- scripts/qapi/introspect.py | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
scripts/qapi/introspect.py +14 -8
@@ -10,7 +10,7 @@ See the COPYING file in the top-level directory. """ -from typing import (NamedTuple, Optional, Sequence) +from typing import (NamedTuple, List, Optional, Sequence) from .common import ( c_name,
@@ -20,7 +20,7 @@ ) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) + QAPISchemaFeature, QAPISchemaType) class Extra(NamedTuple):
@@ -31,12 +31,10 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -def _make_tree(obj, ifcond, features, +def _make_tree(obj, ifcond, extra: Optional[Extra] = None): comment = extra.comment if extra else None extra = Extra(comment, ifcond) - if features: - obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features] return (obj, extra)
@@ -169,6 +167,10 @@ def _use_type(self, typ): return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) + @classmethod + def _gen_features(cls, features: List[QAPISchemaFeature]): + return [_make_tree(f.name, f.ifcond) for f in features] + def _gen_tree(self, name, mtype, obj, ifcond, features): extra = None if mtype not in ('command', 'event', 'builtin', 'array'):
@@ -179,13 +181,17 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): name = self._name(name) obj['name'] = name obj['meta-type'] = mtype - self._trees.append(_make_tree(obj, ifcond, features, extra)) + if features: + obj['features'] = self._gen_features(features) + self._trees.append(_make_tree(obj, ifcond, extra)) def _gen_member(self, member): obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None - return _make_tree(obj, member.ifcond, member.features) + if member.features: + obj['features'] = self._gen_features(member.features) + return _make_tree(obj, member.ifcond) def _gen_variants(self, tag_name, variants): return {'tag': tag_name,
@@ -193,7 +199,7 @@ def _gen_variants(self, tag_name, variants): def _gen_variant(self, variant): obj = {'case': variant.name, 'type': self._use_type(variant.type)} - return _make_tree(obj, variant.ifcond, None) + return _make_tree(obj, variant.ifcond) def visit_builtin_type(self, name, info, json_type): self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
Signature · 1 line
2.26.2
Re: [PATCH v2 31/38] qapi/introspect.py: add _gen_features helper
Quoted 33 lines · John Snow
_make_treedoesn't know if it is receiving an object or some other type; adding features information should arguably be performed by the caller.This will help us refactor
_make_treemore gracefully in the next patch.Signed-off-by John Snow <[email protected]>--- scripts/qapi/introspect.py | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)scripts/qapi/introspect.py +2
-2@@ -10,7 +10,7 @@ See the COPYING file in the top-level directory. """ -from typing import (NamedTuple, Optional, Sequence) +from typing import (NamedTuple, List, Optional, Sequence) from .common import ( c_name,@@ -20,7 +20,7 @@ ) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) + QAPISchemaFeature, QAPISchemaType) class Extra(NamedTuple):
scripts/qapi/introspect.py class Extra(NamedTuple): · line 37
Quoted 8 lines
scripts/qapi/introspect.py +1
-1@@ -31,12 +31,10 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -def _make_tree(obj, ifcond, features, +def _make_tree(obj, ifcond, extra: Optional[Extra] = None): comment = extra.comment if extra else None extra = Extra(comment, ifcond)
I believe these two lines above should be removed, as suggested in patch 30, but let's ignore that for now.
scripts/qapi/introspect.py class Extra(NamedTuple): · line 39
Quoted 2 lines
scripts/qapi/introspect.py +0
-2- if features: - obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
I can't say I understand completely why moving these two lines outside _make_tree() is useful, but if it makes the cleanup work you did easier, I trust this is the right thing to do. The changes look correct.
Quoted 46 lines
scripts/qapi/introspect.py +11
-3return (obj, extra)@@ -169,6 +167,10 @@ def _use_type(self, typ): return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) + @classmethod + def _gen_features(cls, features: List[QAPISchemaFeature]): + return [_make_tree(f.name, f.ifcond) for f in features] + def _gen_tree(self, name, mtype, obj, ifcond, features): extra = None if mtype not in ('command', 'event', 'builtin', 'array'):@@ -179,13 +181,17 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): name = self._name(name) obj['name'] = name obj['meta-type'] = mtype - self._trees.append(_make_tree(obj, ifcond, features, extra)) + if features: + obj['features'] = self._gen_features(features) + self._trees.append(_make_tree(obj, ifcond, extra)) def _gen_member(self, member): obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None - return _make_tree(obj, member.ifcond, member.features) + if member.features: + obj['features'] = self._gen_features(member.features) + return _make_tree(obj, member.ifcond) def _gen_variants(self, tag_name, variants): return {'tag': tag_name,@@ -193,7 +199,7 @@ def _gen_variants(self, tag_name, variants): def _gen_variant(self, variant): obj = {'case': variant.name, 'type': self._use_type(variant.type)} - return _make_tree(obj, variant.ifcond, None) + return _make_tree(obj, variant.ifcond) def visit_builtin_type(self, name, info, json_type): self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 31/38] qapi/introspect.py: add _gen_features helper
Quoted 3 lines · Eduardo Habkost
I believe these two lines above should be removed, as suggested in patch 30, but let's ignore that for now.
Yup, headed there.
Quoted 6 lines
scripts/qapi/introspect.py class Extra(NamedTuple): · line 39
Quoted 2 lines
scripts/qapi/introspect.py +0
-2- if features: - obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]I can't say I understand completely why moving these two lines outside
_make_tree()is useful, but if it makes the cleanup work you did easier, I trust this is the right thing to do. The changes look correct.
The basic premise is:
Why pass information you want to add to obj['features'] to a function to make that assignment, when you could just perform that assignment yourself?
Otherwise, _make_tree, which accepts any arbitrary object (not just dicts!) has to interrogate its arguments to make sure you gave it a dict when you give it a features argument.
Type-wise, it's cleaner to perform this transformation when we KNOW we have an object than it is to defer to a more abstracted function and assert/downcast back to more specific types.
Re: [PATCH v2 31/38] qapi/introspect.py: add _gen_features helper
Quoted 14 lines · John Snow
Quoted 3 lines · Eduardo Habkost
I believe these two lines above should be removed, as suggested in patch 30, but let's ignore that for now.
Yup, headed there.
Quoted 6 lines
scripts/qapi/introspect.py class Extra(NamedTuple): · line 39
Quoted 2 lines
scripts/qapi/introspect.py +0
-2- if features: - obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]I can't say I understand completely why moving these two lines outside
_make_tree()is useful, but if it makes the cleanup work you did easier, I trust this is the right thing to do. The changes look correct.
Quoted 8 lines
The basic premise is:
Why pass information you want to add to obj['features'] to a function to make that assignment, when you could just perform that assignment yourself?
Otherwise,
_make_tree, which accepts any arbitrary object (not just dicts!) has to interrogate its arguments to make sure you gave it a dict when you give it a features argument.
Oh, I was not aware that obj could be not a dictionary. In this case, it makes lots of sense to move the magic outside the function.
Quoted 4 lines
Type-wise, it's cleaner to perform this transformation when we KNOW we have an object than it is to defer to a more abstracted function and assert/downcast back to more specific types.
Signature · 1 line
Eduardo
PATCH v2 32/38 qapi/introspect.py: create a typed 'Node' data structure
In reply to John SnowReplacing the un-typed tuple, add a typed Node that we can add typed metadata to.
--- scripts/qapi/introspect.py | 53 ++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 25 deletions(-)
scripts/qapi/introspect.py +28 -25
@@ -31,11 +31,18 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -def _make_tree(obj, ifcond, - extra: Optional[Extra] = None): - comment = extra.comment if extra else None - extra = Extra(comment, ifcond) - return (obj, extra) +class Node: + """ + Node generally contains a SchemaInfo-like type (as a dict), + But it also used to wrap comments/ifconds around leaf value types. + """ + # Remove after 3.7 adds @dataclass: + # pylint: disable=too-few-public-methods + def __init__(self, data, ifcond: List[str], + extra: Optional[Extra] = None): + self.data = data + comment = extra.comment if extra else None + self.extra = Extra(comment, ifcond) def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -43,18 +50,15 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False): def indent(level): return level * 4 * ' ' - if isinstance(obj, tuple): - ifobj, extra = obj - ifcond = extra.ifcond - comment = extra.comment + if isinstance(obj, Node): ret = '' - if comment: - ret += indent(level) + '/* %s */\n' % comment - if ifcond: - ret += gen_if(ifcond) - ret += _tree_to_qlit(ifobj, level) - if ifcond: - ret += '\n' + gen_endif(ifcond) + if obj.extra.comment: + ret += indent(level) + '/* %s */\n' % obj.extra.comment + if obj.extra.ifcond: + ret += gen_if(obj.extra.ifcond) + ret += _tree_to_qlit(obj.data, level) + if obj.extra.ifcond: + ret += '\n' + gen_endif(obj.extra.ifcond) return ret ret = ''
@@ -169,7 +173,7 @@ def _use_type(self, typ): @classmethod def _gen_features(cls, features: List[QAPISchemaFeature]): - return [_make_tree(f.name, f.ifcond) for f in features] + return [Node(f.name, f.ifcond) for f in features] def _gen_tree(self, name, mtype, obj, ifcond, features): extra = None
@@ -183,7 +187,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): obj['meta-type'] = mtype if features: obj['features'] = self._gen_features(features) - self._trees.append(_make_tree(obj, ifcond, extra)) + self._trees.append(Node(obj, ifcond, extra)) def _gen_member(self, member): obj = {'name': member.name, 'type': self._use_type(member.type)}
@@ -191,7 +195,7 @@ def _gen_member(self, member): obj['default'] = None if member.features: obj['features'] = self._gen_features(member.features) - return _make_tree(obj, member.ifcond) + return Node(obj, member.ifcond) def _gen_variants(self, tag_name, variants): return {'tag': tag_name,
@@ -199,15 +203,14 @@ def _gen_variants(self, tag_name, variants): def _gen_variant(self, variant): obj = {'case': variant.name, 'type': self._use_type(variant.type)} - return _make_tree(obj, variant.ifcond) + return Node(obj, variant.ifcond) def visit_builtin_type(self, name, info, json_type): self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) def visit_enum_type(self, name, info, ifcond, features, members, prefix): self._gen_tree(name, 'enum', - {'values': [_make_tree(m.name, m.ifcond, None) - for m in members]}, + {'values': [Node(m.name, m.ifcond) for m in members]}, ifcond, features) def visit_array_type(self, name, info, ifcond, element_type):
@@ -227,9 +230,9 @@ def visit_object_type_flat(self, name, info, ifcond, features, def visit_alternate_type(self, name, info, ifcond, features, variants): self._gen_tree(name, 'alternate', {'members': [ - _make_tree({'type': self._use_type(m.type)}, - m.ifcond, None) - for m in variants.variants]}, + Node({'type': self._use_type(m.type)}, m.ifcond) + for m in variants.variants + ]}, ifcond, features) def visit_command(self, name, info, ifcond, features,
Signature · 1 line
2.26.2
Re: [PATCH v2 32/38] qapi/introspect.py: create a typed 'Node' data structure
Quoted 4 lines · John Snow
Replacing the un-typed tuple, add a typed Node that we can add typed metadata to.
Signed-off-by John Snow <[email protected]>
This is the most complex patch so far, and it's very hard to understand what it does without type annotations.
Have you consider adding type annotations to the code before patch 30/38 (even if using object in some parts), so we can make this easier to review?
In case it's useful, below is an attempt to add type annotations to the old code. It can be applied after patch 29/38. It reuses portions of patch 33/38.
--- scripts/qapi/introspect.py | 138 ++++++++++++++++++++++++++----------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 100 insertions(+), 45 deletions(-)
scripts/qapi/introspect.py +99 -39
@@ -10,6 +10,17 @@ This work is licensed under the terms of the GNU GPL, version 2. See the COPYING file in the top-level directory. """ +from typing import ( + Dict, + Generic, + List, + NamedTuple, + Optional, + Sequence, + TypeVar, + Tuple +) + from .common import ( c_name, gen_endif,
@@ -17,25 +28,48 @@ from .common import ( mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) - - -def _make_tree(obj, ifcond, features, extra=None): +from .schema import ( + QAPISchema, + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, +) +from .source import QAPISourceInfo + +T = TypeVar('T') +# this should actually be: Union[str, list, dict, bool, 'AnnotatedNode'] +# but mypy doesn't support recursive types +TreeNode = object +TreeDict = Dict[str, TreeNode] +Extra = Dict[str, object] +AnnotatedNode = Tuple[T, Extra] + +def _make_tree(obj: TreeDict, ifcond: List[str], + features: List[QAPISchemaFeature], + extra: Optional[Extra] = None) -> TreeNode: if extra is None: extra = {} if ifcond: extra['if'] = ifcond if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] + obj['features'] = ([(f.name, {'if': f.ifcond}) for f in features]) if extra: return (obj, extra) return obj -def _tree_to_qlit(obj, level=0, suppress_first_indent=False): +def _tree_to_qlit(obj: TreeNode, + level: int = 0, + suppress_first_indent : bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, tuple):
@@ -85,21 +119,20 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[TreeNode] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h"
@@ -107,10 +140,10 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self)
@@ -132,18 +165,18 @@ const QLitObject %(c_name)s = %(c_string)s; self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: # Map the various integer types to plain int if typ.json_type() == 'int': typ = self._schema.lookup_type('int')
@@ -162,8 +195,10 @@ const QLitObject %(c_name)s = %(c_string)s; return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) - def _gen_tree(self, name, mtype, obj, ifcond, features): - extra = None + def _gen_tree(self, name: str, mtype: str, obj: TreeDict, + ifcond: List[str], + features: Optional[List[QAPISchemaFeature]]) -> None: + extra: Extra = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask: # Output a comment to make it easy to map masked names
@@ -174,44 +209,60 @@ const QLitObject %(c_name)s = %(c_string)s; obj['meta-type'] = mtype self._trees.append(_make_tree(obj, ifcond, features, extra)) - def _gen_member(self, member): + def _gen_member(self, + member: QAPISchemaObjectTypeMember) -> TreeNode: obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None return _make_tree(obj, member.ifcond, member.features) - def _gen_variants(self, tag_name, variants): + def _gen_variants(self, tag_name: str, + variants: List[QAPISchemaVariant]) -> TreeDict: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant): + def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode: obj = {'case': variant.name, 'type': self._use_type(variant.type)} return _make_tree(obj, variant.ifcond, None) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, + info: Optional[QAPISourceInfo], + json_type: str) -> None: self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree(name, 'enum', {'values': [_make_tree(m.name, m.ifcond, None) for m in members]}, ifcond, features) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + features: List[QAPISchemaFeature], + members: Sequence[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: + obj: TreeDict = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name, variants.variants)) self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: + self._gen_tree(name, 'alternate', {'members': [ _make_tree({'type': self._use_type(m.type)},
@@ -219,24 +270,33 @@ const QLitObject %(c_name)s = %(c_string)s; for m in variants.variants]}, ifcond, features) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + ret_type: Optional[QAPISchemaType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool) -> None: + arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: TreeDict = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, boxed: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/schema.py +1 -1
@@ -28,7 +28,7 @@ from .parser import QAPISchemaParser class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature)
Signature · 4 lines
2.26.2 -- Eduardo
Re: [PATCH v2 32/38] qapi/introspect.py: create a typed 'Node' data structure
Quoted 9 lines · Eduardo Habkost
Quoted 4 lines · John Snow
Replacing the un-typed tuple, add a typed Node that we can add typed metadata to.
Signed-off-by John Snow <[email protected]>This is the most complex patch so far, and it's very hard to understand what it does without type annotations.
Mmm, sorry about that.
Quoted 4 lines
Have you consider adding type annotations to the code before patch 30/38 (even if using
objectin some parts), so we can make this easier to review?
If you feel like it isn't just noise to learn types we're about to destroy with something else, I'll try.
Quoted 4 lines
In case it's useful, below is an attempt to add type annotations to the old code. It can be applied after patch 29/38. It reuses portions of patch 33/38.
OK. End result is I will squash the Extra and Node types together, but I need to figure out how to make it look nice for the review history.
Gimmie a few to make a mess and I'll try to put it back together afterward.
Quoted 305 lines
Signed-off-by John Snow <[email protected]>Signed-off-by Eduardo Habkost <[email protected]>--- scripts/qapi/introspect.py | 138 ++++++++++++++++++++++++++----------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 100 insertions(+), 45 deletions(-)scripts/qapi/introspect.py +99
-39@@ -10,6 +10,17 @@ This work is licensed under the terms of the GNU GPL, version 2. See the COPYING file in the top-level directory. """ +from typing import ( + Dict, + Generic, + List, + NamedTuple, + Optional, + Sequence, + TypeVar, + Tuple +) + from .common import ( c_name, gen_endif,@@ -17,25 +28,48 @@ from .common import ( mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) - - -def _make_tree(obj, ifcond, features, extra=None): +from .schema import ( + QAPISchema, + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, +) +from .source import QAPISourceInfo + +T = TypeVar('T') +# this should actually be: Union[str, list, dict, bool, 'AnnotatedNode'] +# but mypy doesn't support recursive types +TreeNode = object +TreeDict = Dict[str, TreeNode] +Extra = Dict[str, object] +AnnotatedNode = Tuple[T, Extra] + +def _make_tree(obj: TreeDict, ifcond: List[str], + features: List[QAPISchemaFeature], + extra: Optional[Extra] = None) -> TreeNode: if extra is None: extra = {} if ifcond: extra['if'] = ifcond if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] + obj['features'] = ([(f.name, {'if': f.ifcond}) for f in features]) if extra: return (obj, extra) return obj -def _tree_to_qlit(obj, level=0, suppress_first_indent=False): +def _tree_to_qlit(obj: TreeNode, + level: int = 0, + suppress_first_indent : bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, tuple):@@ -85,21 +119,20 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[TreeNode] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h"@@ -107,10 +140,10 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self)@@ -132,18 +165,18 @@ const QLitObject %(c_name)s = %(c_string)s; self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: # Map the various integer types to plain int if typ.json_type() == 'int': typ = self._schema.lookup_type('int')@@ -162,8 +195,10 @@ const QLitObject %(c_name)s = %(c_string)s; return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) - def _gen_tree(self, name, mtype, obj, ifcond, features): - extra = None + def _gen_tree(self, name: str, mtype: str, obj: TreeDict, + ifcond: List[str], + features: Optional[List[QAPISchemaFeature]]) -> None: + extra: Extra = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask: # Output a comment to make it easy to map masked names@@ -174,44 +209,60 @@ const QLitObject %(c_name)s = %(c_string)s; obj['meta-type'] = mtype self._trees.append(_make_tree(obj, ifcond, features, extra)) - def _gen_member(self, member): + def _gen_member(self, + member: QAPISchemaObjectTypeMember) -> TreeNode: obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None return _make_tree(obj, member.ifcond, member.features) - def _gen_variants(self, tag_name, variants): + def _gen_variants(self, tag_name: str, + variants: List[QAPISchemaVariant]) -> TreeDict: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant): + def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode: obj = {'case': variant.name, 'type': self._use_type(variant.type)} return _make_tree(obj, variant.ifcond, None) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, + info: Optional[QAPISourceInfo], + json_type: str) -> None: self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree(name, 'enum', {'values': [_make_tree(m.name, m.ifcond, None) for m in members]}, ifcond, features) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + features: List[QAPISchemaFeature], + members: Sequence[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: + obj: TreeDict = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name, variants.variants)) self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: + self._gen_tree(name, 'alternate', {'members': [ _make_tree({'type': self._use_type(m.type)},@@ -219,24 +270,33 @@ const QLitObject %(c_name)s = %(c_string)s; for m in variants.variants]}, ifcond, features) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + ret_type: Optional[QAPISchemaType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool) -> None: + arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: TreeDict = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, boxed: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir)scripts/qapi/mypy.ini +0
-5@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/schema.py +1
-1@@ -28,7 +28,7 @@ from .parser import QAPISchemaParser class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature)
Re: [PATCH v2 32/38] qapi/introspect.py: create a typed 'Node' data structure
In reply to Eduardo HabkostQuoted 70 lines · Eduardo Habkost
Quoted 4 lines · John Snow
Replacing the un-typed tuple, add a typed Node that we can add typed metadata to.
Signed-off-by John Snow <[email protected]>This is the most complex patch so far, and it's very hard to understand what it does without type annotations.
Have you consider adding type annotations to the code before patch 30/38 (even if using
objectin some parts), so we can make this easier to review?In case it's useful, below is an attempt to add type annotations to the old code. It can be applied after patch 29/38. It reuses portions of patch 33/38.
Signed-off-by John Snow <[email protected]>Signed-off-by Eduardo Habkost <[email protected]>--- scripts/qapi/introspect.py | 138 ++++++++++++++++++++++++++----------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 100 insertions(+), 45 deletions(-)scripts/qapi/introspect.py +25
-5@@ -10,6 +10,17 @@ This work is licensed under the terms of the GNU GPL, version 2. See the COPYING file in the top-level directory. """ +from typing import ( + Dict, + Generic, + List, + NamedTuple, + Optional, + Sequence, + TypeVar, + Tuple +) + from .common import ( c_name, gen_endif,@@ -17,25 +28,48 @@ from .common import ( mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) - - -def _make_tree(obj, ifcond, features, extra=None): +from .schema import ( + QAPISchema, + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, +) +from .source import QAPISourceInfo
scripts/qapi/introspect.py from .common import ( · line 54
Quoted 10 lines
scripts/qapi/introspect.py +10
-0+ +T = TypeVar('T') +# this should actually be: Union[str, list, dict, bool, 'AnnotatedNode'] +# but mypy doesn't support recursive types +TreeNode = object +TreeDict = Dict[str, TreeNode] +Extra = Dict[str, object] +AnnotatedNode = Tuple[T, Extra] + +def _make_tree(obj: TreeDict, ifcond: List[str],
^^^^^^^^
Technically untrue! obj may be a SchemaInfo-like dict, or a string. (And I will tell you why mypy didn't catch this!)
Quoted 127 lines
scripts/qapi/introspect.py +30
-21+ features: List[QAPISchemaFeature], + extra: Optional[Extra] = None) -> TreeNode: if extra is None: extra = {} if ifcond: extra['if'] = ifcond if features: - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features] + obj['features'] = ([(f.name, {'if': f.ifcond}) for f in features]) if extra: return (obj, extra) return obj -def _tree_to_qlit(obj, level=0, suppress_first_indent=False): +def _tree_to_qlit(obj: TreeNode, + level: int = 0, + suppress_first_indent : bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, tuple):@@ -85,21 +119,20 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[TreeNode] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h"@@ -107,10 +140,10 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self)@@ -132,18 +165,18 @@ const QLitObject %(c_name)s = %(c_string)s; self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: # Map the various integer types to plain int if typ.json_type() == 'int': typ = self._schema.lookup_type('int')@@ -162,8 +195,10 @@ const QLitObject %(c_name)s = %(c_string)s; return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) - def _gen_tree(self, name, mtype, obj, ifcond, features): - extra = None + def _gen_tree(self, name: str, mtype: str, obj: TreeDict, + ifcond: List[str], + features: Optional[List[QAPISchemaFeature]]) -> None: + extra: Extra = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask: # Output a comment to make it easy to map masked names@@ -174,44 +209,60 @@ const QLitObject %(c_name)s = %(c_string)s; obj['meta-type'] = mtype self._trees.append(_make_tree(obj, ifcond, features, extra)) - def _gen_member(self, member): + def _gen_member(self, + member: QAPISchemaObjectTypeMember) -> TreeNode: obj = {'name': member.name, 'type': self._use_type(member.type)} if member.optional: obj['default'] = None return _make_tree(obj, member.ifcond, member.features) - def _gen_variants(self, tag_name, variants): + def _gen_variants(self, tag_name: str, + variants: List[QAPISchemaVariant]) -> TreeDict: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant): + def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode: obj = {'case': variant.name, 'type': self._use_type(variant.type)} return _make_tree(obj, variant.ifcond, None) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, + info: Optional[QAPISourceInfo], + json_type: str) -> None:
scripts/qapi/introspect.py const QLitObject %(c_name)s = %(c_string)s; · line 239
Quoted 10 lines
scripts/qapi/introspect.py +4
-1self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree(name, 'enum', {'values': [_make_tree(m.name, m.ifcond, None) for m in members]},
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Over here, Members is List[QAPISchemaEnumMember], which is true enough, but we don't yet have a typing for the 'str' field. mypy fills in with the "Any" type, which is *assumed* to fill the type we want.
Nope! It's a string. Once you add the type annotations for schema.py, this would fail.
Quoted 105 lines
scripts/qapi/introspect.py +30
-12ifcond, features) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + features: List[QAPISchemaFeature], + members: Sequence[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: + obj: TreeDict = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name, variants.variants)) self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: + self._gen_tree(name, 'alternate', {'members': [ _make_tree({'type': self._use_type(m.type)},@@ -219,24 +270,33 @@ const QLitObject %(c_name)s = %(c_string)s; for m in variants.variants]}, ifcond, features) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + ret_type: Optional[QAPISchemaType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool) -> None: + arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: TreeDict = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, boxed: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir)scripts/qapi/mypy.ini +0
-5@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/schema.py +1
-1@@ -28,7 +28,7 @@ from .parser import QAPISchemaParser class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature)
PATCH v2 33/38 qapi/introspect.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/introspect.py | 135 ++++++++++++++++++++++++++----------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 96 insertions(+), 46 deletions(-)
scripts/qapi/introspect.py +95 -40
@@ -10,7 +10,15 @@ See the COPYING file in the top-level directory. """ -from typing import (NamedTuple, List, Optional, Sequence) +from typing import ( + Dict, + Generic, + List, + NamedTuple, + Optional, + Sequence, + TypeVar, +) from .common import ( c_name,
@@ -19,8 +27,24 @@ mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaFeature, QAPISchemaType) +from .schema import ( + QAPISchema, + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, +) +from .source import QAPISourceInfo + + +_NodeType = TypeVar('_NodeType') +_DObject = Dict[str, object] class Extra(NamedTuple):
@@ -31,23 +55,24 @@ class Extra(NamedTuple): ifcond: Sequence[str] = tuple() -class Node: +class Node(Generic[_NodeType]): """ Node generally contains a SchemaInfo-like type (as a dict), But it also used to wrap comments/ifconds around leaf value types. """ # Remove after 3.7 adds @dataclass: # pylint: disable=too-few-public-methods - def __init__(self, data, ifcond: List[str], + def __init__(self, data: _NodeType, ifcond: List[str], extra: Optional[Extra] = None): self.data = data comment = extra.comment if extra else None self.extra = Extra(comment, ifcond) -def _tree_to_qlit(obj, level=0, suppress_first_indent=False): +def _tree_to_qlit(obj: object, level: int = 0, + suppress_first_indent: bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, Node):
@@ -94,21 +119,20 @@ def indent(level): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[Node[_DObject]] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h"
@@ -116,10 +140,10 @@ def __init__(self, prefix, unmask): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self)
@@ -141,18 +165,18 @@ def visit_end(self): self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: # Map the various integer types to plain int if typ.json_type() == 'int': typ = self._schema.lookup_type('int')
@@ -172,10 +196,13 @@ def _use_type(self, typ): return self._name(typ.name) @classmethod - def _gen_features(cls, features: List[QAPISchemaFeature]): + def _gen_features(cls, + features: List[QAPISchemaFeature]) -> List[Node[str]]: return [Node(f.name, f.ifcond) for f in features] - def _gen_tree(self, name, mtype, obj, ifcond, features): + def _gen_tree(self, name: str, mtype: str, obj: _DObject, + ifcond: List[str], + features: Optional[List[QAPISchemaFeature]]) -> None: extra = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask:
@@ -189,45 +216,65 @@ def _gen_tree(self, name, mtype, obj, ifcond, features): obj['features'] = self._gen_features(features) self._trees.append(Node(obj, ifcond, extra)) - def _gen_member(self, member): - obj = {'name': member.name, 'type': self._use_type(member.type)} + def _gen_member(self, + member: QAPISchemaObjectTypeMember) -> Node[_DObject]: + obj: _DObject = { + 'name': member.name, + 'type': self._use_type(member.type) + } if member.optional: obj['default'] = None if member.features: obj['features'] = self._gen_features(member.features) return Node(obj, member.ifcond) - def _gen_variants(self, tag_name, variants): + def _gen_variants(self, tag_name: str, + variants: List[QAPISchemaVariant]) -> _DObject: return {'tag': tag_name, 'variants': [self._gen_variant(v) for v in variants]} - def _gen_variant(self, variant): - obj = {'case': variant.name, 'type': self._use_type(variant.type)} + def _gen_variant(self, variant: QAPISchemaVariant) -> Node[_DObject]: + obj: _DObject = { + 'case': variant.name, + 'type': self._use_type(variant.type) + } return Node(obj, variant.ifcond) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo], + json_type: str) -> None: self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree(name, 'enum', {'values': [Node(m.name, m.ifcond) for m in members]}, ifcond, features) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: List[str], + features: List[QAPISchemaFeature], + members: Sequence[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: + obj: _DObject = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name, variants.variants)) self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: self._gen_tree(name, 'alternate', {'members': [ Node({'type': self._use_type(m.type)}, m.ifcond)
@@ -235,24 +282,32 @@ def visit_alternate_type(self, name, info, ifcond, features, variants): ]}, ifcond, features) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + ret_type: Optional[QAPISchemaType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: _DObject = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: QAPISourceInfo, + ifcond: List[str], features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, boxed: bool) -> None: arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -19,11 +19,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/schema.py +1 -1
@@ -28,7 +28,7 @@ class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature)
Signature · 1 line
2.26.2
PATCH v2 34/38 qapi/types.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++----------- 2 files changed, 64 insertions(+), 27 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -29,11 +29,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.types] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.visit] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/types.py +64 -22
207 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v2 34/38] qapi/types.py: add type hint annotations
Quoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 34/38] qapi/types.py: add type hint annotations
In reply to John SnowQuoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
PATCH v2 35/38 qapi/types.py: remove one-letter variables
In reply to John Snow"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/types.py | 29 +++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -7,7 +7,6 @@ ignore-patterns=doc.py, expr.py, parser.py, schema.py, - types.py, visit.py,
scripts/qapi/types.py +15 -14
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str, .array = (const char *const[]) { ''', c_name=c_name(name)) - for m in members: - ret += gen_if(m.ifcond) - index = c_enum_const(name, m.name, prefix) + for member in members: + ret += gen_if(member.ifcond) + index = c_enum_const(name, member.name, prefix) ret += mcgen(''' [%(index)s] = "%(name)s", ''', - index=index, name=m.name) - ret += gen_endif(m.ifcond) + index=index, name=member.name) + ret += gen_endif(member.ifcond) ret += mcgen(''' },
@@ -79,13 +79,13 @@ def gen_enum(name: str, ''', c_name=c_name(name)) - for m in enum_members: - ret += gen_if(m.ifcond) + for member in enum_members: + ret += gen_if(member.ifcond) ret += mcgen(''' %(c_enum)s, ''', - c_enum=c_enum_const(name, m.name, prefix)) - ret += gen_endif(m.ifcond) + c_enum=c_enum_const(name, member.name, prefix)) + ret += gen_endif(member.ifcond) ret += mcgen(''' } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name) ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for variant in variants.variants if variants else (): + obj = variant.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''
Signature · 1 line
2.26.2
Re: [PATCH v2 35/38] qapi/types.py: remove one-letter variables
Quoted 7 lines · John Snow
"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
Signed-off-by John Snow <[email protected]>---
[...]
scripts/qapi/types.py def gen_object(name: str, ifcond: List[str], · line 151
Quoted 10 lines
scripts/qapi/types.py +1
-5@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name) ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for variant in variants.variants if variants else ():
I'm not sure I like this weird expression, but I believe asking for a 120-patch cleanup series to be respun because of a tiny style issue would be counterproductive, so:
Quoted 11 lines
scripts/qapi/types.py +5
-0+ obj = variant.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''Signature · 1 line
2.26.2
Signature · 1 line
Eduardo
Re: [PATCH v2 35/38] qapi/types.py: remove one-letter variables
Quoted 5 lines · Eduardo Habkost
I'm not sure I like this weird expression, but I believe asking for a 120-patch cleanup series to be respun because of a tiny style issue would be counterproductive, so:
Reviewed-by Eduardo Habkost<[email protected]>
I was trying to reduce the indent level to accommodate the longer names, but python ternaries *are* pretty weird.
It'd be nice to enforce always having a variants object instead (even if it's empty!) and then add __bool__ and __iter__ methods to QAPISchemaVariants such that you could always do:
"if variants"
or
"for variant in variants"
but we're not there just yet... should I just put it back the way it was, with the deep nesting?
--js
Re: [PATCH v2 35/38] qapi/types.py: remove one-letter variables
Quoted 14 lines · John Snow
Quoted 5 lines · Eduardo Habkost
I'm not sure I like this weird expression, but I believe asking for a 120-patch cleanup series to be respun because of a tiny style issue would be counterproductive, so:
Reviewed-by Eduardo Habkost<[email protected]>I was trying to reduce the indent level to accommodate the longer names, but python ternaries *are* pretty weird.
It'd be nice to enforce always having a variants object instead (even if it's empty!) and then add
__bool__and__iter__methods to QAPISchemaVariants such that you could always do:
Quoted 8 lines
"if variants"
or
"for variant in variants"
but we're not there just yet... should I just put it back the way it was, with the deep nesting?
I don't have a strong opinion. I got used to it after seeing the same pattern being used a few times.
Signature · 1 line
Eduardo
Re: [PATCH v2 35/38] qapi/types.py: remove one-letter variables
In reply to John SnowQuoted 3 lines · John Snow
"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
:D
Quoted 3 lines
Now that this file is clean, enable pylint checks on this file.
Signed-off-by John Snow <[email protected]>
PATCH v2 36/38 qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
In reply to John Snow--- scripts/qapi/visit.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +10 -5
@@ -22,7 +22,10 @@ indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import ( + QAPISchemaEnumType, + QAPISchemaObjectType, +) def gen_visit_decl(name, scalar=False):
@@ -84,15 +87,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType) + ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Signature · 1 line
2.26.2
Re: [PATCH v2 36/38] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>
This for making mypy happy, correct? An explanation in the commit message would be nice.
Signature · 1 line
Eduardo
Re: [PATCH v2 36/38] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 8 lines · Eduardo Habkost
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>This for making mypy happy, correct? An explanation in the commit message would be nice.
Reviewed-by Eduardo Habkost <[email protected]>
Yes, it's for mypy -- but it's a runtime visible change. Technically our type system isn't mature enough to express this constraint natively, so it's being carried around as developer knowledge.
This formalizes that knowledge, albeit in a very crude way.
I've amended the commit msg.
Re: [PATCH v2 36/38] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 10 lines · John Snow
Quoted 8 lines · Eduardo Habkost
Quoted 1 line · John Snow
Signed-off-by John Snow <[email protected]>This for making mypy happy, correct? An explanation in the commit message would be nice.
Reviewed-by Eduardo Habkost <[email protected]>
Quoted 7 lines
Yes, it's for mypy -- but it's a runtime visible change. Technically our type system isn't mature enough to express this constraint natively, so it's being carried around as developer knowledge.
This formalizes that knowledge, albeit in a very crude way.
I've amended the commit msg.
OK, this answers my previous question about why it was handled as an assert.
- Cleber.
Re: [PATCH v2 36/38] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
In reply to John SnowQuoted 22 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/visit.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)scripts/qapi/visit.py +4
-1@@ -22,7 +22,10 @@ indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import ( + QAPISchemaEnumType, + QAPISchemaObjectType, +) def gen_visit_decl(name, scalar=False):
scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 92
Quoted 6 lines
scripts/qapi/visit.py +3
-0@@ -84,15 +87,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType) +
I'd be interested in knowing why this wasn't left to be handled by the type checking only. Anyway,
Re: [PATCH v2 36/38] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 25 lines · Cleber Rosa
Quoted 24 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/visit.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)scripts/qapi/visit.py +4
-1@@ -22,7 +22,10 @@ indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import ( + QAPISchemaEnumType, + QAPISchemaObjectType, +) def gen_visit_decl(name, scalar=False):@@ -84,15 +87,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond)
Quoted 8 lines
scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 92
Quoted 4 lines
scripts/qapi/visit.py +3
-0if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType) +I'd be interested in knowing why this wasn't left to be handled by the type checking only. Anyway,
QAPISchemaVariants is a container type that is used to house a number of QAPISchemaVariant types; but it (can) also take a tag_member to identify one of the fields in the variants that can be used to differentiate them.
Now, we assert that tag_member must be a QAPISchemaObjectTypeMember. QAPISchemaVariant is also a QAPISchemaObjectTypeMember.
a QAPISchemaObjectTypeMember is a QAPISchemaMember. a QAPISchemaMember describes one 'member' of either an enum, a features list, or an object member.
Now, the QAPISchemaObjectTypeMember (and not the QAPISchemaMember!) serves as a container for a QAPISchemaType -- this is a wrapper type, effectively. That contained type can be *anything*, because object members can be *anything*.
Oops, but if we zoom back out, we are only able to constrain tag_member to being a QAPISchemaObjectTypeMember, we have no expressive power over its contained type.
That's why you need the assertion here; because of a loss of specificity when we declare tag_member.
"Wow, John, it sounds like you should use a Generic type to be able to describe the inner type of a QAPISchemaObjectTypeMember?"
Uh, yup, you're right! I should. But it's complicated, because QAPISchemaMember does not have a contained type. Further, the contained type of a Member may or may not be known at construction time right now.
It's fixable, and probably involves adding something like a "string constant" dummy type to allow QAPISchemaMember to have a contained type.
"Hey, all of that sounds very messy. What if we just added in a few assertions for right now while we get the preliminary types in, and then we can make adjustments based on what makes the code prettier?"
Sounds like a plan, hypothetical quote person.
Quoted 2 lines
Reviewed-by Cleber Rosa <[email protected]>
Re: [PATCH v2 36/38] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 63 lines · John Snow
Quoted 21 lines · Cleber Rosa
Quoted 20 lines · John Snow
Signed-off-by John Snow <[email protected]>--- scripts/qapi/visit.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)scripts/qapi/visit.py +4
-1@@ -22,7 +22,10 @@ indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import ( + QAPISchemaEnumType, + QAPISchemaObjectType, +) def gen_visit_decl(name, scalar=False):@@ -84,15 +87,17 @@ def gen_visit_object_members(name, base, members, variants):Quoted 9 lines
scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 92
Quoted 5 lines
scripts/qapi/visit.py +3
-0ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType) +I'd be interested in knowing why this wasn't left to be handled by the type checking only. Anyway,
QAPISchemaVariants is a container type that is used to house a number of QAPISchemaVariant types; but it (can) also take a
tag_memberto identify one of the fields in the variants that can be used to differentiate them.Now, we assert that
tag_membermust be a QAPISchemaObjectTypeMember. QAPISchemaVariant is also a QAPISchemaObjectTypeMember.a QAPISchemaObjectTypeMember is a QAPISchemaMember. a QAPISchemaMember describes one 'member' of either an enum, a features list, or an object member.
Now, the QAPISchemaObjectTypeMember (and not the QAPISchemaMember!) serves as a container for a QAPISchemaType -- this is a wrapper type, effectively. That contained type can be *anything*, because object members can be *anything*.
Oops, but if we zoom back out, we are only able to constrain
tag_memberto being a QAPISchemaObjectTypeMember, we have no expressive power over its contained type.That's why you need the assertion here; because of a loss of specificity when we declare
tag_member."Wow, John, it sounds like you should use a Generic type to be able to describe the inner type of a QAPISchemaObjectTypeMember?"
Uh, yup, you're right! I should. But it's complicated, because QAPISchemaMember does not have a contained type. Further, the contained type of a Member may or may not be known at construction time right now.
Quoted 11 lines
It's fixable, and probably involves adding something like a "string constant" dummy type to allow QAPISchemaMember to have a contained type.
"Hey, all of that sounds very messy. What if we just added in a few assertions for right now while we get the preliminary types in, and then we can make adjustments based on what makes the code prettier?"
Sounds like a plan, hypothetical quote person.
Quoted 2 lines
Reviewed-by Cleber Rosa <[email protected]>
I did not attempt to learn the type names by heart (mental sanity first) but I get the big picture. Thanks!
- Cleber.
PATCH v2 37/38 qapi/visit.py: remove unused parameters from gen_visit_object
In reply to John SnowAnd this fixes the pylint report for this file, so make sure we check this in the future, too.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/visit.py | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -7,7 +7,6 @@ ignore-patterns=doc.py, expr.py, parser.py, schema.py, - visit.py, [MESSAGES CONTROL]
scripts/qapi/visit.py +2 -2
@@ -253,7 +253,7 @@ def gen_visit_alternate(name, variants): return ret -def gen_visit_object(name, base, members, variants): +def gen_visit_object(name): return mcgen(''' bool visit_type_%(c_name)s
@@ -346,7 +346,7 @@ def visit_object_type(self, name, info, ifcond, features, if not name.startswith('q_'): # only explicit types need an allocating visit self._genh.add(gen_visit_decl(name)) - self._genc.add(gen_visit_object(name, base, members, variants)) + self._genc.add(gen_visit_object(name)) def visit_alternate_type(self, name, info, ifcond, features, variants): with ifcontext(ifcond, self._genh, self._genc):
Signature · 1 line
2.26.2
Re: [PATCH v2 37/38] qapi/visit.py: remove unused parameters from gen_visit_object
Quoted 4 lines · John Snow
And this fixes the pylint report for this file, so make sure we check this in the future, too.
Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 37/38] qapi/visit.py: remove unused parameters from gen_visit_object
In reply to John SnowQuoted 4 lines · John Snow
And this fixes the pylint report for this file, so make sure we check this in the future, too.
Signed-off-by John Snow <[email protected]>
LGTM, and pylint is happy.
PATCH v2 38/38 qapi/visit.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 ---- scripts/qapi/visit.py | 68 +++++++++++++++++++++++++++++++++---------- 2 files changed, 52 insertions(+), 21 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -28,8 +28,3 @@ check_untyped_defs = False disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False - -[mypy-qapi.visit] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False
scripts/qapi/visit.py +52 -16
162 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v2 38/38] qapi/visit.py: add type hint annotations
Quoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo
Re: [PATCH v2 38/38] qapi/visit.py: add type hint annotations
In reply to John SnowQuoted 4 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.Signed-off-by John Snow <[email protected]>
Quoted 3 lines · John Snow
Hi, this series adds static type hints to the QAPI module. This is part one!
Hi, I will be sending out a v3 shortly.
The status of this series is presently as follows:
+ [01] do-not-merge-qapi-add #
+ [02] qapi-gen-separate-arg-parsing # [SOB] JS [RB] EH
+ [03] qapi-move-generator-entrypoint # [SOB] JS [RB] CR,EH [TB] CR
+ [04] qapi-prefer-explicit-relative # [SOB] JS [RB] CR,EH + [05] qapi-remove-wildcard-includes # [SOB] JS [RB] CR,EH + [06] qapi-enforce-import-order # [SOB] JS + [07] qapi-delint-using-flake8 # [SOB] JS [RB] CR,EH + [08] qapi-add-pylintrc # [SOB] JS [TB] CR,EH [RB] CR + [09] qapi-common-py-remove-python # [SOB] JS [RB] CR,EH + [10] qapi-common-add-indent-manager # [SOB] JS [RB] CR,EH + [11] qapi-common-py-delint-with # [SOB] JS [RB] CR,EH + [12] replace-c-by-char # [SOB] JS [RB] CR,EH + [13] qapi-common-py-check-with # [SOB] JS [RB] CR [TB] CR,EH + [14] qapi-common-py-add-notational # [SOB] JS [RB] CR,EH + [15] qapi-common-move-comments-into # [SOB] JS [RB] CR,EH + [16] qapi-split-build_params-into # [SOB] JS [RB] CR,EH + [17] qapi-establish-mypy-type # [SOB] JS [TB] EH + [18] qapi-events-py-add-notational # [SOB] JS [RB] CR,EH + [19] qapi-events-move-comments-into # [SOB] JS [RB] CR,EH + [20] qapi-commands-py-don-t-re-bind # [SOB] JS [RB] CR,EH + [21] qapi-commands-py-add # [SOB] JS [RB] CR,EH + [22] qapi-commands-py-enable # [SOB] JS [RB] CR,EH + [23] qapi-source-py-add-notational # [SOB] JS [RB] EH + [24] qapi-source-py-delint-with # [SOB] JS [RB] CR,EH [TB] CR + [25] qapi-gen-py-fix-edge-case-of # + [26] qapi-gen-py-add-notational # [SOB] JS [RB] CR,EH + [27] qapi-gen-py-enable-checking # [SOB] JS [RB] CR,EH [TB] CR + [28] qapi-gen-py-remove-unused # [SOB] JS [RB] CR,EH + [29] qapi-gen-py-update-write-to-be # [SOB] JS [RB] CR,EH + [30] qapi-gen-py-delint-with-pylint # [SOB] JS [RB] CR,EH + [31] qapi-introspect-py-assert-obj # + [32] qapi-introspect-py-create-a # [SOB] EH,JS + [33] qapi-introspect-py-add # + [34] qapi-introspect-py-unify # + [35] qapi-introspect-py-replace # + [36] qapi-introspect-py-create-a-0 # + [37] qapi-types-py-add-type-hint # [SOB] JS [RB] CR,EH + [38] qapi-types-py-remove-one # [SOB] JS [RB] CR,EH + [39] qapi-visit-py-assert # [SOB] JS [RB] CR,EH + [40] qapi-visit-py-remove-unused # [SOB] JS [RB] CR,EH [TB] CR > [41] qapi-visit-py-add-notational # [SOB] JS [RB] CR,EH [TB] CR
Thank you for your help so far, everyone!
--js