qapi: static typing conversion, pt1
v5 cover qapi: static typing conversion, pt1
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)
In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
Notes:
- After patch 07, isort -c should pass 100% on this and every future commit.
- After patch 08, flake8 qapi/ should pass 100% on this and every future commit.
- After patch 09, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.
- After patch 18, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.
Review Status:
[01] docs-repair-broken-references # [02] qapi-modify-docstrings-to-be # [03] qapi-gen-separate-arg-parsing # [RB] CR,EH [TB] CR [SOB] JS [04] qapi-move-generator-entrypoint # [RB] CR,EH [TB] CR [SOB] JS [05] qapi-prefer-explicit-relative # [RB] CR,EH [SOB] JS [06] qapi-remove-wildcard-includes # [RB] CR,EH [SOB] JS [07] qapi-enforce-import-order # [RB] CR [TB] CR [SOB] JS [08] qapi-delint-using-flake8 # [RB] CR,EH [SOB] JS [09] qapi-add-pylintrc # [RB] CR [TB] CR,EH [SOB] JS [10] qapi-common-py-remove-python # [RB] CR,EH [SOB] JS [11] qapi-common-add-indent-manager # [RB] CR,EH [SOB] JS [12] qapi-common-py-delint-with # [RB] CR,EH [SOB] JS [13] replace-c-by-char # [RB] CR,EH [SOB] JS [14] qapi-common-py-check-with # [RB] CR [TB] CR,EH [SOB] JS [15] qapi-common-py-add-notational # [RB] CR,EH [SOB] JS [16] qapi-common-move-comments-into # [RB] CR,EH [SOB] JS [17] qapi-split-build_params-into # [RB] CR,EH [SOB] JS [18] qapi-establish-mypy-type # [RB] CR [TB] CR,EH [SOB] JS [19] qapi-events-py-add-notational # [RB] CR,EH [SOB] JS [20] qapi-events-move-comments-into # [RB] CR,EH [SOB] JS [21] qapi-commands-py-don-t-re-bind # [RB] CR,EH [SOB] JS [22] qapi-commands-py-add # [RB] CR,EH [SOB] JS [23] qapi-commands-py-enable # [RB] CR,EH [SOB] JS [24] qapi-source-py-add-notational # [RB] CR,EH [TB] CR [SOB] JS [25] qapi-source-py-delint-with # [RB] CR,EH [TB] CR [SOB] JS [26] qapi-gen-py-fix-edge-case-of # [RB] CR,EH [SOB] JS [27] qapi-gen-py-add-notational # [RB] CR,EH [SOB] JS [28] qapi-gen-py-enable-checking # [RB] CR,EH [TB] CR [SOB] JS [29] qapi-gen-py-remove-unused # [RB] CR,EH [SOB] JS [30] qapi-gen-py-update-write-to-be # [RB] CR,EH [SOB] JS [31] qapi-gen-py-delint-with-pylint # [RB] CR,EH [SOB] JS [32] qapi-types-py-add-type-hint # [RB] CR,EH [SOB] JS [33] qapi-types-py-remove-one # [RB] CR,EH [SOB] JS [34] qapi-visit-py-assert # [RB] CR,EH [SOB] JS [35] qapi-visit-py-remove-unused # [RB] CR,EH [TB] CR [SOB] JS [36] qapi-visit-py-add-notational # [RB] CR,EH [TB] CR [SOB] JS
Changelog:
002/36:[0012] [FC] 'qapi: modify docstrings to be sphinx-compatible' 027/36:[----] [-C] 'qapi/gen.py: add type hint annotations'
V5:
- Remove DO-NOT-MERGE patches (Now in Part2)
- Remove introspect.py patches (Now in Part2)
- 02: Docstring formatting, more commit message (Markus)
V4:
- Rebase on Peter Maydell's work;
- 05: Context differences from Peter Maydell's work
- 06: Remove qapi.doc
- 07: Remove qapi.doc, remove superfluous "if match"
- 09: Remove qapi.doc
- 13: remove qapi.doc
- 18: remove qapi.doc
- 22: remove qapi.doc
- 31: remove QAPIGenDoc changes
- Minor adjustments from list feedback;
- 01: More backticks for QAPI json files, now that they are in Sphinx-exe
- 02: Use :ref: role for the docker-ref cross-reference
- 04: Remove doc.py work changes; add references for start_if/end_if
- 10: Changes to accommodate isort
- 11: Added lines_after_imports=2
- 16: isort whitespace changes
- 24: Take Markus's docstring phrasing
- 34: add comment explaining os.open()
- 37: isort differences
V3:
- Use isort to enforce import consistency
- Use sphinx apidoc to check docstring format
V2:
- Removed Python3.6 patch in favor of Thomas Huth's
- Addressed (most) feedback from Markus
- Renamed type hint annotation commits
John Snow (36):
docs: repair broken references
qapi: modify docstrings to be sphinx-compatible
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into module
qapi: Prefer explicit relative imports
qapi: Remove wildcard includes
qapi: enforce import order/styling with isort
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable qapi/common.py: check with pylint
qapi/common.py: add type hint annotations qapi/common.py: Convert comments into docstrings, and elaborate qapi/common.py: move build_params into gen.py
qapi: establish mypy type-checking baseline qapi/events.py: add type hint annotations qapi/events.py: Move comments into docstrings qapi/commands.py: Don't re-bind to variable of different type qapi/commands.py: add type hint annotations qapi/commands.py: enable checking with mypy qapi/source.py: add type hint annotations qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module qapi/gen.py: add type hint annotations
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic qapi/gen.py: delint with pylint
qapi/types.py: add type hint annotations qapi/types.py: remove one-letter variables qapi/visit.py: assert tag_member contains a QAPISchemaEnumType qapi/visit.py: remove unused parameters from gen_visit_object qapi/visit.py: add type hint annotations
docs/devel/multi-thread-tcg.rst | 2 +- docs/devel/testing.rst | 2 +- scripts/qapi-gen.py | 57 ++-------- scripts/qapi/.flake8 | 2 + scripts/qapi/.isort.cfg | 7 ++ scripts/qapi/commands.py | 87 +++++++++++---- scripts/qapi/common.py | 163 +++++++++++++++------------- scripts/qapi/events.py | 58 +++++++--- scripts/qapi/expr.py | 7 +- scripts/qapi/gen.py | 182 ++++++++++++++++++++------------ scripts/qapi/introspect.py | 16 ++- scripts/qapi/main.py | 89 ++++++++++++++++ scripts/qapi/mypy.ini | 30 ++++++ scripts/qapi/parser.py | 7 +- scripts/qapi/pylintrc | 70 ++++++++++++ scripts/qapi/schema.py | 33 +++--- scripts/qapi/source.py | 34 +++--- scripts/qapi/types.py | 125 +++++++++++++++------- scripts/qapi/visit.py | 116 ++++++++++++++------ 19 files changed, 752 insertions(+), 335 deletions(-) create mode 100644 scripts/qapi/.flake8 create mode 100644 scripts/qapi/.isort.cfg create mode 100644 scripts/qapi/main.py create mode 100644 scripts/qapi/mypy.ini create mode 100644 scripts/qapi/pylintrc
Signature · 1 line
2.26.2
PATCH v5 1/36 docs: repair broken references
In two different places, we are not making a cross-reference to some resource correctly.
--- docs/devel/multi-thread-tcg.rst | 2 +- docs/devel/testing.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
docs/devel/multi-thread-tcg.rst +1 -1
@@ -267,7 +267,7 @@ of view of external observers (e.g. another processor core). They can apply to any memory operations as well as just loads or stores. The Linux kernel has an excellent `write-up -<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>` +<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`_ on the various forms of memory barrier and the guarantees they can provide.
docs/devel/testing.rst +1 -1
@@ -953,7 +953,7 @@ compiler flags are needed to build for a given target. If you have the ability to run containers as the user you can also take advantage of the build systems "Docker" support. It will then use containers to build any test case for an enabled guest where there is -no system compiler available. See :ref: `_docker-ref` for details. +no system compiler available. See :ref:`docker-ref` for details. Running subset of tests -----------------------
Signature · 1 line
2.26.2
PATCH v5 2/36 qapi: modify docstrings to be sphinx-compatible
In reply to John SnowA precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.
A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
- Additional examples, diagrams, or other metadata follows below.
See also:
https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
--- scripts/qapi/gen.py | 6 ++++-- scripts/qapi/parser.py | 1 + 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -2
@@ -154,9 +154,11 @@ def _bottom(self): @contextmanager def ifcontext(ifcond, *args): - """A 'with' statement context manager to wrap with start_if()/end_if() + """ + A with-statement context manager that wraps with `start_if()` / `end_if()`. - *args: any number of QAPIGenCCode + :param ifcond: A list of conditionals, passed to `start_if()`. + :param args: any number of `QAPIGenCCode`. Example::
scripts/qapi/parser.py +1 -0
@@ -381,6 +381,7 @@ def append(self, line): The way that the line is dealt with depends on which part of the documentation we're parsing right now: + * The body section: ._append_line is ._append_body_line * An argument section: ._append_line is ._append_args_line * A features section: ._append_line is ._append_features_line
Signature · 1 line
2.26.2
Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible
Quoted 8 lines · John Snow
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.
A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
Quoted 7 lines
- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
Mandatory when they apply?
Quoted 35 lines
- Additional examples, diagrams, or other metadata follows below.
See also:
https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-listsSigned-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 6 ++++-- scripts/qapi/parser.py | 1 + 2 files changed, 5 insertions(+), 2 deletions(-)scripts/qapi/gen.py +4
-2@@ -154,9 +154,11 @@ def _bottom(self): @contextmanager def ifcontext(ifcond, *args): - """A 'with' statement context manager to wrap with start_if()/end_if() + """ + A with-statement context manager that wraps with `start_if()` / `end_if()`. - *args: any number of QAPIGenCCode + :param ifcond: A list of conditionals, passed to `start_if()`. + :param args: any number of `QAPIGenCCode`. Example::scripts/qapi/parser.py +0
-0
scripts/qapi/parser.py def append(self, line): · line 387
Quoted 7 lines
scripts/qapi/parser.py +1
-0@@ -381,6 +381,7 @@ def append(self, line): The way that the line is dealt with depends on which part of the documentation we're parsing right now: + * The body section: ._append_line is ._append_body_line * An argument section: ._append_line is ._append_args_line * A features section: ._append_line is ._append_features_line
I'm asking because you're not adding :param line: here.
Same for several other functions in this file.
In schema.py:
class QAPISchemaMember:
""" Represents object members, enum members and features """
Are the spaces next to """ okay?
Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible
Quoted 18 lines · Markus Armbruster
Quoted 8 lines · John Snow
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.
A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
Quoted 7 lines
- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
Quoted 2 lines
Mandatory when they apply?
Subject of debate...
- Some people really hate obvious docstring comments.
- Some people really like the consistency.
Which type of developer am I? Guess it depends on when you ask.
Figured we'd hash that out when I go to write a style guide document.
Quoted 36 lines
Quoted 36 lines
- Additional examples, diagrams, or other metadata follows below.
See also:
https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-listsSigned-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 6 ++++-- scripts/qapi/parser.py | 1 + 2 files changed, 5 insertions(+), 2 deletions(-)scripts/qapi/gen.py +4
-2@@ -154,9 +154,11 @@ def _bottom(self): @contextmanager def ifcontext(ifcond, *args): - """A 'with' statement context manager to wrap with start_if()/end_if() + """ + A with-statement context manager that wraps with `start_if()` / `end_if()`. - *args: any number of QAPIGenCCode + :param ifcond: A list of conditionals, passed to `start_if()`. + :param args: any number of `QAPIGenCCode`. Example::scripts/qapi/parser.py +0
-0@@ -381,6 +381,7 @@ def append(self, line):
Quoted 9 lines
scripts/qapi/parser.py def append(self, line): · line 387
Quoted 6 lines
scripts/qapi/parser.py +1
-0The way that the line is dealt with depends on which part of the documentation we're parsing right now: + * The body section: ._append_line is ._append_body_line * An argument section: ._append_line is ._append_args_line * A features section: ._append_line is ._append_features_lineI'm asking because you're not adding :param line: here.
Yeah, it's not necessary to test the syntax of what else I've written with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't required at all and could be deleted. You can do so if you'd like; it will just show up later in some other patch or series more designed to fix formatting.
Quoted 9 lines
Same for several other functions in this file.
In
schema.py:class QAPISchemaMember:
""" Represents object members, enum members and features """Are the spaces next to """ okay?
Ideally cleaned up, but that's not a goal of this patch or series.
--js
Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible
Quoted 21 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.
A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
Quoted 9 lines
Quoted 7 lines
- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.Mandatory when they apply?
Quoted 8 lines
Subject of debate...
- Some people really hate obvious docstring comments.
- Some people really like the consistency.Which type of developer am I? Guess it depends on when you ask.
Figured we'd hash that out when I go to write a style guide document.
Fair enough.
If I stop reading after the first paragraph, the patch matches expectations built by the commit message.
If I speed-read, the first paragraph barely registers, but the second makes me slow down, giving me the mistaken idea that this patch is about converting to a preliminary style guide. It's not, it's about getting Sphinx errors out of the way.
I figure you didn't stop after the first paragraph because you felt a need to explain why you resolve the "obvious errors" the way you do.
Perhaps:
qapi: modify docstrings to be sphinx-compatible
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. For now, merely avoid obvious errors that cause Sphinx documentation build problems, using a style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
[...]
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
- Additional examples, diagrams, or other metadata follows below.
See also:
https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Blank line here, by convention.
Quoted 35 lines
Quoted 25 lines
Quoted 25 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 6 ++++-- scripts/qapi/parser.py | 1 + 2 files changed, 5 insertions(+), 2 deletions(-)scripts/qapi/gen.py +4
-1@@ -154,9 +154,11 @@ def _bottom(self): @contextmanager def ifcontext(ifcond, *args): - """A 'with' statement context manager to wrap with start_if()/end_if() + """ + A with-statement context manager that wraps with `start_if()` / `end_if()`. - *args: any number of QAPIGenCCode + :param ifcond: A list of conditionals, passed to `start_if()`. + :param args: any number of `QAPIGenCCode`. Example:: diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.pyindex
9d1a3e2eea9..31bc2e6dca9100644scripts/qapi/parser.py +0
-0@@ -381,6 +381,7 @@ def append(self, line):Quoted 9 lines
Quoted 7 lines
scripts/qapi/parser.py +0
-0The way that the line is dealt with depends on whichpart of
the documentation we're parsing right now:
+
* The body section: ._append_line is ._append_body_line
* An argument section: ._append_line is ._append_args_line
* A features section: ._append_line is ._append_features_lineI'm asking because you're not adding :param line: here.
Quoted 5 lines
Yeah, it's not necessary to test the syntax of what else I've written with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't required at all and could be deleted. You can do so if you'd like; it will just show up later in some other patch or series more designed to fix formatting.
I recommend (but do not demand) to strictly limit this commit to "avoiding obvious errors that cause sphinx documentation build problems."
Quoted 8 lines
Quoted 6 lines
Same for several other functions in this file.
Inschema.py:
class QAPISchemaMember:
""" Represents object members, enum members and features """
Are the spaces next to """ okay?Ideally cleaned up, but that's not a goal of this patch or series.
Got it.
Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible
Quoted 28 lines · Markus Armbruster
Quoted 26 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.
A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
Quoted 9 lines
Quoted 7 lines
- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.Mandatory when they apply?
Subject of debate...
- Some people really hate obvious docstring comments.
- Some people really like the consistency.
Quoted 9 lines
Quoted 3 lines
Which type of developer am I? Guess it depends on when you ask.
Figured we'd hash that out when I go to write a style guide document.
Fair enough.
If I stop reading after the first paragraph, the patch matches expectations built by the commit message.
(:
Quoted 22 lines
If I speed-read, the first paragraph barely registers, but the second makes me slow down, giving me the mistaken idea that this patch is about converting to a preliminary style guide. It's not, it's about getting Sphinx errors out of the way.
I figure you didn't stop after the first paragraph because you felt a need to explain why you resolve the "obvious errors" the way you do.
Perhaps:
qapi: modify docstrings to be sphinx-compatible
A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. For now, merely avoid obvious errors that cause Sphinx documentation build problems, using a style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.
[...]
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
- Additional examples, diagrams, or other metadata follows below.
See also:
https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-listsBlank line here, by convention.
Wonder why my script didn't do that. Eh.
Quoted 41 lines
Quoted 35 lines
Quoted 25 lines
Quoted 25 lines
Signed-off-by John Snow <[email protected]>--- scripts/qapi/gen.py | 6 ++++-- scripts/qapi/parser.py | 1 + 2 files changed, 5 insertions(+), 2 deletions(-)scripts/qapi/gen.py +4
-1@@ -154,9 +154,11 @@ def _bottom(self): @contextmanager def ifcontext(ifcond, *args): - """A 'with' statement context manager to wrap with start_if()/end_if() + """ + A with-statement context manager that wraps with `start_if()` / `end_if()`. - *args: any number of QAPIGenCCode + :param ifcond: A list of conditionals, passed to `start_if()`. + :param args: any number of `QAPIGenCCode`. Example:: diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.pyindex
9d1a3e2eea9..31bc2e6dca9100644scripts/qapi/parser.py +0
-0@@ -381,6 +381,7 @@ def append(self, line):Quoted 9 lines
Quoted 7 lines
scripts/qapi/parser.py +0
-0The way that the line is dealt with depends on whichpart of
the documentation we're parsing right now:
+
* The body section: ._append_line is ._append_body_line
* An argument section: ._append_line is ._append_args_line
* A features section: ._append_line is ._append_features_lineI'm asking because you're not adding :param line: here.
Quoted 5 lines
Yeah, it's not necessary to test the syntax of what else I've written with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't required at all and could be deleted. You can do so if you'd like; it will just show up later in some other patch or series more designed to fix formatting.
Quoted 4 lines
I recommend (but do not demand) to strictly limit this commit to "avoiding obvious errors that cause sphinx documentation build problems."
OK, I'll drop this bit for now, but I will keep the new annotations for ifcontext, because... well. Why do it twice.
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Same for several other functions in this file.
Inschema.py:
class QAPISchemaMember:
""" Represents object members, enum members and features """
Are the spaces next to """ okay?Ideally cleaned up, but that's not a goal of this patch or series.
Got it.
PATCH v5 3/36 qapi-gen: Separate arg-parsing from generation
In reply to John SnowThis is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.
--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +62 -23
@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',
@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())
Signature · 1 line
2.26.2
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 31 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +6
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re
scripts/qapi-gen.py line 22
Quoted 10 lines
scripts/qapi-gen.py +2
-1import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit
Unrelated cleanup. Okay.
Quoted 18 lines
scripts/qapi-gen.py +15
-1-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.
scripts/qapi-gen.py line 49
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)
Uh...
$ python3 scripts/qapi-gen.py --prefix=@ x
scripts/qapi-gen.py: : funny character '@' in prefix '@'Unwanted " :".
This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
Quoted 18 lines
scripts/qapi-gen.py +16
-0+ + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema')
scripts/qapi-gen.py line 75
Quoted 10 lines
scripts/qapi-gen.py +4
-2parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols")
I don't like the changes to default=, because:
1. They are only losely related to the patch's purpose.
2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.
3. The defaults will not change, and nothing else uses the constants.
Quoted 22 lines
scripts/qapi-gen.py +5
-9parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr)
scripts/qapi-gen.py def main(argv): · line 91
Quoted 10 lines
scripts/qapi-gen.py +3
-7- exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0
Subtle change: you move the gen_FOO() into the try ... except. Okay; they don't raise QAPIError, but perhaps worth a mention in the commit message.
scripts/qapi-gen.py def main(argv): · line 95
Quoted 5 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())
"Python was designed to be easy to understand and fun to use." Ha ha ha.
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 77 lines · Markus Armbruster
Quoted 31 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +6
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import rescripts/qapi-gen.py line 22
Quoted 10 lines
scripts/qapi-gen.py +2
-1import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visitUnrelated cleanup. Okay.
Quoted 18 lines
scripts/qapi-gen.py +15
-1-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.scripts/qapi-gen.py line 49
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)Uh...
Quoted 8 lines
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: : funny character '@' in prefix '@'Unwanted " :".
This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).
Quack indeed - why does our base error class require so much information from a specific part of the generation process?
Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.
You are witnessing some more future-bleed.
Quoted 3 lines
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?
The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.
Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.
Quoted 31 lines
Quoted 18 lines
scripts/qapi-gen.py +16
-0+ + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema')scripts/qapi-gen.py line 75
Quoted 10 lines
scripts/qapi-gen.py +4
-2parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols")I don't like the changes to default=, because:
Quoted 2 lines
1. They are only losely related to the patch's purpose.
Subjective, but OK.
Quoted 3 lines
2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.
All of it is in main.py, though! If you were to, say, move generate() elsewhere, it'd look pretty compact as just the CLI frontend, no?
Quoted 2 lines
3. The defaults will not change, and nothing else uses the constants.
But, fine. Cleber had the same comment but I wasn't fully on-board, but two folks saying the same thing ...
Quoted 28 lines
Quoted 28 lines
scripts/qapi-gen.py +5
-15parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix)
Quoted 9 lines
scripts/qapi-gen.py def main(argv): · line 91
Quoted 4 lines
scripts/qapi-gen.py +3
-1- gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0Subtle change: you move the
gen_FOO()into the try ... except. Okay; they don't raise QAPIError, but perhaps worth a mention in the commit message.
Forbidden future knowledge; I intend them to.
Quoted 9 lines
scripts/qapi-gen.py def main(argv): · line 95
Quoted 5 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())"Python was designed to be easy to understand and fun to use." Ha ha ha.
I mean, I'm having fun, aren't you?
--js
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 79 lines · John Snow
Quoted 70 lines · Markus Armbruster
Quoted 28 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +5
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparsescripts/qapi-gen.py line 19
Quoted 10 lines
scripts/qapi-gen.py +2
-1import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visitUnrelated cleanup. Okay.
Quoted 17 lines
scripts/qapi-gen.py +15
-1-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.scripts/qapi-gen.py line 49
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)Uh...
Quoted 7 lines
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: : funny character '@' in prefix '@' Unwanted " :". This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).
Quoted 2 lines
Quack indeed - why does our base error class require so much information from a specific part of the generation process?
Because it's not "a base error class", it's a base error class for the QAPI schema compiler frontend.
Quoted 2 lines
Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.
Evolving it to satisfy a need for a more widely usable error class is okay.
Quoted 7 lines
You are witnessing some more future-bleed.
Quoted 3 lines
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?
Yes, because that's where you can emit the better error message more easily.
$ python3 scripts/qapi-gen.py --prefix=@ x
scripts/qapi-gen.py: 'funny character '@' in argument of --prefixis better than
$ python3 scripts/qapi-gen.py --prefix=@ x
scripts/qapi-gen.py: funny character '@' in prefix '@'In generate(), the knowledge where the offending prefix value comes from is no longer available.
To emit this error message, you'd have to raise a sufficiently distinct error in generate, catch it in main(), then put the error message together somehow. Bah.
Aside: there's a stray ' in the old error message.
Quoted 3 lines
The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.
YAGNI.
Quoted 2 lines
Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.
"@prefix is sane" is a precondition of generate().
When there's a real risk of preconditions getting violated, or readers getting confused about preconditions, check them with assert.
Quoted 32 lines
Quoted 22 lines
Quoted 22 lines
scripts/qapi-gen.py +17
-1+ + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store',Quoted 9 lines
scripts/qapi-gen.py line 75
Quoted 6 lines
scripts/qapi-gen.py +3
-1+ default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols")I don't like the changes to default=, because:
1. They are only losely related to the patch's purpose.
Quoted 8 lines
Subjective, but OK.
Quoted 3 lines
2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.
All of it is in
main.py, though! If you were to, say, movegenerate()elsewhere, it'd look pretty compact as just the CLI frontend, no?
Same statement is more compact than same screenful is more compact than same file :)
Quoted 44 lines
Quoted 2 lines
3. The defaults will not change, and nothing else uses the constants.
But, fine. Cleber had the same comment but I wasn't fully on-board, but two folks saying the same thing ...
Quoted 28 lines
Quoted 28 lines
scripts/qapi-gen.py +5
-14parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix)Quoted 9 lines
scripts/qapi-gen.py def main(argv): · line 91
Quoted 4 lines
scripts/qapi-gen.py +3
-1- gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0Subtle change: you move the
gen_FOO()into the try ... except. Okay;
they don't raise QAPIError, but perhaps worth a mention in the commit message.
Quoted 1 line
Forbidden future knowledge; I intend them to.
I don't mind the move.
Quoted 9 lines
Quoted 7 lines
scripts/qapi-gen.py def main(argv): · line 95
Quoted 4 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())"Python was designed to be easy to understand and fun to use." Ha ha ha.
I mean, I'm having fun, aren't you?
So many kinds of fun! The fun I'm having with this patch hunk is mocking "easy and fun" Python for requiring such an elaborate menuett just to express "can run as program".
This emperor has no clothes, either. And that's funny, isn't it?
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 81 lines · Markus Armbruster
Quoted 79 lines · John Snow
Quoted 77 lines · Markus Armbruster
Quoted 28 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +5
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparsescripts/qapi-gen.py line 19
Quoted 10 lines
scripts/qapi-gen.py +2
-1import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visitUnrelated cleanup. Okay.
Quoted 17 lines
scripts/qapi-gen.py +15
-1-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.scripts/qapi-gen.py line 49
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)Uh...
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: : funny character '@' in prefix '@' Unwanted " :". This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).
Quoted 6 lines
Quoted 2 lines
Quack indeed - why does our base error class require so much information from a specific part of the generation process?
Because it's not "a base error class", it's a base error class for the QAPI schema compiler frontend.
Well. It's the base for every error we /had/.
Quoted 6 lines
Quoted 2 lines
Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.
Evolving it to satisfy a need for a more widely usable error class is okay.
Yep. It's helpful to keep a very generic form on which we grow other errors from, so that things like the entry point can be written legibly.
Quoted 26 lines
Quoted 7 lines
You are witnessing some more future-bleed.
Quoted 3 lines
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?
Yes, because that's where you can emit the better error message more easily.
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: 'funny character '@' in argument of --prefixis better than
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: funny character '@' in prefix '@'In
generate(), the knowledge where the offending prefix value comes from is no longer available.To emit this error message, you'd have to raise a sufficiently distinct error in generate, catch it in
main(), then put the error message together somehow. Bah.
Quoted 8 lines
Aside: there's a stray ' in the old error message.
Quoted 3 lines
The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.
YAGNI.
It's useful for testing and debugging to be able to just call it outside of the CLI, though. Maybe you won't use it, but I will.
I could always add the prefix check into a tiny function and give the good error message in main(), and just assert in generate() if you insist on the slightly more specific error message from the CLI script.
Quoted 116 lines
Quoted 2 lines
Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.
"@prefix is sane" is a precondition of
generate().When there's a real risk of preconditions getting violated, or readers getting confused about preconditions, check them with assert.
Quoted 32 lines
Quoted 22 lines
Quoted 22 lines
scripts/qapi-gen.py +17
-1+ + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store',Quoted 9 lines
scripts/qapi-gen.py line 75
Quoted 6 lines
scripts/qapi-gen.py +3
-1+ default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols")I don't like the changes to default=, because:
1. They are only losely related to the patch's purpose.Quoted 8 lines
Subjective, but OK.
Quoted 3 lines
2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.
All of it is in
main.py, though! If you were to, say, movegenerate()elsewhere, it'd look pretty compact as just the CLI frontend, no?Same statement is more compact than same screenful is more compact than same file :)
Quoted 44 lines
Quoted 2 lines
3. The defaults will not change, and nothing else uses the constants.
But, fine. Cleber had the same comment but I wasn't fully on-board, but two folks saying the same thing ...
Quoted 28 lines
Quoted 28 lines
scripts/qapi-gen.py +5
-14parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix)Quoted 9 lines
scripts/qapi-gen.py def main(argv): · line 91
Quoted 4 lines
scripts/qapi-gen.py +3
-1- gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0Subtle change: you move the
gen_FOO()into the try ... except. Okay;
they don't raise QAPIError, but perhaps worth a mention in the commit message.Quoted 1 line
Forbidden future knowledge; I intend them to.
I don't mind the move.
Quoted 9 lines
Quoted 7 lines
scripts/qapi-gen.py def main(argv): · line 95
Quoted 4 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())"Python was designed to be easy to understand and fun to use." Ha ha ha.
I mean, I'm having fun, aren't you?
So many kinds of fun! The fun I'm having with this patch hunk is mocking "easy and fun" Python for requiring such an elaborate menuett just to express "can run as program".
This emperor has no clothes, either. And that's funny, isn't it?
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 81 lines · John Snow
Quoted 80 lines · Markus Armbruster
Quoted 78 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 28 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +5
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparsescripts/qapi-gen.py line 19
Quoted 10 lines
scripts/qapi-gen.py +2
-1import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visitUnrelated cleanup. Okay.
Quoted 16 lines
scripts/qapi-gen.py +15
-0-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.scripts/qapi-gen.py line 49
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)Uh...
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: : funny character '@' in prefix '@' Unwanted " :". This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quack indeed - why does our base error class require so much information from a specific part of the generation process?
Because it's not "a base error class", it's a base error class for the
QAPI schema compiler frontend.Well. It's the base for every error we /had/.
You asked why the class has the init it has, and I answered :)
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.
Evolving it to satisfy a need for a more widely usable error class is
okay.Yep. It's helpful to keep a very generic form on which we grow other errors from, so that things like the entry point can be written legibly.
If you have a non-trivial error message format convention, you have a use for a function formatting error messages.
If you have a separation between diagnose and report of errors, you you have a use for a transport from diagnose to report. In Python, that's raise.
The existing error message in main() has neither.
The existing error class QAPIError caters for the existing users.
Quoted 23 lines
Quoted 23 lines
Quoted 7 lines
You are witnessing some more future-bleed.
Quoted 3 lines
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?
Yes, because that's where you can emit the better error message more easily.
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: 'funny character '@' in argument of --prefix is better than $ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: funny character '@' in prefix '@' In generate(), the knowledge where the offending prefix value comes from is no longer available. To emit this error message, you'd have to raise a sufficiently distinct error in generate, catch it in main(), then put the error message together somehow. Bah. Aside: there's a stray ' in the old error message.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.
YAGNI.
It's useful for testing and debugging to be able to just call it outside of the CLI, though. Maybe you won't use it, but I will.
For testing and debugging, treating "prefix is sane" as a precondition is fine. I wouldn't even bother checking it. A check would catch accidents, and these accidents seem vanishingly unlikely to me.
Evidence: we did without *any* prefix checking for *years*. I added it in commit 1cf47a15f18 just for completeness.
Quoted 3 lines
I could always add the prefix check into a tiny function and give the good error message in
main(), and just assert ingenerate()if you insist on the slightly more specific error message from the CLI script.
If you genuinely think a check is needed there, that's the way to go.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.
"@prefix is sane" is a precondition of
generate(). When there's a real risk of preconditions getting violated, or readers
getting confused about preconditions, check them with assert.
[...]
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 152 lines · Markus Armbruster
Quoted 81 lines · John Snow
Quoted 80 lines · Markus Armbruster
Quoted 78 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 28 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +5
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparsescripts/qapi-gen.py line 19
Quoted 10 lines
scripts/qapi-gen.py +2
-1import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visitUnrelated cleanup. Okay.
Quoted 16 lines
scripts/qapi-gen.py +15
-0-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.scripts/qapi-gen.py line 49
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)Uh...
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: : funny character '@' in prefix '@' Unwanted " :". This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quack indeed - why does our base error class require so much information from a specific part of the generation process?
Because it's not "a base error class", it's a base error class for the
QAPI schema compiler frontend.Well. It's the base for every error we /had/.
You asked why the class has the init it has, and I answered :)
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.
Evolving it to satisfy a need for a more widely usable error class is
okay.Yep. It's helpful to keep a very generic form on which we grow other errors from, so that things like the entry point can be written legibly.
If you have a non-trivial error message format convention, you have a use for a function formatting error messages.
If you have a separation between diagnose and report of errors, you you have a use for a transport from diagnose to report. In Python, that's raise.
The existing error message in
main()has neither.The existing error class QAPIError caters for the existing users.
Quoted 23 lines
Quoted 23 lines
Quoted 7 lines
You are witnessing some more future-bleed.
Quoted 3 lines
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?
Yes, because that's where you can emit the better error message more easily.
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: 'funny character '@' in argument of --prefix is better than $ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: funny character '@' in prefix '@' In generate(), the knowledge where the offending prefix value comes from is no longer available. To emit this error message, you'd have to raise a sufficiently distinct error in generate, catch it in main(), then put the error message together somehow. Bah. Aside: there's a stray ' in the old error message.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.
YAGNI.
It's useful for testing and debugging to be able to just call it outside of the CLI, though. Maybe you won't use it, but I will.
For testing and debugging, treating "prefix is sane" as a precondition is fine. I wouldn't even bother checking it. A check would catch accidents, and these accidents seem vanishingly unlikely to me.
Quoted 9 lines
Evidence: we did without *any* prefix checking for *years*. I added it in commit
1cf47a15f18just for completeness.Quoted 3 lines
I could always add the prefix check into a tiny function and give the good error message in
main(), and just assert ingenerate()if you insist on the slightly more specific error message from the CLI script.If you genuinely think a check is needed there, that's the way to go.
A goal of this patch in particular is the clean separation of the CLI frontend from the implementation logic behind it, such that you have:
(1) The CLI parsing, and
(2) A function implementing a task the package is capable of performing
The CLI parsing method converts sys.argv arguments into function parameter data that can be acted upon by another function in the package.
If checks are exclusive to the process of converting flags to argument data, they should stay in the CLI parsing function.
If checks prevent you from passing bad data lower into the program, I prefer pushing that check lower into the stack instead, such that the module is equivalently useful without the CLI frontend.
This conceptualization might be different from how you perceive QAPI. In my case, I consider the public functions and modules themselves to be interface that should behave reasonably and with some defensiveness. You might only count the CLI script itself in that category.
In this case, the prefix is passed to many different generators, so it seems good to check it before we hand it off to all those different places.
You don't like the loss of precision that causes for the CLI frontend error, which is reasonable.
I consider the function interface and the CLI frontend "equivalent", and so I found it odd to reject bad data in one, but not the other. IOW, I do not consider the CLI frontend the only user of generate().
I think it's minor in this case, sure. I am fine with saying "A valid prefix is a precondition", but then I want to document and enforce that precondition -- so I did wind up using an assert in my pending respin.
--js
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.
"@prefix is sane" is a precondition of
generate(). When there's a real risk of preconditions getting violated, or readers
getting confused about preconditions, check them with assert.[...]
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
In reply to Markus ArmbrusterQuoted 165 lines · Markus Armbruster
Quoted 31 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +6
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import rescripts/qapi-gen.py line 22
Quoted 10 lines
scripts/qapi-gen.py +2
-1import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visitUnrelated cleanup. Okay.
Quoted 18 lines
scripts/qapi-gen.py +15
-1-def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names.scripts/qapi-gen.py line 47
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg)Uh...
$ python3 scripts/qapi-gen.py --prefix=@ x scripts/qapi-gen.py: : funny character '@' in prefix '@'Unwanted " :".
This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).
Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.
Quoted 18 lines
scripts/qapi-gen.py +16
-0+ + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint. + Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema')scripts/qapi-gen.py line 73
Quoted 10 lines
scripts/qapi-gen.py +4
-2parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols")I don't like the changes to default=, because:
1. They are only losely related to the patch's purpose.
2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.
3. The defaults will not change, and nothing else uses the constants.
Quoted 22 lines
scripts/qapi-gen.py +5
-9parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr)scripts/qapi-gen.py def main(argv): · line 91
Quoted 10 lines
scripts/qapi-gen.py +3
-7- exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0Subtle change: you move the
gen_FOO()into the try ... except. Okay; they don't raise QAPIError, but perhaps worth a mention in the commit message.Quoted 2 lines
Quoted 7 lines
scripts/qapi-gen.py
Quoted 3 lines
if
__name__== '__main__':scripts/qapi-gen.py +1
-1- main(sys.argv) + sys.exit(main())"Python was designed to be easy to understand and fun to use." Ha ha ha.
Here's a Fixup that requires more (but obvious) Fixups to later patches in this series. (The next patch needs the same changes re-applied to the now-moved file.)
A note: I add a FIXME, but when I respin my error series, it will take care of this piece. This is just the quickest, dumbest way to victory in the interim. It is not worth engineering a better solution, because there is one waiting in part 4 already.
(Future knowledge: I add a QAPIError that does not define its arguments that inherits Exception and is used as the basis for all error classes in QAPI; and the contextual error that requires "info" is changed to inherit from QAPIError. This is useful as a "package-level" error, one in which that *all* errors in this package inherit from. We already achieve that, but we needed a more general form to serve as the root of that inheritance tree.)
scripts/qapi-gen.py +2 -6
@@ -22,10 +22,6 @@ from qapi.visit import gen_visit -DEFAULT_OUTPUT_DIR = '' -DEFAULT_PREFIX = '' - - def generate(schema_file: str, output_dir: str, prefix: str,
@@ -68,10 +64,10 @@ def main() -> int: parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") parser.add_argument('-o', '--output-dir', action='store', - default=DEFAULT_OUTPUT_DIR, + default='', help="write output to directory OUTPUT_DIR") parser.add_argument('-p', '--prefix', action='store', - default=DEFAULT_PREFIX, + default='', help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names',
action='store_true',
dest='unmask',
scripts/qapi/error.py +3 -0
@@ -21,6 +21,9 @@ def __init__(self, info, col, msg): def __str__(self): + # FIXME: We need a general-purpose no-context error class. + if not self.info: + return self.msg loc = str(self.info) if self.col is not None: assert self.info.line is not None loc += ':%s' % self.col
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
In reply to John SnowQuoted 71 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +35
-4@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file)
scripts/qapi-gen.py line 61
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint.
What's a "shell script entrypoint"?
Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html
Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts
Quoted 43 lines
scripts/qapi-gen.py +13
-16+ Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix)
scripts/qapi-gen.py def main(argv): · line 95
Quoted 10 lines
scripts/qapi-gen.py +4
-3- gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0 if __name__ == '__main__': - main(sys.argv) + sys.exit(main())
What does sys.exit() really buy us here? I'm asking because both spots in the Python docs I referenced above do without.
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 86 lines · Markus Armbruster
Quoted 71 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +35
-4@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file)scripts/qapi-gen.py line 61
Quoted 10 lines
scripts/qapi-gen.py +10
-0+ gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + + +def main() -> int: + """ + gapi-gen shell script entrypoint.What's a "shell script entrypoint"?
Quoted 6 lines
Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.htmlSimilar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts
"entrypoint" is Python garble for a function that can be registered as a callable from the command line.
So in a theoretical setup.py, you'd do something like:
'entry_points': {
'console_scripts': [
'qapi-gen = qapi.main:main',
]
}
so when I say "shell script entrypoint", I am referring to a shell script (I mean: it has a shebang and can be executed by an interactive shell process) that calls the entrypoint.
Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to just generate that script.
(i.e. doing pip install qemu.qapi will install a 'qapi-gen' CLI script for you. this is how packages like sphinx create the 'sphinx-build' script, etc.)
Quoted 50 lines
Quoted 50 lines
scripts/qapi-gen.py +16
-18+ Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0
Quoted 7 lines
scripts/qapi-gen.py def main(argv): · line 95
Quoted 3 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())What does
sys.exit()really buy us here? I'm asking because both spots in the Python docs I referenced above do without.
It just pushes the sys.exit out of the main function so it can be invoked by other machinery. (And takes the return code from main and turns it into the return code for the process.)
I don't think it winds up mattering for simple "console_script" entry points, but you don't want the called function to exit and deny the caller the chance to do their own tidying post-call.
You've already offered a "YAGNI", but it's just the convention I tend to stick to for how to structure entry points.
--js
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 98 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 74 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +41
-4@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + +Quoted 9 lines
scripts/qapi-gen.py line 61
Quoted 3 lines
scripts/qapi-gen.py +3
-0+def main() -> int: + """ + gapi-gen shell script entrypoint.What's a "shell script entrypoint"?
Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html
Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts"entrypoint" is Python garble for a function that can be registered as a callable from the command line.
So in a theoretical
setup.py, you'd do something like:'
entry_points': {
'console_scripts': [
'qapi-gen = qapi.main:main',
]
}
Quoted 3 lines
so when I say "shell script entrypoint", I am referring to a shell script (I mean: it has a shebang and can be executed by an interactive shell process) that calls the entrypoint.
It can be executed by any process. See execve(2):
pathname must be either a binary executable, or a script starting with a line of the form:
#!interpreter [optional-arg]
For details of the latter case, see "Interpreter scripts" below.
"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.
Quoted 68 lines
Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to just generate that script.
(i.e. doing
pip install qemu.qapiwill install a 'qapi-gen' CLI script for you. this is how packages like sphinx create the 'sphinx-build' script, etc.)Quoted 49 lines
Quoted 49 lines
scripts/qapi-gen.py +16
-17+ Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0Quoted 7 lines
scripts/qapi-gen.py def main(argv): · line 95
Quoted 3 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())What does
sys.exit()really buy us here? I'm asking because both spots
in the Python docs I referenced above do without.It just pushes the sys.exit out of the main function so it can be invoked by other machinery. (And takes the return code from main and turns it into the return code for the process.)
Quoted 6 lines
I don't think it winds up mattering for simple "
console_script" entry points, but you don't want the called function to exit and deny the caller the chance to do their own tidying post-call.You've already offered a "YAGNI", but it's just the convention I tend to stick to for how to structure entry points.
I'm not questioning the conventional if __name__ == '__main__' menuett. I wonder why *we* need sys.exit() where the examples in the Python docs don't.
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 189 lines · Markus Armbruster
Quoted 98 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 74 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +41
-4@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema. +""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory. + + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """ + match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) + if match.end() != len(prefix): + msg = "funny character '{:s}' in prefix '{:s}'".format( + prefix[match.end()], prefix) + raise QAPIError('', None, msg) + + schema = QAPISchema(schema_file) + gen_types(schema, output_dir, prefix, builtins) + gen_visit(schema, output_dir, prefix, builtins) + gen_commands(schema, output_dir, prefix) + gen_events(schema, output_dir, prefix) + gen_introspect(schema, output_dir, prefix, unmask) + +Quoted 9 lines
scripts/qapi-gen.py line 61
Quoted 3 lines
scripts/qapi-gen.py +3
-0+def main() -> int: + """ + gapi-gen shell script entrypoint.What's a "shell script entrypoint"?
Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html
Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts"entrypoint" is Python garble for a function that can be registered as a callable from the command line.
So in a theoretical
setup.py, you'd do something like:'
entry_points': {
'console_scripts': [
'qapi-gen = qapi.main:main',
]
}Quoted 3 lines
so when I say "shell script entrypoint", I am referring to a shell script (I mean: it has a shebang and can be executed by an interactive shell process) that calls the entrypoint.
It can be executed by any process. See execve(2):
pathname must be either a binary executable, or a script starting with a line of the form:
#!interpreter [optional-arg]
For details of the latter case, see "Interpreter scripts" below.
"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.
Quoted 72 lines
Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to just generate that script.
(i.e. doing
pip install qemu.qapiwill install a 'qapi-gen' CLI script for you. this is how packages like sphinx create the 'sphinx-build' script, etc.)Quoted 49 lines
Quoted 49 lines
scripts/qapi-gen.py +16
-17+ Expects arguments via sys.argv, see --help for details. + + :return: int, 0 on success, 1 on failure. + """ parser = argparse.ArgumentParser( description='Generate code from a QAPI schema') parser.add_argument('-b', '--builtins', action='store_true', help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', default='', + parser.add_argument('-o', '--output-dir', action='store', + default=DEFAULT_OUTPUT_DIR, help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', default='', + parser.add_argument('-p', '--prefix', action='store', + default=DEFAULT_PREFIX, help="prefix for symbols") parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', dest='unmask',@@ -32,25 +79,17 @@ def main(argv): parser.add_argument('schema', action='store') args = parser.parse_args() - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix) - if match.end() != len(args.prefix): - print("%s: 'funny character '%s' in argument of --prefix" - % (sys.argv[0], args.prefix[match.end()]), - file=sys.stderr) - sys.exit(1) - try: - schema = QAPISchema(args.schema) + generate(args.schema, + output_dir=args.output_dir, + prefix=args.prefix, + unmask=args.unmask, + builtins=args.builtins) except QAPIError as err: - print(err, file=sys.stderr) - exit(1) - - gen_types(schema, args.output_dir, args.prefix, args.builtins) - gen_visit(schema, args.output_dir, args.prefix, args.builtins) - gen_commands(schema, args.output_dir, args.prefix) - gen_events(schema, args.output_dir, args.prefix) - gen_introspect(schema, args.output_dir, args.prefix, args.unmask) + print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) + return 1 + return 0Quoted 7 lines
scripts/qapi-gen.py def main(argv): · line 95
Quoted 3 lines
scripts/qapi-gen.py +1
-1if __name__ == '__main__': - main(sys.argv) + sys.exit(main())What does
sys.exit()really buy us here? I'm asking because both spots
in the Python docs I referenced above do without.It just pushes the sys.exit out of the main function so it can be invoked by other machinery. (And takes the return code from main and turns it into the return code for the process.)
I don't think it winds up mattering for simple "
console_script" entry points, but you don't want the called function to exit and deny the caller the chance to do their own tidying post-call.
Quoted 7 lines
Quoted 2 lines
You've already offered a "YAGNI", but it's just the convention I tend to stick to for how to structure entry points.
I'm not questioning the conventional if
__name__== '__main__' menuett. I wonder why *we* needsys.exit()where the examples in the Python docs don't.
I assume they don't care about setting that return code manually. By default it's going to be 0 on normal exit, and non-zero if it raises or aborts.
We catch the error to pretty-print it to console instead, and want to avoid the stack trace so we return a status code instead.
I assume that's the only difference, really.
--js
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
In reply to Markus ArmbrusterQuoted 5 lines · Markus Armbruster
It can be executed by any process. See execve(2):
pathname must be either a binary executable, or a script starting with a line of the form:
Quoted 7 lines
#!interpreter [optional-arg]
For details of the latter case, see "Interpreter scripts" below.
"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.
You know, I don't think I was actually explicitly aware that the #! shebang was not something the shell actually processed itself. Always learning new things.
(No, I don't think I have ever execve'd something that wasn't a binary.)
"entry point" is a little vague, an entry point for what? by whom? I was trying to call attention to the idea specifically that main() was intended as python's "console script entry point", but used the word "shell" instead.
"console script entrypoint" is also a lot of jargon. What I really want to communicate is: "When you run qapi-gen on your command-line, this is the function that runs!"
So I guess something like:
"qapi-gen executable entry point." will suffice. Please further adjust to your liking when staging.
--js
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 13 lines · John Snow
Quoted 11 lines · Markus Armbruster
It can be executed by any process. See execve(2):
pathname must be either a binary executable, or a script starting with a line of the form:
#!interpreter [optional-arg]
For details of the latter case, see "Interpreter scripts"
below."Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.
Quoted 5 lines
You know, I don't think I was actually explicitly aware that the #! shebang was not something the shell actually processed itself. Always learning new things.
(No, I don't think I have ever execve'd something that wasn't a binary.)
I'm sure you've done it countless times, just not knowingly :)
Quoted 6 lines
"entry point" is a little vague, an entry point for what? by whom? I was trying to call attention to the idea specifically that
main()was intended as python's "console script entry point", but used the word "shell" instead."console script entrypoint" is also a lot of jargon. What I really
It is.
If I didn't find "console script" so thoroughly misguided, I'd advie you to stick to it just because it's what we found in Python's docs. It's misguided, because this entry point is the one and only entry point for *any* kind of Python executable program, be it console, GUI, or whatever.
Quoted 7 lines
want to communicate is: "When you run
qapi-genon your command-line, this is the function that runs!"So I guess something like:
"qapi-gen executable entry point." will suffice. Please further adjust to your liking when staging.
Works for me.
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
In reply to John SnowI keep stumbling over things in later patches that turn out to go back to this one.
Quoted 17 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +0
-0@@ -1,30 +1,77 @@ #!/usr/bin/env python3
scripts/qapi-gen.py line 9
Quoted 10 lines
scripts/qapi-gen.py +5
-2-# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema.
PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.
Quoted 18 lines
scripts/qapi-gen.py +4
-2+""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = ''
scripts/qapi-gen.py line 35
Quoted 10 lines
scripts/qapi-gen.py +10
-0+DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory.
PEP 257: The docstring is a phrase ending in a period. It prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...".
Suggest
Generate C code for the given schema into the target directory.
Quoted 9 lines
scripts/qapi-gen.py +9
-0+ + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """
[...]
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 28 lines · Markus Armbruster
I keep stumbling over things in later patches that turn out to go back to this one.
Quoted 23 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +1
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory.
Quoted 9 lines
scripts/qapi-gen.py line 9
Quoted 4 lines
scripts/qapi-gen.py +4
-0+""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema.PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.
Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.
("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")
Quoted 33 lines
Quoted 18 lines
scripts/qapi-gen.py +4
-2+""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = ''scripts/qapi-gen.py line 35
Quoted 10 lines
scripts/qapi-gen.py +10
-0+DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """ + generate uses a given schema to produce C code in the target directory.PEP 257: The docstring is a phrase ending in a period. It prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...".
Quoted 4 lines
Suggest
Generate C code for the given schema into the target directory.
OK. I don't mind trying to foster a consistent tone. I clearly didn't. I will add a note to my style guide todo.
I give you permission to change the voice in any of my docstrings, or to adjust the phrasing to be more technically accurate as you see fit. You are the primary maintainer of this code, of course, and you will know best.
It will be quicker to give you full permission to just change any of the docstrings as you see fit than it will be to play review-respin ping-pong.
Quoted 11 lines
Quoted 9 lines
scripts/qapi-gen.py +9
-0+ + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """[...]
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 36 lines · John Snow
Quoted 25 lines · Markus Armbruster
I keep stumbling over things in later patches that turn out to go back to this one.
Quoted 21 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +1
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later.Quoted 9 lines
scripts/qapi-gen.py line 9
Quoted 5 lines
scripts/qapi-gen.py +3
-0# See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema.PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.
Quoted 8 lines
Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.
("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")
Emacs Python mode is close enough by default: fill-paragraph (bound to M-q) uses variable fill-column, which defaults to 70. If you want the extra two columns PEP 8 grants you, I can show you how to bump it to 72 just for Python mode.
You can use fill-paragraph for code, too. I don't myself, because I disagree with its line breaking decisions too often (and so does PEP 8). A better Python mode would break code lines more neatly, and with the width defaulting to 79.
Quoted 37 lines
Quoted 24 lines
Quoted 24 lines
scripts/qapi-gen.py +13
-2+""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """Quoted 9 lines
scripts/qapi-gen.py line 35
Quoted 1 line
scripts/qapi-gen.py +1
-0+ generate uses a given schema to produce C code in the target directory.PEP 257: The docstring is a phrase ending in a period. It prescribes
the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...". Suggest
Generate C code for the given schema into the target
directory.OK. I don't mind trying to foster a consistent tone. I clearly didn't. I will add a note to my style guide todo.
Quoted 8 lines
I give you permission to change the voice in any of my docstrings, or to adjust the phrasing to be more technically accurate as you see fit. You are the primary maintainer of this code, of course, and you will know best.
It will be quicker to give you full permission to just change any of the docstrings as you see fit than it will be to play review-respin ping-pong.
Me rewriting your commits without your consent is putting words in your mouth, which I don't want to do.
We can still reduce ping-pong: whenever I can, I don't just say "this needs improvement", I propose improvements. If you disagree, we talk. Else, if you have to respin, you make a reasonable effort to take them. Else, the remaining improvements are trivial (because no respin), and I'll make them in my tree.
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
scripts/qapi-gen.py +9
-0+ + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """[...]
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 52 lines · Markus Armbruster
Quoted 36 lines · John Snow
Quoted 25 lines · Markus Armbruster
I keep stumbling over things in later patches that turn out to go back to this one.
Quoted 21 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +1
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later.Quoted 9 lines
scripts/qapi-gen.py line 9
Quoted 5 lines
scripts/qapi-gen.py +3
-0# See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema.PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.
Quoted 8 lines
Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.
("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")
Emacs Python mode is close enough by default: fill-paragraph (bound to M-q) uses variable fill-column, which defaults to 70. If you want the extra two columns PEP 8 grants you, I can show you how to bump it to 72 just for Python mode.
Quoted 5 lines
You can use fill-paragraph for code, too. I don't myself, because I disagree with its line breaking decisions too often (and so does PEP 8). A better Python mode would break code lines more neatly, and with the width defaulting to 79.
Yeah, how do I set the reflow to 72 for specific modes?
I tend to do a lot of refactoring and "prototyping" in Pycharm, but when it comes to bread and butter edits I still prefer emacs. I kinda bounce between 'em a lot. Having emacs DTRT is still useful to me.
Quoted 46 lines
Quoted 37 lines
Quoted 24 lines
Quoted 24 lines
scripts/qapi-gen.py +13
-2+""" import argparse import re import sys from qapi.commands import gen_commands +from qapi.error import QAPIError from qapi.events import gen_events from qapi.introspect import gen_introspect -from qapi.schema import QAPIError, QAPISchema +from qapi.schema import QAPISchema from qapi.types import gen_types from qapi.visit import gen_visit -def main(argv): +DEFAULT_OUTPUT_DIR = '' +DEFAULT_PREFIX = '' + + +def generate(schema_file: str, + output_dir: str, + prefix: str, + unmask: bool = False, + builtins: bool = False) -> None: + """Quoted 9 lines
scripts/qapi-gen.py line 35
Quoted 1 line
scripts/qapi-gen.py +1
-0+ generate uses a given schema to produce C code in the target directory.PEP 257: The docstring is a phrase ending in a period. It prescribes
the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...". Suggest
Generate C code for the given schema into the target
directory.OK. I don't mind trying to foster a consistent tone. I clearly didn't. I will add a note to my style guide todo.
Quoted 8 lines
I give you permission to change the voice in any of my docstrings, or to adjust the phrasing to be more technically accurate as you see fit. You are the primary maintainer of this code, of course, and you will know best.
It will be quicker to give you full permission to just change any of the docstrings as you see fit than it will be to play review-respin ping-pong.
Quoted 9 lines
Me rewriting your commits without your consent is putting words in your mouth, which I don't want to do.
We can still reduce ping-pong: whenever I can, I don't just say "this needs improvement", I propose improvements. If you disagree, we talk. Else, if you have to respin, you make a reasonable effort to take them. Else, the remaining improvements are trivial (because no respin), and I'll make them in my tree.
I appreciate the consideration. They're not just "my" words, though, they are "our" words!
I do give you permission to touch up things like punctuation, voice, etc. If I dislike the changes I can always yelp at you post-hoc when you post the PR email.
So I'll reiterate that I am definitely happy with such changes -- the precision of the technical writing here is not my strong suit as I do not know QAPI as intimately as you, and it's not the focus of this series. I don't think of it as a personal offense that someone would copy-edit these things.
If you're not comfortable doing it, that's OK too, but I'm definitely okay with things like comment and docstring edits in particular. It's just something that feels hard to predict.
(Anyway, I made this change. There are likely other voice and line length changes needed, but I think I will try to address those systematically when I lift missing-docstring for pylint. Probably I will craft a series that creates a style guide, enables sphinx, lifts missing-docstring, and performs this same style of cleanup, going module-by-module.)
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
scripts/qapi-gen.py +9
-0+ + :param schema_file: The primary QAPI schema file. + :param output_dir: The output directory to store generated code. + :param prefix: Optional C-code prefix for symbol names. + :param unmask: Expose non-ABI names through introspection? + :param builtins: Generate code for built-in types? + + :raise QAPIError: On failures. + """[...]
Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation
Quoted 58 lines · John Snow
Quoted 52 lines · Markus Armbruster
Quoted 36 lines · John Snow
Quoted 25 lines · Markus Armbruster
I keep stumbling over things in later patches that turn out to go back to this one.
Quoted 21 lines · John Snow
This is a minor re-work of the entrypoint script. It isolates a
generate()method from the actual command-line mechanism.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 23 deletions(-)scripts/qapi-gen.py +1
-2@@ -1,30 +1,77 @@ #!/usr/bin/env python3 -# QAPI generator -# + # This work is licensed under the terms of the GNU GPL, version 2 or later.Quoted 9 lines
scripts/qapi-gen.py line 9
Quoted 5 lines
scripts/qapi-gen.py +3
-0# See the COPYING file in the top-level directory. +""" +QAPI Generator + +This script is the main entry point for generating C code from the QAPI schema.PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.
Quoted 8 lines
Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.
("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")
Emacs Python mode is close enough by default: fill-paragraph (bound to M-q) uses variable fill-column, which defaults to 70. If you want the extra two columns PEP 8 grants you, I can show you how to bump it to 72 just for Python mode.
Quoted 4 lines
You can use fill-paragraph for code, too. I don't myself, because I disagree with its line breaking decisions too often (and so does PEP 8). A better Python mode would break code lines more neatly, and with the width defaulting to 79.
Quoted 5 lines
Yeah, how do I set the reflow to 72 for specific modes?
I tend to do a lot of refactoring and "prototyping" in Pycharm, but when it comes to bread and butter edits I still prefer emacs. I kinda bounce between 'em a lot. Having emacs DTRT is still useful to me.
In your .emacs:
(add-hook 'python-mode-hook
(lambda () (setq fill-column 72)))Happy to explain this in detail if you're curious.
[...]
PATCH v5 4/36 qapi: move generator entrypoint into module
In reply to John SnowAs part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the package, only leaving a very tiny entrypoint shim outside of the module.
--- scripts/qapi-gen.py | 88 +++---------------------------------------- scripts/qapi/main.py | 89 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 82 deletions(-) create mode 100644 scripts/qapi/main.py
scripts/qapi-gen.py +6 -82
@@ -4,92 +4,16 @@ # See the COPYING file in the top-level directory. """ -QAPI Generator +QAPI code generation execution shim. -This script is the main entry point for generating C code from the QAPI schema. +This standalone script exists primarily to facilitate the running of the QAPI +code generator without needing to install the python module to the current +execution environment. """ -import argparse -import re import sys -from qapi.commands import gen_commands -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit - - -DEFAULT_OUTPUT_DIR = '' -DEFAULT_PREFIX = '' - - -def generate(schema_file: str, - output_dir: str, - prefix: str, - unmask: bool = False, - builtins: bool = False) -> None: - """ - generate uses a given schema to produce C code in the target directory. - - :param schema_file: The primary QAPI schema file. - :param output_dir: The output directory to store generated code. - :param prefix: Optional C-code prefix for symbol names. - :param unmask: Expose non-ABI names through introspection? - :param builtins: Generate code for built-in types? - - :raise QAPIError: On failures. - """ - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) - if match.end() != len(prefix): - msg = "funny character '{:s}' in prefix '{:s}'".format( - prefix[match.end()], prefix) - raise QAPIError('', None, msg) - - schema = QAPISchema(schema_file) - gen_types(schema, output_dir, prefix, builtins) - gen_visit(schema, output_dir, prefix, builtins) - gen_commands(schema, output_dir, prefix) - gen_events(schema, output_dir, prefix) - gen_introspect(schema, output_dir, prefix, unmask) - - -def main() -> int: - """ - gapi-gen shell script entrypoint. - Expects arguments via sys.argv, see --help for details. - - :return: int, 0 on success, 1 on failure. - """ - parser = argparse.ArgumentParser( - description='Generate code from a QAPI schema') - parser.add_argument('-b', '--builtins', action='store_true', - help="generate code for built-in types") - parser.add_argument('-o', '--output-dir', action='store', - default=DEFAULT_OUTPUT_DIR, - help="write output to directory OUTPUT_DIR") - parser.add_argument('-p', '--prefix', action='store', - default=DEFAULT_PREFIX, - help="prefix for symbols") - parser.add_argument('-u', '--unmask-non-abi-names', action='store_true', - dest='unmask', - help="expose non-ABI names in introspection") - parser.add_argument('schema', action='store') - args = parser.parse_args() - - try: - generate(args.schema, - output_dir=args.output_dir, - prefix=args.prefix, - unmask=args.unmask, - builtins=args.builtins) - except QAPIError as err: - print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr) - return 1 - return 0 - +from qapi import main if __name__ == '__main__': - sys.exit(main()) + sys.exit(main.main())
scripts/qapi/main.py new file +89 -0
89 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v5 5/36 qapi: Prefer explicit relative imports
In reply to John SnowAll of the QAPI include statements are changed to be package-aware, as explicit relative imports.
A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.
When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.
This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.
--- scripts/qapi/commands.py | 4 ++-- scripts/qapi/events.py | 8 ++++---- scripts/qapi/expr.py | 4 ++-- scripts/qapi/gen.py | 4 ++-- scripts/qapi/introspect.py | 8 ++++---- scripts/qapi/main.py | 14 +++++++------- scripts/qapi/parser.py | 4 ++-- scripts/qapi/schema.py | 8 ++++---- scripts/qapi/types.py | 6 +++--- scripts/qapi/visit.py | 6 +++--- 10 files changed, 33 insertions(+), 33 deletions(-)
scripts/qapi/commands.py +2 -2
@@ -13,8 +13,8 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import * +from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/events.py +4 -4
@@ -12,10 +12,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember -from qapi.types import gen_enum, gen_enum_lookup +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember +from .types import gen_enum, gen_enum_lookup def build_event_send_proto(name, arg_type, boxed):
scripts/qapi/expr.py +2 -2
@@ -16,8 +16,8 @@ import re from collections import OrderedDict -from qapi.common import c_name -from qapi.error import QAPISemError +from .common import c_name +from .error import QAPISemError # Names must be letters, numbers, -, and _. They must start with letter,
scripts/qapi/gen.py +2 -2
@@ -17,8 +17,8 @@ import re from contextlib import contextmanager -from qapi.common import * -from qapi.schema import QAPISchemaVisitor +from .common import * +from .schema import QAPISchemaVisitor class QAPIGen:
scripts/qapi/introspect.py +4 -4
@@ -10,10 +10,10 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaMonolithicCVisitor -from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .common import * +from .gen import QAPISchemaMonolithicCVisitor +from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, + QAPISchemaType) def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/main.py +7 -7
@@ -11,13 +11,13 @@ import re import sys -from qapi.commands import gen_commands -from qapi.error import QAPIError -from qapi.events import gen_events -from qapi.introspect import gen_introspect -from qapi.schema import QAPISchema -from qapi.types import gen_types -from qapi.visit import gen_visit +from .commands import gen_commands +from .error import QAPIError +from .events import gen_events +from .introspect import gen_introspect +from .schema import QAPISchema +from .types import gen_types +from .visit import gen_visit DEFAULT_OUTPUT_DIR = ''
scripts/qapi/parser.py +2 -2
@@ -18,8 +18,8 @@ import re from collections import OrderedDict -from qapi.error import QAPIParseError, QAPISemError -from qapi.source import QAPISourceInfo +from .error import QAPIParseError, QAPISemError +from .source import QAPISourceInfo class QAPISchemaParser:
scripts/qapi/schema.py +4 -4
@@ -18,10 +18,10 @@ import re from collections import OrderedDict -from qapi.common import c_name, pointer_suffix -from qapi.error import QAPIError, QAPISemError -from qapi.expr import check_exprs -from qapi.parser import QAPISchemaParser +from .common import c_name, pointer_suffix +from .error import QAPIError, QAPISemError +from .expr import check_exprs +from .parser import QAPISchemaParser class QAPISchemaEntity:
scripts/qapi/types.py +3 -3
@@ -13,9 +13,9 @@ # See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaEnumMember, QAPISchemaObjectType # variants must be emitted before their container; track what has already
scripts/qapi/visit.py +3 -3
@@ -13,9 +13,9 @@ See the COPYING file in the top-level directory. """ -from qapi.common import * -from qapi.gen import QAPISchemaModularCVisitor, ifcontext -from qapi.schema import QAPISchemaObjectType +from .common import * +from .gen import QAPISchemaModularCVisitor, ifcontext +from .schema import QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
Signature · 1 line
2.26.2
Re: [PATCH v5 05/36] qapi: Prefer explicit relative imports
Quoted 33 lines · John Snow
All of the QAPI include statements are changed to be package-aware, as explicit relative imports.
A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.
When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.
This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>
Quoted 12 lines
--- scripts/qapi/commands.py | 4 ++-- scripts/qapi/events.py | 8 ++++---- scripts/qapi/expr.py | 4 ++-- scripts/qapi/gen.py | 4 ++-- scripts/qapi/introspect.py | 8 ++++---- scripts/qapi/main.py | 14 +++++++------- scripts/qapi/parser.py | 4 ++-- scripts/qapi/schema.py | 8 ++++---- scripts/qapi/types.py | 6 +++--- scripts/qapi/visit.py | 6 +++--- 10 files changed, 33 insertions(+), 33 deletions(-)
PATCH v5 6/36 qapi: Remove wildcard includes
In reply to John SnowWildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
--- scripts/qapi/commands.py | 2 +- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 38 insertions(+), 8 deletions(-)
scripts/qapi/commands.py +1 -1
@@ -13,7 +13,7 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import build_params, c_name, mcgen from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
scripts/qapi/events.py +6 -1
@@ -12,7 +12,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + build_params, + c_enum_const, + c_name, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +9 -3
@@ -11,13 +11,19 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. - +from contextlib import contextmanager import errno import os import re -from contextlib import contextmanager -from .common import * +from .common import ( + c_fname, + gen_endif, + gen_if, + guardend, + guardstart, + mcgen, +) from .schema import QAPISchemaVisitor
scripts/qapi/introspect.py +6 -1
@@ -10,7 +10,12 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaMonolithicCVisitor from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, QAPISchemaType)
scripts/qapi/types.py +7 -1
@@ -13,7 +13,13 @@ # See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
scripts/qapi/visit.py +9 -1
@@ -13,7 +13,15 @@ See the COPYING file in the top-level directory. """ -from .common import * +from .common import ( + c_enum_const, + c_name, + gen_endif, + gen_if, + mcgen, + pop_indent, + push_indent, +) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
Signature · 1 line
2.26.2
Re: [PATCH v5 06/36] qapi: Remove wildcard includes
Quoted 11 lines · John Snow
Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>
Quoted 8 lines
--- scripts/qapi/commands.py | 2 +- scripts/qapi/events.py | 7 ++++++- scripts/qapi/gen.py | 12 +++++++++--- scripts/qapi/introspect.py | 7 ++++++- scripts/qapi/types.py | 8 +++++++- scripts/qapi/visit.py | 10 +++++++++- 6 files changed, 38 insertions(+), 8 deletions(-)
PATCH v5 7/36 qapi: enforce import order/styling with isort
In reply to John SnowWhile we're mucking around with imports, we might as well formalize the style we use. Let's use isort to do it for us.
lines_after_imports=2: Use two lines after imports, to match PEP8's desire to have "two lines before and after" class definitions, which are likely to start immediately after imports.
force_sort_within_sections: Intermingles "from x" and "import x" style statements, such that sorting is always performed strictly on the module name itself.
force_grid_wrap=4: Four or more imports from a single module will force the one-per-line style that's more git-friendly. This will generally happen for 'typing' imports.
multi_line_output=3: Uses the one-per-line indented style for long imports.
include_trailing_comma: Adds a comma to the last import in a group, which makes git conflicts nicer to deal with, generally.
line_length: 72 is chosen to match PEP8's "docstrings and comments" line length limit. If you have a single line import that exceeds 72 characters, your names are too long!
--- scripts/qapi/.isort.cfg | 7 +++++++ scripts/qapi/expr.py | 3 ++- scripts/qapi/introspect.py | 7 +++++-- scripts/qapi/parser.py | 2 +- scripts/qapi/schema.py | 2 +- 5 files changed, 16 insertions(+), 5 deletions(-) create mode 100644 scripts/qapi/.isort.cfg
scripts/qapi/.isort.cfg new file +7 -0
@@ -0,0 +1,7 @@ +[settings] +force_grid_wrap=4 +force_sort_within_sections=True +include_trailing_comma=True +line_length=72 +lines_after_imports=2 +multi_line_output=3
scripts/qapi/expr.py +2 -1
@@ -14,8 +14,9 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -import re from collections import OrderedDict +import re + from .common import c_name from .error import QAPISemError
scripts/qapi/introspect.py +5 -2
@@ -17,8 +17,11 @@ mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .schema import ( + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaType, +) def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/parser.py +1 -1
@@ -14,9 +14,9 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. +from collections import OrderedDict import os import re -from collections import OrderedDict from .error import QAPIParseError, QAPISemError from .source import QAPISourceInfo
scripts/qapi/schema.py +1 -1
@@ -14,9 +14,9 @@ # TODO catching name collisions in generated code would be nice +from collections import OrderedDict import os import re -from collections import OrderedDict from .common import c_name, pointer_suffix from .error import QAPIError, QAPISemError
Signature · 1 line
2.26.2
Re: [PATCH v5 07/36] qapi: enforce import order/styling with isort
Quoted 44 lines · John Snow
While we're mucking around with imports, we might as well formalize the style we use. Let's use isort to do it for us.
lines_after_imports=2: Use two lines after imports, to match PEP8's desire to have "two lines before and after" class definitions, which are likely to start immediately after imports.
force_sort_within_sections: Intermingles "from x" and "import x" style statements, such that sorting is always performed strictly on the module name itself.force_grid_wrap=4: Four or more imports from a single module will force the one-per-line style that's more git-friendly. This will generally happen for 'typing' imports.
multi_line_output=3: Uses the one-per-line indented style for long imports.
include_trailing_comma: Adds a comma to the last import in a group, which makes git conflicts nicer to deal with, generally.
line_length: 72 is chosen to match PEP8's "docstrings and comments" line length limit. If you have a single line import that exceeds 72 characters, your names are too long!Suggested-by Cleber Rosa <[email protected]>Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/.isort.cfg | 7 +++++++ scripts/qapi/expr.py | 3 ++- scripts/qapi/introspect.py | 7 +++++-- scripts/qapi/parser.py | 2 +- scripts/qapi/schema.py | 2 +- 5 files changed, 16 insertions(+), 5 deletions(-) create mode 100644 scripts/qapi/.isort.cfgscripts/qapi/.isort.cfg new file +0
-0
scripts/qapi/.isort.cfg line 7
Quoted 8 lines
scripts/qapi/.isort.cfg new file +7
-0@@ -0,0 +1,7 @@ +[settings] +force_grid_wrap=4 +force_sort_within_sections=True +include_trailing_comma=True +line_length=72 +lines_after_imports=2 +multi_line_output=3
Add one, please :)
Quoted 53 lines
scripts/qapi/expr.py +2
-1@@ -14,8 +14,9 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -import re from collections import OrderedDict +import re + from .common import c_name from .error import QAPISemErrorscripts/qapi/introspect.py +5
-2@@ -17,8 +17,11 @@ mcgen, ) from .gen import QAPISchemaMonolithicCVisitor -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType, - QAPISchemaType) +from .schema import ( + QAPISchemaArrayType, + QAPISchemaBuiltinType, + QAPISchemaType, +) def _make_tree(obj, ifcond, features, extra=None):scripts/qapi/parser.py +1
-1@@ -14,9 +14,9 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. +from collections import OrderedDict import os import re -from collections import OrderedDict from .error import QAPIParseError, QAPISemError from .source import QAPISourceInfoscripts/qapi/schema.py +0
-0
scripts/qapi/schema.py line 22
Quoted 10 lines
scripts/qapi/schema.py +1
-1@@ -14,9 +14,9 @@ # TODO catching name collisions in generated code would be nice +from collections import OrderedDict import os import re -from collections import OrderedDict from .common import c_name, pointer_suffix from .error import QAPIError, QAPISemError
PATCH v5 8/36 qapi: delint using flake8
In reply to John SnowPetty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 16 +++++++++++----- 4 files changed, 20 insertions(+), 9 deletions(-) create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Prefer pylint's bare-except checks to flake8's
scripts/qapi/commands.py +2 -1
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, + QObject **ret_out, Error **errp) { Visitor *v;
scripts/qapi/schema.py +5 -3
@@ -536,7 +536,7 @@ def set_defined_in(self, name): v.set_defined_in(name) def check(self, schema, seen): - if not self.tag_member: # flat union + if not self.tag_member: # flat union self.tag_member = seen.get(c_name(self._tag_name)) base = "'base'" # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname): self._entity_dict = {} self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) - self._make_module(None) # built-ins + self._make_module(None) # built-ins self._make_module(fname) self._predefining = True self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members): # But it's not tight: the disjunction need not imply it. We # may end up compiling useless wrapper types. # TODO kill simple unions or implement the disjunction - assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access + + # pylint: disable=protected-access + assert (ifcond or []) == typ._ifcond else: self._def_entity(QAPISchemaObjectType( name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +11 -5
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False): if not scalar: c_type += '*' return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp); + +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_type)sobj, Error **errp); ''', c_name=c_name(name), c_type=c_type)
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants): def gen_visit_list(name, element_type): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false; %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type): def gen_visit_enum(name): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s *obj, Error **errp) { int value = *obj; bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ def gen_visit_enum(name): def gen_visit_alternate(name, variants): ret = mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp) { bool ok = false;
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants): def gen_visit_object(name, base, members, variants): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp) { bool ok = false;
Signature · 1 line
2.26.2
Re: [PATCH v5 08/36] qapi: delint using flake8
Quoted 110 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 16 +++++++++++----- 4 files changed, 20 insertions(+), 9 deletions(-) create mode 100644 scripts/qapi/.flake8scripts/qapi/.flake8 new file +2
-0@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Prefer pylint's bare-except checks to flake8'sscripts/qapi/commands.py +2
-1@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, + QObject **ret_out, Error **errp) { Visitor *v;scripts/qapi/schema.py +5
-3@@ -536,7 +536,7 @@ def set_defined_in(self, name): v.set_defined_in(name) def check(self, schema, seen): - if not self.tag_member: # flat union + if not self.tag_member: # flat union self.tag_member = seen.get(c_name(self._tag_name)) base = "'base'" # Pointing to the base type when not implicit would be@@ -821,7 +821,7 @@ def __init__(self, fname): self._entity_dict = {} self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) - self._make_module(None) # built-ins + self._make_module(None) # built-ins self._make_module(fname) self._predefining = True self._def_predefineds()@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members): # But it's not tight: the disjunction need not imply it. We # may end up compiling useless wrapper types. # TODO kill simple unions or implement the disjunction - assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access + + # pylint: disable=protected-access + assert (ifcond or []) == typ._ifcond else: self._def_entity(QAPISchemaObjectType( name, info, None, ifcond, None, None, members, None))scripts/qapi/visit.py +7
-3@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False): if not scalar: c_type += '*' return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp); + +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_type)sobj, Error **errp); ''', c_name=c_name(name), c_type=c_type)@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants): def gen_visit_list(name, element_type): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false; %(c_name)s *tail;@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type): def gen_visit_enum(name): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s *obj, Error **errp) { int value = *obj; bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
scripts/qapi/visit.py def gen_visit_enum(name): · line 180
Quoted 6 lines
scripts/qapi/visit.py +2
-1@@ -172,7 +176,8 @@ def gen_visit_enum(name): def gen_visit_alternate(name, variants): ret = mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
Please break this line the same way as the ones above:
bool visit_type_%(c_name)s(Visitor *v, const char *name,
%(c_name)s **obj, Error **errp)
scripts/qapi/visit.py def gen_visit_alternate(name, variants): · line 258
Quoted 12 lines
scripts/qapi/visit.py +2
-1{ bool ok = false;@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants): def gen_visit_object(name, base, members, variants): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp) { bool ok = false;
Likewise.
Re: [PATCH v5 08/36] qapi: delint using flake8
Quoted 131 lines · Markus Armbruster
Quoted 110 lines · John Snow
Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/.flake8 | 2 ++ scripts/qapi/commands.py | 3 ++- scripts/qapi/schema.py | 8 +++++--- scripts/qapi/visit.py | 16 +++++++++++----- 4 files changed, 20 insertions(+), 9 deletions(-)create mode 100644
scripts/qapi/.flake8scripts/qapi/.flake8 new file +2
-0@@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Prefer pylint's bare-except checks to flake8'sscripts/qapi/commands.py +2
-1@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type): def gen_marshal_output(ret_type): return mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, + QObject **ret_out, Error **errp) { Visitor *v;scripts/qapi/schema.py +5
-3@@ -536,7 +536,7 @@ def set_defined_in(self, name): v.set_defined_in(name) def check(self, schema, seen): - if not self.tag_member: # flat union + if not self.tag_member: # flat union self.tag_member = seen.get(c_name(self._tag_name)) base = "'base'" # Pointing to the base type when not implicit would be@@ -821,7 +821,7 @@ def __init__(self, fname): self._entity_dict = {} self._module_dict = OrderedDict() self._schema_dir = os.path.dirname(fname) - self._make_module(None) # built-ins + self._make_module(None) # built-ins self._make_module(fname) self._predefining = True self._def_predefineds()@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members): # But it's not tight: the disjunction need not imply it. We # may end up compiling useless wrapper types. # TODO kill simple unions or implement the disjunction - assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access + + # pylint: disable=protected-access + assert (ifcond or []) == typ._ifcond else: self._def_entity(QAPISchemaObjectType( name, info, None, ifcond, None, None, members, None))scripts/qapi/visit.py +7
-3@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False): if not scalar: c_type += '*' return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp); + +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_type)sobj, Error **errp); ''', c_name=c_name(name), c_type=c_type)@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants): def gen_visit_list(name, element_type): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s **obj, Error **errp) { bool ok = false; %(c_name)s *tail;@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type): def gen_visit_enum(name): return mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp) +bool visit_type_%(c_name)s(Visitor *v, const char *name, + %(c_name)s *obj, Error **errp) { int value = *obj; bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);scripts/qapi/visit.py def gen_visit_enum(name): · line 180
Quoted 6 lines
scripts/qapi/visit.py +2
-1@@ -172,7 +176,8 @@ def gen_visit_enum(name): def gen_visit_alternate(name, variants): ret = mcgen(''' -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)Please break this line the same way as the ones above:
bool visit_type_%(c_name)s(Visitor *v, const char *name,
%(c_name)s **obj, Error **errp)Quoted 7 lines
scripts/qapi/visit.py +0
-0{ bool ok = false;@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants): def gen_visit_object(name, base, members, variants): return mcgen('''
Quoted 8 lines
scripts/qapi/visit.py def gen_visit_enum(name): · line 182
Quoted 5 lines
scripts/qapi/visit.py +2
-1-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp) +bool visit_type_%(c_name)s + (Visitor *v, const char *name, %(c_name)s **obj, Error **errp) { bool ok = false;Likewise.
Whupps. Bad conflict resolution on my part. Didn't mean to do this.
--js
PATCH v5 9/36 qapi: add pylintrc
In reply to John SnowUsing pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.
I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is released), I aim to have gitlab CI running the specific targeted versions of pylint, mypy, flake8, etc in a job.
2.5.x will work if you additionally pass --disable=bad-whitespace. This warning was removed from 2.6.x, for lack of consistent support.
Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5+, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.
Therefore, to run it, you must:
Quoted 1 line
pylint
scripts/qapi/--rcfile=scripts/qapi/pylintrc
--- scripts/qapi/pylintrc | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +73 -0
@@ -0,0 +1,73 @@ +[MASTER] + +# Add files or directories matching the regex patterns to the ignore list. +# The regex matches against base names, not paths. +ignore-patterns=common.py, + error.py, + expr.py, + gen.py, + parser.py, + schema.py, + source.py, + types.py, + visit.py, + + +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=fixme, + missing-docstring, + too-many-arguments, + too-many-branches, + too-many-statements, + too-many-instance-attributes, + +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _ + +[VARIABLES] + +[STRING] + +[SPELLING] + +[FORMAT] + +[SIMILARITIES] + +# Ignore import statements themselves when computing similarities. +ignore-imports=yes + +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]
Signature · 1 line
2.26.2
PATCH v5 10/36 qapi/common.py: Remove python compatibility workaround
In reply to John Snow--- scripts/qapi/common.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +1 -4
@@ -119,10 +119,7 @@ def cgen(code, **kwds): raw = code % kwds if indent_level: indent = genindent(indent_level) - # re.subn() lacks flags support before Python 2.7, use re.compile() - raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE), - indent, raw) - raw = raw[0] + raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
Signature · 1 line
2.26.2
PATCH v5 11/36 qapi/common.py: Add indent manager
In reply to John SnowCode style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
--- scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++-------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 36 insertions(+), 20 deletions(-)
scripts/qapi/common.py +33 -16
@@ -93,33 +93,50 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> None: + """Increase the indentation level by ``amount``, default 4.""" + self._level += amount + + def decrease(self, amount: int = 4) -> None: + """Decrease the indentation level by ``amount``, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}") + self._level -= amount + + +indent = Indentation() # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey indent, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if indent: + raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)
scripts/qapi/visit.py +3 -4
@@ -18,9 +18,8 @@ c_name, gen_endif, gen_if, + indent, mcgen, - pop_indent, - push_indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + indent.increase() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + indent.decrease() ret += mcgen(''' } ''')
Signature · 1 line
2.26.2
Re: [PATCH v5 11/36] qapi/common.py: Add indent manager
Quoted 10 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>
Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.
Quoted 47 lines
--- scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++-------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 36 insertions(+), 20 deletions(-)scripts/qapi/common.py +17
-11@@ -93,33 +93,50 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level) -def pop_indent(indent_amount=4): - global indent_level
scripts/qapi/common.py def c_name(name, protect=True): · line 127
Quoted 10 lines
scripts/qapi/common.py +9
-1- indent_level -= indent_amount + def increase(self, amount: int = 4) -> None: + """Increase the indentation level by ``amount``, default 4.""" + self._level += amount + + def decrease(self, amount: int = 4) -> None: + """Decrease the indentation level by ``amount``, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}")
Raise a fancy error when there's an actual need for it. You're not coding a framework thousands of people you never heard of will put to uses you cannot imagine.
Quoted 52 lines
scripts/qapi/common.py +7
-4+ self._level -= amount + + +indent = Indentation() # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey indent, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if indent: + raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)scripts/qapi/visit.py +3
-4@@ -18,9 +18,8 @@ c_name, gen_endif, gen_if, + indent, mcgen, - pop_indent, - push_indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + indent.increase() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + indent.decrease() ret += mcgen(''' } ''')
Re: [PATCH v5 11/36] qapi/common.py: Add indent manager
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Quoted 9 lines
Quoted 5 lines
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.
I agree, but refactoring it properly is beyond my capacity right now.
This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.
Creating a stateful object was the fastest way from A to B.
Quoted 45 lines
Quoted 45 lines
--- scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++-------------- scripts/qapi/visit.py | 7 +++--- 2 files changed, 36 insertions(+), 20 deletions(-)scripts/qapi/common.py +17
-9@@ -93,33 +93,50 @@ def c_name(name, protect=True): pointer_suffix = ' *' + eatspace -def genindent(count): - ret = '' - for _ in range(count): - ret += ' ' - return ret +class Indentation: + """ + Indentation level management. + :param initial: Initial number of spaces, default 0. + """ + def __init__(self, initial: int = 0) -> None: + self._level = initial -indent_level = 0 + def __int__(self) -> int: + return self._level + def __repr__(self) -> str: + return "{}({:d})".format(type(self).__name__, self._level) -def push_indent(indent_amount=4): - global indent_level - indent_level += indent_amount + def __str__(self) -> str: + """Return the current indentation as a string of spaces.""" + return ' ' * self._level + def __bool__(self) -> bool: + """True when there is a non-zero indentation.""" + return bool(self._level)
Quoted 17 lines
scripts/qapi/common.py def c_name(name, protect=True): · line 127
Quoted 12 lines
scripts/qapi/common.py +9
-3-def pop_indent(indent_amount=4): - global indent_level - indent_level -= indent_amount + def increase(self, amount: int = 4) -> None: + """Increase the indentation level by ``amount``, default 4.""" + self._level += amount + + def decrease(self, amount: int = 4) -> None: + """Decrease the indentation level by ``amount``, default 4.""" + if self._level < amount: + raise ArithmeticError( + f"Can't remove {amount:d} spaces from {self!r}")Raise a fancy error when there's an actual need for it. You're not coding a framework thousands of people you never heard of will put to uses you cannot imagine.
It's not fancy, it's just a normal built-in exception, like AssertionError or any other:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ArithmeticError: Can't remove 4 spaces from Indent(0)
vs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError: Can't remove 4 spaces from Indent(0)
I feel like it's kind of a lateral move? I realize you feel this is an overfancy class doing what we hope is a temporary job, and that I have overengineered the hell out of a tiny do-nothing class... but I suppose that's also why I feel weird changing it around so much to accomplish so little.
Differences in style, I suppose.
Feel free to change it around to suit your tastes, I don't think it's worth spending a lot of ping-pong time on this paintsink in particular.
--js
Quoted 52 lines
Quoted 52 lines
scripts/qapi/common.py +7
-4+ self._level -= amount + + +indent = Indentation() # Generate @code with @kwds interpolated. -# Obey indent_level, and strip eatspace. +# Obey indent, and strip eatspace. def cgen(code, **kwds): raw = code % kwds - if indent_level: - indent = genindent(indent_level) - raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE) + if indent: + raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(eatspace) + r' *', '', raw)scripts/qapi/visit.py +3
-4@@ -18,9 +18,8 @@ c_name, gen_endif, gen_if, + indent, mcgen, - pop_indent, - push_indent, ) from .gen import QAPISchemaModularCVisitor, ifcontext from .schema import QAPISchemaObjectType@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants): if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) { ''', name=memb.name, c_name=c_name(memb.name)) - push_indent() + indent.increase() ret += mcgen(''' if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) { return false;@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants): c_type=memb.type.c_name(), name=memb.name, c_name=c_name(memb.name)) if memb.optional: - pop_indent() + indent.decrease() ret += mcgen(''' } ''')
Re: [PATCH v5 11/36] qapi/common.py: Add indent manager
Quoted 18 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Quoted 9 lines
Quoted 5 lines
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.
Quoted 6 lines
I agree, but refactoring it properly is beyond my capacity right now.
This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.
Creating a stateful object was the fastest way from A to B.
An even dumber solution could be:
indent_prefixes = []
def push_indent(amount=4):
"""Add amount spaces to indentation prefix""" indent_prefixes.push(' '*amount)
def pop_indent():
"""Revert the most recent push_indent() call""" indent_prefixes.pop()
def genindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
No global keyword involved, and the only stateful object is a dumb list.
Signature · 1 line
Eduardo
Re: [PATCH v5 11/36] qapi/common.py: Add indent manager
Quoted 39 lines · Eduardo Habkost
Quoted 18 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Quoted 9 lines
Quoted 5 lines
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.
Quoted 6 lines
I agree, but refactoring it properly is beyond my capacity right now.
This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.
Creating a stateful object was the fastest way from A to B.
An even dumber solution could be:
indent_prefixes= []
defpush_indent(amount=4):
"""Addamountspaces to indentation prefix"""indent_prefixes.push(' '*amount)
defpop_indent():
"""Revert the most recentpush_indent()call"""indent_prefixes.pop()
defgenindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
Quoted 2 lines
No global keyword involved, and the only stateful object is a dumb list.
Ha, this is Dumb with a capital D! I respect that :)
John, I'm not asking you to switch. Use your judgement.
Re: [PATCH v5 11/36] qapi/common.py: Add indent manager
Quoted 41 lines · Markus Armbruster
Quoted 39 lines · Eduardo Habkost
Quoted 18 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow
Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.
Quoted 9 lines
Quoted 5 lines
Make a little indent level manager instead.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.
Quoted 6 lines
I agree, but refactoring it properly is beyond my capacity right now.
This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.
Creating a stateful object was the fastest way from A to B.
An even dumber solution could be:
indent_prefixes= []
defpush_indent(amount=4):
"""Addamountspaces to indentation prefix"""indent_prefixes.push(' '*amount)
defpop_indent():
"""Revert the most recentpush_indent()call"""indent_prefixes.pop()
defgenindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)
Quoted 7 lines
Quoted 2 lines
No global keyword involved, and the only stateful object is a dumb list.
Ha, this is Dumb with a capital D! I respect that :)
John, I'm not asking you to switch. Use your judgement.
It's something we'll revisit soon, I'm sure. it's not good to be managing indent in so many different ways in so many different places.
(I prefer to leave it alone for now to try and press forward with accomplishing regression checks on strict typing.)
--js
PATCH v5 12/36 qapi/common.py: delint with pylint
In reply to John SnowAt this point, that just means using a consistent strategy for constant names. constants get UPPER_CASE and names not used externally get a leading underscore.
As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.
--- scripts/qapi/common.py | 18 ++++++++---------- scripts/qapi/schema.py | 14 +++++++------- 2 files changed, 15 insertions(+), 17 deletions(-)
scripts/qapi/common.py +8 -10
@@ -14,6 +14,11 @@ import re +EATSPACE = '\033EATSPACE.' +POINTER_SUFFIX = ' *' + EATSPACE +_C_NAME_TRANS = str.maketrans('.-', '__') + + # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None): return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -c_name_trans = str.maketrans('.-', '__') - - # Map @name to a valid C identifier. # If @protect, avoid returning certain ticklish identifiers (like # C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True): 'not_eq', 'or', 'or_eq', 'xor', 'xor_eq']) # namespace pollution: polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386']) - name = name.translate(c_name_trans) + name = name.translate(_C_NAME_TRANS) if protect and (name in c89_words | c99_words | c11_words | gcc_words | cpp_words | polluted_words): return 'q_' + name return name -eatspace = '\033EATSPACE.' -pointer_suffix = ' *' + eatspace - - class Indentation: """ Indentation level management.
@@ -132,12 +130,12 @@ def decrease(self, amount: int = 4) -> None: # Generate @code with @kwds interpolated. -# Obey indent, and strip eatspace. +# Obey indent, and strip EATSPACE. def cgen(code, **kwds): raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) - return re.sub(re.escape(eatspace) + r' *', '', raw) + return re.sub(re.escape(EATSPACE) + r' *', '', raw) def mcgen(code, **kwds):
scripts/qapi/schema.py +7 -7
@@ -18,7 +18,7 @@ import os import re -from .common import c_name, pointer_suffix +from .common import POINTER_SUFFIX, c_name from .error import QAPIError, QAPISemError from .expr import check_exprs from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self): return True def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'array'
@@ -430,7 +430,7 @@ def c_name(self): def c_type(self): assert not self.is_implicit() - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def c_unboxed_type(self): return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None): v.connect_doc(doc) def c_type(self): - return c_name(self.name) + pointer_suffix + return c_name(self.name) + POINTER_SUFFIX def json_type(self): return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type): self._make_array_type(name, None) def _def_predefineds(self): - for t in [('str', 'string', 'char' + pointer_suffix), + for t in [('str', 'string', 'char' + POINTER_SUFFIX), ('number', 'number', 'double'), ('int', 'int', 'int64_t'), ('int8', 'int', 'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self): ('uint64', 'int', 'uint64_t'), ('size', 'int', 'uint64_t'), ('bool', 'boolean', 'bool'), - ('any', 'value', 'QObject' + pointer_suffix), - ('null', 'null', 'QNull' + pointer_suffix)]: + ('any', 'value', 'QObject' + POINTER_SUFFIX), + ('null', 'null', 'QNull' + POINTER_SUFFIX)]: self._def_builtin_type(*t) self.the_empty_object_type = QAPISchemaObjectType( 'q_empty', None, None, None, None, None, [], None)
Signature · 1 line
2.26.2
PATCH v5 13/36 qapi/common.py: Replace one-letter 'c' variable
In reply to John Snow--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_': + char = c_fun_str[i] + # When char is upper case and no '_' appears before, do more checks + if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_': if i < length - 1 and c_fun_str[i + 1].islower(): new_name += '_' elif c_fun_str[i - 1].isdigit(): new_name += '_' - new_name += c + new_name += char return new_name.lstrip('_').upper()
Signature · 1 line
2.26.2
Re: [PATCH v5 13/36] qapi/common.py: Replace one-letter 'c' variable
Quoted 18 lines · John Snow
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>--- scripts/qapi/common.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)scripts/qapi/common.py +0
-3@@ -30,14 +30,14 @@ def camel_to_upper(value): new_name = '' length = len(c_fun_str) for i in range(length): - c = c_fun_str[i] - # When c is upper and no '_' appears before, do more checks - if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
scripts/qapi/common.py def camel_to_upper(value): · line 41
Quoted 10 lines
scripts/qapi/common.py +4
-1+ char = c_fun_str[i] + # When char is upper case and no '_' appears before, do more checks + if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_': if i < length - 1 and c_fun_str[i + 1].islower(): new_name += '_' elif c_fun_str[i - 1].isdigit(): new_name += '_' - new_name += c + new_name += char return new_name.lstrip('_').upper()
PATCH v5 14/36 qapi/common.py: check with pylint
In reply to John SnowRemove qapi/common.py from the pylintrc ignore list.
--- scripts/qapi/pylintrc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi/pylintrc +1 -2
@@ -2,8 +2,7 @@ # Add files or directories matching the regex patterns to the ignore list. # The regex matches against base names, not paths. -ignore-patterns=common.py, - error.py, +ignore-patterns=error.py, expr.py, gen.py, parser.py,
Signature · 1 line
2.26.2
PATCH v5 15/36 qapi/common.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +16 -11
@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Sequence EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None: # Generate @code with @kwds interpolated. # Obey indent, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: object) -> str: raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: object) -> str: if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''
@@ -183,7 +186,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:
Signature · 1 line
2.26.2
Re: [PATCH v5 15/36] qapi/common.py: add type hint annotations
Quoted 108 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +13
-10@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Sequence EATSPACE = '\033EATSPACE.'@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None: # Generate @code with @kwds interpolated. # Obey indent, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: object) -> str: raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: object) -> str: if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s@@ -157,7 +160,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */@@ -165,7 +168,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''@@ -174,7 +177,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''
scripts/qapi/common.py def gen_endif(ifcond): · line 194
Quoted 10 lines
scripts/qapi/common.py +3
-1@@ -183,7 +186,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:
@arg_type is the only parameter left unannotated. Scratching head... aha:
qapi/common.py: move build_params into gen.py
Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.
Let's amend the commit message:
Note that build_params() cannot be fully annotated due to import dependency issues. The commit after next will take care of it.
If we swap the next two commits, the fix follows immediately. Better, but only worth it if swapping is pretty much effortless.
Re: [PATCH v5 15/36] qapi/common.py: add type hint annotations
Quoted 131 lines · Markus Armbruster
Quoted 108 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/common.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)scripts/qapi/common.py +13
-10@@ -12,6 +12,7 @@ # See the COPYING file in the top-level directory. import re +from typing import Optional, Sequence EATSPACE = '\033EATSPACE.'@@ -22,7 +23,7 @@ # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 # ENUM24_Name -> ENUM24_NAME -def camel_to_upper(value): +def camel_to_upper(value: str) -> str: c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -41,7 +42,9 @@ def camel_to_upper(value): return new_name.lstrip('_').upper() -def c_enum_const(type_name, const_name, prefix=None): +def c_enum_const(type_name: str, + const_name: str, + prefix: Optional[str] = None) -> str: if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None): # into substrings of a generated C function name. # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' -def c_name(name, protect=True): +def c_name(name: str, protect: bool = True) -> str: # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None: # Generate @code with @kwds interpolated. # Obey indent, and strip EATSPACE. -def cgen(code, **kwds): +def cgen(code: str, **kwds: object) -> str: raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE) return re.sub(re.escape(EATSPACE) + r' *', '', raw) -def mcgen(code, **kwds): +def mcgen(code: str, **kwds: object) -> str: if code[0] == '\n': code = code[1:] return cgen(code, **kwds) -def c_fname(filename): +def c_fname(filename: str) -> str: return re.sub(r'[^A-Za-z0-9_]', '_', filename) -def guardstart(name): +def guardstart(name: str) -> str: return mcgen(''' #ifndef %(name)s #define %(name)s@@ -157,7 +160,7 @@ def guardstart(name): name=c_fname(name).upper()) -def guardend(name): +def guardend(name: str) -> str: return mcgen(''' #endif /* %(name)s */@@ -165,7 +168,7 @@ def guardend(name): name=c_fname(name).upper()) -def gen_if(ifcond): +def gen_if(ifcond: Sequence[str]) -> str: ret = '' for ifc in ifcond: ret += mcgen('''@@ -174,7 +177,7 @@ def gen_if(ifcond): return ret -def gen_endif(ifcond): +def gen_endif(ifcond: Sequence[str]) -> str: ret = '' for ifc in reversed(ifcond): ret += mcgen('''scripts/qapi/common.py def gen_endif(ifcond): · line 194
Quoted 10 lines
scripts/qapi/common.py +3
-1@@ -183,7 +186,9 @@ def gen_endif(ifcond): return ret -def build_params(arg_type, boxed, extra=None): +def build_params(arg_type, + boxed: bool, + extra: Optional[str] = None) -> str: ret = '' sep = '' if boxed:@arg_type is the only parameter left unannotated. Scratching head... aha:
qapi/common.py: movebuild_paramsintogen.pyIncluding it in
common.pycreates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.
Quoted 8 lines
Let's amend the commit message:
Note that
build_params()cannot be fully annotated due to import dependency issues. The commit after next will take care of it.If we swap the next two commits, the fix follows immediately. Better, but only worth it if swapping is pretty much effortless.
The reason they're not flipped was to avoid giving the impression that docstrings were what was preventing us from enabling the mypy type checking.
The reason I don't flip the comments and the type hints is because that's a lot of rebase pain for perceptibly little benefit.
I amended the commit message.
PATCH v5 16/36 qapi/common.py: Convert comments into docstrings, and elaborate
In reply to John SnowAs docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Primarily, we use :param X: descr, :return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.
(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)
--- scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 14 deletions(-)
scripts/qapi/common.py +39 -14
@@ -15,15 +15,24 @@ from typing import Optional, Sequence +#: Sentinel value that causes all space to its right to be removed. EATSPACE = '\033EATSPACE.' POINTER_SUFFIX = ' *' + EATSPACE _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """ + Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """ c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str
@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map ``name`` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None: self._level -= amount +#: Global, current indent level for code generation. indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate ``code`` with ``kwds`` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """ raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
Signature · 1 line
2.26.2
Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 31 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-listsPrimarily, we use
:param X: descr,:return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 14 deletions(-)scripts/qapi/common.py +0
-0
scripts/qapi/common.py line 18
Quoted 4 lines
scripts/qapi/common.py +1
-0@@ -15,15 +15,24 @@ from typing import Optional, Sequence +#: Sentinel value that causes all space to its right to be removed.
What's the purpose of : after # ?
I'm not sure this is a "sentinel value". Wikipedia:
In computer programming, a sentinel value (also referred to as a flag value, trip value, rogue value, signal value, or dummy data)[1] is a special value in the context of an algorithm which uses its presence as a condition of termination, typically in a loop or recursive algorithm.
https://en.wikipedia.org/wiki/Sentinel_value
Perhaps
# Magic string value that gets removed along with all space to the
# right.
Quoted 10 lines
scripts/qapi/common.py +1
-3EATSPACE = '\033EATSPACE.' POINTER_SUFFIX = ' *' + EATSPACE _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """
scripts/qapi/common.py line 35
Quoted 10 lines
scripts/qapi/common.py +10
-0+ Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """
I wonder whether these indented lines get wrapped into one unintelligible parapgraph.
Have you eyeballed the output of Sphinx?
Quoted 65 lines
scripts/qapi/common.py +27
-11c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map ``name`` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None: self._level -= amount +#: Global, current indent level for code generation. indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate ``code`` with ``kwds`` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """ raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 36 lines · Markus Armbruster
Quoted 34 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-listsPrimarily, we use
:param X: descr,:return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 14 deletions(-)scripts/qapi/common.py +0
-0@@ -15,15 +15,24 @@ from typing import Optional, Sequence
Quoted 4 lines
scripts/qapi/common.py line 18
Quoted 1 line
scripts/qapi/common.py +1
-0+#: Sentinel value that causes all space to its right to be removed.What's the purpose of : after # ?
Documents this name in Sphinx. We had a small discussion about it, I think; "Does using this special form or the docstring make the comment visible in any IDE?" (No.)
There's no Python-AST way to document these, but there is a Sphinx way to document them, so I did that.
(Doing it like this allows EATSPACE to be used as a cross-reference.)
Quoted 10 lines
I'm not sure this is a "sentinel value". Wikipedia:
In computer programming, a sentinel value (also referred to as a flag value, trip value, rogue value, signal value, or dummy data)[1] is a special value in the context of an algorithm which uses its presence as a condition of termination, typically in a loop or recursive algorithm.
I really should try to learn English as a second language so I know what any of the words I use mean, I guess. I had slipped to a less strict usage where it meant more like "placeholder".
Quoted 5 lines
Perhaps
# Magic string value that gets removed along with all space to the # right.
This can be written on one line if we gently disregard the 72 column limit. (Maybe you already did when you wrote it and my client wrapped it. Who knows!)
Quoted 24 lines
Quoted 10 lines
scripts/qapi/common.py +1
-3EATSPACE = '\033EATSPACE.' POINTER_SUFFIX = ' *' + EATSPACE _C_NAME_TRANS = str.maketrans('.-', '__') -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1 -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2 -# ENUM24_Name -> ENUM24_NAME def camel_to_upper(value: str) -> str: + """scripts/qapi/common.py line 35
Quoted 10 lines
scripts/qapi/common.py +10
-0+ Converts CamelCase to CAMEL_CASE. + + Examples: + ENUMName -> ENUM_NAME + EnumName1 -> ENUM_NAME1 + ENUM_NAME -> ENUM_NAME + ENUM_NAME1 -> ENUM_NAME1 + ENUM_Name2 -> ENUM_NAME2 + ENUM24_Name -> ENUM24_NAME + """I wonder whether these indented lines get wrapped into one unintelligible parapgraph.
Quoted 2 lines
Have you eyeballed the output of Sphinx?
Eyeballed, but didn't validate this specific one. Yeah, it's nonsense.
Examples::
ENUMName -> ENUM_NAME
etc. works better.
Quoted 65 lines
Quoted 65 lines
scripts/qapi/common.py +27
-11c_fun_str = c_name(value, False) if value.isupper(): return c_fun_str@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str: def c_enum_const(type_name: str, const_name: str, prefix: Optional[str] = None) -> str: + """ + Generate a C enumeration constant name. + + :param type_name: The name of the enumeration. + :param const_name: The name of this constant. + :param prefix: Optional, prefix that overrides the type_name. + """ if prefix is not None: type_name = prefix return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper() -# Map @name to a valid C identifier. -# If @protect, avoid returning certain ticklish identifiers (like -# C keywords) by prepending 'q_'. -# -# Used for converting 'name' from a 'name':'type' qapi definition -# into a generated struct member, as well as converting type names -# into substrings of a generated C function name. -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' def c_name(name: str, protect: bool = True) -> str: + """ + Map ``name`` to a valid C identifier. + + Used for converting 'name' from a 'name':'type' qapi definition + into a generated struct member, as well as converting type names + into substrings of a generated C function name. + + '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo' + protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int' + + :param name: The name to map. + :param protect: If true, avoid returning certain ticklish identifiers + (like C keywords) by prepending ``q_``. + """ # ANSI X3J11/88-090, 3.1.1 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue', 'default', 'do', 'double', 'else', 'enum', 'extern',@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None: self._level -= amount +#: Global, current indent level for code generation. indent = Indentation() -# Generate @code with @kwds interpolated. -# Obey indent, and strip EATSPACE. def cgen(code: str, **kwds: object) -> str: + """ + Generate ``code`` with ``kwds`` interpolated. + + Obey `indent`, and strip `EATSPACE`. + """ raw = code % kwds if indent: raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate
Quoted 40 lines · John Snow
Quoted 35 lines · Markus Armbruster
Quoted 33 lines · John Snow
As docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-listsPrimarily, we use
:param X: descr,:return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 14 deletions(-)scripts/qapi/common.py +0
-0@@ -15,15 +15,24 @@ from typing import Optional, SequenceQuoted 3 lines
scripts/qapi/common.py line 18
Quoted 1 line
scripts/qapi/common.py +1
-0+#: Sentinel value that causes all space to its right to be removed.What's the purpose of : after # ?
Quoted 8 lines
Documents this name in Sphinx. We had a small discussion about it, I think; "Does using this special form or the docstring make the comment visible in any IDE?" (No.)
There's no Python-AST way to document these, but there is a Sphinx way to document them, so I did that.
(Doing it like this allows
EATSPACEto be used as a cross-reference.)
Thanks.
Consider pointing this out when you write the comment & doc string part of our Python style guide.
Quoted 19 lines
Quoted 8 lines
I'm not sure this is a "sentinel value". Wikipedia:
In computer programming, a sentinel value (also referred to as a flag value, trip value, rogue value, signal value, or dummy data)[1] is a special value in the context of an algorithm which uses its presence as a condition of termination, typically in a loop or recursive algorithm.
https://en.wikipedia.org/wiki/Sentinel_valueI really should try to learn English as a second language so I know what any of the words I use mean, I guess. I had slipped to a less strict usage where it meant more like "placeholder".
Quoted 5 lines
Perhaps
# Magic string value that gets removed along with all space to
the# right.
Quoted 3 lines
This can be written on one line if we gently disregard the 72 column limit. (Maybe you already did when you wrote it and my client wrapped it. Who knows!)
Drop the period and it fits ;-P
You could also drop "value" without loss.
[...]
PATCH v5 17/36 qapi/common.py: move build_params into gen.py
In reply to John SnowIncluding it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.
--- scripts/qapi/commands.py | 9 +++++++-- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 9 ++------- scripts/qapi/gen.py | 31 +++++++++++++++++++++++++++++-- 4 files changed, 38 insertions(+), 34 deletions(-)
scripts/qapi/commands.py +7 -2
@@ -13,8 +13,13 @@ See the COPYING file in the top-level directory. """ -from .common import build_params, c_name, mcgen -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import c_name, mcgen +from .gen import ( + QAPIGenCCode, + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'
scripts/qapi/events.py +2 -7
@@ -12,13 +12,8 @@ See the COPYING file in the top-level directory. """ -from .common import ( - build_params, - c_enum_const, - c_name, - mcgen, -) -from .gen import QAPISchemaModularCVisitor, ifcontext +from .common import c_enum_const, c_name, mcgen +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +29 -2
@@ -2,9 +2,11 @@ # # QAPI code generation # -# Copyright (c) 2018-2019 Red Hat Inc. +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2019 Red Hat Inc. # # Authors: +# Anthony Liguori <[email protected]> # Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #
@@ -15,16 +17,18 @@ import errno import os import re +from typing import Optional from .common import ( c_fname, + c_name, gen_endif, gen_if, guardend, guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import QAPISchemaObjectType, QAPISchemaVisitor class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after): return out +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void' + + class QAPIGenCCode(QAPIGen): def __init__(self, fname):
Signature · 1 line
2.26.2
Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py
Quoted 93 lines · John Snow
Including it in
common.pycreates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/commands.py | 9 +++++++-- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 9 ++------- scripts/qapi/gen.py | 31 +++++++++++++++++++++++++++++-- 4 files changed, 38 insertions(+), 34 deletions(-)scripts/qapi/commands.py +7
-2@@ -13,8 +13,13 @@ See the COPYING file in the top-level directory. """ -from .common import build_params, c_name, mcgen -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import c_name, mcgen +from .gen import ( + QAPIGenCCode, + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) def gen_command_decl(name, arg_type, boxed, ret_type):scripts/qapi/common.py +0
-23@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'scripts/qapi/events.py +2
-7@@ -12,13 +12,8 @@ See the COPYING file in the top-level directory. """ -from .common import ( - build_params, - c_enum_const, - c_name, - mcgen, -) -from .gen import QAPISchemaModularCVisitor, ifcontext +from .common import c_enum_const, c_name, mcgen +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookupscripts/qapi/gen.py +0
-0@@ -2,9 +2,11 @@ # # QAPI code generation
scripts/qapi/gen.py line 12
Quoted 10 lines
scripts/qapi/gen.py +3
-1# -# Copyright (c) 2018-2019 Red Hat Inc. +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2019 Red Hat Inc. # # Authors: +# Anthony Liguori <[email protected]> # Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #
The code you move into this file is actually Red Hat's:
$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date' Author: Markus Armbruster <[email protected]> Date: Wed Aug 15 21:37:36 2018 +0800 Author: Marc-André Lureau <[email protected]> Date: Thu Jun 1 16:41:41 2017 +0400 Author: Eric Blake <[email protected]> Date: Wed Jul 13 21:50:20 2016 -0600 Author: Eric Blake <[email protected]> Date: Wed Jul 13 21:50:19 2016 -0600 Author: Eric Blake <[email protected]> Date: Thu Mar 17 16:48:28 2016 -0600 Author: Markus Armbruster <[email protected]> Date: Wed Sep 16 13:06:20 2015 +0200
So the correct update is simply
#
# QAPI code generation
#
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
#
# Authors:
# Markus Armbruster <[email protected]>
# Marc-André Lureau <[email protected]> #
Quoted 50 lines
scripts/qapi/gen.py +26
-1@@ -15,16 +17,18 @@ import errno import os import re +from typing import Optional from .common import ( c_fname, + c_name, gen_endif, gen_if, guardend, guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import QAPISchemaObjectType, QAPISchemaVisitor class QAPIGen:@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after): return out +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void' + + class QAPIGenCCode(QAPIGen): def __init__(self, fname):
Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py
Quoted 127 lines · Markus Armbruster
Quoted 93 lines · John Snow
Including it in
common.pycreates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/commands.py | 9 +++++++-- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 9 ++------- scripts/qapi/gen.py | 31 +++++++++++++++++++++++++++++-- 4 files changed, 38 insertions(+), 34 deletions(-)scripts/qapi/commands.py +7
-2@@ -13,8 +13,13 @@ See the COPYING file in the top-level directory. """ -from .common import build_params, c_name, mcgen -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import c_name, mcgen +from .gen import ( + QAPIGenCCode, + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) def gen_command_decl(name, arg_type, boxed, ret_type):scripts/qapi/common.py +0
-23@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'scripts/qapi/events.py +2
-7@@ -12,13 +12,8 @@ See the COPYING file in the top-level directory. """ -from .common import ( - build_params, - c_enum_const, - c_name, - mcgen, -) -from .gen import QAPISchemaModularCVisitor, ifcontext +from .common import c_enum_const, c_name, mcgen +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookupscripts/qapi/gen.py +0
-0@@ -2,9 +2,11 @@ # # QAPI code generationscripts/qapi/gen.py line 12
Quoted 10 lines
scripts/qapi/gen.py +3
-1# -# Copyright (c) 2018-2019 Red Hat Inc. +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2019 Red Hat Inc. # # Authors: +# Anthony Liguori <[email protected]> # Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #The code you move into this file is actually Red Hat's:
$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date' Author: Markus Armbruster <[email protected]> Date: Wed Aug 15 21:37:36 2018 +0800 Author: Marc-André Lureau <[email protected]> Date: Thu Jun 1 16:41:41 2017 +0400 Author: Eric Blake <[email protected]> Date: Wed Jul 13 21:50:20 2016 -0600 Author: Eric Blake <[email protected]> Date: Wed Jul 13 21:50:19 2016 -0600 Author: Eric Blake <[email protected]> Date: Thu Mar 17 16:48:28 2016 -0600 Author: Markus Armbruster <[email protected]> Date: Wed Sep 16 13:06:20 2015 +0200So the correct update is simply
# # QAPI code generation #
Quoted 8 lines
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
#
# Authors:
# Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #
OK, thank you. I did the dumbest possibly correct thing. My appetite for doing line ownership analysis is ... low.
Quoted 50 lines
Quoted 50 lines
scripts/qapi/gen.py +26
-1@@ -15,16 +17,18 @@ import errno import os import re +from typing import Optional from .common import ( c_fname, + c_name, gen_endif, gen_if, guardend, guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import QAPISchemaObjectType, QAPISchemaVisitor class QAPIGen:@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after): return out +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void' + + class QAPIGenCCode(QAPIGen): def __init__(self, fname):
Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py
In reply to Markus ArmbrusterQuoted 127 lines · Markus Armbruster
Quoted 93 lines · John Snow
Including it in
common.pycreates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/commands.py | 9 +++++++-- scripts/qapi/common.py | 23 ----------------------- scripts/qapi/events.py | 9 ++------- scripts/qapi/gen.py | 31 +++++++++++++++++++++++++++++-- 4 files changed, 38 insertions(+), 34 deletions(-)scripts/qapi/commands.py +7
-2@@ -13,8 +13,13 @@ See the COPYING file in the top-level directory. """ -from .common import build_params, c_name, mcgen -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext +from .common import c_name, mcgen +from .gen import ( + QAPIGenCCode, + QAPISchemaModularCVisitor, + build_params, + ifcontext, +) def gen_command_decl(name, arg_type, boxed, ret_type):scripts/qapi/common.py +0
-23@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret - - -def build_params(arg_type, - boxed: bool, - extra: Optional[str] = None) -> str: - ret = '' - sep = '' - if boxed: - assert arg_type - ret += '%s arg' % arg_type.c_param_type() - sep = ', ' - elif arg_type: - assert not arg_type.variants - for memb in arg_type.members: - ret += sep - sep = ', ' - if memb.optional: - ret += 'bool has_%s, ' % c_name(memb.name) - ret += '%s %s' % (memb.type.c_param_type(), - c_name(memb.name)) - if extra: - ret += sep + extra - return ret if ret else 'void'scripts/qapi/events.py +2
-7@@ -12,13 +12,8 @@ See the COPYING file in the top-level directory. """ -from .common import ( - build_params, - c_enum_const, - c_name, - mcgen, -) -from .gen import QAPISchemaModularCVisitor, ifcontext +from .common import c_enum_const, c_name, mcgen +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext from .schema import QAPISchemaEnumMember from .types import gen_enum, gen_enum_lookupscripts/qapi/gen.py +0
-0@@ -2,9 +2,11 @@ # # QAPI code generationscripts/qapi/gen.py line 12
Quoted 10 lines
scripts/qapi/gen.py +3
-1# -# Copyright (c) 2018-2019 Red Hat Inc. +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2019 Red Hat Inc. # # Authors: +# Anthony Liguori <[email protected]> # Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #The code you move into this file is actually Red Hat's:
$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date' Author: Markus Armbruster <[email protected]> Date: Wed Aug 15 21:37:36 2018 +0800 Author: Marc-André Lureau <[email protected]> Date: Thu Jun 1 16:41:41 2017 +0400 Author: Eric Blake <[email protected]> Date: Wed Jul 13 21:50:20 2016 -0600 Author: Eric Blake <[email protected]> Date: Wed Jul 13 21:50:19 2016 -0600 Author: Eric Blake <[email protected]> Date: Thu Mar 17 16:48:28 2016 -0600 Author: Markus Armbruster <[email protected]> Date: Wed Sep 16 13:06:20 2015 +0200So the correct update is simply
# # QAPI code generation #
Quoted 7 lines
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
#
# Authors:
# Markus Armbruster <[email protected]>
# Marc-André Lureau <[email protected]> #
I am pretty sure build_params() below is a derivative work of generate_command_decl(), added in commit c17d9908a942 ("qapi: add qapi-commands.py code generator"), copyright IBM Corp. I'm not a lawyer, though.
Because figuring that out is complicated, when I move or copy code between files, I either:
(a) copy no copyright notices from the other file, or
(b) copy all copyright notices from the other files.
(The above is a description of what I do, but it is not a suggestion on how to proceed on this case.)
Quoted 51 lines
Quoted 51 lines
scripts/qapi/gen.py +26
-1@@ -15,16 +17,18 @@ import errno import os import re +from typing import Optional from .common import ( c_fname, + c_name, gen_endif, gen_if, guardend, guardstart, mcgen, ) -from .schema import QAPISchemaVisitor +from .schema import QAPISchemaObjectType, QAPISchemaVisitor class QAPIGen:@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after): return out +def build_params(arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + extra: Optional[str] = None) -> str: + ret = '' + sep = '' + if boxed: + assert arg_type + ret += '%s arg' % arg_type.c_param_type() + sep = ', ' + elif arg_type: + assert not arg_type.variants + for memb in arg_type.members: + ret += sep + sep = ', ' + if memb.optional: + ret += 'bool has_%s, ' % c_name(memb.name) + ret += '%s %s' % (memb.type.c_param_type(), + c_name(memb.name)) + if extra: + ret += sep + extra + return ret if ret else 'void' + + class QAPIGenCCode(QAPIGen): def __init__(self, fname):
Signature · 1 line
Eduardo
PATCH v5 18/36 qapi: establish mypy type-checking baseline
In reply to John SnowFix two very minor issues, and then establish a mypy type-checking baseline.
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
This is designed and tested for mypy 0.770 or greater.
--- scripts/qapi/mypy.ini | 60 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 ++- 2 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/mypy.ini new file +60 -0
@@ -0,0 +1,60 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False
scripts/qapi/schema.py +2 -1
@@ -17,6 +17,7 @@ from collections import OrderedDict import os import re +from typing import Optional from .common import POINTER_SUFFIX, c_name from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Signature · 1 line
2.26.2
Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline
Quoted 1 line · John Snow
Fix two very minor issues,
What are the two issues? I'm asking because I can see only one.
Quoted 2 lines
and then establish a mypy type-checking
baseline.
Any particular reason this can't go before PATCH 15 "qapi/common.py: add type hint annotations".
Quoted 103 lines
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
This is designed and tested for mypy 0.770 or greater.
Signed-off-by John Snow <[email protected]>Tested-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 60 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 ++- 2 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 scripts/qapi/mypy.iniscripts/qapi/mypy.ini new file +60
-0@@ -0,0 +1,60 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = Falsescripts/qapi/schema.py +2
-1@@ -17,6 +17,7 @@ from collections import OrderedDict import os import re +from typing import Optional from .common import POINTER_SUFFIX, c_name from .error import QAPIError, QAPISemError@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline
Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow
Fix two very minor issues,
What are the two issues? I'm asking because I can see only one.
Bitrot! The other one got fixed elsewhere and rebase didn't conflict, so I didn't update the commit message.
Quoted 6 lines
Quoted 2 lines
and then establish a mypy type-checking
baseline.Any particular reason this can't go before PATCH 15 "
qapi/common.py: add type hint annotations".
Just the way the chips fell. common.py does not check until this very commit.
Adding a mypy config can be shuffled up earlier and earlier by determining which checks to ignore and adding them, but I'm not sure who that benefits.
Quoted 103 lines
Quoted 103 lines
Like pylint, this should be run from the folder above:
Quoted 1 line
mypy --config-file=qapi/mypy.ini qapi/
This is designed and tested for mypy 0.770 or greater.
Signed-off-by John Snow <[email protected]>Tested-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 60 ++++++++++++++++++++++++++++++++++++++++++ scripts/qapi/schema.py | 3 ++- 2 files changed, 62 insertions(+), 1 deletion(-)create mode 100644
scripts/qapi/mypy.iniscripts/qapi/mypy.ini new file +60
-0@@ -0,0 +1,60 @@ +[mypy] +strict = True +strict_optional = False +disallow_untyped_calls = False +python_version = 3.6 + +[mypy-qapi.commands] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.error] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.events] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.expr] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.gen] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.introspect] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.parser] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.schema] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.source] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.types] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + +[mypy-qapi.visit] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = Falsescripts/qapi/schema.py +2
-1@@ -17,6 +17,7 @@ from collections import OrderedDict import os import re +from typing import Optional from .common import POINTER_SUFFIX, c_name from .error import QAPIError, QAPISemError@@ -25,7 +26,7 @@ class QAPISchemaEntity: - meta = None + meta: Optional[str] = None def __init__(self, name, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str)
Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline
Quoted 16 lines · John Snow
Quoted 5 lines · Markus Armbruster
Quoted 1 line · John Snow
Fix two very minor issues,
What are the two issues? I'm asking because I can see only one.
Bitrot! The other one got fixed elsewhere and rebase didn't conflict, so I didn't update the commit message.
Quoted 5 lines
Quoted 2 lines
and then establish a mypy type-checking
baseline.Any particular reason this can't go before PATCH 15 "
qapi/common.py: add type hint annotations".
Quoted 6 lines
Just the way the chips fell.
common.pydoes not check until this very commit.Adding a mypy config can be shuffled up earlier and earlier by determining which checks to ignore and adding them, but I'm not sure who that benefits.
I like symmetry, but it's okay as is. I asked to make sure I'm not missing something more subtle.
[...]
PATCH v5 19/36 qapi/events.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 ----- 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +35 -11
@@ -12,19 +12,31 @@ See the COPYING file in the top-level directory. """ +from typing import List + from .common import c_enum_const, c_name, mcgen from .gen import QAPISchemaModularCVisitor, build_params, ifcontext -from .schema import QAPISchemaEnumMember +from .schema import ( + QAPISchema, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, +) +from .source import QAPISourceInfo from .types import gen_enum, gen_enum_lookup -def build_event_send_proto(name, arg_type, boxed): +def build_event_send_proto(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return 'void qapi_event_send_%(c_name)s(%(param)s)' % { 'c_name': c_name(name.lower()), 'param': build_params(arg_type, boxed)} -def gen_event_send_decl(name, arg_type, boxed): +def gen_event_send_decl(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return mcgen(''' %(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed): # Declare and initialize an object 'qapi' using parameters from build_params() -def gen_param_var(typ): +def gen_param_var(typ: QAPISchemaObjectType) -> str: assert not typ.variants ret = mcgen(''' %(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ): return ret -def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): +def gen_event_send(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool, + event_enum_name: str, + event_emit: str) -> str: # FIXME: Our declaration of local variables (and of 'errp' in the # parameter list) can collide with exploded members of the event's # data type passed in as parameters. If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-events', ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) - self._event_enum_members = [] + self._event_enum_members: List[QAPISchemaEnumMember] = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit') - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: events = self._module_basename('qapi-events', name) types = self._module_basename('qapi-types', name) visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('emit', ' * QAPI Events emission') self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h"
@@ -189,7 +205,13 @@ def visit_end(self): event_emit=self._event_emit_name, event_enum=self._event_enum_name)) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: QAPISchemaObjectType, + boxed: bool) -> None: with ifcontext(ifcond, self._genh, self._genc): self._genh.add(gen_event_send_decl(name, arg_type, boxed)) self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -200,7 +222,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed): self._event_enum_members.append(QAPISchemaEnumMember(name, None)) -def gen_events(schema, output_dir, prefix): +def gen_events(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenEventVisitor(prefix) schema.visit(vis) vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
@@ -14,11 +14,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.events] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.expr] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 92 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 ----- 2 files changed, 35 insertions(+), 16 deletions(-)scripts/qapi/events.py +24
-8@@ -12,19 +12,31 @@ See the COPYING file in the top-level directory. """ +from typing import List + from .common import c_enum_const, c_name, mcgen from .gen import QAPISchemaModularCVisitor, build_params, ifcontext -from .schema import QAPISchemaEnumMember +from .schema import ( + QAPISchema, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, +) +from .source import QAPISourceInfo from .types import gen_enum, gen_enum_lookup -def build_event_send_proto(name, arg_type, boxed): +def build_event_send_proto(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return 'void qapi_event_send_%(c_name)s(%(param)s)' % { 'c_name': c_name(name.lower()), 'param': build_params(arg_type, boxed)} -def gen_event_send_decl(name, arg_type, boxed): +def gen_event_send_decl(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool) -> str: return mcgen(''' %(proto)s;@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed): # Declare and initialize an object 'qapi' using parameters from build_params() -def gen_param_var(typ): +def gen_param_var(typ: QAPISchemaObjectType) -> str: assert not typ.variants ret = mcgen(''' %(c_name)s param = {@@ -61,7 +73,11 @@ def gen_param_var(typ): return ret -def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): +def gen_event_send(name: str, + arg_type: QAPISchemaObjectType, + boxed: bool, + event_enum_name: str, + event_emit: str) -> str: # FIXME: Our declaration of local variables (and of 'errp' in the # parameter list) can collide with exploded members of the event's # data type passed in as parameters. If this collision ever hits in@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-events', ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) - self._event_enum_members = [] + self._event_enum_members: List[QAPISchemaEnumMember] = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit') - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: events = self._module_basename('qapi-events', name) types = self._module_basename('qapi-types', name) visit = self._module_basename('qapi-visit', name)
scripts/qapi/events.py def _begin_user_module(self, name): · line 187
Quoted 5 lines
scripts/qapi/events.py +1
-1@@ -168,7 +184,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None:
Ignorant question: what's the difference between -> None (like here) and nothing (like __init__() above?
Quoted 3 lines
scripts/qapi/events.py +0
-0self._add_system_module('emit', ' * QAPI Events emission') self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h"
[...]
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 18 lines · Markus Armbruster
Quoted 16 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 ----- 2 files changed, 35 insertions(+), 16 deletions(-)scripts/qapi/events.py +0
-0@@ -12,19 +12,31 @@
[...]
Quoted 23 lines
Quoted 23 lines
scripts/qapi/events.py +3
-3@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit): class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-events', ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) - self._event_enum_members = [] + self._event_enum_members: List[QAPISchemaEnumMember] = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit') - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: events = self._module_basename('qapi-events', name) types = self._module_basename('qapi-types', name) visit = self._module_basename('qapi-visit', name)@@ -168,7 +184,7 @@ def _begin_user_module(self, name): ''', types=types))
Quoted 5 lines
scripts/qapi/events.py def _begin_user_module(self, name): · line 187
Quoted 2 lines
scripts/qapi/events.py +1
-1- def visit_end(self): + def visit_end(self) -> None:Ignorant question: what's the difference between -> None (like here) and nothing (like
__init__()above?
Looks like commit message of PATCH 24 has an answer.
Copy to all the commits that omit -> None similarly?
Quoted 4 lines
Quoted 3 lines
scripts/qapi/events.py +0
-0self._add_system_module('emit', ' * QAPI Events emission') self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h"[...]
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 3 lines · Markus Armbruster
Looks like commit message of PATCH 24 has an answer.
Copy to all the commits that omit -> None similarly?
Probably not needed.
It's covered by the class basics section in the mypy manual; https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
and if you should happen to omit annotations for __init__ entirely as a novice, you will be treated to messages such as these:
qapi/source.py:18: error: Function is missing a return type annotation qapi/source.py:18: note: Use "-> None" if function does not return a value Found 1 error in 1 file (checked 14 source files)
Pretty good error!
There's no error if you DO explicitly add a -> None from __init__, but at worst it's just extraneous (but correct) information.
I could add a note to the style guide that I prefer omitting the return from __init__. I like omitting as much as I possibly can.
(You'll notice I don't always type every local, either -- when local inference is accurate, I leave it alone.)
--js
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 12 lines · John Snow
Quoted 2 lines · Markus Armbruster
Looks like commit message of PATCH 24 has an answer. Copy to all the commits that omit -> None similarly?
Probably not needed.
It's covered by the class basics section in the mypy manual;
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methodsand if you should happen to omit annotations for
__init__entirely as a novice, you will be treated to messages such as these:
Quoted 8 lines
qapi/source.py:18: error: Function is missing a return type annotationqapi/source.py:18: note: Use "-> None" if function does not return a value Found 1 error in 1 file (checked 14 source files)Pretty good error!
There's no error if you DO explicitly add a -> None from
__init__, but at worst it's just extraneous (but correct) information.
Let me apply the zero-one-infinity rule:
* Zero: explain it in none of the commit messages, i.e. dumb down PATCH 24.
* One: explain it in one. Do it in the first one, please (here, I think).
* Infinity: explain it in every one.
Up to you!
Quoted 5 lines
I could add a note to the style guide that I prefer omitting the return from
__init__. I like omitting as much as I possibly can.(You'll notice I don't always type every local, either -- when local inference is accurate, I leave it alone.)
Type inference can save us from repeating the obvious over and over, and that's lovely.
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 28 lines · Markus Armbruster
Quoted 12 lines · John Snow
Quoted 2 lines · Markus Armbruster
Looks like commit message of PATCH 24 has an answer. Copy to all the commits that omit -> None similarly?
Probably not needed.
It's covered by the class basics section in the mypy manual;
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methodsand if you should happen to omit annotations for
__init__entirely as a novice, you will be treated to messages such as these:Quoted 8 lines
qapi/source.py:18: error: Function is missing a return type annotationqapi/source.py:18: note: Use "-> None" if function does not return a value Found 1 error in 1 file (checked 14 source files)Pretty good error!
There's no error if you DO explicitly add a -> None from
__init__, but at worst it's just extraneous (but correct) information.Let me apply the zero-one-infinity rule:
* Zero: explain it in none of the commit messages, i.e. dumb down PATCH 24.
Quoted 7 lines
* One: explain it in one. Do it in the first one, please (here, I think).
* Infinity: explain it in every one.
Up to you!
I'm just bad at predicting which things people will want explained. I know people don't read the cover letters already, so I'd rather go for less instead of more.
I think you and Cleber each noticed a different angle of this: Cleber noticed the first time I *did* annotate __init__'s return and you noticed the first time I *didn't*.
I'll just add it here too, but I have doubts it will be useful reference once it's merged. I guess it doesn't hurt to add it either, I just find it difficult to predict what reviewers will want.
Quoted 9 lines
Quoted 5 lines
I could add a note to the style guide that I prefer omitting the return from
__init__. I like omitting as much as I possibly can.(You'll notice I don't always type every local, either -- when local inference is accurate, I leave it alone.)
Type inference can save us from repeating the obvious over and over, and that's lovely.
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
John Snow <[email protected]> writes:
[...]
Quoted 1 line
I just find it difficult to predict what reviewers will want.
Capricious bunch, those reviewers.
[...]
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
In reply to Markus ArmbrusterQuoted 2 lines · Markus Armbruster
Ignorant question: what's the difference between -> None (like here) and nothing (like
__init__()above?
This came up in Cleber's review, too.
Mypy supports a gradual typing paradigm; it is designed to facilitate what we are doing here: the gradual adding of types until we are able to enforce static typing everywhere.
To that end, mypy uses a simple heuristic to determine if a function is "typed" or "untyped": did you use any type annotations for it?
Meanwhile, __init__ never returns anything. You do not need to annotate its return type. mypy knows what the return type is and must be. In the case of __init__ with no parameters, it is both untyped and strictly typed!
Annotating the return type for no-parameter init methods convinces mypy that this is a strictly typed method. It doesn't do this automatically, because doing so might enable more checks than you were ready for simply because mypy was able to accurately surmise the typing of just __init__.
So it's just a little flip switch to enable strict typing, really.
--js
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 24 lines · John Snow
Quoted 2 lines · Markus Armbruster
Ignorant question: what's the difference between -> None (like here) and nothing (like
__init__()above?This came up in Cleber's review, too.
Mypy supports a gradual typing paradigm; it is designed to facilitate what we are doing here: the gradual adding of types until we are able to enforce static typing everywhere.
To that end, mypy uses a simple heuristic to determine if a function is "typed" or "untyped": did you use any type annotations for it?
Meanwhile,
__init__never returns anything. You do not need to annotate its return type. mypy knows what the return type is and must be. In the case of__init__with no parameters, it is both untyped and strictly typed!Annotating the return type for no-parameter init methods convinces mypy that this is a strictly typed method. It doesn't do this automatically, because doing so might enable more checks than you were ready for simply because mypy was able to accurately surmise the typing of just
__init__.
Quoted 5 lines
So it's just a little flip switch to enable strict typing, really.
--js
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
I now understand we can omit -> None, except when it's the only type hint, because omitting it then would make it untyped, which is not what we want.
Is this just for __init__(), or is it for any function that doesn't return anything?
Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · John Snow
Quoted 2 lines · Markus Armbruster
Ignorant question: what's the difference between -> None (like here) and nothing (like
__init__()above?This came up in Cleber's review, too.
Mypy supports a gradual typing paradigm; it is designed to facilitate what we are doing here: the gradual adding of types until we are able to enforce static typing everywhere.
To that end, mypy uses a simple heuristic to determine if a function is "typed" or "untyped": did you use any type annotations for it?
Meanwhile,
__init__never returns anything. You do not need to annotate its return type. mypy knows what the return type is and must be. In the case of__init__with no parameters, it is both untyped and strictly typed!Annotating the return type for no-parameter init methods convinces mypy that this is a strictly typed method. It doesn't do this automatically, because doing so might enable more checks than you were ready for simply because mypy was able to accurately surmise the typing of just
__init__.So it's just a little flip switch to enable strict typing, really.
--js
Quoted 9 lines
I now understand we can omit -> None, except when it's the only type hint, because omitting it then would make it untyped, which is not what we want.
Is this just for
__init__(), or is it for any function that doesn't return anything?
*just* __init__. Some other dunders have types that could be assumed, but aren't. I asked about this for some other well-known but hard-to-type dunders like __exit__ on the Typing-SIG list. Not a strong response yet, I might push again later.
--js
PATCH v5 20/36 qapi/events.py: Move comments into docstrings
In reply to John SnowClarify them while we're here.
--- scripts/qapi/events.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/qapi/events.py +5 -1
@@ -44,8 +44,12 @@ def gen_event_send_decl(name: str, proto=build_event_send_proto(name, arg_type, boxed)) -# Declare and initialize an object 'qapi' using parameters from build_params() def gen_param_var(typ: QAPISchemaObjectType) -> str: + """ + Generate a struct variable holding the event parameters. + + Initialize it with the function arguments defined in `gen_event_send`. + """ assert not typ.variants ret = mcgen(''' %(c_name)s param = {
Signature · 1 line
2.26.2
PATCH v5 21/36 qapi/commands.py: Don't re-bind to variable of different type
In reply to John SnowMypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
--- scripts/qapi/commands.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +1 -3
@@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): if not options: options = ['QCO_NO_OPTIONS'] - options = " | ".join(options) - ret = mcgen(''' qmp_register_command(cmds, "%(name)s", qmp_marshal_%(c_name)s, %(opts)s); ''', name=name, c_name=c_name(name), - opts=options) + opts=" | ".join(options)) return ret
Signature · 1 line
2.26.2
Re: [PATCH v5 21/36] qapi/commands.py: Don't re-bind to variable of different type
Quoted 18 lines · John Snow
Mypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/commands.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)scripts/qapi/commands.py +0
-0@@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): if not options: options = ['QCO_NO_OPTIONS']
scripts/qapi/commands.py def gen_register_command(name, success_response, allow_oob, allow_preconfig): · line 204
Quoted 10 lines
scripts/qapi/commands.py +1
-3- options = " | ".join(options) - ret = mcgen(''' qmp_register_command(cmds, "%(name)s", qmp_marshal_%(c_name)s, %(opts)s); ''', name=name, c_name=c_name(name), - opts=options) + opts=" | ".join(options)) return ret
Definitely neater.
PATCH v5 22/36 qapi/commands.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/commands.py | 71 ++++++++++++++++++++++++++++++---------- 1 file changed, 54 insertions(+), 17 deletions(-)
scripts/qapi/commands.py +54 -17
@@ -13,16 +13,34 @@ See the COPYING file in the top-level directory. """ +from typing import ( + Dict, + List, + Optional, + Set, +) + from .common import c_name, mcgen from .gen import ( + QAPIGenC, QAPIGenCCode, QAPISchemaModularCVisitor, build_params, ifcontext, ) +from .schema import ( + QAPISchema, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaType, +) +from .source import QAPISourceInfo -def gen_command_decl(name, arg_type, boxed, ret_type): +def gen_command_decl(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: return mcgen(''' %(c_type)s qmp_%(c_name)s(%(params)s); ''',
@@ -31,7 +49,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type): params=build_params(arg_type, boxed, 'Error **errp')) -def gen_call(name, arg_type, boxed, ret_type): +def gen_call(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: ret = '' argstr = ''
@@ -67,7 +88,7 @@ def gen_call(name, arg_type, boxed, ret_type): return ret -def gen_marshal_output(ret_type): +def gen_marshal_output(ret_type: QAPISchemaType) -> str: return mcgen(''' static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -88,19 +109,22 @@ def gen_marshal_output(ret_type): c_type=ret_type.c_type(), c_name=ret_type.c_name()) -def build_marshal_proto(name): +def build_marshal_proto(name: str) -> str: return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)) -def gen_marshal_decl(name): +def gen_marshal_decl(name: str) -> str: return mcgen(''' %(proto)s; ''', proto=build_marshal_proto(name)) -def gen_marshal(name, arg_type, boxed, ret_type): +def gen_marshal(name: str, + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ret_type: Optional[QAPISchemaType]) -> str: have_args = boxed or (arg_type and not arg_type.is_empty()) ret = mcgen('''
@@ -182,7 +206,10 @@ def gen_marshal(name, arg_type, boxed, ret_type): return ret -def gen_register_command(name, success_response, allow_oob, allow_preconfig): +def gen_register_command(name: str, + success_response: bool, + allow_oob: bool, + allow_preconfig: bool) -> str: options = [] if not success_response:
@@ -204,7 +231,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig): return ret -def gen_registry(registry, prefix): +def gen_registry(registry: str, prefix: str) -> str: ret = mcgen(''' void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -221,15 +248,14 @@ def gen_registry(registry, prefix): class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): - - def __init__(self, prefix): + def __init__(self, prefix: str): super().__init__( prefix, 'qapi-commands', ' * Schema-defined QAPI/QMP commands', None, __doc__) self._regy = QAPIGenCCode(None) - self._visited_ret_types = {} + self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {} - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: self._visited_ret_types[self._genc] = set() commands = self._module_basename('qapi-commands', name) types = self._module_basename('qapi-types', name)
@@ -253,7 +279,7 @@ def _begin_user_module(self, name): ''', types=types)) - def visit_end(self): + def visit_end(self) -> None: self._add_system_module('init', ' * QAPI Commands initialization') self._genh.add(mcgen(''' #include "qapi/qmp/dispatch.h"
@@ -269,9 +295,18 @@ def visit_end(self): prefix=self._prefix)) self._genc.add(gen_registry(self._regy.get_content(), self._prefix)) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig): + def visit_command(self, + name: str, + info: QAPISourceInfo, + ifcond: List[str], + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool) -> None: if not gen: return # FIXME: If T is a user-defined type, the user is responsible
@@ -292,7 +327,9 @@ def visit_command(self, name, info, ifcond, features, allow_oob, allow_preconfig)) -def gen_commands(schema, output_dir, prefix): +def gen_commands(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: vis = QAPISchemaGenCommandVisitor(prefix) schema.visit(vis) vis.write(output_dir)
Signature · 1 line
2.26.2
PATCH v5 23/36 qapi/commands.py: enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.error] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy
Quoted 13 lines · John Snow
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)scripts/qapi/mypy.ini +0
-0@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False
scripts/qapi/mypy.ini strict_optional = False · line 9
Quoted 10 lines
scripts/qapi/mypy.ini +0
-5python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.error] disallow_untyped_defs = False disallow_incomplete_defs = False
The equivalent change for events.py you squashed into the commit adding type hints. Any particular reason for not doing the same here?
Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy
Quoted 17 lines · Markus Armbruster
Quoted 15 lines · John Snow
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)scripts/qapi/mypy.ini +0
-0@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6
Quoted 12 lines
scripts/qapi/mypy.ini strict_optional = False · line 9
Quoted 8 lines
scripts/qapi/mypy.ini +0
-5-[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.error] disallow_untyped_defs = False disallow_incomplete_defs = FalseThe equivalent change for
events.pyyou squashed into the commit adding type hints. Any particular reason for not doing the same here?
Just making my life easier for re-arranging commits and rebasing. By separating them out whenever I had > 1 fix patch prior, I was able to freely re-arrange and re-order the prior fix patches.
They can be squashed on commit if desired, but for my own sake and inability to predict review comments, they are separate.
--js
Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy
Quoted 30 lines · John Snow
Quoted 19 lines · Markus Armbruster
Quoted 17 lines · John Snow
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)scripts/qapi/mypy.ini +0
-2@@ -4,11 +4,6 @@ strict_optional = False disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.commands] -disallow_untyped_defs = False -disallow_incomplete_defs = FalseQuoted 9 lines
scripts/qapi/mypy.ini strict_optional = False · line 9
Quoted 5 lines
scripts/qapi/mypy.ini +0
-2-check_untyped_defs = False - [mypy-qapi.error] disallow_untyped_defs = False disallow_incomplete_defs = FalseThe equivalent change for
events.pyyou squashed into the commit adding
type hints. Any particular reason for not doing the same here?
Quoted 6 lines
Just making my life easier for re-arranging commits and rebasing. By separating them out whenever I had > 1 fix patch prior, I was able to freely re-arrange and re-order the prior fix patches.
They can be squashed on commit if desired, but for my own sake and inability to predict review comments, they are separate.
I only asked because some are separate, and some are not.
Squashing on commit is an easy way toward consistency here.
PATCH v5 24/36 qapi/source.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior.
This commit *only* adds annotations.
A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.
Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.
--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -34,11 +34,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/source.py +18 -13
@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]): self.fname = fname self.line = line self.parent = parent - self.pragma = parent.pragma if parent else QAPISchemaPragma() - self.defn_meta = None - self.defn_name = None + self.pragma: QAPISchemaPragma = ( + parent.pragma if parent else QAPISchemaPragma() + ) + self.defn_meta: Optional[str] = None + self.defn_name: Optional[str] = None - def set_defn(self, meta, name): + def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self): + def next_line(self: T) -> T: info = copy.copy(self) info.line += 1 return info - def loc(self): + def loc(self) -> str: if self.fname is None: return sys.argv[0] ret = self.fname
@@ -49,13 +54,13 @@ def loc(self): ret += ':%d' % self.line return ret - def in_defn(self): + def in_defn(self) -> str: if self.defn_name: return "%s: In %s '%s':\n" % (self.fname, self.defn_meta, self.defn_name) return '' - def include_path(self): + def include_path(self) -> str: ret = '' parent = self.parent while parent:
@@ -63,5 +68,5 @@ def include_path(self): parent = parent.parent return ret - def __str__(self): + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc()
Signature · 1 line
2.26.2
Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations
Quoted 54 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.A note on typing of
__init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.Since
__init__may never return any value, it is preferred to omit the return annotation whenever possible.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-5@@ -34,11 +34,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +3
-2@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = []
scripts/qapi/source.py line 30
Quoted 10 lines
scripts/qapi/source.py +4
-2# Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):
More ignorant questions (I'm abusing the review process to learn Python type hints)...
Why do you need to annotate self here, but not elsewhere?
Why do you use T instead of QAPISourceInfo?
scripts/qapi/source.py line 36
Quoted 9 lines
scripts/qapi/source.py +3
-3self.fname = fname self.line = line self.parent = parent - self.pragma = parent.pragma if parent else QAPISchemaPragma() - self.defn_meta = None - self.defn_name = None + self.pragma: QAPISchemaPragma = ( + parent.pragma if parent else QAPISchemaPragma() + )
Type inference fail?
Quoted 42 lines
scripts/qapi/source.py +8
-6+ self.defn_meta: Optional[str] = None + self.defn_name: Optional[str] = None - def set_defn(self, meta, name): + def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self): + def next_line(self: T) -> T: info = copy.copy(self) info.line += 1 return info - def loc(self): + def loc(self) -> str: if self.fname is None: return sys.argv[0] ret = self.fname@@ -49,13 +54,13 @@ def loc(self): ret += ':%d' % self.line return ret - def in_defn(self): + def in_defn(self) -> str: if self.defn_name: return "%s: In %s '%s':\n" % (self.fname, self.defn_meta, self.defn_name) return '' - def include_path(self): + def include_path(self) -> str: ret = '' parent = self.parent while parent:@@ -63,5 +68,5 @@ def include_path(self): parent = parent.parent return ret - def __str__(self): + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc()
Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations
Quoted 70 lines · Markus Armbruster
Quoted 54 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.A note on typing of
__init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.Since
__init__may never return any value, it is preferred to omit the return annotation whenever possible.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-5@@ -34,11 +34,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +3
-2@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = [] + self.returns_whitelist: List[str] = []scripts/qapi/source.py line 30
Quoted 10 lines
scripts/qapi/source.py +4
-2# Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):More ignorant questions (I'm abusing the review process to learn Python type hints)...
Quoted 2 lines
Why do you need to annotate self here, but not elsewhere?
This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.
This is a pattern that allows for subclassing. I am stating that this __init__ method takes a parent of the same type as itself, whatever that happens to actually be.
T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.
(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)
Quoted 12 lines
Why do you use T instead of QAPISourceInfo?
scripts/qapi/source.py line 36
Quoted 9 lines
scripts/qapi/source.py +3
-3self.fname = fname self.line = line self.parent = parent - self.pragma = parent.pragma if parent else QAPISchemaPragma() - self.defn_meta = None - self.defn_name = None + self.pragma: QAPISchemaPragma = ( + parent.pragma if parent else QAPISchemaPragma() + )
Quoted 2 lines
Type inference fail?
Yes:
qapi/source.py:34: error: Cannot determine type of 'pragma' Found 1 error in 1 file (checked 14 source files)
Quoted 42 lines
Quoted 42 lines
scripts/qapi/source.py +8
-6+ self.defn_meta: Optional[str] = None + self.defn_name: Optional[str] = None - def set_defn(self, meta, name): + def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self): + def next_line(self: T) -> T: info = copy.copy(self) info.line += 1 return info - def loc(self): + def loc(self) -> str: if self.fname is None: return sys.argv[0] ret = self.fname@@ -49,13 +54,13 @@ def loc(self): ret += ':%d' % self.line return ret - def in_defn(self): + def in_defn(self) -> str: if self.defn_name: return "%s: In %s '%s':\n" % (self.fname, self.defn_meta, self.defn_name) return '' - def include_path(self): + def include_path(self) -> str: ret = '' parent = self.parent while parent:@@ -63,5 +68,5 @@ def include_path(self): parent = parent.parent return ret - def __str__(self): + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc()
Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations
Quoted 82 lines · John Snow
Quoted 66 lines · Markus Armbruster
Quoted 50 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.A note on typing of
__init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.Since
__init__may never return any value, it is preferred to omit the return annotation whenever possible.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-4@@ -34,11 +34,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +2
-2@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = []scripts/qapi/source.py line 27
Quoted 10 lines
scripts/qapi/source.py +5
-2+ self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):More ignorant questions (I'm abusing the review process to learn Python type hints)...
Quoted 1 line
Why do you need to annotate self here, but not elsewhere?
This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.
This is a pattern that allows for subclassing. I am stating that this
__init__method takes a parent of the same type as itself, whatever that happens to actually be.T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.
Quoted 5 lines
(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)
Say I define class QSISub as a direct subclass of QAPISourceInfo, and let it inherit __init__(). What's the type of QSISub.__init__()'s parameter @parent?
According to my reading of your explanation, it's QSISub. Correct?
If we used QAPISourceInfo instead of T for @parent, then it would be QAPISourceInfo. Correct?
Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really needs to be a QSISub. We can't know when we write QAPISourceInfo. But we don't *have* to get this exactly right for all future subclasses, because I can always override __init__() when inheritance doesn't give me the __init__() I want. Correct?
Say I override __init__(), and have it call super().__init__(). I have to pass it a QAPISourceInfo @parent. A QSISub will do (it's a subtype). Correct?
One more: is bound='QAPISourceInfo' strictly needed?
Quoted 1 line
Quoted 1 line
Why do you use T instead of QAPISourceInfo?
[...]
Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations
Quoted 90 lines · Markus Armbruster
Quoted 82 lines · John Snow
Quoted 66 lines · Markus Armbruster
Quoted 50 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.A note on typing of
__init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.Since
__init__may never return any value, it is preferred to omit the return annotation whenever possible.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++------------- 2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-4@@ -34,11 +34,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.source] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.types] disallow_untyped_defs = False disallow_incomplete_defs = Falsescripts/qapi/source.py +2
-2@@ -11,37 +11,42 @@ import copy import sys +from typing import List, Optional, TypeVar class QAPISchemaPragma: - def __init__(self): + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False # Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = []scripts/qapi/source.py line 27
Quoted 10 lines
scripts/qapi/source.py +5
-2+ self.returns_whitelist: List[str] = [] # Whitelist of entities allowed to violate case conventions - self.name_case_whitelist = [] + self.name_case_whitelist: List[str] = [] class QAPISourceInfo: - def __init__(self, fname, line, parent): + T = TypeVar('T', bound='QAPISourceInfo') + + def __init__(self: T, fname: str, line: int, parent: Optional[T]):More ignorant questions (I'm abusing the review process to learn Python type hints)...
Quoted 1 line
Why do you need to annotate self here, but not elsewhere?
This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.
This is a pattern that allows for subclassing. I am stating that this
__init__method takes a parent of the same type as itself, whatever that happens to actually be.T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.
Quoted 5 lines
(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)
Quoted 6 lines
Say I define class QSISub as a direct subclass of QAPISourceInfo, and let it inherit
__init__(). What's the type of QSISub.__init__()'s parameter @parent?According to my reading of your explanation, it's QSISub. Correct?
That's right.
(I'm realizing that this is maybe not a constraint that we should even anticipate here, because maybe we don't wish to say that the parent should always be of the same type. but hey, it led to a good mypy lesson.
I'm going to edit it to do the simpler thing for now and leave well enough alone. There's another chance to see an interesting pattern of TypeVars in the error series in part 4 that I think is actually more explicitly appropriate.)
Quoted 3 lines
If we used QAPISourceInfo instead of T for @parent, then it would be QAPISourceInfo. Correct?
Yup!
Here's a little sample program that shows what this kind of typing does:
```
from typing import TypeVar, Optional
class Example:
T = TypeVar('T', bound='Example')
def __init__(self: T, parent: Optional[T] = None):
self.parent = parent
class SubExample(Example):
pass
x = Example()
y = Example(x)
z = SubExample()
a = Example(x) # OK
b = Example(z) # OK
c = SubExample(x) # BZZZT
d = SubExample(z) # OK
```If you check this with mypy, you'll get this error:
``` test.py:17: error: Argument 1 to "SubExample" has incompatible type "Example"; expected "Optional[SubExample]" Found 1 error in 1 file (checked 1 source file) ```
Quoted 6 lines
Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really needs to be a QSISub. We can't know when we write QAPISourceInfo. But we don't *have* to get this exactly right for all future subclasses, because I can always override
__init__()when inheritance doesn't give me the__init__()I want. Correct?
You could, but I suggested on IRC the other day that I am not fully comfortable with the LSP rules that mypy (sometimes?) enforces. I tend not to want to override init with narrower types if I can avoid it, but it is true that we do this quite a lot in the codebase already.
(I believe I have seen mypy throw errors about this on occasion, but I can't pinpoint the exact circumstances in which it does. It's a point of confusion for me.)
Quoted 6 lines
Say I override
__init__(), and have it call super().__init__(). I have to pass it a QAPISourceInfo @parent. A QSISub will do (it's a subtype). Correct?One more: is bound='QAPISourceInfo' strictly needed?
I'm not sure if bound is strictly required or not. mypy docs just use it outright and don't mention why:
https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self
Generally, this constrains a TypeVar to some upper bound, see --
https://mypy.readthedocs.io/en/stable/generics.html#type-variables-with-upper-bounds
-- but in this case, we're only using that TypeVar for an init method that only exists for this type and below, so it might be redundant.
I modified my example program to remove the bound and it appears to fail in the same exact way, so it might be pointless in this case. It might have a stronger use if you want to re-use the 'T' typevar elsewhere where it isn't implicitly bound by the 'self' argument. Maybe it has implications for multi-inheritance too, I've not tested it to that level.
Thanks for the good question!
--js
Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations
Quoted 164 lines · John Snow
Quoted 89 lines · Markus Armbruster
Quoted 81 lines · John Snow
Quoted 65 lines · Markus Armbruster
Quoted 51 lines · John Snow
Annotations do not change runtime behavior.
This commit *only* adds annotations.A note on typing of
__init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.Since
__init__may never return any value, it is preferred to omit the return annotation whenever possible.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>---
scripts/qapi/mypy.ini | 5 -----scripts/qapi/source.py| 31 ++++++++++++++++++-------------2 files changed, 18 insertions(+), 18 deletions(-)scripts/qapi/mypy.ini +0
-0@@ -34,11 +34,6 @@ disallow_untyped_defs = False
disallow_incomplete_defs= False
check_untyped_defs= False
-[mypy-qapi.source]scripts/qapi/mypy.ini +0
-4-disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False -[mypy-qapi.types]
disallow_untyped_defs= False
disallow_incomplete_defs= Falsescripts/qapi/source.py +0
-0@@ -11,37 +11,42 @@import copy
import sys
+from typing import List, Optional, TypeVar
class QAPISchemaPragma:
- def__init__(self):
+ def__init__(self)-> None:
# Are documentation comments required? self.doc_required = False
# Whitelist of commands allowed to return a non-dictionary - self.returns_whitelist = []
+ self.returns_whitelist: List[str] = []
# Whitelist of entities allowed to violate case conventionsQuoted 8 lines
- self.name_case_whitelist = []
+ self.name_case_whitelist: List[str] = [] class QAPISourceInfo:
- def__init__(self, fname, line, parent):
+ T =TypeVar('T', bound='QAPISourceInfo')
+
+ def __init__(self: T, fname: str, line: int, parent:
Optional[T]):More ignorant questions (I'm abusing the review process to learn Python type hints)...
Quoted 1 line
Why do you need to annotate self here, but not elsewhere?
This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.
This is a pattern that allows for subclassing. I am stating that this
__init__method takes a parent of the same type as itself, whatever that happens to actually be.T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.
Quoted 5 lines
(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)
Quoted 6 lines
Say I define class QSISub as a direct subclass of QAPISourceInfo, and let it inherit __init__(). What's the type of QSISub.__init__()'s parameter @parent?
According to my reading of your explanation, it's QSISub. Correct?
That's right.
(I'm realizing that this is maybe not a constraint that we should even anticipate here, because maybe we don't wish to say that the parent should always be of the same type. but hey, it led to a good mypy lesson.
I'm going to edit it to do the simpler thing for now and leave well enough alone. There's another chance to see an interesting pattern of TypeVars in the error series in part 4 that I think is actually more explicitly appropriate.)
Quoted 3 lines
If we used QAPISourceInfo instead of T for @parent, then it would be QAPISourceInfo. Correct?
Yup!
Here's a little sample program that shows what this kind of typing does:
``` from typing import TypeVar, Optional class Example: T = TypeVar('T', bound='Example') def __init__(self: T, parent: Optional[T] = None): self.parent = parent class SubExample(Example): pass x = Example() y = Example(x) z = SubExample() a = Example(x) # OK b = Example(z) # OK c = SubExample(x) # BZZZT d = SubExample(z) # OK ```If you check this with mypy, you'll get this error:
``` test.py:17: error: Argument 1 to "SubExample" has incompatible type "Example"; expected "Optional[SubExample]" Found 1 error in 1 file (checked 1 source file) ```Quoted 6 lines
Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really needs to be a QSISub. We can't know when we write QAPISourceInfo. But we don't *have* to get this exactly right for all future subclasses, because I can always override
__init__()when inheritance doesn't give me the__init__()I want. Correct?You could, but I suggested on IRC the other day that I am not fully comfortable with the LSP rules that mypy (sometimes?) enforces. I tend not to want to override init with narrower types if I can avoid it, but it is true that we do this quite a lot in the codebase already.
(I believe I have seen mypy throw errors about this on occasion, but I can't pinpoint the exact circumstances in which it does. It's a point of confusion for me.)
Quoted 6 lines
Say I override
__init__(), and have it call super().__init__(). I have to pass it a QAPISourceInfo @parent. A QSISub will do (it's a subtype). Correct?One more: is bound='QAPISourceInfo' strictly needed?
I'm not sure if bound is strictly required or not. mypy docs just use it outright and don't mention why:
https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self
Generally, this constrains a TypeVar to some upper bound, see --
Quoted 8 lines
https://mypy.readthedocs.io/en/stable/generics.html#type-variables-with-upper-bounds
-- but in this case, we're only using that TypeVar for an init method that only exists for this type and below, so it might be redundant.
I modified my example program to remove the bound and it appears to fail in the same exact way, so it might be pointless in this case. It might have a stronger use if you want to re-use the 'T' typevar elsewhere where it isn't implicitly bound by the 'self' argument. Maybe it has implications for multi-inheritance too, I've not tested it to that level.
SPOKE TOO SOON, ADDENDUM:
Adding bound= is still necessary to allow type inference to assume the minimal base form. It's not needed for my toy example because I don't do anything *with* the variable.
In this case, though, the next_line method needs help:
def next_line(self: T) -> T:
info = copy.copy(self)
info.line += 1
return infoNow, when we return a copy of ourselves, we are going to return something that is the same type as ourselves. If we remove the bound from the TypeVar here, "info.line" will become an error because mypy cannot infer the minimum form.
(It doesn't seem smart enough to notice that the class it is defined in does not inherit from anywhere, so it MUST be at least a QAPISourceInfo.)
So I am going to:
(1) Make __init__'s typing less dynamic, because it's not really needed.
(2) Leave the TypeVar in for the benefit of next_line().
--js
PATCH v5 25/36 qapi/source.py: delint with pylint
In reply to John SnowShush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/source.py | 3 +++ 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/pylintrc +0 -1
@@ -7,7 +7,6 @@ ignore-patterns=error.py, gen.py, parser.py, schema.py, - source.py, types.py, visit.py,
scripts/qapi/source.py +3 -0
@@ -15,6 +15,9 @@ class QAPISchemaPragma: + # Replace with @dataclass in Python 3.7+ + # pylint: disable=too-few-public-methods + def __init__(self) -> None: # Are documentation comments required? self.doc_required = False
Signature · 1 line
2.26.2
PATCH v5 26/36 qapi/gen.py: Fix edge-case of _is_user_module
In reply to John SnowThe edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +1 -1
@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./')) @staticmethod def _is_builtin_module(name):
Signature · 1 line
2.26.2
Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 9 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module
In reply to John SnowQuoted 2 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
The edge case is impossible, as discussed in review of v2. I figure the type checker can't see that, so we need to help it some. Can we mention this in the commit message?
Quoted 13 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0
scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 246
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./'))
return not (name is None or name.startswith('./')
Looks slightly clearer to me.
Quoted 3 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):
Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module
Quoted 9 lines · Markus Armbruster
Quoted 2 lines · John Snow
The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.
The edge case is impossible, as discussed in review of v2. I figure the type checker can't see that, so we need to help it some. Can we mention this in the commit message?
Sure. It's quite possible we will be able to better model and constrain these types, but that will come after all of these patches.
Mypy just has no way of knowing that '' is forbidden. That constraint does not exist in the type system we have here at present.
You could, by the way, create a type called NonEmptyString and create a function that casts from str to NonEmptyString by means of an assertion and then annotating the return type.
Anyway, I'll update the commit message for now.
Quoted 25 lines
Quoted 13 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>--- scripts/qapi/gen.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/qapi/gen.py +0
-0scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 246
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): @staticmethod def _is_user_module(name): - return name and not name.startswith('./') + return bool(name and not name.startswith('./'))return not (name is None or
name.startswith('./')Looks slightly clearer to me.
Quoted 3 lines
scripts/qapi/gen.py +0
-0@staticmethod def _is_builtin_module(name):
PATCH v5 27/36 qapi/gen.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +57 -47
@@ -17,7 +17,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,
@@ -29,31 +35,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.
@@ -78,7 +84,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif
@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]): super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str): super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
@@ -167,7 +171,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */
@@ -177,16 +181,15 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]: """ A with-statement context manager that wraps with `start_if()` / `end_if()`.
@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args): class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + blurb: str, + pydoc: str): self._prefix = prefix self._what = what self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc): self._genh = QAPIGenH(self._prefix + self._what + '.h', blurb, pydoc) - def write(self, output_dir): + def write(self, output_dir: str) -> None: self._genc.write(output_dir) self._genh.write(output_dir) class QAPISchemaModularCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + user_blurb: str, + builtin_blurb: Optional[str], + pydoc: str): self._prefix = prefix self._what = what self._user_blurb = user_blurb self._builtin_blurb = builtin_blurb self._pydoc = pydoc - self._genc = None - self._genh = None - self._module = {} - self._main_module = None + self._genc: Optional[QAPIGenC] = None + self._genh: Optional[QAPIGenH] = None + self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {} + self._main_module: Optional[str] = None @staticmethod - def _is_user_module(name): + def _is_user_module(name: Optional[str]) -> bool: return bool(name and not name.startswith('./')) @staticmethod - def _is_builtin_module(name): + def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what, name): + def _module_dirname(self, what: str, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return '' - def _module_basename(self, what, name): + def _module_basename(self, what: str, name: Optional[str]) -> str: ret = '' if self._is_builtin_module(name) else self._prefix if self._is_user_module(name): basename = os.path.basename(name)
@@ -266,27 +276,27 @@ def _module_basename(self, what, name): ret += re.sub(r'-', '-' + name + '-', what) return ret - def _module_filename(self, what, name): + def _module_filename(self, what: str, name: Optional[str]) -> str: return os.path.join(self._module_dirname(what, name), self._module_basename(what, name)) - def _add_module(self, name, blurb): + def _add_module(self, name: Optional[str], blurb: str) -> None: basename = self._module_filename(self._what, name) genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) self._genc, self._genh = self._module[name] - def _add_user_module(self, name, blurb): + def _add_user_module(self, name: str, blurb: str) -> None: assert self._is_user_module(name) if self._main_module is None: self._main_module = name self._add_module(name, blurb) - def _add_system_module(self, name, blurb): + def _add_system_module(self, name: Optional[str], blurb: str) -> None: self._add_module(name and './' + name, blurb) - def write(self, output_dir, opt_builtins=False): + def write(self, output_dir: str, opt_builtins: bool = False) -> None: for name in self._module: if self._is_builtin_module(name) and not opt_builtins: continue
@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False): genc.write(output_dir) genh.write(output_dir) - def _begin_system_module(self, name): + def _begin_system_module(self, name: None) -> None: pass - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: pass - def visit_module(self, name): + def visit_module(self, name: Optional[str]) -> None: if name is None: if self._builtin_blurb: self._add_system_module(None, self._builtin_blurb)
@@ -314,7 +324,7 @@ def visit_module(self, name): self._add_user_module(name, self._user_blurb) self._begin_user_module(name) - def visit_include(self, name, info): + def visit_include(self, name: str, info: QAPISourceInfo) -> None: relname = os.path.relpath(self._module_filename(self._what, name), os.path.dirname(self._genh.fname)) self._genh.preamble_add(mcgen('''
Signature · 1 line
2.26.2
Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations
Quoted 30 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)scripts/qapi/gen.py +7
-1@@ -17,7 +17,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,
scripts/qapi/gen.py line 42
Quoted 10 lines
scripts/qapi/gen.py +2
-2@@ -29,31 +35,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfo class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]):
I'd expect fname: str. Can you point me to the spot that passes None?
Quoted 39 lines
scripts/qapi/gen.py +7
-7self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.@@ -78,7 +84,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endif
scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 127
Quoted 6 lines
scripts/qapi/gen.py +1
-2@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]):
Likewise.
Quoted 22 lines
scripts/qapi/gen.py +4
-4super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble)
scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 152
Quoted 10 lines
scripts/qapi/gen.py +2
-3- def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content() class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str):
Here it's just str.
Quoted 26 lines
scripts/qapi/gen.py +3
-4super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */@@ -167,7 +171,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */@@ -177,16 +181,15 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str:
scripts/qapi/gen.py def _bottom(self): · line 192
Quoted 10 lines
scripts/qapi/gen.py +2
-2return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname) @contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:
Oh, the type hint for a *args is QAPIGenCCode, even though args is a tuple of excess positional arguments (which are all QAPIGenCCode).
Quoted 105 lines
scripts/qapi/gen.py +25
-18""" A with-statement context manager that wraps with `start_if()` / `end_if()`.@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args): class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + blurb: str, + pydoc: str): self._prefix = prefix self._what = what self._genc = QAPIGenC(self._prefix + self._what + '.c',@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc): self._genh = QAPIGenH(self._prefix + self._what + '.h', blurb, pydoc) - def write(self, output_dir): + def write(self, output_dir: str) -> None: self._genc.write(output_dir) self._genh.write(output_dir) class QAPISchemaModularCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + user_blurb: str, + builtin_blurb: Optional[str], + pydoc: str): self._prefix = prefix self._what = what self._user_blurb = user_blurb self._builtin_blurb = builtin_blurb self._pydoc = pydoc - self._genc = None - self._genh = None - self._module = {} - self._main_module = None + self._genc: Optional[QAPIGenC] = None + self._genh: Optional[QAPIGenH] = None + self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {} + self._main_module: Optional[str] = None @staticmethod - def _is_user_module(name): + def _is_user_module(name: Optional[str]) -> bool: return bool(name and not name.startswith('./')) @staticmethod - def _is_builtin_module(name): + def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what, name): + def _module_dirname(self, what: str, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return '' - def _module_basename(self, what, name): + def _module_basename(self, what: str, name: Optional[str]) -> str: ret = '' if self._is_builtin_module(name) else self._prefix if self._is_user_module(name): basename = os.path.basename(name)@@ -266,27 +276,27 @@ def _module_basename(self, what, name): ret += re.sub(r'-', '-' + name + '-', what) return ret - def _module_filename(self, what, name): + def _module_filename(self, what: str, name: Optional[str]) -> str: return os.path.join(self._module_dirname(what, name), self._module_basename(what, name)) - def _add_module(self, name, blurb): + def _add_module(self, name: Optional[str], blurb: str) -> None: basename = self._module_filename(self._what, name) genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) self._genc, self._genh = self._module[name] - def _add_user_module(self, name, blurb): + def _add_user_module(self, name: str, blurb: str) -> None: assert self._is_user_module(name) if self._main_module is None: self._main_module = name self._add_module(name, blurb) - def _add_system_module(self, name, blurb): + def _add_system_module(self, name: Optional[str], blurb: str) -> None: self._add_module(name and './' + name, blurb) - def write(self, output_dir, opt_builtins=False): + def write(self, output_dir: str, opt_builtins: bool = False) -> None: for name in self._module: if self._is_builtin_module(name) and not opt_builtins: continue
scripts/qapi/gen.py def write(self, output_dir, opt_builtins=False): · line 307
Quoted 5 lines
scripts/qapi/gen.py +1
-1@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False): genc.write(output_dir) genh.write(output_dir) - def _begin_system_module(self, name): + def _begin_system_module(self, name: None) -> None:
I figure the type hint None signals a simplifcation opportunity. No need to worry about it now.
Quoted 20 lines
scripts/qapi/gen.py +3
-3pass - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: pass - def visit_module(self, name): + def visit_module(self, name: Optional[str]) -> None: if name is None: if self._builtin_blurb: self._add_system_module(None, self._builtin_blurb)@@ -314,7 +324,7 @@ def visit_module(self, name): self._add_user_module(name, self._user_blurb) self._begin_user_module(name) - def visit_include(self, name, info): + def visit_include(self, name: str, info: QAPISourceInfo) -> None: relname = os.path.relpath(self._module_filename(self._what, name), os.path.dirname(self._genh.fname)) self._genh.preamble_add(mcgen('''
Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations
Quoted 38 lines · Markus Armbruster
Quoted 36 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)scripts/qapi/gen.py +8
-1@@ -17,7 +17,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,@@ -29,31 +35,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfo
Quoted 7 lines
scripts/qapi/gen.py line 42
Quoted 4 lines
scripts/qapi/gen.py +1
-2class QAPIGen: - - def __init__(self, fname): + def __init__(self, fname: Optional[str]):I'd expect fname: str. Can you point me to the spot that passes None?
qapi/commands.py: self._regy = QAPIGenCCode(None)
Good time to mention again: I am disabling strict none checks for now in this conversion.
That means we can pass Optional[T] to functions expecting T and mypy will not complain. This should obviously be fixed long-term, but the cleanups involve things that are a higher class of finnicky.
If this alarms you, good! We'll fix it in time, but it doesn't break anything any worse than it already was.
If this check was disabled, you would be able to edit the Optional[str] to str and see what broke. But because I use this affordance for now, you would see no difference.
Quoted 76 lines
Quoted 39 lines
scripts/qapi/gen.py +7
-7self.fname = fname self._preamble = '' self._body = '' - def preamble_add(self, text): + def preamble_add(self, text: str) -> None: self._preamble += text - def add(self, text): + def add(self, text: str) -> None: self._body += text - def get_content(self): + def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() - def _top(self): + def _top(self) -> str: return '' - def _bottom(self): + def _bottom(self) -> str: return '' - def write(self, output_dir): + def write(self, output_dir: str) -> None: # Include paths starting with ../ are used to reuse modules of the main # schema in specialised schemas. Don't overwrite the files that are # already generated for the main schema.@@ -78,7 +84,7 @@ def write(self, output_dir): f.close() -def _wrap_ifcond(ifcond, before, after): +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str: if before == after: return after # suppress empty #if ... #endifscripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 127
Quoted 6 lines
scripts/qapi/gen.py +1
-2@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType], class QAPIGenCCode(QAPIGen): - - def __init__(self, fname): + def __init__(self, fname: Optional[str]):Likewise.
Quoted 28 lines
scripts/qapi/gen.py +5
-5super().__init__(fname) - self._start_if = None + self._start_if: Optional[Tuple[List[str], str, str]] = None - def start_if(self, ifcond): + def start_if(self, ifcond: List[str]) -> None: assert self._start_if is None self._start_if = (ifcond, self._body, self._preamble) - def end_if(self): + def end_if(self) -> None: assert self._start_if self._wrap_ifcond() self._start_if = None - def _wrap_ifcond(self): + def _wrap_ifcond(self) -> None: self._body = _wrap_ifcond(self._start_if[0], self._start_if[1], self._body) self._preamble = _wrap_ifcond(self._start_if[0], self._start_if[2], self._preamble) - def get_content(self): + def get_content(self) -> str: assert self._start_if is None return super().get_content()
Quoted 7 lines
scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 152
Quoted 4 lines
scripts/qapi/gen.py +1
-2class QAPIGenC(QAPIGenCCode): - - def __init__(self, fname, blurb, pydoc): + def __init__(self, fname: str, blurb: str, pydoc: str):Here it's just str.
That's right.
Quoted 33 lines
Quoted 33 lines
scripts/qapi/gen.py +4
-5super().__init__(fname) self._blurb = blurb self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc, re.MULTILINE)) - def _top(self): + def _top(self) -> str: return mcgen(''' /* AUTOMATICALLY GENERATED, DO NOT MODIFY */@@ -167,7 +171,7 @@ def _top(self): ''', blurb=self._blurb, copyright=self._copyright) - def _bottom(self): + def _bottom(self) -> str: return mcgen(''' /* Dummy declaration to prevent empty .o file */@@ -177,16 +181,15 @@ def _bottom(self): class QAPIGenH(QAPIGenC): - - def _top(self): + def _top(self) -> str: return super()._top() + guardstart(self.fname) - def _bottom(self): + def _bottom(self) -> str: return guardend(self.fname)
Quoted 7 lines
scripts/qapi/gen.py def _bottom(self): · line 192
Quoted 3 lines
scripts/qapi/gen.py +1
-1@contextmanager -def ifcontext(ifcond, *args): +def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:Oh, the type hint for a *args is QAPIGenCCode, even though args is a tuple of excess positional arguments (which are all QAPIGenCCode).
Yeah, it's the way type hints interact with the special '*' syntax. This is an Iterable of QAPIGenCCode.
**kwargs works the same way: you annotate the values, and the keys are assumed to be str.
Quoted 108 lines
Quoted 108 lines
scripts/qapi/gen.py +25
-18""" A with-statement context manager that wraps with `start_if()` / `end_if()`.@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args): class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + blurb: str, + pydoc: str): self._prefix = prefix self._what = what self._genc = QAPIGenC(self._prefix + self._what + '.c',@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc): self._genh = QAPIGenH(self._prefix + self._what + '.h', blurb, pydoc) - def write(self, output_dir): + def write(self, output_dir: str) -> None: self._genc.write(output_dir) self._genh.write(output_dir) class QAPISchemaModularCVisitor(QAPISchemaVisitor): - - def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): + def __init__(self, + prefix: str, + what: str, + user_blurb: str, + builtin_blurb: Optional[str], + pydoc: str): self._prefix = prefix self._what = what self._user_blurb = user_blurb self._builtin_blurb = builtin_blurb self._pydoc = pydoc - self._genc = None - self._genh = None - self._module = {} - self._main_module = None + self._genc: Optional[QAPIGenC] = None + self._genh: Optional[QAPIGenH] = None + self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {} + self._main_module: Optional[str] = None @staticmethod - def _is_user_module(name): + def _is_user_module(name: Optional[str]) -> bool: return bool(name and not name.startswith('./')) @staticmethod - def _is_builtin_module(name): + def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what, name): + def _module_dirname(self, what: str, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return '' - def _module_basename(self, what, name): + def _module_basename(self, what: str, name: Optional[str]) -> str: ret = '' if self._is_builtin_module(name) else self._prefix if self._is_user_module(name): basename = os.path.basename(name)@@ -266,27 +276,27 @@ def _module_basename(self, what, name): ret += re.sub(r'-', '-' + name + '-', what) return ret - def _module_filename(self, what, name): + def _module_filename(self, what: str, name: Optional[str]) -> str: return os.path.join(self._module_dirname(what, name), self._module_basename(what, name)) - def _add_module(self, name, blurb): + def _add_module(self, name: Optional[str], blurb: str) -> None: basename = self._module_filename(self._what, name) genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) self._genc, self._genh = self._module[name] - def _add_user_module(self, name, blurb): + def _add_user_module(self, name: str, blurb: str) -> None: assert self._is_user_module(name) if self._main_module is None: self._main_module = name self._add_module(name, blurb) - def _add_system_module(self, name, blurb): + def _add_system_module(self, name: Optional[str], blurb: str) -> None: self._add_module(name and './' + name, blurb) - def write(self, output_dir, opt_builtins=False): + def write(self, output_dir: str, opt_builtins: bool = False) -> None: for name in self._module: if self._is_builtin_module(name) and not opt_builtins: continue@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False): genc.write(output_dir) genh.write(output_dir)
Quoted 6 lines
scripts/qapi/gen.py def write(self, output_dir, opt_builtins=False): · line 307
Quoted 2 lines
scripts/qapi/gen.py +1
-1- def _begin_system_module(self, name): + def _begin_system_module(self, name: None) -> None:I figure the type hint None signals a simplifcation opportunity. No need to worry about it now.
Yes, the module name stuff in general has a smell to it. I didn't look deeper, but there's an opportunity for cleaning it up. I realize the type signature here has parity with _begin_user_module, but I noticed it was *never* called with anything that wasn't a literal None, so I added the stricter type.
Quoted 20 lines
Quoted 20 lines
scripts/qapi/gen.py +3
-3pass - def _begin_user_module(self, name): + def _begin_user_module(self, name: str) -> None: pass - def visit_module(self, name): + def visit_module(self, name: Optional[str]) -> None: if name is None: if self._builtin_blurb: self._add_system_module(None, self._builtin_blurb)@@ -314,7 +324,7 @@ def visit_module(self, name): self._add_user_module(name, self._user_blurb) self._begin_user_module(name) - def visit_include(self, name, info): + def visit_include(self, name: str, info: QAPISourceInfo) -> None: relname = os.path.relpath(self._module_filename(self._what, name), os.path.dirname(self._genh.fname)) self._genh.preamble_add(mcgen('''
Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations
In reply to John SnowQuoted 30 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)scripts/qapi/gen.py +7
-1@@ -17,7 +17,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,
scripts/qapi/gen.py line 38
Quoted 4 lines
scripts/qapi/gen.py +1
-0@@ -29,31 +35,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfo
PATCH 03 has a similar cleanup. Are there more? Perhaps a separate patch doing just this kind of cleanup would make sense. Up to you.
[...]
Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations
Quoted 40 lines · Markus Armbruster
Quoted 30 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)scripts/qapi/gen.py +7
-1@@ -17,7 +17,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,scripts/qapi/gen.py line 38
Quoted 4 lines
scripts/qapi/gen.py +1
-0@@ -29,31 +35,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfoPATCH 03 has a similar cleanup. Are there more? Perhaps a separate patch doing just this kind of cleanup would make sense. Up to you.
Quoted 2 lines
[...]
This isn't a cleanup, I am just importing QAPISourceInfo to use for an annotation. It's relevant and required for this patch, and doesn't make sense on its own.
Patch 03 ... Oh, you mean identifying the correct location of QAPIError. Uh... nah? I think that was the only case of that one changing. Not worth pulling out or naming, I think.
Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations
Quoted 41 lines · John Snow
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 104 ++++++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 47 deletions(-)scripts/qapi/gen.py +7
-1@@ -17,7 +17,13 @@ import errno import os import re -from typing import Optional +from typing import ( + Dict, + Iterator, + List, + Optional, + Tuple, +) from .common import ( c_fname,Quoted 9 lines
scripts/qapi/gen.py line 38
Quoted 5 lines
scripts/qapi/gen.py +1
-0@@ -29,31 +35,31 @@ mcgen, ) from .schema import QAPISchemaObjectType, QAPISchemaVisitor +from .source import QAPISourceInfoPATCH 03 has a similar cleanup. Are there more? Perhaps a separate patch doing just this kind of cleanup would make sense. Up to you. [...]
Quoted 3 lines
This isn't a cleanup, I am just importing QAPISourceInfo to use for an annotation. It's relevant and required for this patch, and doesn't make sense on its own.
I was mistaken. A case of patch-review-eye...
Quoted 3 lines
Patch 03 ... Oh, you mean identifying the correct location of QAPIError. Uh... nah? I think that was the only case of that one changing. Not worth pulling out or naming, I think.
Agree.
PATCH v5 28/36 qapi/gen.py: Enable checking with mypy
In reply to John Snow--- scripts/qapi/mypy.ini | 5 ----- 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -14,11 +14,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.gen] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.introspect] disallow_untyped_defs = False disallow_incomplete_defs = False
Signature · 1 line
2.26.2
PATCH v5 29/36 qapi/gen.py: Remove unused parameter
In reply to John Snowmodule_basename doesn't use the 'what' argument, so remove it.
--- scripts/qapi/gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool: def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what: str, name: Optional[str]) -> str: + def _module_dirname(self, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return ''
@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str: return ret def _module_filename(self, what: str, name: Optional[str]) -> str: - return os.path.join(self._module_dirname(what, name), + return os.path.join(self._module_dirname(name), self._module_basename(what, name)) def _add_module(self, name: Optional[str], blurb: str) -> None:
Signature · 1 line
2.26.2
Re: [PATCH v5 29/36] qapi/gen.py: Remove unused parameter
Quoted 1 line · John Snow
module_basenamedoesn't use the 'what' argument, so remove it.
_module_dirname(), I suppose.
Quoted 30 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/qapi/gen.py +2
-2@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool: def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what: str, name: Optional[str]) -> str: + def _module_dirname(self, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return ''@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str: return ret def _module_filename(self, what: str, name: Optional[str]) -> str: - return os.path.join(self._module_dirname(what, name), + return os.path.join(self._module_dirname(name), self._module_basename(what, name)) def _add_module(self, name: Optional[str], blurb: str) -> None:
Re: [PATCH v5 29/36] qapi/gen.py: Remove unused parameter
Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow
module_basenamedoesn't use the 'what' argument, so remove it.
_module_dirname(), I suppose.
My brain just translates stuff I didn't ask it to all the time. (Fixed.)
Quoted 30 lines
Quoted 30 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/qapi/gen.py +2
-2@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool: def _is_builtin_module(name: Optional[str]) -> bool: return not name - def _module_dirname(self, what: str, name: Optional[str]) -> str: + def _module_dirname(self, name: Optional[str]) -> str: if self._is_user_module(name): return os.path.dirname(name) return ''@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str: return ret def _module_filename(self, what: str, name: Optional[str]) -> str: - return os.path.join(self._module_dirname(what, name), + return os.path.join(self._module_dirname(name), self._module_basename(what, name)) def _add_module(self, name: Optional[str], blurb: str) -> None:
PATCH v5 30/36 qapi/gen.py: update write() to be more idiomatic
In reply to John SnowMake the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.
--- scripts/qapi/gen.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +11 -14
@@ -14,7 +14,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import (
@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None: return pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try: - os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True) + + # use os.open for O_CREAT to create and read a non-existant file fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close() + with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text) def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Signature · 1 line
2.26.2
Re: [PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic
Quoted 28 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-)scripts/qapi/gen.py +0
-1@@ -14,7 +14,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import (@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None: return
scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 71
Quoted 10 lines
scripts/qapi/gen.py +2
-5pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try: - os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True)
I wouldn't call this part "heavily subjective". When wrote the old code, exist_ok was still off limits (it's new in 3.2).
Quoted 11 lines
scripts/qapi/gen.py +2
-8+ + # use os.open for O_CREAT to create and read a non-existant file fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close()
scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 84
Quoted 10 lines
scripts/qapi/gen.py +7
-0+ with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text) def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Re: [PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic
Quoted 35 lines · Markus Armbruster
Quoted 33 lines · John Snow
Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception, use
fdopen()to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/gen.py | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-)scripts/qapi/gen.py +1
-2@@ -14,7 +14,6 @@ # See the COPYING file in the top-level directory. from contextlib import contextmanager -import errno import os import re from typing import (@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None: return pathname = os.path.join(output_dir, self.fname) odir = os.path.dirname(pathname) + if odir: - try:
Quoted 9 lines
scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 71
Quoted 5 lines
scripts/qapi/gen.py +1
-4- os.makedirs(odir) - except os.error as e: - if e.errno != errno.EEXIST: - raise + os.makedirs(odir, exist_ok=True)I wouldn't call this part "heavily subjective". When wrote the old code,
exist_okwas still off limits (it's new in 3.2).
It's cool if you agree, I just realize that what people consider idiomatic is subjective unless it's enforcable by a tool. This isn't.
"I made this look more like if I wrote it, which caused Dopamine" is a bad commit message. (But more true.)
Quoted 20 lines
Quoted 20 lines
scripts/qapi/gen.py +9
-8+ + # use os.open for O_CREAT to create and read a non-existant file fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f = open(fd, 'r+', encoding='utf-8') - text = self.get_content() - oldtext = f.read(len(text) + 1) - if text != oldtext: - f.seek(0) - f.truncate(0) - f.write(text) - f.close() + with os.fdopen(fd, 'r+', encoding='utf-8') as fp: + text = self.get_content() + oldtext = fp.read(len(text) + 1) + if text != oldtext: + fp.seek(0) + fp.truncate(0) + fp.write(text)
Quoted 4 lines
scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 84
Quoted 1 line
scripts/qapi/gen.py +0
-0def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:Reviewed-by Markus Armbruster <[email protected]>
Thanks, though :)
PATCH v5 31/36 qapi/gen.py: delint with pylint
In reply to John Snow'fp' and 'fd' are self-evident in context, add them to the list of OK names.
_top and _bottom also need to stay standard methods because some users override the method and need to use self. Tell pylint to shush.
--- scripts/qapi/gen.py | 2 ++ scripts/qapi/pylintrc | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -0
@@ -53,9 +53,11 @@ def get_content(self) -> str: return self._top() + self._preamble + self._body + self._bottom() def _top(self) -> str: + # pylint: disable=no-self-use return '' def _bottom(self) -> str: + # pylint: disable=no-self-use return '' def write(self, output_dir: str) -> None:
scripts/qapi/pylintrc +3 -2
@@ -4,7 +4,6 @@ # The regex matches against base names, not paths. ignore-patterns=error.py, expr.py, - gen.py, parser.py, schema.py, types.py,
@@ -45,7 +44,9 @@ good-names=i, k, ex, Run, - _ + _, + fp, # fp = open(...) + fd, # fd = os.open(...) [VARIABLES]
Signature · 1 line
2.26.2
PATCH v5 32/36 qapi/types.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++----------- 2 files changed, 64 insertions(+), 27 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -29,11 +29,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.types] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.visit] disallow_untyped_defs = False disallow_incomplete_defs = False
scripts/qapi/types.py +64 -22
207 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v5 33/36 qapi/types.py: remove one-letter variables
In reply to John Snow"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/types.py | 29 +++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -6,7 +6,6 @@ ignore-patterns=error.py, expr.py, parser.py, schema.py, - types.py, visit.py,
scripts/qapi/types.py +15 -14
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str, .array = (const char *const[]) { ''', c_name=c_name(name)) - for m in members: - ret += gen_if(m.ifcond) - index = c_enum_const(name, m.name, prefix) + for member in members: + ret += gen_if(member.ifcond) + index = c_enum_const(name, member.name, prefix) ret += mcgen(''' [%(index)s] = "%(name)s", ''', - index=index, name=m.name) - ret += gen_endif(m.ifcond) + index=index, name=member.name) + ret += gen_endif(member.ifcond) ret += mcgen(''' },
@@ -79,13 +79,13 @@ def gen_enum(name: str, ''', c_name=c_name(name)) - for m in enum_members: - ret += gen_if(m.ifcond) + for member in enum_members: + ret += gen_if(member.ifcond) ret += mcgen(''' %(c_enum)s, ''', - c_enum=c_enum_const(name, m.name, prefix)) - ret += gen_endif(m.ifcond) + c_enum=c_enum_const(name, member.name, prefix)) + ret += gen_endif(member.ifcond) ret += mcgen(''' } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name) ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for variant in variants.variants if variants else (): + obj = variant.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''
Signature · 1 line
2.26.2
Re: [PATCH v5 33/36] qapi/types.py: remove one-letter variables
Quoted 30 lines · John Snow
"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/pylintrc | 1 - scripts/qapi/types.py | 29 +++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)scripts/qapi/pylintrc +0
-1@@ -6,7 +6,6 @@ ignore-patterns=error.py, expr.py, parser.py, schema.py, - types.py, visit.py,scripts/qapi/types.py +0
-0
scripts/qapi/types.py def gen_enum_lookup(name: str, · line 52
Quoted 7 lines
scripts/qapi/types.py +1
-3@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str, .array = (const char *const[]) { ''', c_name=c_name(name)) - for m in members: - ret += gen_if(m.ifcond) - index = c_enum_const(name, m.name, prefix) + for member in members:
Let's use @memb. It's more visually distinct from @members, and gen_struct_members() already uses it.
Quoted 31 lines
scripts/qapi/types.py +8
-6+ ret += gen_if(member.ifcond) + index = c_enum_const(name, member.name, prefix) ret += mcgen(''' [%(index)s] = "%(name)s", ''', - index=index, name=m.name) - ret += gen_endif(m.ifcond) + index=index, name=member.name) + ret += gen_endif(member.ifcond) ret += mcgen(''' },@@ -79,13 +79,13 @@ def gen_enum(name: str, ''', c_name=c_name(name)) - for m in enum_members: - ret += gen_if(m.ifcond) + for member in enum_members: + ret += gen_if(member.ifcond) ret += mcgen(''' %(c_enum)s, ''', - c_enum=c_enum_const(name, m.name, prefix)) - ret += gen_endif(m.ifcond) + c_enum=c_enum_const(name, member.name, prefix)) + ret += gen_endif(member.ifcond) ret += mcgen(''' } %(c_name)s;
scripts/qapi/types.py def gen_object(name: str, ifcond: List[str], · line 151
Quoted 9 lines
scripts/qapi/types.py +1
-5@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name) ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for variant in variants.variants if variants else ():
Let's use @var, for consistency with gen_variants().
Quoted 7 lines
scripts/qapi/types.py +5
-0+ obj = variant.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''
Re: [PATCH v5 33/36] qapi/types.py: remove one-letter variables
Quoted 27 lines · Markus Armbruster
Quoted 25 lines · John Snow
"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/pylintrc | 1 - scripts/qapi/types.py | 29 +++++++++++++++-------------- 2 files changed, 15 insertions(+), 15 deletions(-)scripts/qapi/pylintrc +0
-1@@ -6,7 +6,6 @@ ignore-patterns=error.py, expr.py, parser.py, schema.py, - types.py, visit.py,
Quoted 16 lines
scripts/qapi/types.py def gen_enum_lookup(name: str, · line 52
Quoted 12 lines
scripts/qapi/types.py +1
-3@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str, .array = (const char *const[]) { ''', c_name=c_name(name)) - for m in members: - ret += gen_if(m.ifcond) - index = c_enum_const(name, m.name, prefix) + for member in members:Let's use @memb. It's more visually distinct from @members, and
gen_struct_members()already uses it.
OK.
Quoted 33 lines
Quoted 33 lines
scripts/qapi/types.py +8
-6+ ret += gen_if(member.ifcond) + index = c_enum_const(name, member.name, prefix) ret += mcgen(''' [%(index)s] = "%(name)s", ''', - index=index, name=m.name) - ret += gen_endif(m.ifcond) + index=index, name=member.name) + ret += gen_endif(member.ifcond) ret += mcgen(''' },@@ -79,13 +79,13 @@ def gen_enum(name: str, ''', c_name=c_name(name)) - for m in enum_members: - ret += gen_if(m.ifcond) + for member in enum_members: + ret += gen_if(member.ifcond) ret += mcgen(''' %(c_enum)s, ''', - c_enum=c_enum_const(name, m.name, prefix)) - ret += gen_endif(m.ifcond) + c_enum=c_enum_const(name, member.name, prefix)) + ret += gen_endif(member.ifcond) ret += mcgen(''' } %(c_name)s;@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str], objects_seen.add(name)
Quoted 10 lines
scripts/qapi/types.py def gen_object(name: str, ifcond: List[str], · line 151
Quoted 7 lines
scripts/qapi/types.py +1
-5ret = '' - if variants: - for v in variants.variants: - if isinstance(v.type, QAPISchemaObjectType): - ret += gen_object(v.type.name, v.type.ifcond, v.type.base, - v.type.local_members, v.type.variants) + for variant in variants.variants if variants else ():Let's use @var, for consistency with
gen_variants().
You're the boss.
(Said only when I don't actually still kinda sorta want to do it my way. Then you're the boss, but in the bad way. Like I am the petulant teenager working at a Burger King and I am having a disagreement with the Assistant Manager over the proper way to stock the ketchup packets.)
Quoted 7 lines
Quoted 7 lines
scripts/qapi/types.py +5
-0+ obj = variant.type + if not isinstance(obj, QAPISchemaObjectType): + continue + ret += gen_object(obj.name, obj.ifcond, obj.base, + obj.local_members, obj.variants) ret += mcgen('''
PATCH v5 34/36 qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
In reply to John SnowThis is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.
--- scripts/qapi/visit.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +7 -5
@@ -22,7 +22,7 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import QAPISchemaEnumType, QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType) + ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Signature · 1 line
2.26.2
Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 24 lines · John Snow
This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/visit.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)scripts/qapi/visit.py +1
-1@@ -22,7 +22,7 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import QAPISchemaEnumType, QAPISchemaObjectType def gen_visit_decl(name, scalar=False):
scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 88
Quoted 5 lines
scripts/qapi/visit.py +2
-0@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType)
I'm curious: do you need the local variable to make the assertion stick?
Quoted 16 lines
scripts/qapi/visit.py +4
-4+ ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 28 lines · Markus Armbruster
Quoted 26 lines · John Snow
This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/visit.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)scripts/qapi/visit.py +1
-1@@ -22,7 +22,7 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import QAPISchemaEnumType, QAPISchemaObjectType def gen_visit_decl(name, scalar=False):@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond)
Quoted 6 lines
scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 88
Quoted 3 lines
scripts/qapi/visit.py +2
-0if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType)I'm curious: do you need the local variable to make the assertion stick?
No, but it only sticks to the binding and not the data. i.e. assertions to downcast work on the *name*.
(This comes up somewhere in the schema.py patches where I make a change that looks completely pointless, but it makes mypy happy.)
I could have left it alone. I just saw a lot of repeated multi-dots and habitually created a temporary local for the purpose.
Quoted 16 lines
Quoted 16 lines
scripts/qapi/visit.py +4
-4+ ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 34 lines · John Snow
Quoted 23 lines · Markus Armbruster
Quoted 21 lines · John Snow
This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/visit.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)scripts/qapi/visit.py +1
-1@@ -22,7 +22,7 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import QAPISchemaEnumType, QAPISchemaObjectTypeQuoted 9 lines
scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 87
Quoted 6 lines
scripts/qapi/visit.py +2
-0def gen_visit_decl(name, scalar=False):@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType)I'm curious: do you need the local variable to make the assertion stick?
Quoted 2 lines
No, but it only sticks to the binding and not the data. i.e. assertions to downcast work on the *name*.
Would it stick to variants.tag_member, too?
I'm asking to learn, not to find a reason to make you change your patch.
Quoted 5 lines
(This comes up somewhere in the
schema.pypatches where I make a change that looks completely pointless, but it makes mypy happy.)I could have left it alone. I just saw a lot of repeated multi-dots and habitually created a temporary local for the purpose.
Matter of taste. Long chains of dots make the code hard to read because they are so long. Temporary variable make it hard to read because you have to remember what they mean. Tradeoff. I come up with cases I find hard to decide all too often.
In case the local variable isn't needed for mypy: when you throw in something that isn't needed for the patch's stated purpose, it's best to mention it in the commit message, because not mentioning it is a review comment magnet :)
Put yourself in the reviewers shoes. Your lovingly crafted commit message puts him into a positive mood. He nods along while reading your obvious patch at a good pace. And then he runs smack into the unexpected unrelated part, and stops: oh, what's going on here? Back up some and read more slowly to make sure I understand.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
scripts/qapi/visit.py +4
-4+ ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
Quoted 36 lines · Markus Armbruster
Quoted 34 lines · John Snow
Quoted 32 lines · Markus Armbruster
Quoted 23 lines · John Snow
This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>--- scripts/qapi/visit.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)scripts/qapi/visit.py +1
-1@@ -22,7 +22,7 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaObjectType +from .schema import QAPISchemaEnumType, QAPISchemaObjectType def gen_visit_decl(name, scalar=False):scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 87
Quoted 4 lines
scripts/qapi/visit.py +2
-0@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants): ret += gen_endif(memb.ifcond) if variants: + tag_member = variants.tag_member + assert isinstance(tag_member.type, QAPISchemaEnumType)I'm curious: do you need the local variable to make the assertion stick?
Quoted 5 lines
Quoted 2 lines
No, but it only sticks to the binding and not the data. i.e. assertions to downcast work on the *name*.
Would it stick to variants.tag_member, too?
As long as variants isn't re-bound in that scope, it should stick within that scope, yeah. (I think. Didn't test. It's my expectation.)
Quoted 18 lines
I'm asking to learn, not to find a reason to make you change your patch.
Quoted 5 lines
(This comes up somewhere in the
schema.pypatches where I make a change that looks completely pointless, but it makes mypy happy.)I could have left it alone. I just saw a lot of repeated multi-dots and habitually created a temporary local for the purpose.
Matter of taste. Long chains of dots make the code hard to read because they are so long. Temporary variable make it hard to read because you have to remember what they mean. Tradeoff. I come up with cases I find hard to decide all too often.
In case the local variable isn't needed for mypy: when you throw in something that isn't needed for the patch's stated purpose, it's best to mention it in the commit message, because not mentioning it is a review comment magnet :)
Quoted 6 lines
Put yourself in the reviewers shoes. Your lovingly crafted commit message puts him into a positive mood. He nods along while reading your obvious patch at a good pace. And then he runs smack into the unexpected unrelated part, and stops: oh, what's going on here? Back up some and read more slowly to make sure I understand.
Yeah, understood. Well, part of it is knowing the review style of your reviewer too. It's been a while since we've personally swapped patches.
The actual process for much of this was: "There is an error! let me fix that error."
And then I do that, but I do so using idioms and patterns I'm familiar or comfortable with. And then post-hoc, not 100% of them were necessarily required, strictly, to fix the problem. Most of this series was written "one file at a time", and then split out post-fact into little per-warning changes.
I didn't even send part 1 until I finished typing the *entire* package, to make sure that things were as correct as possible from the very first commit.
I often don't really even notice that little changes aren't strictly necessary until they're challenged, it's usually not a conscious choice to try and sneak stuff in.
Still always trying to find a balance between "Easy to maintain and iterate" and "easy to review." Tough line for me.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
scripts/qapi/visit.py +4
-4+ ret += mcgen(''' switch (obj->%(c_name)s) { ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(tag_member.name)) for var in variants.variants: - case_str = c_enum_const(variants.tag_member.type.name, - var.name, - variants.tag_member.type.prefix) + case_str = c_enum_const(tag_member.type.name, var.name, + tag_member.type.prefix) ret += gen_if(var.ifcond) if var.type.name == 'q_empty': # valid variant and nothing to do
Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
John Snow <[email protected]> writes:
[...]
Quoted 2 lines
Still always trying to find a balance between "Easy to maintain and iterate" and "easy to review." Tough line for me.
I assure you it's a tough line for everyone.
[...]
PATCH v5 35/36 qapi/visit.py: remove unused parameters from gen_visit_object
In reply to John SnowAnd this fixes the pylint report for this file, so make sure we check this in the future, too.
--- scripts/qapi/pylintrc | 1 - scripts/qapi/visit.py | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/qapi/pylintrc +0 -1
@@ -6,7 +6,6 @@ ignore-patterns=error.py, expr.py, parser.py, schema.py, - visit.py, [MESSAGES CONTROL]
scripts/qapi/visit.py +2 -2
@@ -250,7 +250,7 @@ def gen_visit_alternate(name, variants): return ret -def gen_visit_object(name, base, members, variants): +def gen_visit_object(name): return mcgen(''' bool visit_type_%(c_name)s
@@ -343,7 +343,7 @@ def visit_object_type(self, name, info, ifcond, features, if not name.startswith('q_'): # only explicit types need an allocating visit self._genh.add(gen_visit_decl(name)) - self._genc.add(gen_visit_object(name, base, members, variants)) + self._genc.add(gen_visit_object(name)) def visit_alternate_type(self, name, info, ifcond, features, variants): with ifcontext(ifcond, self._genh, self._genc):
Signature · 1 line
2.26.2
Re: [PATCH v5 35/36] qapi/visit.py: remove unused parameters from gen_visit_object
Quoted 11 lines · John Snow
And this fixes the pylint report for this file, so make sure we check this in the future, too.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/pylintrc | 1 - scripts/qapi/visit.py | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-)
PATCH v5 36/36 qapi/visit.py: add type hint annotations
In reply to John SnowAnnotations do not change runtime behavior. This commit *only* adds annotations.
--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++---------- 2 files changed, 56 insertions(+), 22 deletions(-)
scripts/qapi/mypy.ini +0 -5
@@ -28,8 +28,3 @@ check_untyped_defs = False disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False - -[mypy-qapi.visit] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False
scripts/qapi/visit.py +56 -17
164 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v5 36/36] qapi/visit.py: add type hint annotations
Quoted 48 lines · John Snow
Annotations do not change runtime behavior. This commit *only* adds annotations.
Signed-off-by John Snow <[email protected]>Reviewed-by Eduardo Habkost <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Tested-by Cleber Rosa <[email protected]>--- scripts/qapi/mypy.ini | 5 --- scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++---------- 2 files changed, 56 insertions(+), 22 deletions(-)scripts/qapi/mypy.ini +0
-5@@ -28,8 +28,3 @@ check_untyped_defs = False disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False - -[mypy-qapi.visit] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = Falsescripts/qapi/visit.py +7
-1@@ -13,6 +13,8 @@ See the COPYING file in the top-level directory. """ +from typing import List, Optional + from .common import ( c_enum_const, c_name,@@ -22,10 +24,20 @@ mcgen, ) from .gen import QAPISchemaModularCVisitor, ifcontext -from .schema import QAPISchemaEnumType, QAPISchemaObjectType +from .schema import ( + QAPISchema, + QAPISchemaEnumMember, + QAPISchemaEnumType, + QAPISchemaFeature,
scripts/qapi/visit.py line 40
Quoted 10 lines
scripts/qapi/visit.py +7
-1+ QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariants, +) +from .source import QAPISourceInfo -def gen_visit_decl(name, scalar=False): +def gen_visit_decl(name: str, scalar: bool = False) -> str:
More abuse of review to learn: even through we initialize with False, we still need the type hint?
Quoted 3 lines
scripts/qapi/visit.py +0
-0c_type = c_name(name) + ' *' if not scalar: c_type += '*'
[...]
Re: [PATCH v5 36/36] qapi/visit.py: add type hint annotations
Quoted 2 lines · Markus Armbruster
More abuse of review to learn: even through we initialize with False, we still need the type hint?
Dunno.
jsnow@scv ~/s/q/scripts (python-qapi-cleanup-pt1)> mypy --config-file=qapi/mypy.ini qapi/; qapi/visit.py:40: error: Function is missing a type annotation for one or more arguments Found 1 error in 1 file (checked 14 source files)
...yup.
It also serves the purpose of constraining the type to also avoid unions with things that might be false-ish. Looks silly, but maybe future mypy installments will afford us more chances to use inference. It doesn't right now.
--js
Quoted 14 lines · John Snow
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Quoted 9 lines
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
Notes:
- After patch 07,
isort -cshould pass 100% on this and every future commit.
FIY, I went on to replicate/validate your testing, and I ran on patches 07 an later:
isort --check-only --settings-path ./scripts/qapi/.isort.cfg --recursive ./scripts/qapi
With success.
Quoted 3 lines
- After patch 08,
flake8 qapi/should pass 100% on this and every future commit.
Here on patches 08 an later, I've run:
flake8 ./scripts/qapi
And starting on patch 24 ("qapi/source.py: add type hint annotations") it complained about:
/scripts/qapi/source.py:44:31: F821 undefined name 'T'
Quoted 3 lines
- After patch 09,
pylint --rcfile=qapi/pylintrc qapi/should pass 100% on this and every future commit.
Here I ran on patches 09 and later:
pylint --rcfile=./scripts/qapi/pylintrc ./scripts/qapi/
And all succeeded.
Quoted 3 lines
- After patch 18,
mypy --config-file=qapi/mypy.ini qapi/should pass 100% on this and every future commit.
Here I ran on patches 18 and later:
mypy --config-file=./scripts/qapi/mypy.ini ./scripts/qapi/
And all succeeded.
- Cleber.
Quoted 8 lines · Cleber Rosa
Here on patches 08 an later, I've run:
flake8
./scripts/qapiAnd starting on patch 24 ("
qapi/source.py: add type hint annotations") it complained about:/scripts/qapi/source.py:44:31: F821 undefined name 'T'
I think that must be flake8 < 3.8.0. Try using >= 3.8.0.
(Yes, still working on getting proper pipenv/requirements/something up for these, sorry.)
Quoted 12 lines · John Snow
Quoted 8 lines · Cleber Rosa
Here on patches 08 an later, I've run:
flake8
./scripts/qapiAnd starting on patch 24 ("
qapi/source.py: add type hint annotations") it complained about:/scripts/qapi/source.py:44:31: F821 undefined name 'T'I think that must be flake8 < 3.8.0. Try using >= 3.8.0.
ACK, I was on 3.7.9. They're all passing now with 3.8.4.
- Cleber.
Quoted 10 lines · John Snow
Hi, this series adds static type hints to the QAPI module. This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)
Quoted 7 lines
In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.
Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.
Neat and pleasant to review. The care you put into this shows. Thanks!