Skip to contentSkip to conversation

qapi: static typing conversion, pt1

PATCH Superseded John Snow 5 participants 125 messages Oct 5, 2020 – Oct 9, 2020

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

v5 cover qapi: static typing conversion, pt1

Hi, this series adds static type hints to the QAPI module. This is part one!

Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6

- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)

In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.

Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.

Notes:

- After patch 07, isort -c should pass 100% on this and every future commit.

- After patch 08, flake8 qapi/ should pass 100% on this and every future commit.

- After patch 09, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.

- After patch 18, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.

Review Status:

[01] docs-repair-broken-references  #
[02] qapi-modify-docstrings-to-be   #
[03] qapi-gen-separate-arg-parsing  # [RB] CR,EH [TB] CR [SOB] JS
[04] qapi-move-generator-entrypoint # [RB] CR,EH [TB] CR [SOB] JS
[05] qapi-prefer-explicit-relative  # [RB] CR,EH [SOB] JS
[06] qapi-remove-wildcard-includes  # [RB] CR,EH [SOB] JS
[07] qapi-enforce-import-order      # [RB] CR [TB] CR [SOB] JS
[08] qapi-delint-using-flake8       # [RB] CR,EH [SOB] JS
[09] qapi-add-pylintrc              # [RB] CR [TB] CR,EH [SOB] JS
[10] qapi-common-py-remove-python   # [RB] CR,EH [SOB] JS
[11] qapi-common-add-indent-manager # [RB] CR,EH [SOB] JS
[12] qapi-common-py-delint-with     # [RB] CR,EH [SOB] JS
[13] replace-c-by-char              # [RB] CR,EH [SOB] JS
[14] qapi-common-py-check-with      # [RB] CR [TB] CR,EH [SOB] JS
[15] qapi-common-py-add-notational  # [RB] CR,EH [SOB] JS
[16] qapi-common-move-comments-into # [RB] CR,EH [SOB] JS
[17] qapi-split-build_params-into   # [RB] CR,EH [SOB] JS
[18] qapi-establish-mypy-type       # [RB] CR [TB] CR,EH [SOB] JS
[19] qapi-events-py-add-notational  # [RB] CR,EH [SOB] JS
[20] qapi-events-move-comments-into # [RB] CR,EH [SOB] JS
[21] qapi-commands-py-don-t-re-bind # [RB] CR,EH [SOB] JS
[22] qapi-commands-py-add           # [RB] CR,EH [SOB] JS
[23] qapi-commands-py-enable        # [RB] CR,EH [SOB] JS
[24] qapi-source-py-add-notational  # [RB] CR,EH [TB] CR [SOB] JS
[25] qapi-source-py-delint-with     # [RB] CR,EH [TB] CR [SOB] JS
[26] qapi-gen-py-fix-edge-case-of   # [RB] CR,EH [SOB] JS
[27] qapi-gen-py-add-notational     # [RB] CR,EH [SOB] JS
[28] qapi-gen-py-enable-checking    # [RB] CR,EH [TB] CR [SOB] JS
[29] qapi-gen-py-remove-unused      # [RB] CR,EH [SOB] JS
[30] qapi-gen-py-update-write-to-be # [RB] CR,EH [SOB] JS
[31] qapi-gen-py-delint-with-pylint # [RB] CR,EH [SOB] JS
[32] qapi-types-py-add-type-hint    # [RB] CR,EH [SOB] JS
[33] qapi-types-py-remove-one       # [RB] CR,EH [SOB] JS
[34] qapi-visit-py-assert           # [RB] CR,EH [SOB] JS
[35] qapi-visit-py-remove-unused    # [RB] CR,EH [TB] CR [SOB] JS
[36] qapi-visit-py-add-notational   # [RB] CR,EH [TB] CR [SOB] JS

Changelog:

002/36:[0012] [FC] 'qapi: modify docstrings to be sphinx-compatible' 027/36:[----] [-C] 'qapi/gen.py: add type hint annotations'

V5:
- Remove DO-NOT-MERGE patches (Now in Part2)
- Remove introspect.py patches (Now in Part2)
- 02: Docstring formatting, more commit message (Markus)

V4:
- Rebase on Peter Maydell's work;
- 05: Context differences from Peter Maydell's work
- 06: Remove qapi.doc
- 07: Remove qapi.doc, remove superfluous "if match"
- 09: Remove qapi.doc
- 13: remove qapi.doc
- 18: remove qapi.doc
- 22: remove qapi.doc
- 31: remove QAPIGenDoc changes

- Minor adjustments from list feedback;
- 01: More backticks for QAPI json files, now that they are in Sphinx-exe
- 02: Use :ref: role for the docker-ref cross-reference
- 04: Remove doc.py work changes; add references for start_if/end_if
- 10: Changes to accommodate isort
- 11: Added lines_after_imports=2
- 16: isort whitespace changes
- 24: Take Markus's docstring phrasing
- 34: add comment explaining os.open()
- 37: isort differences

V3:
- Use isort to enforce import consistency
- Use sphinx apidoc to check docstring format

V2:
- Removed Python3.6 patch in favor of Thomas Huth's
- Addressed (most) feedback from Markus
- Renamed type hint annotation commits

John Snow (36):
docs: repair broken references
qapi: modify docstrings to be sphinx-compatible
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into module
qapi: Prefer explicit relative imports
qapi: Remove wildcard includes
qapi: enforce import order/styling with isort
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable qapi/common.py: check with pylint
qapi/common.py: add type hint annotations qapi/common.py: Convert comments into docstrings, and elaborate qapi/common.py: move build_params into gen.py
qapi: establish mypy type-checking baseline qapi/events.py: add type hint annotations qapi/events.py: Move comments into docstrings qapi/commands.py: Don't re-bind to variable of different type qapi/commands.py: add type hint annotations qapi/commands.py: enable checking with mypy qapi/source.py: add type hint annotations qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module qapi/gen.py: add type hint annotations
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic qapi/gen.py: delint with pylint
qapi/types.py: add type hint annotations qapi/types.py: remove one-letter variables qapi/visit.py: assert tag_member contains a QAPISchemaEnumType qapi/visit.py: remove unused parameters from gen_visit_object qapi/visit.py: add type hint annotations

 docs/devel/multi-thread-tcg.rst |   2 +-
 docs/devel/testing.rst          |   2 +-
 scripts/qapi-gen.py             |  57 ++--------
 scripts/qapi/.flake8            |   2 +
 scripts/qapi/.isort.cfg         |   7 ++
 scripts/qapi/commands.py        |  87 +++++++++++----
 scripts/qapi/common.py          | 163 +++++++++++++++-------------
 scripts/qapi/events.py          |  58 +++++++---
 scripts/qapi/expr.py            |   7 +-
 scripts/qapi/gen.py             | 182 ++++++++++++++++++++------------
 scripts/qapi/introspect.py      |  16 ++-
 scripts/qapi/main.py            |  89 ++++++++++++++++
 scripts/qapi/mypy.ini           |  30 ++++++
 scripts/qapi/parser.py          |   7 +-
 scripts/qapi/pylintrc           |  70 ++++++++++++
 scripts/qapi/schema.py          |  33 +++---
 scripts/qapi/source.py          |  34 +++---
 scripts/qapi/types.py           | 125 +++++++++++++++-------
 scripts/qapi/visit.py           | 116 ++++++++++++++------
 19 files changed, 752 insertions(+), 335 deletions(-)
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/.isort.cfg
 create mode 100644 scripts/qapi/main.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/pylintrc
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v5 1/36 docs: repair broken references

In two different places, we are not making a cross-reference to some resource correctly.

---
 docs/devel/multi-thread-tcg.rst | 2 +-
 docs/devel/testing.rst          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
docs/devel/multi-thread-tcg.rst +1 -1
index 21483870dbc..92a9eba13c9 100644
@@ -267,7 +267,7 @@ of view of external observers (e.g. another processor core). They can
 apply to any memory operations as well as just loads or stores.
 
 The Linux kernel has an excellent `write-up
-<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`
+<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`_
 on the various forms of memory barrier and the guarantees they can
 provide.
 
docs/devel/testing.rst +1 -1
index bd64c1bdcdd..8875a40a2b6 100644
@@ -953,7 +953,7 @@ compiler flags are needed to build for a given target.
 If you have the ability to run containers as the user you can also
 take advantage of the build systems "Docker" support. It will then use
 containers to build any test case for an enabled guest where there is
-no system compiler available. See :ref: `_docker-ref` for details.
+no system compiler available. See :ref:`docker-ref` for details.
 
 Running subset of tests
 -----------------------
Signature · 1 line
2.26.2
author
#2 lore raw

PATCH v5 2/36 qapi: modify docstrings to be sphinx-compatible

In reply to John Snow

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.

A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
- Additional examples, diagrams, or other metadata follows below.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

---
 scripts/qapi/gen.py    | 6 ++++--
 scripts/qapi/parser.py | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -2
index ca66c82b5b8..dc7b94aa115 100644
@@ -154,9 +154,11 @@ def _bottom(self):
 
 @contextmanager
 def ifcontext(ifcond, *args):
-    """A 'with' statement context manager to wrap with start_if()/end_if()
+    """
+    A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
-    *args: any number of QAPIGenCCode
+    :param ifcond: A list of conditionals, passed to `start_if()`.
+    :param args: any number of `QAPIGenCCode`.
 
     Example::
 
scripts/qapi/parser.py +1 -0
index 9d1a3e2eea9..31bc2e6dca9 100644
@@ -381,6 +381,7 @@ def append(self, line):
 
         The way that the line is dealt with depends on which part of
         the documentation we're parsing right now:
+
         * The body section: ._append_line is ._append_body_line
         * An argument section: ._append_line is ._append_args_line
         * A features section: ._append_line is ._append_features_line
Signature · 1 line
2.26.2
#3 lore raw

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

Quoted 8 lines · John Snow

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.

A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

Quoted 7 lines

- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.

Mandatory when they apply?

Quoted 35 lines

- Additional examples, diagrams, or other metadata follows below.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

---
 scripts/qapi/gen.py    | 6 ++++--
 scripts/qapi/parser.py | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -2
index ca66c82b5b8..dc7b94aa115 100644
@@ -154,9 +154,11 @@ def _bottom(self):
 
 @contextmanager
 def ifcontext(ifcond, *args):
-    """A 'with' statement context manager to wrap with start_if()/end_if()
+    """
+    A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
-    *args: any number of QAPIGenCCode
+    :param ifcond: A list of conditionals, passed to `start_if()`.
+    :param args: any number of `QAPIGenCCode`.
 
     Example::
 
scripts/qapi/parser.py +0 -0
index 9d1a3e2eea9..31bc2e6dca9 100644

scripts/qapi/parser.py def append(self, line): · line 387

Quoted 7 lines
scripts/qapi/parser.py +1 -0
@@ -381,6 +381,7 @@ def append(self, line):
 
         The way that the line is dealt with depends on which part of
         the documentation we're parsing right now:
+
         * The body section: ._append_line is ._append_body_line
         * An argument section: ._append_line is ._append_args_line
         * A features section: ._append_line is ._append_features_line

I'm asking because you're not adding :param line: here.

Same for several other functions in this file.

In schema.py:

class QAPISchemaMember:
""" Represents object members, enum members and features """

Are the spaces next to """ okay?

author
#4 lore raw

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

Quoted 18 lines · Markus Armbruster
Quoted 8 lines · John Snow

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.

A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

Quoted 7 lines

- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.

Quoted 2 lines

Mandatory when they apply?

Subject of debate...

- Some people really hate obvious docstring comments.
- Some people really like the consistency.

Which type of developer am I? Guess it depends on when you ask.

Figured we'd hash that out when I go to write a style guide document.

Quoted 36 lines
Quoted 36 lines

- Additional examples, diagrams, or other metadata follows below.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

---
  scripts/qapi/gen.py    | 6 ++++--
  scripts/qapi/parser.py | 1 +
  2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -2
index ca66c82b5b8..dc7b94aa115 100644
@@ -154,9 +154,11 @@ def _bottom(self):
 
  @contextmanager
  def ifcontext(ifcond, *args):
-    """A 'with' statement context manager to wrap with start_if()/end_if()
+    """
+    A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
-    *args: any number of QAPIGenCCode
+    :param ifcond: A list of conditionals, passed to `start_if()`.
+    :param args: any number of `QAPIGenCCode`.
 
      Example::
 
scripts/qapi/parser.py +0 -0
index 9d1a3e2eea9..31bc2e6dca9 100644
@@ -381,6 +381,7 @@ def append(self, line):
 
Quoted 9 lines

scripts/qapi/parser.py def append(self, line): · line 387

Quoted 6 lines
scripts/qapi/parser.py +1 -0
          The way that the line is dealt with depends on which part of
          the documentation we're parsing right now:
+
          * The body section: ._append_line is ._append_body_line
          * An argument section: ._append_line is ._append_args_line
          * A features section: ._append_line is ._append_features_line

I'm asking because you're not adding :param line: here.

Yeah, it's not necessary to test the syntax of what else I've written with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't required at all and could be deleted. You can do so if you'd like; it will just show up later in some other patch or series more designed to fix formatting.

Quoted 9 lines

Same for several other functions in this file.

In schema.py:

class QAPISchemaMember:
""" Represents object members, enum members and features """

Are the spaces next to """ okay?

Ideally cleaned up, but that's not a goal of this patch or series.

--js

#5 lore raw

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

Quoted 21 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.

A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

Quoted 9 lines
Quoted 7 lines

- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.

Mandatory when they apply?

Quoted 8 lines

Subject of debate...

- Some people really hate obvious docstring comments.
- Some people really like the consistency.

Which type of developer am I? Guess it depends on when you ask.

Figured we'd hash that out when I go to write a style guide document.

Fair enough.

If I stop reading after the first paragraph, the patch matches expectations built by the commit message.

If I speed-read, the first paragraph barely registers, but the second makes me slow down, giving me the mistaken idea that this patch is about converting to a preliminary style guide. It's not, it's about getting Sphinx errors out of the way.

I figure you didn't stop after the first paragraph because you felt a need to explain why you resolve the "obvious errors" the way you do.

Perhaps:

qapi: modify docstrings to be sphinx-compatible

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. For now, merely avoid obvious errors that cause Sphinx documentation build problems, using a style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

[...]

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

- Additional examples, diagrams, or other metadata follows below.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Blank line here, by convention.

Quoted 35 lines
Quoted 25 lines
Quoted 25 lines
---
  scripts/qapi/gen.py    | 6 ++++--
  scripts/qapi/parser.py | 1 +
  2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -1
index ca66c82b5b8..dc7b94aa115 100644
@@ -154,9 +154,11 @@ def _bottom(self):
    @contextmanager
  def ifcontext(ifcond, *args):
-    """A 'with' statement context manager to wrap with start_if()/end_if()
+    """
+    A with-statement context manager that wraps with `start_if()` / `end_if()`.
  -    *args: any number of QAPIGenCCode
+    :param ifcond: A list of conditionals, passed to `start_if()`.
+    :param args: any number of `QAPIGenCCode`.
        Example::
  diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py

index 9d1a3e2eea9..31bc2e6dca9 100644

scripts/qapi/parser.py +0 -0
@@ -381,6 +381,7 @@ def append(self, line):
Quoted 9 lines
Quoted 7 lines
scripts/qapi/parser.py +0 -0
            The way that the line is dealt with depends on which

part of
the documentation we're parsing right now:
+
* The body section: ._append_line is ._append_body_line
* An argument section: ._append_line is ._append_args_line
* A features section: ._append_line is ._append_features_line

I'm asking because you're not adding :param line: here.

Quoted 5 lines

Yeah, it's not necessary to test the syntax of what else I've written with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't required at all and could be deleted. You can do so if you'd like; it will just show up later in some other patch or series more designed to fix formatting.

I recommend (but do not demand) to strictly limit this commit to "avoiding obvious errors that cause sphinx documentation build problems."

Quoted 8 lines
Quoted 6 lines

Same for several other functions in this file.
In schema.py:
class QAPISchemaMember:
""" Represents object members, enum members and features """
Are the spaces next to """ okay?

Ideally cleaned up, but that's not a goal of this patch or series.

Got it.

author
#6 lore raw

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

Quoted 28 lines · Markus Armbruster
Quoted 26 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · John Snow

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. At present, we are avoiding obvious errors that cause sphinx documentation build problems.

A preliminary style guide is loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

Quoted 9 lines
Quoted 7 lines

- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs, contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.

Mandatory when they apply?

Subject of debate...

- Some people really hate obvious docstring comments.
- Some people really like the consistency.

Quoted 9 lines
Quoted 3 lines

Which type of developer am I? Guess it depends on when you ask.

Figured we'd hash that out when I go to write a style guide document.

Fair enough.

If I stop reading after the first paragraph, the patch matches expectations built by the commit message.

(:

Quoted 22 lines

If I speed-read, the first paragraph barely registers, but the second makes me slow down, giving me the mistaken idea that this patch is about converting to a preliminary style guide. It's not, it's about getting Sphinx errors out of the way.

I figure you didn't stop after the first paragraph because you felt a need to explain why you resolve the "obvious errors" the way you do.

Perhaps:

qapi: modify docstrings to be sphinx-compatible

A precise style guide and a package-wide overhaul is forthcoming pending further discussion and consensus. For now, merely avoid obvious errors that cause Sphinx documentation build problems, using a style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability with our existing Sphinx framework, and because it has loose recognition in the Pycharm IDE.

[...]

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

- Additional examples, diagrams, or other metadata follows below.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Blank line here, by convention.

Wonder why my script didn't do that. Eh.

Quoted 41 lines
Quoted 35 lines
Quoted 25 lines
Quoted 25 lines
---
   scripts/qapi/gen.py    | 6 ++++--
   scripts/qapi/parser.py | 1 +
   2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +4 -1
index ca66c82b5b8..dc7b94aa115 100644
@@ -154,9 +154,11 @@ def _bottom(self):
     @contextmanager
   def ifcontext(ifcond, *args):
-    """A 'with' statement context manager to wrap with start_if()/end_if()
+    """
+    A with-statement context manager that wraps with `start_if()` / `end_if()`.
   -    *args: any number of QAPIGenCCode
+    :param ifcond: A list of conditionals, passed to `start_if()`.
+    :param args: any number of `QAPIGenCCode`.
         Example::
   diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py

index 9d1a3e2eea9..31bc2e6dca9 100644

scripts/qapi/parser.py +0 -0
@@ -381,6 +381,7 @@ def append(self, line):
Quoted 9 lines
Quoted 7 lines
scripts/qapi/parser.py +0 -0
             The way that the line is dealt with depends on which

part of
the documentation we're parsing right now:
+
* The body section: ._append_line is ._append_body_line
* An argument section: ._append_line is ._append_args_line
* A features section: ._append_line is ._append_features_line

I'm asking because you're not adding :param line: here.

Quoted 5 lines

Yeah, it's not necessary to test the syntax of what else I've written with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't required at all and could be deleted. You can do so if you'd like; it will just show up later in some other patch or series more designed to fix formatting.

Quoted 4 lines

I recommend (but do not demand) to strictly limit this commit to "avoiding obvious errors that cause sphinx documentation build problems."

OK, I'll drop this bit for now, but I will keep the new annotations for ifcontext, because... well. Why do it twice.

Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

Same for several other functions in this file.
In schema.py:
class QAPISchemaMember:
""" Represents object members, enum members and features """
Are the spaces next to """ okay?

Ideally cleaned up, but that's not a goal of this patch or series.

Got it.

author
#7 lore raw

PATCH v5 3/36 qapi-gen: Separate arg-parsing from generation

In reply to John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
 scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
 1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +62 -23
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
 import sys
 
 from qapi.commands import gen_commands
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
     parser = argparse.ArgumentParser(
         description='Generate code from a QAPI schema')
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                         help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                         help="prefix for symbols")
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
     try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
     except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
 
 if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())
Signature · 1 line
2.26.2
#8 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 31 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
 scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
 1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +6 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re

scripts/qapi-gen.py line 22

Quoted 10 lines
scripts/qapi-gen.py +2 -1
 import sys
 
 from qapi.commands import gen_commands
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 18 lines
scripts/qapi-gen.py +15 -1
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 49

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

    $ python3 scripts/qapi-gen.py --prefix=@ x
    scripts/qapi-gen.py: : funny character '@' in prefix '@'

Unwanted " :".

This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

Quoted 18 lines
scripts/qapi-gen.py +16 -0
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
     parser = argparse.ArgumentParser(
         description='Generate code from a QAPI schema')

scripts/qapi-gen.py line 75

Quoted 10 lines
scripts/qapi-gen.py +4 -2
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                         help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                         help="prefix for symbols")

I don't like the changes to default=, because:

1. They are only losely related to the patch's purpose.

2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.

3. The defaults will not change, and nothing else uses the constants.

Quoted 22 lines
scripts/qapi-gen.py +5 -9
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
     try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
     except QAPIError as err:
-        print(err, file=sys.stderr)

scripts/qapi-gen.py def main(argv): · line 91

Quoted 10 lines
scripts/qapi-gen.py +3 -7
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0

Subtle change: you move the gen_FOO() into the try ... except. Okay; they don't raise QAPIError, but perhaps worth a mention in the commit message.

scripts/qapi-gen.py def main(argv): · line 95

Quoted 5 lines
scripts/qapi-gen.py +1 -1
 if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

"Python was designed to be easy to understand and fun to use." Ha ha ha.

author
#9 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 77 lines · Markus Armbruster
Quoted 31 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +6 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
  # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
  import argparse
  import re

scripts/qapi-gen.py line 22

Quoted 10 lines
scripts/qapi-gen.py +2 -1
  import sys
 
  from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 18 lines
scripts/qapi-gen.py +15 -1
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 49

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

Quoted 8 lines
     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: : funny character '@' in prefix '@'

Unwanted " :".

This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).

Quack indeed - why does our base error class require so much information from a specific part of the generation process?

Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.

You are witnessing some more future-bleed.

Quoted 3 lines

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?

The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.

Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.

Quoted 31 lines
Quoted 18 lines
scripts/qapi-gen.py +16 -0
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
      parser = argparse.ArgumentParser(
          description='Generate code from a QAPI schema')

scripts/qapi-gen.py line 75

Quoted 10 lines
scripts/qapi-gen.py +4 -2
      parser.add_argument('-b', '--builtins', action='store_true',
                          help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                          help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                          help="prefix for symbols")

I don't like the changes to default=, because:

Quoted 2 lines

1. They are only losely related to the patch's purpose.

Subjective, but OK.

Quoted 3 lines

2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.

All of it is in main.py, though! If you were to, say, move generate() elsewhere, it'd look pretty compact as just the CLI frontend, no?

Quoted 2 lines

3. The defaults will not change, and nothing else uses the constants.

But, fine. Cleber had the same comment but I wasn't fully on-board, but two folks saying the same thing ...

Quoted 28 lines
Quoted 28 lines
scripts/qapi-gen.py +5 -15
      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                          dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
      parser.add_argument('schema', action='store')
      args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
      try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
      except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
Quoted 9 lines

scripts/qapi-gen.py def main(argv): · line 91

Quoted 4 lines
scripts/qapi-gen.py +3 -1
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0

Subtle change: you move the gen_FOO() into the try ... except. Okay; they don't raise QAPIError, but perhaps worth a mention in the commit message.

Forbidden future knowledge; I intend them to.

Quoted 9 lines

scripts/qapi-gen.py def main(argv): · line 95

Quoted 5 lines
scripts/qapi-gen.py +1 -1
  if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

"Python was designed to be easy to understand and fun to use." Ha ha ha.

I mean, I'm having fun, aren't you?

--js

#10 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 79 lines · John Snow
Quoted 70 lines · Markus Armbruster
Quoted 28 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +5 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
  # See the COPYING file in the top-level directory.
  +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
    import argparse

scripts/qapi-gen.py line 19

Quoted 10 lines
scripts/qapi-gen.py +2 -1
  import re
  import sys
    from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 17 lines
scripts/qapi-gen.py +15 -1
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 49

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

Quoted 7 lines
     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: : funny character '@' in prefix '@'
Unwanted " :".
This is due to a hack: you pass '' for info (*quack*).  Everything
else
passes QAPISourceInfo (I believe).
Quoted 2 lines

Quack indeed - why does our base error class require so much information from a specific part of the generation process?

Because it's not "a base error class", it's a base error class for the QAPI schema compiler frontend.

Quoted 2 lines

Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.

Evolving it to satisfy a need for a more widely usable error class is okay.

Quoted 7 lines

You are witnessing some more future-bleed.

Quoted 3 lines

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?

Yes, because that's where you can emit the better error message more easily.

    $ python3 scripts/qapi-gen.py --prefix=@ x
    scripts/qapi-gen.py: 'funny character '@' in argument of --prefix

is better than

    $ python3 scripts/qapi-gen.py --prefix=@ x
    scripts/qapi-gen.py: funny character '@' in prefix '@'

In generate(), the knowledge where the offending prefix value comes from is no longer available.

To emit this error message, you'd have to raise a sufficiently distinct error in generate, catch it in main(), then put the error message together somehow. Bah.

Aside: there's a stray ' in the old error message.

Quoted 3 lines

The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.

YAGNI.

Quoted 2 lines

Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.

"@prefix is sane" is a precondition of generate().

When there's a real risk of preconditions getting violated, or readers getting confused about preconditions, check them with assert.

Quoted 32 lines
Quoted 22 lines
Quoted 22 lines
scripts/qapi-gen.py +17 -1
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
      parser = argparse.ArgumentParser(
          description='Generate code from a QAPI schema')
      parser.add_argument('-b', '--builtins', action='store_true',
                          help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
Quoted 9 lines

scripts/qapi-gen.py line 75

Quoted 6 lines
scripts/qapi-gen.py +3 -1
+                        default=DEFAULT_OUTPUT_DIR,
                          help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                          help="prefix for symbols")

I don't like the changes to default=, because:
1. They are only losely related to the patch's purpose.

Quoted 8 lines

Subjective, but OK.

Quoted 3 lines

2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.

All of it is in main.py, though! If you were to, say, move generate() elsewhere, it'd look pretty compact as just the CLI frontend, no?

Same statement is more compact than same screenful is more compact than same file :)

Quoted 44 lines
Quoted 2 lines

3. The defaults will not change, and nothing else uses the constants.

But, fine. Cleber had the same comment but I wasn't fully on-board, but two folks saying the same thing ...

Quoted 28 lines
Quoted 28 lines
scripts/qapi-gen.py +5 -14
      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                          dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
      parser.add_argument('schema', action='store')
      args = parser.parse_args()
  -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
 args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
      try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
      except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
Quoted 9 lines

scripts/qapi-gen.py def main(argv): · line 91

Quoted 4 lines
scripts/qapi-gen.py +3 -1
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0

Subtle change: you move the gen_FOO() into the try ... except. Okay;
they don't raise QAPIError, but perhaps worth a mention in the commit message.

Quoted 1 line

Forbidden future knowledge; I intend them to.

I don't mind the move.

Quoted 9 lines
Quoted 7 lines

scripts/qapi-gen.py def main(argv): · line 95

Quoted 4 lines
scripts/qapi-gen.py +1 -1
  if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

"Python was designed to be easy to understand and fun to use." Ha ha ha.

I mean, I'm having fun, aren't you?

So many kinds of fun! The fun I'm having with this patch hunk is mocking "easy and fun" Python for requiring such an elaborate menuett just to express "can run as program".

This emperor has no clothes, either. And that's funny, isn't it?

author
#11 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 81 lines · Markus Armbruster
Quoted 79 lines · John Snow
Quoted 77 lines · Markus Armbruster
Quoted 28 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
   1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +5 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
   #!/usr/bin/env python3
-# QAPI generator
-#
+
   # This work is licensed under the terms of the GNU GPL, version 2 or later.
   # See the COPYING file in the top-level directory.
   +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
     import argparse

scripts/qapi-gen.py line 19

Quoted 10 lines
scripts/qapi-gen.py +2 -1
   import re
   import sys
     from qapi.commands import gen_commands
+from qapi.error import QAPIError
   from qapi.events import gen_events
   from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
   from qapi.types import gen_types
   from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 17 lines
scripts/qapi-gen.py +15 -1
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 49

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

      $ python3 scripts/qapi-gen.py --prefix=@ x
      scripts/qapi-gen.py: : funny character '@' in prefix '@'
Unwanted " :".
This is due to a hack: you pass '' for info (*quack*).  Everything
else
passes QAPISourceInfo (I believe).
Quoted 6 lines
Quoted 2 lines

Quack indeed - why does our base error class require so much information from a specific part of the generation process?

Because it's not "a base error class", it's a base error class for the QAPI schema compiler frontend.

Well. It's the base for every error we /had/.

Quoted 6 lines
Quoted 2 lines

Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.

Evolving it to satisfy a need for a more widely usable error class is okay.

Yep. It's helpful to keep a very generic form on which we grow other errors from, so that things like the entry point can be written legibly.

Quoted 26 lines
Quoted 7 lines

You are witnessing some more future-bleed.

Quoted 3 lines

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?

Yes, because that's where you can emit the better error message more easily.

     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: 'funny character '@' in argument of --prefix

is better than

     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: funny character '@' in prefix '@'

In generate(), the knowledge where the offending prefix value comes from is no longer available.

To emit this error message, you'd have to raise a sufficiently distinct error in generate, catch it in main(), then put the error message together somehow. Bah.

Quoted 8 lines

Aside: there's a stray ' in the old error message.

Quoted 3 lines

The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.

YAGNI.

It's useful for testing and debugging to be able to just call it outside of the CLI, though. Maybe you won't use it, but I will.

I could always add the prefix check into a tiny function and give the good error message in main(), and just assert in generate() if you insist on the slightly more specific error message from the CLI script.

Quoted 116 lines
Quoted 2 lines

Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.

"@prefix is sane" is a precondition of generate().

When there's a real risk of preconditions getting violated, or readers getting confused about preconditions, check them with assert.

Quoted 32 lines
Quoted 22 lines
Quoted 22 lines
scripts/qapi-gen.py +17 -1
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
       parser = argparse.ArgumentParser(
           description='Generate code from a QAPI schema')
       parser.add_argument('-b', '--builtins', action='store_true',
                           help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
Quoted 9 lines

scripts/qapi-gen.py line 75

Quoted 6 lines
scripts/qapi-gen.py +3 -1
+                        default=DEFAULT_OUTPUT_DIR,
                           help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                           help="prefix for symbols")

I don't like the changes to default=, because:
1. They are only losely related to the patch's purpose.

Quoted 8 lines

Subjective, but OK.

Quoted 3 lines

2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.

All of it is in main.py, though! If you were to, say, move generate() elsewhere, it'd look pretty compact as just the CLI frontend, no?

Same statement is more compact than same screenful is more compact than same file :)

Quoted 44 lines
Quoted 2 lines

3. The defaults will not change, and nothing else uses the constants.

But, fine. Cleber had the same comment but I wasn't fully on-board, but two folks saying the same thing ...

Quoted 28 lines
Quoted 28 lines
scripts/qapi-gen.py +5 -14
       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                           dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
       parser.add_argument('schema', action='store')
       args = parser.parse_args()
   -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
 args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
       try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
       except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
Quoted 9 lines

scripts/qapi-gen.py def main(argv): · line 91

Quoted 4 lines
scripts/qapi-gen.py +3 -1
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0

Subtle change: you move the gen_FOO() into the try ... except. Okay;
they don't raise QAPIError, but perhaps worth a mention in the commit message.

Quoted 1 line

Forbidden future knowledge; I intend them to.

I don't mind the move.

Quoted 9 lines
Quoted 7 lines

scripts/qapi-gen.py def main(argv): · line 95

Quoted 4 lines
scripts/qapi-gen.py +1 -1
   if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

"Python was designed to be easy to understand and fun to use." Ha ha ha.

I mean, I'm having fun, aren't you?

So many kinds of fun! The fun I'm having with this patch hunk is mocking "easy and fun" Python for requiring such an elaborate menuett just to express "can run as program".

This emperor has no clothes, either. And that's funny, isn't it?

#12 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 81 lines · John Snow
Quoted 80 lines · Markus Armbruster
Quoted 78 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 28 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
   1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +5 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
   #!/usr/bin/env python3
-# QAPI generator
-#
+
   # This work is licensed under the terms of the GNU GPL, version 2 or later.
   # See the COPYING file in the top-level directory.
   +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
     import argparse

scripts/qapi-gen.py line 19

Quoted 10 lines
scripts/qapi-gen.py +2 -1
   import re
   import sys
     from qapi.commands import gen_commands
+from qapi.error import QAPIError
   from qapi.events import gen_events
   from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
   from qapi.types import gen_types
   from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 16 lines
scripts/qapi-gen.py +15 -0
     -def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 49

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

      $ python3 scripts/qapi-gen.py --prefix=@ x
      scripts/qapi-gen.py: : funny character '@' in prefix '@'
Unwanted " :".
This is due to a hack: you pass '' for info (*quack*).  Everything
else
passes QAPISourceInfo (I believe).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Quack indeed - why does our base error class require so much information from a specific part of the generation process?

Because it's not "a base error class", it's a base error class for the
QAPI schema compiler frontend.

Well. It's the base for every error we /had/.

You asked why the class has the init it has, and I answered :)

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.

Evolving it to satisfy a need for a more widely usable error class is
okay.

Yep. It's helpful to keep a very generic form on which we grow other errors from, so that things like the entry point can be written legibly.

If you have a non-trivial error message format convention, you have a use for a function formatting error messages.

If you have a separation between diagnose and report of errors, you you have a use for a transport from diagnose to report. In Python, that's raise.

The existing error message in main() has neither.

The existing error class QAPIError caters for the existing users.

Quoted 23 lines
Quoted 23 lines
Quoted 7 lines

You are witnessing some more future-bleed.

Quoted 3 lines

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?

Yes, because that's where you can emit the better error message more easily.

     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: 'funny character '@' in argument of --prefix
is better than
     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: funny character '@' in prefix '@'
In generate(), the knowledge where the offending prefix value comes
from
is no longer available.
To emit this error message, you'd have to raise a sufficiently
distinct
error in generate, catch it in main(), then put the error message
together somehow.  Bah.
Aside: there's a stray ' in the old error message.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.

YAGNI.

It's useful for testing and debugging to be able to just call it outside of the CLI, though. Maybe you won't use it, but I will.

For testing and debugging, treating "prefix is sane" as a precondition is fine. I wouldn't even bother checking it. A check would catch accidents, and these accidents seem vanishingly unlikely to me.

Evidence: we did without *any* prefix checking for *years*. I added it in commit 1cf47a15f18 just for completeness.

Quoted 3 lines

I could always add the prefix check into a tiny function and give the good error message in main(), and just assert in generate() if you insist on the slightly more specific error message from the CLI script.

If you genuinely think a check is needed there, that's the way to go.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.

"@prefix is sane" is a precondition of generate(). When there's a real risk of preconditions getting violated, or readers
getting confused about preconditions, check them with assert.

[...]

author
#13 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 152 lines · Markus Armbruster
Quoted 81 lines · John Snow
Quoted 80 lines · Markus Armbruster
Quoted 78 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 28 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
    scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
    1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +5 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
    #!/usr/bin/env python3
-# QAPI generator
-#
+
    # This work is licensed under the terms of the GNU GPL, version 2 or later.
    # See the COPYING file in the top-level directory.
    +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
      import argparse

scripts/qapi-gen.py line 19

Quoted 10 lines
scripts/qapi-gen.py +2 -1
    import re
    import sys
      from qapi.commands import gen_commands
+from qapi.error import QAPIError
    from qapi.events import gen_events
    from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
    from qapi.types import gen_types
    from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 16 lines
scripts/qapi-gen.py +15 -0
      -def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 49

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

       $ python3 scripts/qapi-gen.py --prefix=@ x
       scripts/qapi-gen.py: : funny character '@' in prefix '@'
Unwanted " :".
This is due to a hack: you pass '' for info (*quack*).  Everything
else
passes QAPISourceInfo (I believe).
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Quack indeed - why does our base error class require so much information from a specific part of the generation process?

Because it's not "a base error class", it's a base error class for the
QAPI schema compiler frontend.

Well. It's the base for every error we /had/.

You asked why the class has the init it has, and I answered :)

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

Ah, someone changes this in part 4 so that we have a more generic error class to use as a base when we are missing such information.

Evolving it to satisfy a need for a more widely usable error class is
okay.

Yep. It's helpful to keep a very generic form on which we grow other errors from, so that things like the entry point can be written legibly.

If you have a non-trivial error message format convention, you have a use for a function formatting error messages.

If you have a separation between diagnose and report of errors, you you have a use for a transport from diagnose to report. In Python, that's raise.

The existing error message in main() has neither.

The existing error class QAPIError caters for the existing users.

Quoted 23 lines
Quoted 23 lines
Quoted 7 lines

You are witnessing some more future-bleed.

Quoted 3 lines

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

One might also ask: Is it a good idea to only validate this on a frontend, and not in the implementation?

Yes, because that's where you can emit the better error message more easily.

      $ python3 scripts/qapi-gen.py --prefix=@ x
      scripts/qapi-gen.py: 'funny character '@' in argument of --prefix
is better than
      $ python3 scripts/qapi-gen.py --prefix=@ x
      scripts/qapi-gen.py: funny character '@' in prefix '@'
In generate(), the knowledge where the offending prefix value comes
from
is no longer available.
To emit this error message, you'd have to raise a sufficiently
distinct
error in generate, catch it in main(), then put the error message
together somehow.  Bah.
Aside: there's a stray ' in the old error message.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The idea here was to create a function that could be used in a script (for tests, debugging interfaces, other python packages) to do all of the same things that the CLI tool did, just sans the actual CLI.

YAGNI.

It's useful for testing and debugging to be able to just call it outside of the CLI, though. Maybe you won't use it, but I will.

For testing and debugging, treating "prefix is sane" as a precondition is fine. I wouldn't even bother checking it. A check would catch accidents, and these accidents seem vanishingly unlikely to me.

Quoted 9 lines

Evidence: we did without *any* prefix checking for *years*. I added it in commit 1cf47a15f18 just for completeness.

Quoted 3 lines

I could always add the prefix check into a tiny function and give the good error message in main(), and just assert in generate() if you insist on the slightly more specific error message from the CLI script.

If you genuinely think a check is needed there, that's the way to go.

A goal of this patch in particular is the clean separation of the CLI frontend from the implementation logic behind it, such that you have:

(1) The CLI parsing, and
(2) A function implementing a task the package is capable of performing

The CLI parsing method converts sys.argv arguments into function parameter data that can be acted upon by another function in the package.

If checks are exclusive to the process of converting flags to argument data, they should stay in the CLI parsing function.

If checks prevent you from passing bad data lower into the program, I prefer pushing that check lower into the stack instead, such that the module is equivalently useful without the CLI frontend.

This conceptualization might be different from how you perceive QAPI. In my case, I consider the public functions and modules themselves to be interface that should behave reasonably and with some defensiveness. You might only count the CLI script itself in that category.

In this case, the prefix is passed to many different generators, so it seems good to check it before we hand it off to all those different places.

You don't like the loss of precision that causes for the CLI frontend error, which is reasonable.

I consider the function interface and the CLI frontend "equivalent", and so I found it odd to reject bad data in one, but not the other. IOW, I do not consider the CLI frontend the only user of generate().

I think it's minor in this case, sure. I am fine with saying "A valid prefix is a precondition", but then I want to document and enforce that precondition -- so I did wind up using an assert in my pending respin.

--js

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Wouldn't make sense to allow garbage to flow in from one interface but not the other; so the check is here.

"@prefix is sane" is a precondition of generate(). When there's a real risk of preconditions getting violated, or readers
getting confused about preconditions, check them with assert.

[...]

author
#14 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

In reply to Markus Armbruster
Quoted 165 lines · Markus Armbruster
Quoted 31 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +6 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
  # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
  import argparse
  import re

scripts/qapi-gen.py line 22

Quoted 10 lines
scripts/qapi-gen.py +2 -1
  import sys
 
  from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit

Unrelated cleanup. Okay.

Quoted 18 lines
scripts/qapi-gen.py +15 -1
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.

scripts/qapi-gen.py line 47

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)

Uh...

     $ python3 scripts/qapi-gen.py --prefix=@ x
     scripts/qapi-gen.py: : funny character '@' in prefix '@'

Unwanted " :".

This is due to a hack: you pass '' for info (*quack*). Everything else passes QAPISourceInfo (I believe).

Is it really a good idea to do this in generate? It's not about generating code, it's about validating a CLI option.

Quoted 18 lines
scripts/qapi-gen.py +16 -0
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
      parser = argparse.ArgumentParser(
          description='Generate code from a QAPI schema')

scripts/qapi-gen.py line 73

Quoted 10 lines
scripts/qapi-gen.py +4 -2
      parser.add_argument('-b', '--builtins', action='store_true',
                          help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                          help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                          help="prefix for symbols")

I don't like the changes to default=, because:

1. They are only losely related to the patch's purpose.

2. They split the definition of the CLI: most of it is here, except for defaults, which are defined elsewhere.

3. The defaults will not change, and nothing else uses the constants.

Quoted 22 lines
scripts/qapi-gen.py +5 -9
      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                          dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
      parser.add_argument('schema', action='store')
      args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
      try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
      except QAPIError as err:
-        print(err, file=sys.stderr)

scripts/qapi-gen.py def main(argv): · line 91

Quoted 10 lines
scripts/qapi-gen.py +3 -7
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0

Subtle change: you move the gen_FOO() into the try ... except. Okay; they don't raise QAPIError, but perhaps worth a mention in the commit message.

Quoted 2 lines
Quoted 7 lines

scripts/qapi-gen.py

Quoted 3 lines

if __name__ == '__main__':

scripts/qapi-gen.py +1 -1
-    main(sys.argv)
+    sys.exit(main())

"Python was designed to be easy to understand and fun to use." Ha ha ha.

Here's a Fixup that requires more (but obvious) Fixups to later patches in this series. (The next patch needs the same changes re-applied to the now-moved file.)

A note: I add a FIXME, but when I respin my error series, it will take care of this piece. This is just the quickest, dumbest way to victory in the interim. It is not worth engineering a better solution, because there is one waiting in part 4 already.

(Future knowledge: I add a QAPIError that does not define its arguments that inherits Exception and is used as the basis for all error classes in QAPI; and the contextual error that requires "info" is changed to inherit from QAPIError. This is useful as a "package-level" error, one in which that *all* errors in this package inherit from. We already achieve that, but we needed a more general form to serve as the root of that inheritance tree.)

scripts/qapi-gen.py +2 -6
index 117b396a595..c62c4e93323 100644
@@ -22,10 +22,6 @@
 from qapi.visit import gen_visit
 
 
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
 def generate(schema_file: str,
              output_dir: str,
              prefix: str,
@@ -68,10 +64,10 @@ def main() -> int:
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
     parser.add_argument('-o', '--output-dir', action='store',
-                        default=DEFAULT_OUTPUT_DIR,
+                        default='',
                         help="write output to directory OUTPUT_DIR")
     parser.add_argument('-p', '--prefix', action='store',
-                        default=DEFAULT_PREFIX,
+                        default='',
                         help="prefix for symbols")
     parser.add_argument('-u', '--unmask-non-abi-names', 

action='store_true',

                         dest='unmask',
scripts/qapi/error.py +3 -0
index ae60d9e2fe8..f5a818ae35f 100644
@@ -21,6 +21,9 @@ def __init__(self, info, col, msg):
 
     def __str__(self):
+        # FIXME: We need a general-purpose no-context error class.
+        if not self.info:
+            return self.msg
         loc = str(self.info)
         if self.col is not None:
             assert self.info.line is not None
             loc += ':%s' % self.col
#15 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

In reply to John Snow
Quoted 71 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
 scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
 1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +35 -4
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
 import sys
 
 from qapi.commands import gen_commands
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)

scripts/qapi-gen.py line 61

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.

What's a "shell script entrypoint"?

Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html

Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts

Quoted 43 lines
scripts/qapi-gen.py +13 -16
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
     parser = argparse.ArgumentParser(
         description='Generate code from a QAPI schema')
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                         help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                         help="prefix for symbols")
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
     try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
     except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)

scripts/qapi-gen.py def main(argv): · line 95

Quoted 10 lines
scripts/qapi-gen.py +4 -3
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
 
 if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

What does sys.exit() really buy us here? I'm asking because both spots in the Python docs I referenced above do without.

author
#16 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 86 lines · Markus Armbruster
Quoted 71 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +35 -4
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
  # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
  import argparse
  import re
  import sys
 
  from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)

scripts/qapi-gen.py line 61

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.

What's a "shell script entrypoint"?

Quoted 6 lines

Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html

Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts

"entrypoint" is Python garble for a function that can be registered as a callable from the command line.

So in a theoretical setup.py, you'd do something like:

'entry_points': {
'console_scripts': [
'qapi-gen = qapi.main:main',
]
}

so when I say "shell script entrypoint", I am referring to a shell script (I mean: it has a shebang and can be executed by an interactive shell process) that calls the entrypoint.

Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to just generate that script.

(i.e. doing pip install qemu.qapi will install a 'qapi-gen' CLI script for you. this is how packages like sphinx create the 'sphinx-build' script, etc.)

Quoted 50 lines
Quoted 50 lines
scripts/qapi-gen.py +16 -18
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
      parser = argparse.ArgumentParser(
          description='Generate code from a QAPI schema')
      parser.add_argument('-b', '--builtins', action='store_true',
                          help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                          help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                          help="prefix for symbols")
      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                          dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
      parser.add_argument('schema', action='store')
      args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
      try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
      except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
 
Quoted 7 lines

scripts/qapi-gen.py def main(argv): · line 95

Quoted 3 lines
scripts/qapi-gen.py +1 -1
  if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

What does sys.exit() really buy us here? I'm asking because both spots in the Python docs I referenced above do without.

It just pushes the sys.exit out of the main function so it can be invoked by other machinery. (And takes the return code from main and turns it into the return code for the process.)

I don't think it winds up mattering for simple "console_script" entry points, but you don't want the called function to exit and deny the caller the chance to do their own tidying post-call.

You've already offered a "YAGNI", but it's just the convention I tend to stick to for how to structure entry points.

--js

#17 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 98 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 74 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +41 -4
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
  # See the COPYING file in the top-level directory.
  +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
    import argparse
  import re
  import sys
    from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
Quoted 9 lines

scripts/qapi-gen.py line 61

Quoted 3 lines
scripts/qapi-gen.py +3 -0
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.

What's a "shell script entrypoint"?
Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html
Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts

"entrypoint" is Python garble for a function that can be registered as a callable from the command line.

So in a theoretical setup.py, you'd do something like:

'entry_points': {
'console_scripts': [
'qapi-gen = qapi.main:main',
]
}

Quoted 3 lines

so when I say "shell script entrypoint", I am referring to a shell script (I mean: it has a shebang and can be executed by an interactive shell process) that calls the entrypoint.

It can be executed by any process. See execve(2):

pathname must be either a binary executable, or a script starting with a line of the form:

#!interpreter [optional-arg]

For details of the latter case, see "Interpreter scripts" below.

"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.

Quoted 68 lines

Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to just generate that script.

(i.e. doing pip install qemu.qapi will install a 'qapi-gen' CLI script for you. this is how packages like sphinx create the 'sphinx-build' script, etc.)

Quoted 49 lines
Quoted 49 lines
scripts/qapi-gen.py +16 -17
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
      parser = argparse.ArgumentParser(
          description='Generate code from a QAPI schema')
      parser.add_argument('-b', '--builtins', action='store_true',
                          help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                          help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                          help="prefix for symbols")
      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                          dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
      parser.add_argument('schema', action='store')
      args = parser.parse_args()
  -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
 args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
      try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
      except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
Quoted 7 lines

scripts/qapi-gen.py def main(argv): · line 95

Quoted 3 lines
scripts/qapi-gen.py +1 -1
  if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

What does sys.exit() really buy us here? I'm asking because both spots
in the Python docs I referenced above do without.

It just pushes the sys.exit out of the main function so it can be invoked by other machinery. (And takes the return code from main and turns it into the return code for the process.)

Quoted 6 lines

I don't think it winds up mattering for simple "console_script" entry points, but you don't want the called function to exit and deny the caller the chance to do their own tidying post-call.

You've already offered a "YAGNI", but it's just the convention I tend to stick to for how to structure entry points.

I'm not questioning the conventional if __name__ == '__main__' menuett. I wonder why *we* need sys.exit() where the examples in the Python docs don't.

author
#18 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 189 lines · Markus Armbruster
Quoted 98 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 74 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
   1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +41 -4
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
   #!/usr/bin/env python3
-# QAPI generator
-#
+
   # This work is licensed under the terms of the GNU GPL, version 2 or later.
   # See the COPYING file in the top-level directory.
   +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
     import argparse
   import re
   import sys
     from qapi.commands import gen_commands
+from qapi.error import QAPIError
   from qapi.events import gen_events
   from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
   from qapi.types import gen_types
   from qapi.visit import gen_visit
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
Quoted 9 lines

scripts/qapi-gen.py line 61

Quoted 3 lines
scripts/qapi-gen.py +3 -0
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.

What's a "shell script entrypoint"?
Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html
Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts

"entrypoint" is Python garble for a function that can be registered as a callable from the command line.

So in a theoretical setup.py, you'd do something like:

'entry_points': {
'console_scripts': [
'qapi-gen = qapi.main:main',
]
}

Quoted 3 lines

so when I say "shell script entrypoint", I am referring to a shell script (I mean: it has a shebang and can be executed by an interactive shell process) that calls the entrypoint.

It can be executed by any process. See execve(2):

pathname must be either a binary executable, or a script starting with a line of the form:

#!interpreter [optional-arg]

For details of the latter case, see "Interpreter scripts" below.

"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.

Quoted 72 lines

Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to just generate that script.

(i.e. doing pip install qemu.qapi will install a 'qapi-gen' CLI script for you. this is how packages like sphinx create the 'sphinx-build' script, etc.)

Quoted 49 lines
Quoted 49 lines
scripts/qapi-gen.py +16 -17
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
       parser = argparse.ArgumentParser(
           description='Generate code from a QAPI schema')
       parser.add_argument('-b', '--builtins', action='store_true',
                           help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                           help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                           help="prefix for symbols")
       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                           dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
       parser.add_argument('schema', action='store')
       args = parser.parse_args()
   -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
 args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
       try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
       except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
Quoted 7 lines

scripts/qapi-gen.py def main(argv): · line 95

Quoted 3 lines
scripts/qapi-gen.py +1 -1
   if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())

What does sys.exit() really buy us here? I'm asking because both spots
in the Python docs I referenced above do without.

It just pushes the sys.exit out of the main function so it can be invoked by other machinery. (And takes the return code from main and turns it into the return code for the process.)

I don't think it winds up mattering for simple "console_script" entry points, but you don't want the called function to exit and deny the caller the chance to do their own tidying post-call.

Quoted 7 lines
Quoted 2 lines

You've already offered a "YAGNI", but it's just the convention I tend to stick to for how to structure entry points.

I'm not questioning the conventional if __name__ == '__main__' menuett. I wonder why *we* need sys.exit() where the examples in the Python docs don't.

I assume they don't care about setting that return code manually. By default it's going to be 0 on normal exit, and non-zero if it raises or aborts.

We catch the error to pretty-print it to console instead, and want to avoid the stack trace so we return a status code instead.

I assume that's the only difference, really.

--js

author
#19 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

In reply to Markus Armbruster
Quoted 5 lines · Markus Armbruster

It can be executed by any process. See execve(2):

pathname must be either a binary executable, or a script starting with a line of the form:

Quoted 7 lines

#!interpreter [optional-arg]

For details of the latter case, see "Interpreter scripts" below.

"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.

You know, I don't think I was actually explicitly aware that the #! shebang was not something the shell actually processed itself. Always learning new things.

(No, I don't think I have ever execve'd something that wasn't a binary.)

"entry point" is a little vague, an entry point for what? by whom? I was trying to call attention to the idea specifically that main() was intended as python's "console script entry point", but used the word "shell" instead.

"console script entrypoint" is also a lot of jargon. What I really want to communicate is: "When you run qapi-gen on your command-line, this is the function that runs!"

So I guess something like:

"qapi-gen executable entry point." will suffice. Please further adjust to your liking when staging.

--js

#20 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 13 lines · John Snow
Quoted 11 lines · Markus Armbruster

It can be executed by any process. See execve(2):

pathname must be either a binary executable, or a script starting with a line of the form:
#!interpreter [optional-arg]
For details of the latter case, see "Interpreter scripts"
below.

"Entry point" makes sense in Python context, "script entry point" also makes sense (since every Python program is a script, script is redundant, but not wrong). "Shell script entry point" is misleading.

Quoted 5 lines

You know, I don't think I was actually explicitly aware that the #! shebang was not something the shell actually processed itself. Always learning new things.

(No, I don't think I have ever execve'd something that wasn't a binary.)

I'm sure you've done it countless times, just not knowingly :)

Quoted 6 lines

"entry point" is a little vague, an entry point for what? by whom? I was trying to call attention to the idea specifically that main() was intended as python's "console script entry point", but used the word "shell" instead.

"console script entrypoint" is also a lot of jargon. What I really

It is.

If I didn't find "console script" so thoroughly misguided, I'd advie you to stick to it just because it's what we found in Python's docs. It's misguided, because this entry point is the one and only entry point for *any* kind of Python executable program, be it console, GUI, or whatever.

Quoted 7 lines

want to communicate is: "When you run qapi-gen on your command-line, this is the function that runs!"

So I guess something like:

"qapi-gen executable entry point." will suffice. Please further adjust to your liking when staging.

Works for me.

#21 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

In reply to John Snow

I keep stumbling over things in later patches that turn out to go back to this one.

Quoted 17 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
 scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
 1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +0 -0
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
 #!/usr/bin/env python3

scripts/qapi-gen.py line 9

Quoted 10 lines
scripts/qapi-gen.py +5 -2
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.

PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.

Quoted 18 lines
scripts/qapi-gen.py +4 -2
+"""
 
 import argparse
 import re
 import sys
 
 from qapi.commands import gen_commands
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''

scripts/qapi-gen.py line 35

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.

PEP 257: The docstring is a phrase ending in a period. It prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...".

Suggest

Generate C code for the given schema into the target directory.

Quoted 9 lines
scripts/qapi-gen.py +9 -0
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """

[...]

author
#22 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 28 lines · Markus Armbruster

I keep stumbling over things in later patches that turn out to go back to this one.

Quoted 23 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +1 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
  # See the COPYING file in the top-level directory.
 
Quoted 9 lines

scripts/qapi-gen.py line 9

Quoted 4 lines
scripts/qapi-gen.py +4 -0
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.

PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.

Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.

("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")

Quoted 33 lines
Quoted 18 lines
scripts/qapi-gen.py +4 -2
+"""
 
  import argparse
  import re
  import sys
 
  from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''

scripts/qapi-gen.py line 35

Quoted 10 lines
scripts/qapi-gen.py +10 -0
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.

PEP 257: The docstring is a phrase ending in a period. It prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...".

Quoted 4 lines

Suggest

Generate C code for the given schema into the target directory.

OK. I don't mind trying to foster a consistent tone. I clearly didn't. I will add a note to my style guide todo.

I give you permission to change the voice in any of my docstrings, or to adjust the phrasing to be more technically accurate as you see fit. You are the primary maintainer of this code, of course, and you will know best.

It will be quicker to give you full permission to just change any of the docstrings as you see fit than it will be to play review-respin ping-pong.

Quoted 11 lines
Quoted 9 lines
scripts/qapi-gen.py +9 -0
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """

[...]

#23 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 36 lines · John Snow
Quoted 25 lines · Markus Armbruster

I keep stumbling over things in later patches that turn out to go back to this one.

Quoted 21 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
  1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +1 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
  #!/usr/bin/env python3
-# QAPI generator
-#
+
  # This work is licensed under the terms of the GNU GPL, version 2 or later.
Quoted 9 lines

scripts/qapi-gen.py line 9

Quoted 5 lines
scripts/qapi-gen.py +3 -0
  # See the COPYING file in the top-level directory.
  +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.

PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.

Quoted 8 lines

Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.

("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")

Emacs Python mode is close enough by default: fill-paragraph (bound to M-q) uses variable fill-column, which defaults to 70. If you want the extra two columns PEP 8 grants you, I can show you how to bump it to 72 just for Python mode.

You can use fill-paragraph for code, too. I don't myself, because I disagree with its line breaking decisions too often (and so does PEP 8). A better Python mode would break code lines more neatly, and with the width defaulting to 79.

Quoted 37 lines
Quoted 24 lines
Quoted 24 lines
scripts/qapi-gen.py +13 -2
+"""
    import argparse
  import re
  import sys
    from qapi.commands import gen_commands
+from qapi.error import QAPIError
  from qapi.events import gen_events
  from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
  from qapi.types import gen_types
  from qapi.visit import gen_visit
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
Quoted 9 lines

scripts/qapi-gen.py line 35

Quoted 1 line
scripts/qapi-gen.py +1 -0
+    generate uses a given schema to produce C code in the target directory.

PEP 257: The docstring is a phrase ending in a period. It prescribes
the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...". Suggest
Generate C code for the given schema into the target
directory.

OK. I don't mind trying to foster a consistent tone. I clearly didn't. I will add a note to my style guide todo.

Quoted 8 lines

I give you permission to change the voice in any of my docstrings, or to adjust the phrasing to be more technically accurate as you see fit. You are the primary maintainer of this code, of course, and you will know best.

It will be quicker to give you full permission to just change any of the docstrings as you see fit than it will be to play review-respin ping-pong.

Me rewriting your commits without your consent is putting words in your mouth, which I don't want to do.

We can still reduce ping-pong: whenever I can, I don't just say "this needs improvement", I propose improvements. If you disagree, we talk. Else, if you have to respin, you make a reasonable effort to take them. Else, the remaining improvements are trivial (because no respin), and I'll make them in my tree.

Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
scripts/qapi-gen.py +9 -0
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """

[...]

author
#24 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 52 lines · Markus Armbruster
Quoted 36 lines · John Snow
Quoted 25 lines · Markus Armbruster

I keep stumbling over things in later patches that turn out to go back to this one.

Quoted 21 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
   1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +1 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
   #!/usr/bin/env python3
-# QAPI generator
-#
+
   # This work is licensed under the terms of the GNU GPL, version 2 or later.
Quoted 9 lines

scripts/qapi-gen.py line 9

Quoted 5 lines
scripts/qapi-gen.py +3 -0
   # See the COPYING file in the top-level directory.
   +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.

PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.

Quoted 8 lines

Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.

("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")

Emacs Python mode is close enough by default: fill-paragraph (bound to M-q) uses variable fill-column, which defaults to 70. If you want the extra two columns PEP 8 grants you, I can show you how to bump it to 72 just for Python mode.

Quoted 5 lines

You can use fill-paragraph for code, too. I don't myself, because I disagree with its line breaking decisions too often (and so does PEP 8). A better Python mode would break code lines more neatly, and with the width defaulting to 79.

Yeah, how do I set the reflow to 72 for specific modes?

I tend to do a lot of refactoring and "prototyping" in Pycharm, but when it comes to bread and butter edits I still prefer emacs. I kinda bounce between 'em a lot. Having emacs DTRT is still useful to me.

Quoted 46 lines
Quoted 37 lines
Quoted 24 lines
Quoted 24 lines
scripts/qapi-gen.py +13 -2
+"""
     import argparse
   import re
   import sys
     from qapi.commands import gen_commands
+from qapi.error import QAPIError
   from qapi.events import gen_events
   from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
   from qapi.types import gen_types
   from qapi.visit import gen_visit
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
Quoted 9 lines

scripts/qapi-gen.py line 35

Quoted 1 line
scripts/qapi-gen.py +1 -0
+    generate uses a given schema to produce C code in the target directory.

PEP 257: The docstring is a phrase ending in a period. It prescribes
the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...". Suggest
Generate C code for the given schema into the target
directory.

OK. I don't mind trying to foster a consistent tone. I clearly didn't. I will add a note to my style guide todo.

Quoted 8 lines

I give you permission to change the voice in any of my docstrings, or to adjust the phrasing to be more technically accurate as you see fit. You are the primary maintainer of this code, of course, and you will know best.

It will be quicker to give you full permission to just change any of the docstrings as you see fit than it will be to play review-respin ping-pong.

Quoted 9 lines

Me rewriting your commits without your consent is putting words in your mouth, which I don't want to do.

We can still reduce ping-pong: whenever I can, I don't just say "this needs improvement", I propose improvements. If you disagree, we talk. Else, if you have to respin, you make a reasonable effort to take them. Else, the remaining improvements are trivial (because no respin), and I'll make them in my tree.

I appreciate the consideration. They're not just "my" words, though, they are "our" words!

I do give you permission to touch up things like punctuation, voice, etc. If I dislike the changes I can always yelp at you post-hoc when you post the PR email.

So I'll reiterate that I am definitely happy with such changes -- the precision of the technical writing here is not my strong suit as I do not know QAPI as intimately as you, and it's not the focus of this series. I don't think of it as a personal offense that someone would copy-edit these things.

If you're not comfortable doing it, that's OK too, but I'm definitely okay with things like comment and docstring edits in particular. It's just something that feels hard to predict.

(Anyway, I made this change. There are likely other voice and line length changes needed, but I think I will try to address those systematically when I lift missing-docstring for pylint. Probably I will craft a series that creates a style guide, enables sphinx, lifts missing-docstring, and performs this same style of cleanup, going module-by-module.)

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
scripts/qapi-gen.py +9 -0
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """

[...]

#25 lore raw

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

Quoted 58 lines · John Snow
Quoted 52 lines · Markus Armbruster
Quoted 36 lines · John Snow
Quoted 25 lines · Markus Armbruster

I keep stumbling over things in later patches that turn out to go back to this one.

Quoted 21 lines · John Snow

This is a minor re-work of the entrypoint script. It isolates a generate() method from the actual command-line mechanism.

---
   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
   1 file changed, 62 insertions(+), 23 deletions(-)
scripts/qapi-gen.py +1 -2
index 541e8c1f55d..117b396a595 100644
@@ -1,30 +1,77 @@
   #!/usr/bin/env python3
-# QAPI generator
-#
+
   # This work is licensed under the terms of the GNU GPL, version 2 or later.
Quoted 9 lines

scripts/qapi-gen.py line 9

Quoted 5 lines
scripts/qapi-gen.py +3 -0
   # See the COPYING file in the top-level directory.
   +"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.

PEP 8: For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.

Quoted 8 lines

Eugh. OK, but I don't have a good way to check or enforce this, admittedly. I have to change my emacs settings to understand this when I hit the reflow key. I don't know if the python mode has a context-aware reflow length.

("I don't disagree, but I'm not immediately sure right now how I will make sure I, or anyone else, complies with this. Low priority as a result?")

Emacs Python mode is close enough by default: fill-paragraph (bound to M-q) uses variable fill-column, which defaults to 70. If you want the extra two columns PEP 8 grants you, I can show you how to bump it to 72 just for Python mode.

Quoted 4 lines

You can use fill-paragraph for code, too. I don't myself, because I disagree with its line breaking decisions too often (and so does PEP 8). A better Python mode would break code lines more neatly, and with the width defaulting to 79.

Quoted 5 lines

Yeah, how do I set the reflow to 72 for specific modes?

I tend to do a lot of refactoring and "prototyping" in Pycharm, but when it comes to bread and butter edits I still prefer emacs. I kinda bounce between 'em a lot. Having emacs DTRT is still useful to me.

In your .emacs:

    (add-hook 'python-mode-hook
              (lambda () (setq fill-column 72)))

Happy to explain this in detail if you're curious.

[...]

author
#26 lore raw

PATCH v5 4/36 qapi: move generator entrypoint into module

In reply to John Snow

As part of delinting and adding type hints to the QAPI generator, it's helpful for the entrypoint to be part of the package, only leaving a very tiny entrypoint shim outside of the module.

---
 scripts/qapi-gen.py  | 88 +++----------------------------------------
 scripts/qapi/main.py | 89 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 82 deletions(-)
 create mode 100644 scripts/qapi/main.py
scripts/qapi-gen.py +6 -82
index 117b396a595..f3518d29a54 100644
@@ -4,92 +4,16 @@
 # See the COPYING file in the top-level directory.
 
 """
-QAPI Generator
+QAPI code generation execution shim.
 
-This script is the main entry point for generating C code from the QAPI schema.
+This standalone script exists primarily to facilitate the running of the QAPI
+code generator without needing to install the python module to the current
+execution environment.
 """
 
-import argparse
-import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
-def generate(schema_file: str,
-             output_dir: str,
-             prefix: str,
-             unmask: bool = False,
-             builtins: bool = False) -> None:
-    """
-    generate uses a given schema to produce C code in the target directory.
-
-    :param schema_file: The primary QAPI schema file.
-    :param output_dir: The output directory to store generated code.
-    :param prefix: Optional C-code prefix for symbol names.
-    :param unmask: Expose non-ABI names through introspection?
-    :param builtins: Generate code for built-in types?
-
-    :raise QAPIError: On failures.
-    """
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
-    if match.end() != len(prefix):
-        msg = "funny character '{:s}' in prefix '{:s}'".format(
-            prefix[match.end()], prefix)
-        raise QAPIError('', None, msg)
-
-    schema = QAPISchema(schema_file)
-    gen_types(schema, output_dir, prefix, builtins)
-    gen_visit(schema, output_dir, prefix, builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, unmask)
-
-
-def main() -> int:
-    """
-    gapi-gen shell script entrypoint.
-    Expects arguments via sys.argv, see --help for details.
-
-    :return: int, 0 on success, 1 on failure.
-    """
-    parser = argparse.ArgumentParser(
-        description='Generate code from a QAPI schema')
-    parser.add_argument('-b', '--builtins', action='store_true',
-                        help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store',
-                        default=DEFAULT_OUTPUT_DIR,
-                        help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store',
-                        default=DEFAULT_PREFIX,
-                        help="prefix for symbols")
-    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
-                        dest='unmask',
-                        help="expose non-ABI names in introspection")
-    parser.add_argument('schema', action='store')
-    args = parser.parse_args()
-
-    try:
-        generate(args.schema,
-                 output_dir=args.output_dir,
-                 prefix=args.prefix,
-                 unmask=args.unmask,
-                 builtins=args.builtins)
-    except QAPIError as err:
-        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
-        return 1
-    return 0
-
+from qapi import main
 
 if __name__ == '__main__':
-    sys.exit(main())
+    sys.exit(main.main())
scripts/qapi/main.py new file +89 -0

89 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#27 lore raw

PATCH v5 5/36 qapi: Prefer explicit relative imports

In reply to John Snow

All of the QAPI include statements are changed to be package-aware, as explicit relative imports.

A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.

When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.

When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.

This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.

It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.

---
 scripts/qapi/commands.py   |  4 ++--
 scripts/qapi/events.py     |  8 ++++----
 scripts/qapi/expr.py       |  4 ++--
 scripts/qapi/gen.py        |  4 ++--
 scripts/qapi/introspect.py |  8 ++++----
 scripts/qapi/main.py       | 14 +++++++-------
 scripts/qapi/parser.py     |  4 ++--
 scripts/qapi/schema.py     |  8 ++++----
 scripts/qapi/types.py      |  6 +++---
 scripts/qapi/visit.py      |  6 +++---
 10 files changed, 33 insertions(+), 33 deletions(-)
scripts/qapi/commands.py +2 -2
index 3cf9e1110b2..ce5926146a4 100644
@@ -13,8 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import *
+from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/events.py +4 -4
index b544af5a1ce..04672724388 100644
@@ -12,10 +12,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember
-from qapi.types import gen_enum, gen_enum_lookup
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember
+from .types import gen_enum, gen_enum_lookup
 
 
 def build_event_send_proto(name, arg_type, boxed):
scripts/qapi/expr.py +2 -2
index 2942520399a..03b31ecfc19 100644
@@ -16,8 +16,8 @@
 
 import re
 from collections import OrderedDict
-from qapi.common import c_name
-from qapi.error import QAPISemError
+from .common import c_name
+from .error import QAPISemError
 
 
 # Names must be letters, numbers, -, and _.  They must start with letter,
scripts/qapi/gen.py +2 -2
index dc7b94aa115..fc57fdca5b9 100644
@@ -17,8 +17,8 @@
 import re
 from contextlib import contextmanager
 
-from qapi.common import *
-from qapi.schema import QAPISchemaVisitor
+from .common import *
+from .schema import QAPISchemaVisitor
 
 
 class QAPIGen:
scripts/qapi/introspect.py +4 -4
index 23652be8102..2a34cd1e8ea 100644
@@ -10,10 +10,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaMonolithicCVisitor
-from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                         QAPISchemaType)
+from .common import *
+from .gen import QAPISchemaMonolithicCVisitor
+from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
+                     QAPISchemaType)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/main.py +7 -7
index 9210a0e1a80..1b168a9771a 100644
@@ -11,13 +11,13 @@
 import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
+from .commands import gen_commands
+from .error import QAPIError
+from .events import gen_events
+from .introspect import gen_introspect
+from .schema import QAPISchema
+from .types import gen_types
+from .visit import gen_visit
 
 
 DEFAULT_OUTPUT_DIR = ''
scripts/qapi/parser.py +2 -2
index 31bc2e6dca9..68d8a1ce1cf 100644
@@ -18,8 +18,8 @@
 import re
 from collections import OrderedDict
 
-from qapi.error import QAPIParseError, QAPISemError
-from qapi.source import QAPISourceInfo
+from .error import QAPIParseError, QAPISemError
+from .source import QAPISourceInfo
 
 
 class QAPISchemaParser:
scripts/qapi/schema.py +4 -4
index 78309a00f0a..a835ee6fde3 100644
@@ -18,10 +18,10 @@
 import re
 from collections import OrderedDict
 
-from qapi.common import c_name, pointer_suffix
-from qapi.error import QAPIError, QAPISemError
-from qapi.expr import check_exprs
-from qapi.parser import QAPISchemaParser
+from .common import c_name, pointer_suffix
+from .error import QAPIError, QAPISemError
+from .expr import check_exprs
+from .parser import QAPISchemaParser
 
 
 class QAPISchemaEntity:
scripts/qapi/types.py +3 -3
index 3640f17cd67..ca9a5aacb39 100644
@@ -13,9 +13,9 @@
 # See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
 
 # variants must be emitted before their container; track what has already
scripts/qapi/visit.py +3 -3
index cdabc5fa283..7850f6e8480 100644
@@ -13,9 +13,9 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
Signature · 1 line
2.26.2
Reviewed-by on v5 5/36
#28 lore raw

Re: [PATCH v5 05/36] qapi: Prefer explicit relative imports

Quoted 33 lines · John Snow

All of the QAPI include statements are changed to be package-aware, as explicit relative imports.

A quirk of Python packages is that the name of the package exists only *outside* of the package. This means that to a module inside of the qapi folder, there is inherently no such thing as the "qapi" package. The reason these imports work is because the "qapi" package exists in the context of the caller -- the execution shim, where sys.path includes a directory that has a 'qapi' folder in it.

When we write "from qapi import sibling", we are NOT referencing the folder 'qapi', but rather "any package named qapi in sys.path". If you should so happen to have a 'qapi' package in your path, it will use *that* package.

When we write "from .sibling import foo", we always reference explicitly our sibling module; guaranteeing consistency in *where* we are importing these modules from.

This can be useful when working with virtual environments and packages in development mode. In development mode, a package is installed as a series of symlinks that forwards to your same source files. The problem arises because code quality checkers will follow "import qapi.x" to the "installed" version instead of the sibling file and -- even though they are the same file -- they have different module paths, and this causes cyclic import problems, false positive type mismatch errors, and more.

It can also be useful when dealing with hierarchical packages, e.g. if we allow qemu.core.qmp, qemu.qapi.parser, etc.

Quoted 12 lines
---
 scripts/qapi/commands.py   |  4 ++--
 scripts/qapi/events.py     |  8 ++++----
 scripts/qapi/expr.py       |  4 ++--
 scripts/qapi/gen.py        |  4 ++--
 scripts/qapi/introspect.py |  8 ++++----
 scripts/qapi/main.py       | 14 +++++++-------
 scripts/qapi/parser.py     |  4 ++--
 scripts/qapi/schema.py     |  8 ++++----
 scripts/qapi/types.py      |  6 +++---
 scripts/qapi/visit.py      |  6 +++---
 10 files changed, 33 insertions(+), 33 deletions(-)
author
#29 lore raw

PATCH v5 6/36 qapi: Remove wildcard includes

In reply to John Snow

Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.

flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.

Remove them and include things explicitly by name instead.

---
 scripts/qapi/commands.py   |  2 +-
 scripts/qapi/events.py     |  7 ++++++-
 scripts/qapi/gen.py        | 12 +++++++++---
 scripts/qapi/introspect.py |  7 ++++++-
 scripts/qapi/types.py      |  8 +++++++-
 scripts/qapi/visit.py      | 10 +++++++++-
 6 files changed, 38 insertions(+), 8 deletions(-)
scripts/qapi/commands.py +1 -1
index ce5926146a4..64ed5278f93 100644
@@ -13,7 +13,7 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import build_params, c_name, mcgen
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
scripts/qapi/events.py +6 -1
index 04672724388..6b3afa14d72 100644
@@ -12,7 +12,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    build_params,
+    c_enum_const,
+    c_name,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
scripts/qapi/gen.py +9 -3
index fc57fdca5b9..1fed712b43b 100644
@@ -11,13 +11,19 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-
+from contextlib import contextmanager
 import errno
 import os
 import re
-from contextlib import contextmanager
 
-from .common import *
+from .common import (
+    c_fname,
+    gen_endif,
+    gen_if,
+    guardend,
+    guardstart,
+    mcgen,
+)
 from .schema import QAPISchemaVisitor
 
 
scripts/qapi/introspect.py +6 -1
index 2a34cd1e8ea..b036fcf9ce7 100644
@@ -10,7 +10,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
                      QAPISchemaType)
scripts/qapi/types.py +7 -1
index ca9a5aacb39..53b47f9e58a 100644
@@ -13,7 +13,13 @@
 # See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
scripts/qapi/visit.py +9 -1
index 7850f6e8480..ea277e7704b 100644
@@ -13,7 +13,15 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+    pop_indent,
+    push_indent,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
 
Signature · 1 line
2.26.2
Reviewed-by on v5 6/36
#30 lore raw

Re: [PATCH v5 06/36] qapi: Remove wildcard includes

Quoted 11 lines · John Snow

Wildcard includes become hard to manage when refactoring and dealing with circular dependencies with strictly typed mypy.

flake8 also flags each one as a warning, as it is not smart enough to know which names exist in the imported file.

Remove them and include things explicitly by name instead.

Quoted 8 lines
---
 scripts/qapi/commands.py   |  2 +-
 scripts/qapi/events.py     |  7 ++++++-
 scripts/qapi/gen.py        | 12 +++++++++---
 scripts/qapi/introspect.py |  7 ++++++-
 scripts/qapi/types.py      |  8 +++++++-
 scripts/qapi/visit.py      | 10 +++++++++-
 6 files changed, 38 insertions(+), 8 deletions(-)
author
#31 lore raw

PATCH v5 7/36 qapi: enforce import order/styling with isort

In reply to John Snow

While we're mucking around with imports, we might as well formalize the style we use. Let's use isort to do it for us.

lines_after_imports=2: Use two lines after imports, to match PEP8's desire to have "two lines before and after" class definitions, which are likely to start immediately after imports.

force_sort_within_sections: Intermingles "from x" and "import x" style statements, such that sorting is always performed strictly on the module name itself.

force_grid_wrap=4: Four or more imports from a single module will force the one-per-line style that's more git-friendly. This will generally happen for 'typing' imports.

multi_line_output=3: Uses the one-per-line indented style for long imports.

include_trailing_comma: Adds a comma to the last import in a group, which makes git conflicts nicer to deal with, generally.

line_length: 72 is chosen to match PEP8's "docstrings and comments" line length limit. If you have a single line import that exceeds 72 characters, your names are too long!

---
 scripts/qapi/.isort.cfg    | 7 +++++++
 scripts/qapi/expr.py       | 3 ++-
 scripts/qapi/introspect.py | 7 +++++--
 scripts/qapi/parser.py     | 2 +-
 scripts/qapi/schema.py     | 2 +-
 5 files changed, 16 insertions(+), 5 deletions(-)
 create mode 100644 scripts/qapi/.isort.cfg
scripts/qapi/.isort.cfg new file +7 -0
new file mode 100644
index 00000000000..6d0fd6cc0d3
@@ -0,0 +1,7 @@
+[settings]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
\ No newline at end of file
scripts/qapi/expr.py +2 -1
index 03b31ecfc19..e73b65b6a7e 100644
@@ -14,8 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-import re
 from collections import OrderedDict
+import re
+
 from .common import c_name
 from .error import QAPISemError
 
scripts/qapi/introspect.py +5 -2
index b036fcf9ce7..31acd2f230a 100644
@@ -17,8 +17,11 @@
     mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaType)
+from .schema import (
+    QAPISchemaArrayType,
+    QAPISchemaBuiltinType,
+    QAPISchemaType,
+)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/parser.py +1 -1
index 68d8a1ce1cf..477ef25ccdf 100644
@@ -14,9 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .error import QAPIParseError, QAPISemError
 from .source import QAPISourceInfo
scripts/qapi/schema.py +1 -1
index a835ee6fde3..093f7a38d88 100644
@@ -14,9 +14,9 @@
 
 # TODO catching name collisions in generated code would be nice
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .common import c_name, pointer_suffix
 from .error import QAPIError, QAPISemError
Signature · 1 line
2.26.2
Reviewed-by on v5 7/36
#32 lore raw

Re: [PATCH v5 07/36] qapi: enforce import order/styling with isort

Quoted 44 lines · John Snow

While we're mucking around with imports, we might as well formalize the style we use. Let's use isort to do it for us.

lines_after_imports=2: Use two lines after imports, to match PEP8's desire to have "two lines before and after" class definitions, which are likely to start immediately after imports.

force_sort_within_sections: Intermingles "from x" and "import x" style statements, such that sorting is always performed strictly on the module name itself.

force_grid_wrap=4: Four or more imports from a single module will force the one-per-line style that's more git-friendly. This will generally happen for 'typing' imports.

multi_line_output=3: Uses the one-per-line indented style for long imports.

include_trailing_comma: Adds a comma to the last import in a group, which makes git conflicts nicer to deal with, generally.

line_length: 72 is chosen to match PEP8's "docstrings and comments" line length limit. If you have a single line import that exceeds 72 characters, your names are too long!

---
 scripts/qapi/.isort.cfg    | 7 +++++++
 scripts/qapi/expr.py       | 3 ++-
 scripts/qapi/introspect.py | 7 +++++--
 scripts/qapi/parser.py     | 2 +-
 scripts/qapi/schema.py     | 2 +-
 5 files changed, 16 insertions(+), 5 deletions(-)
 create mode 100644 scripts/qapi/.isort.cfg
scripts/qapi/.isort.cfg new file +0 -0
new file mode 100644
index 00000000000..6d0fd6cc0d3

scripts/qapi/.isort.cfg line 7

Quoted 8 lines
scripts/qapi/.isort.cfg new file +7 -0
@@ -0,0 +1,7 @@
+[settings]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
\ No newline at end of file

Add one, please :)

Quoted 53 lines
scripts/qapi/expr.py +2 -1
index 03b31ecfc19..e73b65b6a7e 100644
@@ -14,8 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-import re
 from collections import OrderedDict
+import re
+
 from .common import c_name
 from .error import QAPISemError
 
scripts/qapi/introspect.py +5 -2
index b036fcf9ce7..31acd2f230a 100644
@@ -17,8 +17,11 @@
     mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaType)
+from .schema import (
+    QAPISchemaArrayType,
+    QAPISchemaBuiltinType,
+    QAPISchemaType,
+)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
scripts/qapi/parser.py +1 -1
index 68d8a1ce1cf..477ef25ccdf 100644
@@ -14,9 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .error import QAPIParseError, QAPISemError
 from .source import QAPISourceInfo
scripts/qapi/schema.py +0 -0
index a835ee6fde3..093f7a38d88 100644

scripts/qapi/schema.py line 22

Quoted 10 lines
scripts/qapi/schema.py +1 -1
@@ -14,9 +14,9 @@
 
 # TODO catching name collisions in generated code would be nice
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .common import c_name, pointer_suffix
 from .error import QAPIError, QAPISemError
author
#33 lore raw

PATCH v5 8/36 qapi: delint using flake8

In reply to John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.

---
 scripts/qapi/.flake8     |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +++++---
 scripts/qapi/visit.py    | 16 +++++++++++-----
 4 files changed, 20 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
new file mode 100644
index 00000000000..6b158c68b84
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
scripts/qapi/commands.py +2 -1
index 64ed5278f93..5dc2f5a9fa8 100644
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
+                                QObject **ret_out, Error **errp)
 {
     Visitor *v;
 
scripts/qapi/schema.py +5 -3
index 093f7a38d88..cfc52e1ae44 100644
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if not self.tag_member: # flat union
+        if not self.tag_member:  # flat union
             self.tag_member = seen.get(c_name(self._tag_name))
             base = "'base'"
             # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
         self._entity_dict = {}
         self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
-        self._make_module(None) # built-ins
+        self._make_module(None)  # built-ins
         self._make_module(fname)
         self._predefining = True
         self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
             # But it's not tight: the disjunction need not imply it.  We
             # may end up compiling useless wrapper types.
             # TODO kill simple unions or implement the disjunction
-            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+            # pylint: disable=protected-access
+            assert (ifcond or []) == typ._ifcond
         else:
             self._def_entity(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +11 -5
index ea277e7704b..808410d6f1b 100644
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_type)sobj, Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
     %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
 def gen_visit_enum(name):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s *obj, Error **errp)
 {
     int value = *obj;
     bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ def gen_visit_enum(name):
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
 
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
 def gen_visit_object(name, base, members, variants):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
 
Signature · 1 line
2.26.2
#34 lore raw

Re: [PATCH v5 08/36] qapi: delint using flake8

Quoted 110 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.

---
 scripts/qapi/.flake8     |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +++++---
 scripts/qapi/visit.py    | 16 +++++++++++-----
 4 files changed, 20 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8
scripts/qapi/.flake8 new file +2 -0
new file mode 100644
index 00000000000..6b158c68b84
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
scripts/qapi/commands.py +2 -1
index 64ed5278f93..5dc2f5a9fa8 100644
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
+                                QObject **ret_out, Error **errp)
 {
     Visitor *v;
 
scripts/qapi/schema.py +5 -3
index 093f7a38d88..cfc52e1ae44 100644
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if not self.tag_member: # flat union
+        if not self.tag_member:  # flat union
             self.tag_member = seen.get(c_name(self._tag_name))
             base = "'base'"
             # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
         self._entity_dict = {}
         self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
-        self._make_module(None) # built-ins
+        self._make_module(None)  # built-ins
         self._make_module(fname)
         self._predefining = True
         self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
             # But it's not tight: the disjunction need not imply it.  We
             # may end up compiling useless wrapper types.
             # TODO kill simple unions or implement the disjunction
-            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+            # pylint: disable=protected-access
+            assert (ifcond or []) == typ._ifcond
         else:
             self._def_entity(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +7 -3
index ea277e7704b..808410d6f1b 100644
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_type)sobj, Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
     %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
 def gen_visit_enum(name):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s *obj, Error **errp)
 {
     int value = *obj;
     bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);

scripts/qapi/visit.py def gen_visit_enum(name): · line 180

Quoted 6 lines
scripts/qapi/visit.py +2 -1
@@ -172,7 +176,8 @@ def gen_visit_enum(name):
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)

Please break this line the same way as the ones above:

bool visit_type_%(c_name)s(Visitor *v, const char *name,

                    %(c_name)s **obj, Error **errp)

scripts/qapi/visit.py def gen_visit_alternate(name, variants): · line 258

Quoted 12 lines
scripts/qapi/visit.py +2 -1
 {
     bool ok = false;
 
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
 def gen_visit_object(name, base, members, variants):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
 {
     bool ok = false;

Likewise.

author
#35 lore raw

Re: [PATCH v5 08/36] qapi: delint using flake8

Quoted 131 lines · Markus Armbruster
Quoted 110 lines · John Snow

Petty style guide fixes and line length enforcement. Not a big win, not a big loss, but flake8 passes 100% on the qapi module, which gives us an easy baseline to enforce hereafter.

A note on the flake8 exception: flake8 will warn on *any* bare except, but pylint's is context-aware and will suppress the warning if you re-raise the exception.

---
  scripts/qapi/.flake8     |  2 ++
  scripts/qapi/commands.py |  3 ++-
  scripts/qapi/schema.py   |  8 +++++---
  scripts/qapi/visit.py    | 16 +++++++++++-----
  4 files changed, 20 insertions(+), 9 deletions(-)

create mode 100644 scripts/qapi/.flake8

scripts/qapi/.flake8 new file +2 -0
new file mode 100644
index 00000000000..6b158c68b84
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
scripts/qapi/commands.py +2 -1
index 64ed5278f93..5dc2f5a9fa8 100644
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
  def gen_marshal_output(ret_type):
      return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
+                                QObject **ret_out, Error **errp)
  {
      Visitor *v;
 
scripts/qapi/schema.py +5 -3
index 093f7a38d88..cfc52e1ae44 100644
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
              v.set_defined_in(name)
 
      def check(self, schema, seen):
-        if not self.tag_member: # flat union
+        if not self.tag_member:  # flat union
              self.tag_member = seen.get(c_name(self._tag_name))
              base = "'base'"
              # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
          self._entity_dict = {}
          self._module_dict = OrderedDict()
          self._schema_dir = os.path.dirname(fname)
-        self._make_module(None) # built-ins
+        self._make_module(None)  # built-ins
          self._make_module(fname)
          self._predefining = True
          self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
              # But it's not tight: the disjunction need not imply it.  We
              # may end up compiling useless wrapper types.
              # TODO kill simple unions or implement the disjunction
-            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+            # pylint: disable=protected-access
+            assert (ifcond or []) == typ._ifcond
          else:
              self._def_entity(QAPISchemaObjectType(
                  name, info, None, ifcond, None, None, members, None))
scripts/qapi/visit.py +7 -3
index ea277e7704b..808410d6f1b 100644
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
      if not scalar:
          c_type += '*'
      return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_type)sobj, Error **errp);
  ''',
                   c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
  def gen_visit_list(name, element_type):
      return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s **obj, Error **errp)
  {
      bool ok = false;
      %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
  def gen_visit_enum(name):
      return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s *obj, Error **errp)
  {
      int value = *obj;
      bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);

scripts/qapi/visit.py def gen_visit_enum(name): · line 180

Quoted 6 lines
scripts/qapi/visit.py +2 -1
@@ -172,7 +176,8 @@ def gen_visit_enum(name):
  def gen_visit_alternate(name, variants):
      ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)

Please break this line the same way as the ones above:

bool visit_type_%(c_name)s(Visitor *v, const char *name,
%(c_name)s **obj, Error **errp)

Quoted 7 lines
scripts/qapi/visit.py +0 -0
  {
      bool ok = false;
 
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
  def gen_visit_object(name, base, members, variants):
      return mcgen('''
 
Quoted 8 lines

scripts/qapi/visit.py def gen_visit_enum(name): · line 182

Quoted 5 lines
scripts/qapi/visit.py +2 -1
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
  {
      bool ok = false;

Likewise.

Whupps. Bad conflict resolution on my part. Didn't mean to do this.

--js

author
#36 lore raw

PATCH v5 9/36 qapi: add pylintrc

In reply to John Snow

Using pylint --generate-rcfile > pylintrc, generate a skeleton pylintrc file. Sections that are not presently relevant (by the end of this series) are removed leaving just the empty section as a search engine / documentation hint to future authors.

I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is released), I aim to have gitlab CI running the specific targeted versions of pylint, mypy, flake8, etc in a job.

2.5.x will work if you additionally pass --disable=bad-whitespace. This warning was removed from 2.6.x, for lack of consistent support.

Right now, quite a few modules are ignored as they are known to fail as of this commit. modules will be removed from the known-bad list throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as that folder is shared by many things, it is going inside the module folder (for now). Due to a bug in pylint 2.5+, pylint does not correctly recognize when it is being run from "inside" a package, and must be run *outside* of the package.

Therefore, to run it, you must:

Quoted 1 line

pylint scripts/qapi/ --rcfile=scripts/qapi/pylintrc

---
 scripts/qapi/pylintrc | 73 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100644 scripts/qapi/pylintrc
scripts/qapi/pylintrc new file +73 -0
new file mode 100644
index 00000000000..76d54c30f85
@@ -0,0 +1,73 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the ignore list.
+# The regex matches against base names, not paths.
+ignore-patterns=common.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore import statements themselves when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
Signature · 1 line
2.26.2
author
#37 lore raw

PATCH v5 10/36 qapi/common.py: Remove python compatibility workaround

In reply to John Snow
---
 scripts/qapi/common.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
scripts/qapi/common.py +1 -4
index ba35abea478..cee63eb95c7 100644
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
     raw = code % kwds
     if indent_level:
         indent = genindent(indent_level)
-        # re.subn() lacks flags support before Python 2.7, use re.compile()
-        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-                      indent, raw)
-        raw = raw[0]
+        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
Signature · 1 line
2.26.2
author
#38 lore raw

PATCH v5 11/36 qapi/common.py: Add indent manager

In reply to John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Make a little indent level manager instead.

---
 scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 36 insertions(+), 20 deletions(-)
scripts/qapi/common.py +33 -16
index cee63eb95c7..b35318b72cf 100644
@@ -93,33 +93,50 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indentation:
+    """
+    Indentation level management.
 
+    :param initial: Initial number of spaces, default 0.
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the current indentation as a string of spaces."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indentation."""
+        return bool(self._level)
 
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def increase(self, amount: int = 4) -> None:
+        """Increase the indentation level by ``amount``, default 4."""
+        self._level += amount
+
+    def decrease(self, amount: int = 4) -> None:
+        """Decrease the indentation level by ``amount``, default 4."""
+        if self._level < amount:
+            raise ArithmeticError(
+                f"Can't remove {amount:d} spaces from {self!r}")
+        self._level -= amount
+
+
+indent = Indentation()
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey indent, and strip eatspace.
 def cgen(code, **kwds):
     raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if indent:
+        raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
scripts/qapi/visit.py +3 -4
index 808410d6f1b..14f30c228b7 100644
@@ -18,9 +18,8 @@
     c_name,
     gen_endif,
     gen_if,
+    indent,
     mcgen,
-    pop_indent,
-    push_indent,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
     if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
                          name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            indent.increase()
         ret += mcgen('''
     if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
         return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
         if memb.optional:
-            pop_indent()
+            indent.decrease()
             ret += mcgen('''
     }
 ''')
Signature · 1 line
2.26.2
#39 lore raw

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

Quoted 10 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Make a little indent level manager instead.

Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.

Quoted 47 lines
---
 scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 36 insertions(+), 20 deletions(-)
scripts/qapi/common.py +17 -11
index cee63eb95c7..b35318b72cf 100644
@@ -93,33 +93,50 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indentation:
+    """
+    Indentation level management.
 
+    :param initial: Initial number of spaces, default 0.
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the current indentation as a string of spaces."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indentation."""
+        return bool(self._level)
 
-def pop_indent(indent_amount=4):
-    global indent_level

scripts/qapi/common.py def c_name(name, protect=True): · line 127

Quoted 10 lines
scripts/qapi/common.py +9 -1
-    indent_level -= indent_amount
+    def increase(self, amount: int = 4) -> None:
+        """Increase the indentation level by ``amount``, default 4."""
+        self._level += amount
+
+    def decrease(self, amount: int = 4) -> None:
+        """Decrease the indentation level by ``amount``, default 4."""
+        if self._level < amount:
+            raise ArithmeticError(
+                f"Can't remove {amount:d} spaces from {self!r}")

Raise a fancy error when there's an actual need for it. You're not coding a framework thousands of people you never heard of will put to uses you cannot imagine.

Quoted 52 lines
scripts/qapi/common.py +7 -4
+        self._level -= amount
+
+
+indent = Indentation()
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey indent, and strip eatspace.
 def cgen(code, **kwds):
     raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if indent:
+        raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
scripts/qapi/visit.py +3 -4
index 808410d6f1b..14f30c228b7 100644
@@ -18,9 +18,8 @@
     c_name,
     gen_endif,
     gen_if,
+    indent,
     mcgen,
-    pop_indent,
-    push_indent,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
     if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
                          name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            indent.increase()
         ret += mcgen('''
     if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
         return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
         if memb.optional:
-            pop_indent()
+            indent.decrease()
             ret += mcgen('''
     }
 ''')
author
#40 lore raw

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Quoted 9 lines
Quoted 5 lines

Make a little indent level manager instead.

Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.

I agree, but refactoring it properly is beyond my capacity right now.

This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.

Creating a stateful object was the fastest way from A to B.

Quoted 45 lines
Quoted 45 lines
---
  scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
  scripts/qapi/visit.py  |  7 +++---
  2 files changed, 36 insertions(+), 20 deletions(-)
scripts/qapi/common.py +17 -9
index cee63eb95c7..b35318b72cf 100644
@@ -93,33 +93,50 @@ def c_name(name, protect=True):
  pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indentation:
+    """
+    Indentation level management.
 
+    :param initial: Initial number of spaces, default 0.
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the current indentation as a string of spaces."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indentation."""
+        return bool(self._level)
 
Quoted 17 lines

scripts/qapi/common.py def c_name(name, protect=True): · line 127

Quoted 12 lines
scripts/qapi/common.py +9 -3
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def increase(self, amount: int = 4) -> None:
+        """Increase the indentation level by ``amount``, default 4."""
+        self._level += amount
+
+    def decrease(self, amount: int = 4) -> None:
+        """Decrease the indentation level by ``amount``, default 4."""
+        if self._level < amount:
+            raise ArithmeticError(
+                f"Can't remove {amount:d} spaces from {self!r}")

Raise a fancy error when there's an actual need for it. You're not coding a framework thousands of people you never heard of will put to uses you cannot imagine.

It's not fancy, it's just a normal built-in exception, like AssertionError or any other:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ArithmeticError: Can't remove 4 spaces from Indent(0)

vs

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError: Can't remove 4 spaces from Indent(0)

I feel like it's kind of a lateral move? I realize you feel this is an overfancy class doing what we hope is a temporary job, and that I have overengineered the hell out of a tiny do-nothing class... but I suppose that's also why I feel weird changing it around so much to accomplish so little.

Differences in style, I suppose.

Feel free to change it around to suit your tastes, I don't think it's worth spending a lot of ping-pong time on this paintsink in particular.

--js

Quoted 52 lines
Quoted 52 lines
scripts/qapi/common.py +7 -4
+        self._level -= amount
+
+
+indent = Indentation()
 
 
  # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey indent, and strip eatspace.
  def cgen(code, **kwds):
      raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if indent:
+        raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
      return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
scripts/qapi/visit.py +3 -4
index 808410d6f1b..14f30c228b7 100644
@@ -18,9 +18,8 @@
      c_name,
      gen_endif,
      gen_if,
+    indent,
      mcgen,
-    pop_indent,
-    push_indent,
  )
  from .gen import QAPISchemaModularCVisitor, ifcontext
  from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
      if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
  ''',
                           name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            indent.increase()
          ret += mcgen('''
      if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
          return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
                       c_type=memb.type.c_name(), name=memb.name,
                       c_name=c_name(memb.name))
          if memb.optional:
-            pop_indent()
+            indent.decrease()
              ret += mcgen('''
      }
  ''')
#41 lore raw

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

Quoted 18 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Quoted 9 lines
Quoted 5 lines

Make a little indent level manager instead.

Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.

Quoted 6 lines

I agree, but refactoring it properly is beyond my capacity right now.

This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.

Creating a stateful object was the fastest way from A to B.

An even dumber solution could be:

indent_prefixes = []
def push_indent(amount=4):
"""Add amount spaces to indentation prefix""" indent_prefixes.push(' '*amount)
def pop_indent():
"""Revert the most recent push_indent() call""" indent_prefixes.pop()
def genindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)

No global keyword involved, and the only stateful object is a dumb list.

Signature · 1 line
Eduardo
#42 lore raw

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

Quoted 39 lines · Eduardo Habkost
Quoted 18 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Quoted 9 lines
Quoted 5 lines

Make a little indent level manager instead.

Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.

Quoted 6 lines

I agree, but refactoring it properly is beyond my capacity right now.

This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.

Creating a stateful object was the fastest way from A to B.

An even dumber solution could be:

indent_prefixes = []
def push_indent(amount=4):
"""Add amount spaces to indentation prefix""" indent_prefixes.push(' '*amount)
def pop_indent():
"""Revert the most recent push_indent() call""" indent_prefixes.pop()
def genindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)

Quoted 2 lines

No global keyword involved, and the only stateful object is a dumb list.

Ha, this is Dumb with a capital D! I respect that :)

John, I'm not asking you to switch. Use your judgement.

author
#43 lore raw

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

Quoted 41 lines · Markus Armbruster
Quoted 39 lines · Eduardo Habkost
Quoted 18 lines · John Snow
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · John Snow

Code style tools really dislike the use of global keywords, because it generally involves re-binding the name at runtime which can have strange effects depending on when and how that global name is referenced in other modules.

Quoted 9 lines
Quoted 5 lines

Make a little indent level manager instead.

Intentation is a job for QAPIGen (and its subtypes). But if this patch is easier to achieve this series' goal, I don't mind.

Quoted 6 lines

I agree, but refactoring it properly is beyond my capacity right now.

This was the dumbest thing I could do to get pylint/mypy passing, which required the elimination (or suppression) of the global keyword.

Creating a stateful object was the fastest way from A to B.

An even dumber solution could be:

indent_prefixes = []
def push_indent(amount=4):
"""Add amount spaces to indentation prefix""" indent_prefixes.push(' '*amount)
def pop_indent():
"""Revert the most recent push_indent() call""" indent_prefixes.pop()
def genindent():
"""Return the current indentation prefix""" return ''.join(indent_prefixes)

Quoted 7 lines
Quoted 2 lines

No global keyword involved, and the only stateful object is a dumb list.

Ha, this is Dumb with a capital D! I respect that :)

John, I'm not asking you to switch. Use your judgement.

It's something we'll revisit soon, I'm sure. it's not good to be managing indent in so many different ways in so many different places.

(I prefer to leave it alone for now to try and press forward with accomplishing regression checks on strict typing.)

--js

author
#44 lore raw

PATCH v5 12/36 qapi/common.py: delint with pylint

In reply to John Snow

At this point, that just means using a consistent strategy for constant names. constants get UPPER_CASE and names not used externally get a leading underscore.

As a preference, while renaming constants to be UPPERCASE, move them to the head of the file. Generally, it's nice to be able to audit the code that runs on import in one central place.

---
 scripts/qapi/common.py | 18 ++++++++----------
 scripts/qapi/schema.py | 14 +++++++-------
 2 files changed, 15 insertions(+), 17 deletions(-)
scripts/qapi/common.py +8 -10
index b35318b72cf..a417b6029c8 100644
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+_C_NAME_TRANS = str.maketrans('.-', '__')
+
+
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True):
                      'not_eq', 'or', 'or_eq', 'xor', 'xor_eq'])
     # namespace pollution:
     polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386'])
-    name = name.translate(c_name_trans)
+    name = name.translate(_C_NAME_TRANS)
     if protect and (name in c89_words | c99_words | c11_words | gcc_words
                     | cpp_words | polluted_words):
         return 'q_' + name
     return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indentation:
     """
     Indentation level management.
@@ -132,12 +130,12 @@ def decrease(self, amount: int = 4) -> None:
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent, and strip eatspace.
+# Obey indent, and strip EATSPACE.
 def cgen(code, **kwds):
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):
scripts/qapi/schema.py +7 -7
index cfc52e1ae44..74c6b96d391 100644
@@ -18,7 +18,7 @@
 import os
 import re
 
-from .common import c_name, pointer_suffix
+from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
 from .expr import check_exprs
 from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self):
         return True
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'array'
@@ -430,7 +430,7 @@ def c_name(self):
 
     def c_type(self):
         assert not self.is_implicit()
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def c_unboxed_type(self):
         return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None):
             v.connect_doc(doc)
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type):
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
-        for t in [('str',    'string',  'char' + pointer_suffix),
+        for t in [('str',    'string',  'char' + POINTER_SUFFIX),
                   ('number', 'number',  'double'),
                   ('int',    'int',     'int64_t'),
                   ('int8',   'int',     'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self):
                   ('uint64', 'int',     'uint64_t'),
                   ('size',   'int',     'uint64_t'),
                   ('bool',   'boolean', 'bool'),
-                  ('any',    'value',   'QObject' + pointer_suffix),
-                  ('null',   'null',    'QNull' + pointer_suffix)]:
+                  ('any',    'value',   'QObject' + POINTER_SUFFIX),
+                  ('null',   'null',    'QNull' + POINTER_SUFFIX)]:
             self._def_builtin_type(*t)
         self.the_empty_object_type = QAPISchemaObjectType(
             'q_empty', None, None, None, None, None, [], None)
Signature · 1 line
2.26.2
author
#45 lore raw

PATCH v5 13/36 qapi/common.py: Replace one-letter 'c' variable

In reply to John Snow
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
index a417b6029c8..338adedef4f 100644
@@ -30,14 +30,14 @@ def camel_to_upper(value):
     new_name = ''
     length = len(c_fun_str)
     for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+        char = c_fun_str[i]
+        # When char is upper case and no '_' appears before, do more checks
+        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
             if i < length - 1 and c_fun_str[i + 1].islower():
                 new_name += '_'
             elif c_fun_str[i - 1].isdigit():
                 new_name += '_'
-        new_name += c
+        new_name += char
     return new_name.lstrip('_').upper()
 
 
Signature · 1 line
2.26.2
Reviewed-by on v5 13/36
#46 lore raw

Re: [PATCH v5 13/36] qapi/common.py: Replace one-letter 'c' variable

Quoted 18 lines · John Snow
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +0 -3
index a417b6029c8..338adedef4f 100644
@@ -30,14 +30,14 @@ def camel_to_upper(value):
     new_name = ''
     length = len(c_fun_str)
     for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':

scripts/qapi/common.py def camel_to_upper(value): · line 41

Quoted 10 lines
scripts/qapi/common.py +4 -1
+        char = c_fun_str[i]
+        # When char is upper case and no '_' appears before, do more checks
+        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
             if i < length - 1 and c_fun_str[i + 1].islower():
                 new_name += '_'
             elif c_fun_str[i - 1].isdigit():
                 new_name += '_'
-        new_name += c
+        new_name += char
     return new_name.lstrip('_').upper()
author
#47 lore raw

PATCH v5 14/36 qapi/common.py: check with pylint

In reply to John Snow

Remove qapi/common.py from the pylintrc ignore list.

---
 scripts/qapi/pylintrc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi/pylintrc +1 -2
index 76d54c30f85..507f15537ab 100644
@@ -2,8 +2,7 @@
 
 # Add files or directories matching the regex patterns to the ignore list.
 # The regex matches against base names, not paths.
-ignore-patterns=common.py,
-                error.py,
+ignore-patterns=error.py,
                 expr.py,
                 gen.py,
                 parser.py,
Signature · 1 line
2.26.2
author
#48 lore raw

PATCH v5 15/36 qapi/common.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/common.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +16 -11
index 338adedef4f..74a2c001ed9 100644
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
     return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
 
 # Generate @code with @kwds interpolated.
 # Obey indent, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: object) -> str:
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
     return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: object) -> str:
     if code[0] == '\n':
         code = code[1:]
     return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
     return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name):
                  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
     return mcgen('''
 
 #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name):
                  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in ifcond:
         ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond):
     return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in reversed(ifcond):
         ret += mcgen('''
@@ -183,7 +186,9 @@ def gen_endif(ifcond):
     return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
     ret = ''
     sep = ''
     if boxed:
Signature · 1 line
2.26.2
#49 lore raw

Re: [PATCH v5 15/36] qapi/common.py: add type hint annotations

Quoted 108 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/common.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +13 -10
index 338adedef4f..74a2c001ed9 100644
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
     return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
 
 # Generate @code with @kwds interpolated.
 # Obey indent, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: object) -> str:
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
     return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: object) -> str:
     if code[0] == '\n':
         code = code[1:]
     return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
     return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name):
                  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
     return mcgen('''
 
 #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name):
                  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in ifcond:
         ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond):
     return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in reversed(ifcond):
         ret += mcgen('''

scripts/qapi/common.py def gen_endif(ifcond): · line 194

Quoted 10 lines
scripts/qapi/common.py +3 -1
@@ -183,7 +186,9 @@ def gen_endif(ifcond):
     return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
     ret = ''
     sep = ''
     if boxed:

@arg_type is the only parameter left unannotated. Scratching head... aha:

qapi/common.py: move build_params into gen.py

Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.

Let's amend the commit message:

Note that build_params() cannot be fully annotated due to import dependency issues. The commit after next will take care of it.

If we swap the next two commits, the fix follows immediately. Better, but only worth it if swapping is pretty much effortless.

author
#50 lore raw

Re: [PATCH v5 15/36] qapi/common.py: add type hint annotations

Quoted 131 lines · Markus Armbruster
Quoted 108 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
  scripts/qapi/common.py | 27 ++++++++++++++++-----------
  1 file changed, 16 insertions(+), 11 deletions(-)
scripts/qapi/common.py +13 -10
index 338adedef4f..74a2c001ed9 100644
@@ -12,6 +12,7 @@
  # See the COPYING file in the top-level directory.
 
  import re
+from typing import Optional, Sequence
 
 
  EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
  # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
      c_fun_str = c_name(value, False)
      if value.isupper():
          return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
      return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
      if prefix is not None:
          type_name = prefix
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
  # into substrings of a generated C function name.
  # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
  # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
      # ANSI X3J11/88-090, 3.1.1
      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                       'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
 
  # Generate @code with @kwds interpolated.
  # Obey indent, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: object) -> str:
      raw = code % kwds
      if indent:
          raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
      return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: object) -> str:
      if code[0] == '\n':
          code = code[1:]
      return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
      return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
      return mcgen('''
  #ifndef %(name)s
  #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name):
                   name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
      return mcgen('''
 
  #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name):
                   name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
      ret = ''
      for ifc in ifcond:
          ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond):
      return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
      ret = ''
      for ifc in reversed(ifcond):
          ret += mcgen('''

scripts/qapi/common.py def gen_endif(ifcond): · line 194

Quoted 10 lines
scripts/qapi/common.py +3 -1
@@ -183,7 +186,9 @@ def gen_endif(ifcond):
      return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
      ret = ''
      sep = ''
      if boxed:

@arg_type is the only parameter left unannotated. Scratching head... aha:

qapi/common.py: move build_params into gen.py

Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.

Quoted 8 lines

Let's amend the commit message:

Note that build_params() cannot be fully annotated due to import dependency issues. The commit after next will take care of it.

If we swap the next two commits, the fix follows immediately. Better, but only worth it if swapping is pretty much effortless.

The reason they're not flipped was to avoid giving the impression that docstrings were what was preventing us from enabling the mypy type checking.

The reason I don't flip the comments and the type hints is because that's a lot of rebase pain for perceptibly little benefit.

I amended the commit message.

author
#51 lore raw

PATCH v5 16/36 qapi/common.py: Convert comments into docstrings, and elaborate

In reply to John Snow

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use :param X: descr, :return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)

---
 scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 14 deletions(-)
scripts/qapi/common.py +39 -14
index 74a2c001ed9..0ef38ea5fe0 100644
@@ -15,15 +15,24 @@
 from typing import Optional, Sequence
 
 
+#: Sentinel value that causes all space to its right to be removed.
 EATSPACE = '\033EATSPACE.'
 POINTER_SUFFIX = ' *' + EATSPACE
 _C_NAME_TRANS = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+    """
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
                  const_name: str,
                  prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map ``name`` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending ``q_``.
+    """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
         self._level -= amount
 
 
+#: Global, current indent level for code generation.
 indent = Indentation()
 
 
-# Generate @code with @kwds interpolated.
-# Obey indent, and strip EATSPACE.
 def cgen(code: str, **kwds: object) -> str:
+    """
+    Generate ``code`` with ``kwds`` interpolated.
+
+    Obey `indent`, and strip `EATSPACE`.
+    """
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
Signature · 1 line
2.26.2
#52 lore raw

Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

Quoted 31 lines · John Snow

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use :param X: descr, :return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)

---
 scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 14 deletions(-)
scripts/qapi/common.py +0 -0
index 74a2c001ed9..0ef38ea5fe0 100644

scripts/qapi/common.py line 18

Quoted 4 lines
scripts/qapi/common.py +1 -0
@@ -15,15 +15,24 @@
 from typing import Optional, Sequence
 
 
+#: Sentinel value that causes all space to its right to be removed.

What's the purpose of : after # ?

I'm not sure this is a "sentinel value". Wikipedia:

In computer programming, a sentinel value (also referred to as a flag value, trip value, rogue value, signal value, or dummy data)[1] is a special value in the context of an algorithm which uses its presence as a condition of termination, typically in a loop or recursive algorithm.

https://en.wikipedia.org/wiki/Sentinel_value

Perhaps

# Magic string value that gets removed along with all space to the

   # right.
Quoted 10 lines
scripts/qapi/common.py +1 -3
 EATSPACE = '\033EATSPACE.'
 POINTER_SUFFIX = ' *' + EATSPACE
 _C_NAME_TRANS = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+    """

scripts/qapi/common.py line 35

Quoted 10 lines
scripts/qapi/common.py +10 -0
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """

I wonder whether these indented lines get wrapped into one unintelligible parapgraph.

Have you eyeballed the output of Sphinx?

Quoted 65 lines
scripts/qapi/common.py +27 -11
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
                  const_name: str,
                  prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map ``name`` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending ``q_``.
+    """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
         self._level -= amount
 
 
+#: Global, current indent level for code generation.
 indent = Indentation()
 
 
-# Generate @code with @kwds interpolated.
-# Obey indent, and strip EATSPACE.
 def cgen(code: str, **kwds: object) -> str:
+    """
+    Generate ``code`` with ``kwds`` interpolated.
+
+    Obey `indent`, and strip `EATSPACE`.
+    """
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
author
#53 lore raw

Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

Quoted 36 lines · Markus Armbruster
Quoted 34 lines · John Snow

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use :param X: descr, :return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)

---
  scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
  1 file changed, 39 insertions(+), 14 deletions(-)
scripts/qapi/common.py +0 -0
index 74a2c001ed9..0ef38ea5fe0 100644
@@ -15,15 +15,24 @@
  from typing import Optional, Sequence
 
 
Quoted 4 lines

scripts/qapi/common.py line 18

Quoted 1 line
scripts/qapi/common.py +1 -0
+#: Sentinel value that causes all space to its right to be removed.

What's the purpose of : after # ?

Documents this name in Sphinx. We had a small discussion about it, I think; "Does using this special form or the docstring make the comment visible in any IDE?" (No.)

There's no Python-AST way to document these, but there is a Sphinx way to document them, so I did that.

(Doing it like this allows EATSPACE to be used as a cross-reference.)

Quoted 10 lines

I'm not sure this is a "sentinel value". Wikipedia:

In computer programming, a sentinel value (also referred to as a flag value, trip value, rogue value, signal value, or dummy data)[1] is a special value in the context of an algorithm which uses its presence as a condition of termination, typically in a loop or recursive algorithm.

https://en.wikipedia.org/wiki/Sentinel_value

I really should try to learn English as a second language so I know what any of the words I use mean, I guess. I had slipped to a less strict usage where it meant more like "placeholder".

Quoted 5 lines

Perhaps

# Magic string value that gets removed along with all space to the # right.

This can be written on one line if we gently disregard the 72 column limit. (Maybe you already did when you wrote it and my client wrapped it. Who knows!)

Quoted 24 lines
Quoted 10 lines
scripts/qapi/common.py +1 -3
  EATSPACE = '\033EATSPACE.'
  POINTER_SUFFIX = ' *' + EATSPACE
  _C_NAME_TRANS = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
  def camel_to_upper(value: str) -> str:
+    """

scripts/qapi/common.py line 35

Quoted 10 lines
scripts/qapi/common.py +10 -0
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """

I wonder whether these indented lines get wrapped into one unintelligible parapgraph.

Quoted 2 lines

Have you eyeballed the output of Sphinx?

Eyeballed, but didn't validate this specific one. Yeah, it's nonsense.

Examples::

    ENUMName -> ENUM_NAME

etc. works better.

Quoted 65 lines
Quoted 65 lines
scripts/qapi/common.py +27 -11
      c_fun_str = c_name(value, False)
      if value.isupper():
          return c_fun_str
@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
  def c_enum_const(type_name: str,
                   const_name: str,
                   prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """
      if prefix is not None:
          type_name = prefix
      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
  def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map ``name`` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending ``q_``.
+    """
      # ANSI X3J11/88-090, 3.1.1
      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                       'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
          self._level -= amount
 
 
+#: Global, current indent level for code generation.
  indent = Indentation()
 
 
-# Generate @code with @kwds interpolated.
-# Obey indent, and strip EATSPACE.
  def cgen(code: str, **kwds: object) -> str:
+    """
+    Generate ``code`` with ``kwds`` interpolated.
+
+    Obey `indent`, and strip `EATSPACE`.
+    """
      raw = code % kwds
      if indent:
          raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
#54 lore raw

Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

Quoted 40 lines · John Snow
Quoted 35 lines · Markus Armbruster
Quoted 33 lines · John Snow

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation style. Sphinx uses an extension of ReST with "domains". We use the (implicit) Python domain, which supports a number of custom "info fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use :param X: descr, :return[s]: descr, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx either chokes or succeeds, but does not enforce a standard of what is otherwise inside the docstring. Pycharm does highlight when your param fields are not aligned with the actual fields present. It does not highlight missing return or exception statements. There is no existing style guide I am aware of that covers a standard for a minimally acceptable docstring. I am debating writing one.)

---
  scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
  1 file changed, 39 insertions(+), 14 deletions(-)
scripts/qapi/common.py +0 -0
index 74a2c001ed9..0ef38ea5fe0 100644
@@ -15,15 +15,24 @@
  from typing import Optional, Sequence
 
Quoted 3 lines

scripts/qapi/common.py line 18

Quoted 1 line
scripts/qapi/common.py +1 -0
+#: Sentinel value that causes all space to its right to be removed.

What's the purpose of : after # ?

Quoted 8 lines

Documents this name in Sphinx. We had a small discussion about it, I think; "Does using this special form or the docstring make the comment visible in any IDE?" (No.)

There's no Python-AST way to document these, but there is a Sphinx way to document them, so I did that.

(Doing it like this allows EATSPACE to be used as a cross-reference.)

Thanks.

Consider pointing this out when you write the comment & doc string part of our Python style guide.

Quoted 19 lines
Quoted 8 lines

I'm not sure this is a "sentinel value". Wikipedia:
In computer programming, a sentinel value (also referred to as a flag value, trip value, rogue value, signal value, or dummy data)[1] is a special value in the context of an algorithm which uses its presence as a condition of termination, typically in a loop or recursive algorithm.
https://en.wikipedia.org/wiki/Sentinel_value

I really should try to learn English as a second language so I know what any of the words I use mean, I guess. I had slipped to a less strict usage where it meant more like "placeholder".

Quoted 5 lines

Perhaps
# Magic string value that gets removed along with all space to
the

    # right.
Quoted 3 lines

This can be written on one line if we gently disregard the 72 column limit. (Maybe you already did when you wrote it and my client wrapped it. Who knows!)

Drop the period and it fits ;-P

You could also drop "value" without loss.

[...]

author
#55 lore raw

PATCH v5 17/36 qapi/common.py: move build_params into gen.py

In reply to John Snow

Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.

---
 scripts/qapi/commands.py |  9 +++++++--
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  9 ++-------
 scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
 4 files changed, 38 insertions(+), 34 deletions(-)
scripts/qapi/commands.py +7 -2
index 5dc2f5a9fa8..f67393f8713 100644
@@ -13,8 +13,13 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    build_params,
+    ifcontext,
+)
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
index 0ef38ea5fe0..9ab0685cc51 100644
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
     return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
scripts/qapi/events.py +2 -7
index 6b3afa14d72..f840a62ed92 100644
@@ -12,13 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import (
-    build_params,
-    c_enum_const,
-    c_name,
-    mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
 
scripts/qapi/gen.py +29 -2
index 1fed712b43b..f2e2746fea5 100644
@@ -2,9 +2,11 @@
 #
 # QAPI code generation
 #
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2019 Red Hat Inc.
 #
 # Authors:
+#  Anthony Liguori <[email protected]>
 #  Markus Armbruster <[email protected]>
 #  Marc-André Lureau <[email protected]>
 #
@@ -15,16 +17,18 @@
 import errno
 import os
 import re
+from typing import Optional
 
 from .common import (
     c_fname,
+    c_name,
     gen_endif,
     gen_if,
     guardend,
     guardstart,
     mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
 
 
 class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
     return out
 
 
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
+
+
 class QAPIGenCCode(QAPIGen):
 
     def __init__(self, fname):
Signature · 1 line
2.26.2
#56 lore raw

Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py

Quoted 93 lines · John Snow

Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.

---
 scripts/qapi/commands.py |  9 +++++++--
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  9 ++-------
 scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
 4 files changed, 38 insertions(+), 34 deletions(-)
scripts/qapi/commands.py +7 -2
index 5dc2f5a9fa8..f67393f8713 100644
@@ -13,8 +13,13 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    build_params,
+    ifcontext,
+)
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
index 0ef38ea5fe0..9ab0685cc51 100644
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
     return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
scripts/qapi/events.py +2 -7
index 6b3afa14d72..f840a62ed92 100644
@@ -12,13 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import (
-    build_params,
-    c_enum_const,
-    c_name,
-    mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
 
scripts/qapi/gen.py +0 -0
index 1fed712b43b..f2e2746fea5 100644
@@ -2,9 +2,11 @@
 #
 # QAPI code generation

scripts/qapi/gen.py line 12

Quoted 10 lines
scripts/qapi/gen.py +3 -1
 #
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2019 Red Hat Inc.
 #
 # Authors:
+#  Anthony Liguori <[email protected]>
 #  Markus Armbruster <[email protected]>
 #  Marc-André Lureau <[email protected]>
 #

The code you move into this file is actually Red Hat's:

$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date'
Author: Markus Armbruster <[email protected]>
Date:   Wed Aug 15 21:37:36 2018 +0800
Author: Marc-André Lureau <[email protected]>
Date:   Thu Jun 1 16:41:41 2017 +0400
Author: Eric Blake <[email protected]>
Date:   Wed Jul 13 21:50:20 2016 -0600
Author: Eric Blake <[email protected]>
Date:   Wed Jul 13 21:50:19 2016 -0600
Author: Eric Blake <[email protected]>
Date:   Thu Mar 17 16:48:28 2016 -0600
Author: Markus Armbruster <[email protected]>
Date:   Wed Sep 16 13:06:20 2015 +0200

So the correct update is simply

#
# QAPI code generation
#
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
#
# Authors:
# Markus Armbruster <[email protected]>
# Marc-André Lureau <[email protected]> #

Quoted 50 lines
scripts/qapi/gen.py +26 -1
@@ -15,16 +17,18 @@
 import errno
 import os
 import re
+from typing import Optional
 
 from .common import (
     c_fname,
+    c_name,
     gen_endif,
     gen_if,
     guardend,
     guardstart,
     mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
 
 
 class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
     return out
 
 
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
+
+
 class QAPIGenCCode(QAPIGen):
 
     def __init__(self, fname):
author
#57 lore raw

Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py

Quoted 127 lines · Markus Armbruster
Quoted 93 lines · John Snow

Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.

---
  scripts/qapi/commands.py |  9 +++++++--
  scripts/qapi/common.py   | 23 -----------------------
  scripts/qapi/events.py   |  9 ++-------
  scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
  4 files changed, 38 insertions(+), 34 deletions(-)
scripts/qapi/commands.py +7 -2
index 5dc2f5a9fa8..f67393f8713 100644
@@ -13,8 +13,13 @@
  See the COPYING file in the top-level directory.
  """
 
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    build_params,
+    ifcontext,
+)
 
 
  def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
index 0ef38ea5fe0..9ab0685cc51 100644
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
  #endif /* %(cond)s */
  ''', cond=ifc)
      return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
scripts/qapi/events.py +2 -7
index 6b3afa14d72..f840a62ed92 100644
@@ -12,13 +12,8 @@
  See the COPYING file in the top-level directory.
  """
 
-from .common import (
-    build_params,
-    c_enum_const,
-    c_name,
-    mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
  from .schema import QAPISchemaEnumMember
  from .types import gen_enum, gen_enum_lookup
 
scripts/qapi/gen.py +0 -0
index 1fed712b43b..f2e2746fea5 100644
@@ -2,9 +2,11 @@
  #
  # QAPI code generation

scripts/qapi/gen.py line 12

Quoted 10 lines
scripts/qapi/gen.py +3 -1
  #
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2019 Red Hat Inc.
  #
  # Authors:
+#  Anthony Liguori <[email protected]>
  #  Markus Armbruster <[email protected]>
  #  Marc-André Lureau <[email protected]>
  #

The code you move into this file is actually Red Hat's:

$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date'
Author: Markus Armbruster <[email protected]>
Date:   Wed Aug 15 21:37:36 2018 +0800
Author: Marc-André Lureau <[email protected]>
Date:   Thu Jun 1 16:41:41 2017 +0400
Author: Eric Blake <[email protected]>
Date:   Wed Jul 13 21:50:20 2016 -0600
Author: Eric Blake <[email protected]>
Date:   Wed Jul 13 21:50:19 2016 -0600
Author: Eric Blake <[email protected]>
Date:   Thu Mar 17 16:48:28 2016 -0600
Author: Markus Armbruster <[email protected]>
Date:   Wed Sep 16 13:06:20 2015 +0200

So the correct update is simply

    #
    # QAPI code generation
    #
Quoted 8 lines

-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
#
# Authors:
# Markus Armbruster <[email protected]> # Marc-André Lureau <[email protected]> #

OK, thank you. I did the dumbest possibly correct thing. My appetite for doing line ownership analysis is ... low.

Quoted 50 lines
Quoted 50 lines
scripts/qapi/gen.py +26 -1
@@ -15,16 +17,18 @@
  import errno
  import os
  import re
+from typing import Optional
 
  from .common import (
      c_fname,
+    c_name,
      gen_endif,
      gen_if,
      guardend,
      guardstart,
      mcgen,
  )
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
 
 
  class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
      return out
 
 
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
+
+
  class QAPIGenCCode(QAPIGen):
 
      def __init__(self, fname):
#58 lore raw

Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py

In reply to Markus Armbruster
Quoted 127 lines · Markus Armbruster
Quoted 93 lines · John Snow

Including it in common.py creates a circular import dependency; schema relies on common, but common.build_params requires a type annotation from schema. To type this properly, it needs to be moved outside the cycle.

---
 scripts/qapi/commands.py |  9 +++++++--
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  9 ++-------
 scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
 4 files changed, 38 insertions(+), 34 deletions(-)
scripts/qapi/commands.py +7 -2
index 5dc2f5a9fa8..f67393f8713 100644
@@ -13,8 +13,13 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    build_params,
+    ifcontext,
+)
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
scripts/qapi/common.py +0 -23
index 0ef38ea5fe0..9ab0685cc51 100644
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
     return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
scripts/qapi/events.py +2 -7
index 6b3afa14d72..f840a62ed92 100644
@@ -12,13 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import (
-    build_params,
-    c_enum_const,
-    c_name,
-    mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
 
scripts/qapi/gen.py +0 -0
index 1fed712b43b..f2e2746fea5 100644
@@ -2,9 +2,11 @@
 #
 # QAPI code generation

scripts/qapi/gen.py line 12

Quoted 10 lines
scripts/qapi/gen.py +3 -1
 #
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2019 Red Hat Inc.
 #
 # Authors:
+#  Anthony Liguori <[email protected]>
 #  Markus Armbruster <[email protected]>
 #  Marc-André Lureau <[email protected]>
 #

The code you move into this file is actually Red Hat's:

$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date'
Author: Markus Armbruster <[email protected]>
Date:   Wed Aug 15 21:37:36 2018 +0800
Author: Marc-André Lureau <[email protected]>
Date:   Thu Jun 1 16:41:41 2017 +0400
Author: Eric Blake <[email protected]>
Date:   Wed Jul 13 21:50:20 2016 -0600
Author: Eric Blake <[email protected]>
Date:   Wed Jul 13 21:50:19 2016 -0600
Author: Eric Blake <[email protected]>
Date:   Thu Mar 17 16:48:28 2016 -0600
Author: Markus Armbruster <[email protected]>
Date:   Wed Sep 16 13:06:20 2015 +0200

So the correct update is simply

   #
   # QAPI code generation
   #
Quoted 7 lines

-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
#
# Authors:
# Markus Armbruster <[email protected]>
# Marc-André Lureau <[email protected]> #

I am pretty sure build_params() below is a derivative work of generate_command_decl(), added in commit c17d9908a942 ("qapi: add qapi-commands.py code generator"), copyright IBM Corp. I'm not a lawyer, though.

Because figuring that out is complicated, when I move or copy code between files, I either:
(a) copy no copyright notices from the other file, or
(b) copy all copyright notices from the other files.

(The above is a description of what I do, but it is not a suggestion on how to proceed on this case.)

Quoted 51 lines
Quoted 51 lines
scripts/qapi/gen.py +26 -1
@@ -15,16 +17,18 @@
 import errno
 import os
 import re
+from typing import Optional
 
 from .common import (
     c_fname,
+    c_name,
     gen_endif,
     gen_if,
     guardend,
     guardstart,
     mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
 
 
 class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
     return out
 
 
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
+
+
 class QAPIGenCCode(QAPIGen):
 
     def __init__(self, fname):
Signature · 1 line
Eduardo
author
#59 lore raw

PATCH v5 18/36 qapi: establish mypy type-checking baseline

In reply to John Snow

Fix two very minor issues, and then establish a mypy type-checking baseline.

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

This is designed and tested for mypy 0.770 or greater.

---
 scripts/qapi/mypy.ini  | 60 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py |  3 ++-
 2 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/mypy.ini new file +60 -0
new file mode 100644
index 00000000000..00fac15dc6e
@@ -0,0 +1,60 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
scripts/qapi/schema.py +2 -1
index 74c6b96d391..483b4b68dff 100644
@@ -17,6 +17,7 @@
 from collections import OrderedDict
 import os
 import re
+from typing import Optional
 
 from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
Signature · 1 line
2.26.2
#60 lore raw

Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline

Quoted 1 line · John Snow

Fix two very minor issues,

What are the two issues? I'm asking because I can see only one.

Quoted 2 lines

and then establish a mypy type-checking
baseline.

Any particular reason this can't go before PATCH 15 "qapi/common.py: add type hint annotations".

Quoted 103 lines

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

This is designed and tested for mypy 0.770 or greater.

---
 scripts/qapi/mypy.ini  | 60 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py |  3 ++-
 2 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100644 scripts/qapi/mypy.ini
scripts/qapi/mypy.ini new file +60 -0
new file mode 100644
index 00000000000..00fac15dc6e
@@ -0,0 +1,60 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
scripts/qapi/schema.py +2 -1
index 74c6b96d391..483b4b68dff 100644
@@ -17,6 +17,7 @@
 from collections import OrderedDict
 import os
 import re
+from typing import Optional
 
 from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
author
#61 lore raw

Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline

Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow

Fix two very minor issues,

What are the two issues? I'm asking because I can see only one.

Bitrot! The other one got fixed elsewhere and rebase didn't conflict, so I didn't update the commit message.

Quoted 6 lines
Quoted 2 lines

and then establish a mypy type-checking
baseline.

Any particular reason this can't go before PATCH 15 "qapi/common.py: add type hint annotations".

Just the way the chips fell. common.py does not check until this very commit.

Adding a mypy config can be shuffled up earlier and earlier by determining which checks to ignore and adding them, but I'm not sure who that benefits.

Quoted 103 lines
Quoted 103 lines

Like pylint, this should be run from the folder above:

Quoted 1 line

mypy --config-file=qapi/mypy.ini qapi/

This is designed and tested for mypy 0.770 or greater.

---
  scripts/qapi/mypy.ini  | 60 ++++++++++++++++++++++++++++++++++++++++++
  scripts/qapi/schema.py |  3 ++-
  2 files changed, 62 insertions(+), 1 deletion(-)

create mode 100644 scripts/qapi/mypy.ini

scripts/qapi/mypy.ini new file +60 -0
new file mode 100644
index 00000000000..00fac15dc6e
@@ -0,0 +1,60 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
scripts/qapi/schema.py +2 -1
index 74c6b96d391..483b4b68dff 100644
@@ -17,6 +17,7 @@
  from collections import OrderedDict
  import os
  import re
+from typing import Optional
 
  from .common import POINTER_SUFFIX, c_name
  from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
  class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
      def __init__(self, name, info, doc, ifcond=None, features=None):
          assert name is None or isinstance(name, str)
#62 lore raw

Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline

Quoted 16 lines · John Snow
Quoted 5 lines · Markus Armbruster
Quoted 1 line · John Snow

Fix two very minor issues,

What are the two issues? I'm asking because I can see only one.

Bitrot! The other one got fixed elsewhere and rebase didn't conflict, so I didn't update the commit message.

Quoted 5 lines
Quoted 2 lines

and then establish a mypy type-checking
baseline.

Any particular reason this can't go before PATCH 15 "qapi/common.py: add type hint annotations".

Quoted 6 lines

Just the way the chips fell. common.py does not check until this very commit.

Adding a mypy config can be shuffled up earlier and earlier by determining which checks to ignore and adding them, but I'm not sure who that benefits.

I like symmetry, but it's okay as is. I asked to make sure I'm not missing something more subtle.

[...]

author
#63 lore raw

PATCH v5 19/36 qapi/events.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
 scripts/qapi/mypy.ini  |  5 -----
 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +35 -11
index f840a62ed92..57e0939e963 100644
@@ -12,19 +12,31 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import c_enum_const, c_name, mcgen
 from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+                           arg_type: QAPISchemaObjectType,
+                           boxed: bool) -> str:
     return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
         'c_name': c_name(name.lower()),
         'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+                        arg_type: QAPISchemaObjectType,
+                        boxed: bool) -> str:
     return mcgen('''
 
 %(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+                   arg_type: QAPISchemaObjectType,
+                   boxed: bool,
+                   event_enum_name: str,
+                   event_emit: str) -> str:
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members = []
+        self._event_enum_members: List[QAPISchemaEnumMember] = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         events = self._module_basename('qapi-events', name)
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('emit', ' * QAPI Events emission')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
@@ -189,7 +205,13 @@ def visit_end(self):
                              event_emit=self._event_emit_name,
                              event_enum=self._event_enum_name))
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(self,
+                    name: str,
+                    info: QAPISourceInfo,
+                    ifcond: List[str],
+                    features: List[QAPISchemaFeature],
+                    arg_type: QAPISchemaObjectType,
+                    boxed: bool) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_event_send_decl(name, arg_type, boxed))
             self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -200,7 +222,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed):
         self._event_enum_members.append(QAPISchemaEnumMember(name, None))
 
 
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+               output_dir: str,
+               prefix: str) -> None:
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
scripts/qapi/mypy.ini +0 -5
index 00fac15dc6e..5df11e53fd1 100644
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.events]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.expr]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
Signature · 1 line
2.26.2
#64 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 92 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
 scripts/qapi/mypy.ini  |  5 -----
 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +24 -8
index f840a62ed92..57e0939e963 100644
@@ -12,19 +12,31 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import c_enum_const, c_name, mcgen
 from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+                           arg_type: QAPISchemaObjectType,
+                           boxed: bool) -> str:
     return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
         'c_name': c_name(name.lower()),
         'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+                        arg_type: QAPISchemaObjectType,
+                        boxed: bool) -> str:
     return mcgen('''
 
 %(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+                   arg_type: QAPISchemaObjectType,
+                   boxed: bool,
+                   event_enum_name: str,
+                   event_emit: str) -> str:
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members = []
+        self._event_enum_members: List[QAPISchemaEnumMember] = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         events = self._module_basename('qapi-events', name)
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)

scripts/qapi/events.py def _begin_user_module(self, name): · line 187

Quoted 5 lines
scripts/qapi/events.py +1 -1
@@ -168,7 +184,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:

Ignorant question: what's the difference between -> None (like here) and nothing (like __init__() above?

Quoted 3 lines
scripts/qapi/events.py +0 -0
         self._add_system_module('emit', ' * QAPI Events emission')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"

[...]

#65 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 18 lines · Markus Armbruster
Quoted 16 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
 scripts/qapi/mypy.ini  |  5 -----
 2 files changed, 35 insertions(+), 16 deletions(-)
scripts/qapi/events.py +0 -0
index f840a62ed92..57e0939e963 100644
@@ -12,19 +12,31 @@

[...]

Quoted 23 lines
Quoted 23 lines
scripts/qapi/events.py +3 -3
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members = []
+        self._event_enum_members: List[QAPISchemaEnumMember] = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         events = self._module_basename('qapi-events', name)
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
Quoted 5 lines

scripts/qapi/events.py def _begin_user_module(self, name): · line 187

Quoted 2 lines
scripts/qapi/events.py +1 -1
-    def visit_end(self):
+    def visit_end(self) -> None:

Ignorant question: what's the difference between -> None (like here) and nothing (like __init__() above?

Looks like commit message of PATCH 24 has an answer.

Copy to all the commits that omit -> None similarly?

Quoted 4 lines
Quoted 3 lines
scripts/qapi/events.py +0 -0
         self._add_system_module('emit', ' * QAPI Events emission')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"

[...]

author
#66 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 3 lines · Markus Armbruster

Looks like commit message of PATCH 24 has an answer.

Copy to all the commits that omit -> None similarly?

Probably not needed.

It's covered by the class basics section in the mypy manual; https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

and if you should happen to omit annotations for __init__ entirely as a novice, you will be treated to messages such as these:

qapi/source.py:18: error: Function is missing a return type annotation qapi/source.py:18: note: Use "-> None" if function does not return a value Found 1 error in 1 file (checked 14 source files)

Pretty good error!

There's no error if you DO explicitly add a -> None from __init__, but at worst it's just extraneous (but correct) information.

I could add a note to the style guide that I prefer omitting the return from __init__. I like omitting as much as I possibly can.

(You'll notice I don't always type every local, either -- when local inference is accurate, I leave it alone.)

--js

#67 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 12 lines · John Snow
Quoted 2 lines · Markus Armbruster

Looks like commit message of PATCH 24 has an answer. Copy to all the commits that omit -> None similarly?

Probably not needed.

It's covered by the class basics section in the mypy manual;
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

and if you should happen to omit annotations for __init__ entirely as a novice, you will be treated to messages such as these:

Quoted 8 lines

qapi/source.py:18: error: Function is missing a return type annotation qapi/source.py:18: note: Use "-> None" if function does not return a value Found 1 error in 1 file (checked 14 source files)

Pretty good error!

There's no error if you DO explicitly add a -> None from __init__, but at worst it's just extraneous (but correct) information.

Let me apply the zero-one-infinity rule:

* Zero: explain it in none of the commit messages, i.e. dumb down PATCH 24.

* One: explain it in one. Do it in the first one, please (here, I think).

* Infinity: explain it in every one.

Up to you!

Quoted 5 lines

I could add a note to the style guide that I prefer omitting the return from __init__. I like omitting as much as I possibly can.

(You'll notice I don't always type every local, either -- when local inference is accurate, I leave it alone.)

Type inference can save us from repeating the obvious over and over, and that's lovely.

author
#68 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 28 lines · Markus Armbruster
Quoted 12 lines · John Snow
Quoted 2 lines · Markus Armbruster

Looks like commit message of PATCH 24 has an answer. Copy to all the commits that omit -> None similarly?

Probably not needed.

It's covered by the class basics section in the mypy manual;
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

and if you should happen to omit annotations for __init__ entirely as a novice, you will be treated to messages such as these:

Quoted 8 lines

qapi/source.py:18: error: Function is missing a return type annotation qapi/source.py:18: note: Use "-> None" if function does not return a value Found 1 error in 1 file (checked 14 source files)

Pretty good error!

There's no error if you DO explicitly add a -> None from __init__, but at worst it's just extraneous (but correct) information.

Let me apply the zero-one-infinity rule:

* Zero: explain it in none of the commit messages, i.e. dumb down PATCH 24.

Quoted 7 lines

* One: explain it in one. Do it in the first one, please (here, I think).

* Infinity: explain it in every one.

Up to you!

I'm just bad at predicting which things people will want explained. I know people don't read the cover letters already, so I'd rather go for less instead of more.

I think you and Cleber each noticed a different angle of this: Cleber noticed the first time I *did* annotate __init__'s return and you noticed the first time I *didn't*.

I'll just add it here too, but I have doubts it will be useful reference once it's merged. I guess it doesn't hurt to add it either, I just find it difficult to predict what reviewers will want.

Quoted 9 lines
Quoted 5 lines

I could add a note to the style guide that I prefer omitting the return from __init__. I like omitting as much as I possibly can.

(You'll notice I don't always type every local, either -- when local inference is accurate, I leave it alone.)

Type inference can save us from repeating the obvious over and over, and that's lovely.

#69 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

John Snow <[email protected]> writes:

[...]

Quoted 1 line

I just find it difficult to predict what reviewers will want.

Capricious bunch, those reviewers.

[...]

author
#70 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

In reply to Markus Armbruster
Quoted 2 lines · Markus Armbruster

Ignorant question: what's the difference between -> None (like here) and nothing (like __init__() above?

This came up in Cleber's review, too.

Mypy supports a gradual typing paradigm; it is designed to facilitate what we are doing here: the gradual adding of types until we are able to enforce static typing everywhere.

To that end, mypy uses a simple heuristic to determine if a function is "typed" or "untyped": did you use any type annotations for it?

Meanwhile, __init__ never returns anything. You do not need to annotate its return type. mypy knows what the return type is and must be. In the case of __init__ with no parameters, it is both untyped and strictly typed!

Annotating the return type for no-parameter init methods convinces mypy that this is a strictly typed method. It doesn't do this automatically, because doing so might enable more checks than you were ready for simply because mypy was able to accurately surmise the typing of just __init__.

So it's just a little flip switch to enable strict typing, really.

--js

https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

#71 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 24 lines · John Snow
Quoted 2 lines · Markus Armbruster

Ignorant question: what's the difference between -> None (like here) and nothing (like __init__() above?

This came up in Cleber's review, too.

Mypy supports a gradual typing paradigm; it is designed to facilitate what we are doing here: the gradual adding of types until we are able to enforce static typing everywhere.

To that end, mypy uses a simple heuristic to determine if a function is "typed" or "untyped": did you use any type annotations for it?

Meanwhile, __init__ never returns anything. You do not need to annotate its return type. mypy knows what the return type is and must be. In the case of __init__ with no parameters, it is both untyped and strictly typed!

Annotating the return type for no-parameter init methods convinces mypy that this is a strictly typed method. It doesn't do this automatically, because doing so might enable more checks than you were ready for simply because mypy was able to accurately surmise the typing of just __init__.

Quoted 5 lines

So it's just a little flip switch to enable strict typing, really.

--js

https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

I now understand we can omit -> None, except when it's the only type hint, because omitting it then would make it untyped, which is not what we want.

Is this just for __init__(), or is it for any function that doesn't return anything?

author
#72 lore raw

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

Quoted 30 lines · Markus Armbruster
Quoted 28 lines · John Snow
Quoted 2 lines · Markus Armbruster

Ignorant question: what's the difference between -> None (like here) and nothing (like __init__() above?

This came up in Cleber's review, too.

Mypy supports a gradual typing paradigm; it is designed to facilitate what we are doing here: the gradual adding of types until we are able to enforce static typing everywhere.

To that end, mypy uses a simple heuristic to determine if a function is "typed" or "untyped": did you use any type annotations for it?

Meanwhile, __init__ never returns anything. You do not need to annotate its return type. mypy knows what the return type is and must be. In the case of __init__ with no parameters, it is both untyped and strictly typed!

Annotating the return type for no-parameter init methods convinces mypy that this is a strictly typed method. It doesn't do this automatically, because doing so might enable more checks than you were ready for simply because mypy was able to accurately surmise the typing of just __init__.

So it's just a little flip switch to enable strict typing, really.

--js

Quoted 9 lines
Quoted 1 line

https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

I now understand we can omit -> None, except when it's the only type hint, because omitting it then would make it untyped, which is not what we want.

Is this just for __init__(), or is it for any function that doesn't return anything?

*just* __init__. Some other dunders have types that could be assumed, but aren't. I asked about this for some other well-known but hard-to-type dunders like __exit__ on the Typing-SIG list. Not a strong response yet, I might push again later.

--js

author
#73 lore raw

PATCH v5 20/36 qapi/events.py: Move comments into docstrings

In reply to John Snow

Clarify them while we're here.

---
 scripts/qapi/events.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/qapi/events.py +5 -1
index 57e0939e963..599f3d1f564 100644
@@ -44,8 +44,12 @@ def gen_event_send_decl(name: str,
                  proto=build_event_send_proto(name, arg_type, boxed))
 
 
-# Declare and initialize an object 'qapi' using parameters from build_params()
 def gen_param_var(typ: QAPISchemaObjectType) -> str:
+    """
+    Generate a struct variable holding the event parameters.
+
+    Initialize it with the function arguments defined in `gen_event_send`.
+    """
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
Signature · 1 line
2.26.2
author
#74 lore raw

PATCH v5 21/36 qapi/commands.py: Don't re-bind to variable of different type

In reply to John Snow

Mypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.

---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +1 -3
index f67393f8713..586774a23c7 100644
@@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     if not options:
         options = ['QCO_NO_OPTIONS']
 
-    options = " | ".join(options)
-
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
                          qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
-                opts=options)
+                opts=" | ".join(options))
     return ret
 
 
Signature · 1 line
2.26.2
Reviewed-by on v5 21/36
#75 lore raw

Re: [PATCH v5 21/36] qapi/commands.py: Don't re-bind to variable of different type

Quoted 18 lines · John Snow

Mypy isn't a fan of rebinding a variable with a new data type. It's easy enough to avoid.

---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/qapi/commands.py +0 -0
index f67393f8713..586774a23c7 100644
@@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     if not options:
         options = ['QCO_NO_OPTIONS']
 

scripts/qapi/commands.py def gen_register_command(name, success_response, allow_oob, allow_preconfig): · line 204

Quoted 10 lines
scripts/qapi/commands.py +1 -3
-    options = " | ".join(options)
-
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
                          qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
-                opts=options)
+                opts=" | ".join(options))
     return ret

Definitely neater.

author
#76 lore raw

PATCH v5 22/36 qapi/commands.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/commands.py | 71 ++++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 17 deletions(-)
scripts/qapi/commands.py +54 -17
index 586774a23c7..24f347a3f12 100644
@@ -13,16 +13,34 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import (
+    Dict,
+    List,
+    Optional,
+    Set,
+)
+
 from .common import c_name, mcgen
 from .gen import (
+    QAPIGenC,
     QAPIGenCCode,
     QAPISchemaModularCVisitor,
     build_params,
     ifcontext,
 )
+from .schema import (
+    QAPISchema,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+                     arg_type: Optional[QAPISchemaObjectType],
+                     boxed: bool,
+                     ret_type: Optional[QAPISchemaType]) -> str:
     return mcgen('''
 %(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
@@ -31,7 +49,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
                  params=build_params(arg_type, boxed, 'Error **errp'))
 
 
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+             arg_type: Optional[QAPISchemaObjectType],
+             boxed: bool,
+             ret_type: Optional[QAPISchemaType]) -> str:
     ret = ''
 
     argstr = ''
@@ -67,7 +88,7 @@ def gen_call(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
     return mcgen('''
 
 static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -88,19 +109,22 @@ def gen_marshal_output(ret_type):
                  c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
     return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
             % c_name(name))
 
 
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
     return mcgen('''
 %(proto)s;
 ''',
                  proto=build_marshal_proto(name))
 
 
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+                arg_type: Optional[QAPISchemaObjectType],
+                boxed: bool,
+                ret_type: Optional[QAPISchemaType]) -> str:
     have_args = boxed or (arg_type and not arg_type.is_empty())
 
     ret = mcgen('''
@@ -182,7 +206,10 @@ def gen_marshal(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_register_command(name, success_response, allow_oob, allow_preconfig):
+def gen_register_command(name: str,
+                         success_response: bool,
+                         allow_oob: bool,
+                         allow_preconfig: bool) -> str:
     options = []
 
     if not success_response:
@@ -204,7 +231,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     return ret
 
 
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -221,15 +248,14 @@ def gen_registry(registry, prefix):
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', None, __doc__)
         self._regy = QAPIGenCCode(None)
-        self._visited_ret_types = {}
+        self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         self._visited_ret_types[self._genc] = set()
         commands = self._module_basename('qapi-commands', name)
         types = self._module_basename('qapi-types', name)
@@ -253,7 +279,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('init', ' * QAPI Commands initialization')
         self._genh.add(mcgen('''
 #include "qapi/qmp/dispatch.h"
@@ -269,9 +295,18 @@ def visit_end(self):
                                       prefix=self._prefix))
         self._genc.add(gen_registry(self._regy.get_content(), self._prefix))
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig):
+    def visit_command(self,
+                      name: str,
+                      info: QAPISourceInfo,
+                      ifcond: List[str],
+                      features: List[QAPISchemaFeature],
+                      arg_type: Optional[QAPISchemaObjectType],
+                      ret_type: Optional[QAPISchemaType],
+                      gen: bool,
+                      success_response: bool,
+                      boxed: bool,
+                      allow_oob: bool,
+                      allow_preconfig: bool) -> None:
         if not gen:
             return
         # FIXME: If T is a user-defined type, the user is responsible
@@ -292,7 +327,9 @@ def visit_command(self, name, info, ifcond, features,
                                                 allow_oob, allow_preconfig))
 
 
-def gen_commands(schema, output_dir, prefix):
+def gen_commands(schema: QAPISchema,
+                 output_dir: str,
+                 prefix: str) -> None:
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
Signature · 1 line
2.26.2
author
#77 lore raw

PATCH v5 23/36 qapi/commands.py: enable checking with mypy

In reply to John Snow
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 5df11e53fd1..8ab9ac52cc4 100644
@@ -4,11 +4,6 @@ strict_optional = False
 disallow_untyped_calls = False
 python_version = 3.6
 
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.error]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
Signature · 1 line
2.26.2
#78 lore raw

Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy

Quoted 13 lines · John Snow
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -0
index 5df11e53fd1..8ab9ac52cc4 100644
@@ -4,11 +4,6 @@ strict_optional = False
 disallow_untyped_calls = False

scripts/qapi/mypy.ini strict_optional = False · line 9

Quoted 10 lines
scripts/qapi/mypy.ini +0 -5
 python_version = 3.6
 
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.error]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False

The equivalent change for events.py you squashed into the commit adding type hints. Any particular reason for not doing the same here?

author
#79 lore raw

Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy

Quoted 17 lines · Markus Armbruster
Quoted 15 lines · John Snow
---
  scripts/qapi/mypy.ini | 5 -----
  1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -0
index 5df11e53fd1..8ab9ac52cc4 100644
@@ -4,11 +4,6 @@ strict_optional = False
  disallow_untyped_calls = False
  python_version = 3.6
 
Quoted 12 lines

scripts/qapi/mypy.ini strict_optional = False · line 9

Quoted 8 lines
scripts/qapi/mypy.ini +0 -5
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
  [mypy-qapi.error]
  disallow_untyped_defs = False
  disallow_incomplete_defs = False

The equivalent change for events.py you squashed into the commit adding type hints. Any particular reason for not doing the same here?

Just making my life easier for re-arranging commits and rebasing. By separating them out whenever I had > 1 fix patch prior, I was able to freely re-arrange and re-order the prior fix patches.

They can be squashed on commit if desired, but for my own sake and inability to predict review comments, they are separate.

--js

#80 lore raw

Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy

Quoted 30 lines · John Snow
Quoted 19 lines · Markus Armbruster
Quoted 17 lines · John Snow
---
  scripts/qapi/mypy.ini | 5 -----
  1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -2
index 5df11e53fd1..8ab9ac52cc4 100644
@@ -4,11 +4,6 @@ strict_optional = False
  disallow_untyped_calls = False
  python_version = 3.6
  -[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
Quoted 9 lines

scripts/qapi/mypy.ini strict_optional = False · line 9

Quoted 5 lines
scripts/qapi/mypy.ini +0 -2
-check_untyped_defs = False
-
  [mypy-qapi.error]
  disallow_untyped_defs = False
  disallow_incomplete_defs = False

The equivalent change for events.py you squashed into the commit adding
type hints. Any particular reason for not doing the same here?

Quoted 6 lines

Just making my life easier for re-arranging commits and rebasing. By separating them out whenever I had > 1 fix patch prior, I was able to freely re-arrange and re-order the prior fix patches.

They can be squashed on commit if desired, but for my own sake and inability to predict review comments, they are separate.

I only asked because some are separate, and some are not.

Squashing on commit is an easy way toward consistency here.

author
#81 lore raw

PATCH v5 24/36 qapi/source.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.

---
 scripts/qapi/mypy.ini  |  5 -----
 scripts/qapi/source.py | 31 ++++++++++++++++++-------------
 2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 8ab9ac52cc4..1b8555dfa39 100644
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
scripts/qapi/source.py +18 -13
index e97b9a8e15e..1cc6a5b82dc 100644
@@ -11,37 +11,42 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
         # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []
         # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
         self.fname = fname
         self.line = line
         self.parent = parent
-        self.pragma = parent.pragma if parent else QAPISchemaPragma()
-        self.defn_meta = None
-        self.defn_name = None
+        self.pragma: QAPISchemaPragma = (
+            parent.pragma if parent else QAPISchemaPragma()
+        )
+        self.defn_meta: Optional[str] = None
+        self.defn_name: Optional[str] = None
 
-    def set_defn(self, meta, name):
+    def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self):
+    def next_line(self: T) -> T:
         info = copy.copy(self)
         info.line += 1
         return info
 
-    def loc(self):
+    def loc(self) -> str:
         if self.fname is None:
             return sys.argv[0]
         ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
             ret += ':%d' % self.line
         return ret
 
-    def in_defn(self):
+    def in_defn(self) -> str:
         if self.defn_name:
             return "%s: In %s '%s':\n" % (self.fname,
                                           self.defn_meta, self.defn_name)
         return ''
 
-    def include_path(self):
+    def include_path(self) -> str:
         ret = ''
         parent = self.parent
         while parent:
@@ -63,5 +68,5 @@ def include_path(self):
             parent = parent.parent
         return ret
 
-    def __str__(self):
+    def __str__(self) -> str:
         return self.include_path() + self.in_defn() + self.loc()
Signature · 1 line
2.26.2
#82 lore raw

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

Quoted 54 lines · John Snow

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.

---
 scripts/qapi/mypy.ini  |  5 -----
 scripts/qapi/source.py | 31 ++++++++++++++++++-------------
 2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 8ab9ac52cc4..1b8555dfa39 100644
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
scripts/qapi/source.py +3 -2
index e97b9a8e15e..1cc6a5b82dc 100644
@@ -11,37 +11,42 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
         # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []

scripts/qapi/source.py line 30

Quoted 10 lines
scripts/qapi/source.py +4 -2
         # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):

More ignorant questions (I'm abusing the review process to learn Python type hints)...

Why do you need to annotate self here, but not elsewhere?

Why do you use T instead of QAPISourceInfo?

scripts/qapi/source.py line 36

Quoted 9 lines
scripts/qapi/source.py +3 -3
         self.fname = fname
         self.line = line
         self.parent = parent
-        self.pragma = parent.pragma if parent else QAPISchemaPragma()
-        self.defn_meta = None
-        self.defn_name = None
+        self.pragma: QAPISchemaPragma = (
+            parent.pragma if parent else QAPISchemaPragma()
+        )

Type inference fail?

Quoted 42 lines
scripts/qapi/source.py +8 -6
+        self.defn_meta: Optional[str] = None
+        self.defn_name: Optional[str] = None
 
-    def set_defn(self, meta, name):
+    def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self):
+    def next_line(self: T) -> T:
         info = copy.copy(self)
         info.line += 1
         return info
 
-    def loc(self):
+    def loc(self) -> str:
         if self.fname is None:
             return sys.argv[0]
         ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
             ret += ':%d' % self.line
         return ret
 
-    def in_defn(self):
+    def in_defn(self) -> str:
         if self.defn_name:
             return "%s: In %s '%s':\n" % (self.fname,
                                           self.defn_meta, self.defn_name)
         return ''
 
-    def include_path(self):
+    def include_path(self) -> str:
         ret = ''
         parent = self.parent
         while parent:
@@ -63,5 +68,5 @@ def include_path(self):
             parent = parent.parent
         return ret
 
-    def __str__(self):
+    def __str__(self) -> str:
         return self.include_path() + self.in_defn() + self.loc()
author
#83 lore raw

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

Quoted 70 lines · Markus Armbruster
Quoted 54 lines · John Snow

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.

---
  scripts/qapi/mypy.ini  |  5 -----
  scripts/qapi/source.py | 31 ++++++++++++++++++-------------
  2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 8ab9ac52cc4..1b8555dfa39 100644
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
  disallow_incomplete_defs = False
  check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
  [mypy-qapi.types]
  disallow_untyped_defs = False
  disallow_incomplete_defs = False
scripts/qapi/source.py +3 -2
index e97b9a8e15e..1cc6a5b82dc 100644
@@ -11,37 +11,42 @@
 
  import copy
  import sys
+from typing import List, Optional, TypeVar
 
 
  class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
          # Are documentation comments required?
          self.doc_required = False
          # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []

scripts/qapi/source.py line 30

Quoted 10 lines
scripts/qapi/source.py +4 -2
          # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
 
  class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):

More ignorant questions (I'm abusing the review process to learn Python type hints)...

Quoted 2 lines

Why do you need to annotate self here, but not elsewhere?

This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.

This is a pattern that allows for subclassing. I am stating that this __init__ method takes a parent of the same type as itself, whatever that happens to actually be.

T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.

(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)

Quoted 12 lines

Why do you use T instead of QAPISourceInfo?

scripts/qapi/source.py line 36

Quoted 9 lines
scripts/qapi/source.py +3 -3
          self.fname = fname
          self.line = line
          self.parent = parent
-        self.pragma = parent.pragma if parent else QAPISchemaPragma()
-        self.defn_meta = None
-        self.defn_name = None
+        self.pragma: QAPISchemaPragma = (
+            parent.pragma if parent else QAPISchemaPragma()
+        )
Quoted 2 lines

Type inference fail?

Yes:

qapi/source.py:34: error: Cannot determine type of 'pragma' Found 1 error in 1 file (checked 14 source files)

Quoted 42 lines
Quoted 42 lines
scripts/qapi/source.py +8 -6
+        self.defn_meta: Optional[str] = None
+        self.defn_name: Optional[str] = None
 
-    def set_defn(self, meta, name):
+    def set_defn(self, meta: str, name: str) -> None:
          self.defn_meta = meta
          self.defn_name = name
 
-    def next_line(self):
+    def next_line(self: T) -> T:
          info = copy.copy(self)
          info.line += 1
          return info
 
-    def loc(self):
+    def loc(self) -> str:
          if self.fname is None:
              return sys.argv[0]
          ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
              ret += ':%d' % self.line
          return ret
 
-    def in_defn(self):
+    def in_defn(self) -> str:
          if self.defn_name:
              return "%s: In %s '%s':\n" % (self.fname,
                                            self.defn_meta, self.defn_name)
          return ''
 
-    def include_path(self):
+    def include_path(self) -> str:
          ret = ''
          parent = self.parent
          while parent:
@@ -63,5 +68,5 @@ def include_path(self):
              parent = parent.parent
          return ret
 
-    def __str__(self):
+    def __str__(self) -> str:
          return self.include_path() + self.in_defn() + self.loc()
#84 lore raw

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

Quoted 82 lines · John Snow
Quoted 66 lines · Markus Armbruster
Quoted 50 lines · John Snow

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.

---
  scripts/qapi/mypy.ini  |  5 -----
  scripts/qapi/source.py | 31 ++++++++++++++++++-------------
  2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -4
index 8ab9ac52cc4..1b8555dfa39 100644
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
  disallow_incomplete_defs = False
  check_untyped_defs = False
  -[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
  [mypy-qapi.types]
  disallow_untyped_defs = False
  disallow_incomplete_defs = False
scripts/qapi/source.py +2 -2
index e97b9a8e15e..1cc6a5b82dc 100644
@@ -11,37 +11,42 @@
    import copy
  import sys
+from typing import List, Optional, TypeVar
 
  class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
          # Are documentation comments required?
          self.doc_required = False
          # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []

scripts/qapi/source.py line 27

Quoted 10 lines
scripts/qapi/source.py +5 -2
+        self.returns_whitelist: List[str] = []
          # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
  class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):

More ignorant questions (I'm abusing the review process to learn Python type hints)...

Quoted 1 line

Why do you need to annotate self here, but not elsewhere?

This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.

This is a pattern that allows for subclassing. I am stating that this __init__ method takes a parent of the same type as itself, whatever that happens to actually be.

T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.

Quoted 5 lines

(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)

Say I define class QSISub as a direct subclass of QAPISourceInfo, and let it inherit __init__(). What's the type of QSISub.__init__()'s parameter @parent?

According to my reading of your explanation, it's QSISub. Correct?

If we used QAPISourceInfo instead of T for @parent, then it would be QAPISourceInfo. Correct?

Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really needs to be a QSISub. We can't know when we write QAPISourceInfo. But we don't *have* to get this exactly right for all future subclasses, because I can always override __init__() when inheritance doesn't give me the __init__() I want. Correct?

Say I override __init__(), and have it call super().__init__(). I have to pass it a QAPISourceInfo @parent. A QSISub will do (it's a subtype). Correct?

One more: is bound='QAPISourceInfo' strictly needed?

Quoted 1 line
Quoted 1 line

Why do you use T instead of QAPISourceInfo?

[...]

author
#85 lore raw

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

Quoted 90 lines · Markus Armbruster
Quoted 82 lines · John Snow
Quoted 66 lines · Markus Armbruster
Quoted 50 lines · John Snow

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.

---
   scripts/qapi/mypy.ini  |  5 -----
   scripts/qapi/source.py | 31 ++++++++++++++++++-------------
   2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -4
index 8ab9ac52cc4..1b8555dfa39 100644
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
   disallow_incomplete_defs = False
   check_untyped_defs = False
   -[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
   [mypy-qapi.types]
   disallow_untyped_defs = False
   disallow_incomplete_defs = False
scripts/qapi/source.py +2 -2
index e97b9a8e15e..1cc6a5b82dc 100644
@@ -11,37 +11,42 @@
     import copy
   import sys
+from typing import List, Optional, TypeVar
 
   class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
           # Are documentation comments required?
           self.doc_required = False
           # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []

scripts/qapi/source.py line 27

Quoted 10 lines
scripts/qapi/source.py +5 -2
+        self.returns_whitelist: List[str] = []
           # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
   class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):

More ignorant questions (I'm abusing the review process to learn Python type hints)...

Quoted 1 line

Why do you need to annotate self here, but not elsewhere?

This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.

This is a pattern that allows for subclassing. I am stating that this __init__ method takes a parent of the same type as itself, whatever that happens to actually be.

T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.

Quoted 5 lines

(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)

Quoted 6 lines

Say I define class QSISub as a direct subclass of QAPISourceInfo, and let it inherit __init__(). What's the type of QSISub.__init__()'s parameter @parent?

According to my reading of your explanation, it's QSISub. Correct?

That's right.

(I'm realizing that this is maybe not a constraint that we should even anticipate here, because maybe we don't wish to say that the parent should always be of the same type. but hey, it led to a good mypy lesson.

I'm going to edit it to do the simpler thing for now and leave well enough alone. There's another chance to see an interesting pattern of TypeVars in the error series in part 4 that I think is actually more explicitly appropriate.)

Quoted 3 lines

If we used QAPISourceInfo instead of T for @parent, then it would be QAPISourceInfo. Correct?

Yup!

Here's a little sample program that shows what this kind of typing does:

```
from typing import TypeVar, Optional

class Example:
    T = TypeVar('T', bound='Example')
    def __init__(self: T, parent: Optional[T] = None):
        self.parent = parent

class SubExample(Example):
    pass


x = Example()
y = Example(x)
z = SubExample()
a = Example(x)            # OK
b = Example(z)            # OK
c = SubExample(x)         # BZZZT
d = SubExample(z)         # OK
```

If you check this with mypy, you'll get this error:

```
test.py:17: error: Argument 1 to "SubExample" has incompatible type "Example"; expected "Optional[SubExample]"
Found 1 error in 1 file (checked 1 source file)
```
Quoted 6 lines

Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really needs to be a QSISub. We can't know when we write QAPISourceInfo. But we don't *have* to get this exactly right for all future subclasses, because I can always override __init__() when inheritance doesn't give me the __init__() I want. Correct?

You could, but I suggested on IRC the other day that I am not fully comfortable with the LSP rules that mypy (sometimes?) enforces. I tend not to want to override init with narrower types if I can avoid it, but it is true that we do this quite a lot in the codebase already.

(I believe I have seen mypy throw errors about this on occasion, but I can't pinpoint the exact circumstances in which it does. It's a point of confusion for me.)

Quoted 6 lines

Say I override __init__(), and have it call super().__init__(). I have to pass it a QAPISourceInfo @parent. A QSISub will do (it's a subtype). Correct?

One more: is bound='QAPISourceInfo' strictly needed?

I'm not sure if bound is strictly required or not. mypy docs just use it outright and don't mention why:

https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self

Generally, this constrains a TypeVar to some upper bound, see --

https://mypy.readthedocs.io/en/stable/generics.html#type-variables-with-upper-bounds

-- but in this case, we're only using that TypeVar for an init method that only exists for this type and below, so it might be redundant.

I modified my example program to remove the bound and it appears to fail in the same exact way, so it might be pointless in this case. It might have a stronger use if you want to re-use the 'T' typevar elsewhere where it isn't implicitly bound by the 'self' argument. Maybe it has implications for multi-inheritance too, I've not tested it to that level.

Thanks for the good question!

--js

author
#86 lore raw

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

Quoted 164 lines · John Snow
Quoted 89 lines · Markus Armbruster
Quoted 81 lines · John Snow
Quoted 65 lines · Markus Armbruster
Quoted 51 lines · John Snow

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no parameters to document a return type of None to be considered fully typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the return annotation whenever possible.

---
   scripts/qapi/mypy.ini  |  5 ----- scripts/qapi/source.py | 31 ++++++++++++++++++-------------

   2 files changed, 18 insertions(+), 18 deletions(-)
scripts/qapi/mypy.ini +0 -0
index 8ab9ac52cc4..1b8555dfa39 100644
@@ -34,11 +34,6 @@ disallow_untyped_defs = False

   disallow_incomplete_defs = False
   check_untyped_defs = False
   -[mypy-qapi.source]

scripts/qapi/mypy.ini +0 -4
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-

   [mypy-qapi.types]
   disallow_untyped_defs = False
   disallow_incomplete_defs = False

scripts/qapi/source.py +0 -0
index e97b9a8e15e..1cc6a5b82dc 100644
@@ -11,37 +11,42 @@

     import copy
   import sys
+from typing import List, Optional, TypeVar
   class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
           # Are documentation comments required? self.doc_required = False
           # Whitelist of commands allowed to return a non-dictionary -        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []
           # Whitelist of entities allowed to violate case conventions

Quoted 8 lines

-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = [] class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent:
Optional[T]):

More ignorant questions (I'm abusing the review process to learn Python type hints)...

Quoted 1 line

Why do you need to annotate self here, but not elsewhere?

This is admittedly me being a little extra, but I thought it was a good way to show a pattern for people who maybe hadn't been exposed to it yet.

This is a pattern that allows for subclassing. I am stating that this __init__ method takes a parent of the same type as itself, whatever that happens to actually be.

T is a TypeVar that we can use for Generics. By declaring the type of self as that TypeVar, we bind T to self's type. When we annotate the parent as a T, we are enforcing that the parent we receive is of the same type as ourselves.

Quoted 5 lines

(Yep, we don't actually subclass QAPISourceInfo and I don't have plans to. It felt slightly icky to hard-code the class type name, though. I try to avoid that whenever I can. I'm not sure I would go so far as to call it a code smell or an antipattern, but it's something I tend to avoid anyway.)

Quoted 6 lines

Say I define class QSISub as a direct subclass of QAPISourceInfo, and let it inherit __init__().  What's the type of QSISub.__init__()'s parameter @parent?

According to my reading of your explanation, it's QSISub.  Correct?

That's right.

(I'm realizing that this is maybe not a constraint that we should even anticipate here, because maybe we don't wish to say that the parent should always be of the same type. but hey, it led to a good mypy lesson.

I'm going to edit it to do the simpler thing for now and leave well enough alone. There's another chance to see an interesting pattern of TypeVars in the error series in part 4 that I think is actually more explicitly appropriate.)

Quoted 3 lines

If we used QAPISourceInfo instead of T for @parent, then it would be QAPISourceInfo.  Correct?

Yup!

Here's a little sample program that shows what this kind of typing does:

```
from typing import TypeVar, Optional

class Example:
     T = TypeVar('T', bound='Example')
     def __init__(self: T, parent: Optional[T] = None):
         self.parent = parent

class SubExample(Example):
     pass


x = Example()
y = Example(x)
z = SubExample()
a = Example(x)            # OK
b = Example(z)            # OK
c = SubExample(x)         # BZZZT
d = SubExample(z)         # OK
```

If you check this with mypy, you'll get this error:

```
test.py:17: error: Argument 1 to "SubExample" has incompatible type "Example"; expected "Optional[SubExample]"
Found 1 error in 1 file (checked 1 source file)
```
Quoted 6 lines

Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really needs to be a QSISub.  We can't know when we write QAPISourceInfo.  But we don't *have* to get this exactly right for all future subclasses, because I can always override __init__() when inheritance doesn't give me the __init__() I want.  Correct?

You could, but I suggested on IRC the other day that I am not fully comfortable with the LSP rules that mypy (sometimes?) enforces. I tend not to want to override init with narrower types if I can avoid it, but it is true that we do this quite a lot in the codebase already.

(I believe I have seen mypy throw errors about this on occasion, but I can't pinpoint the exact circumstances in which it does. It's a point of confusion for me.)

Quoted 6 lines

Say I override __init__(), and have it call super().__init__().  I have to pass it a QAPISourceInfo @parent.  A QSISub will do (it's a subtype). Correct?

One more: is bound='QAPISourceInfo' strictly needed?

I'm not sure if bound is strictly required or not. mypy docs just use it outright and don't mention why:

https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self

Generally, this constrains a TypeVar to some upper bound, see --

Quoted 8 lines

https://mypy.readthedocs.io/en/stable/generics.html#type-variables-with-upper-bounds

-- but in this case, we're only using that TypeVar for an init method that only exists for this type and below, so it might be redundant.

I modified my example program to remove the bound and it appears to fail in the same exact way, so it might be pointless in this case. It might have a stronger use if you want to re-use the 'T' typevar elsewhere where it isn't implicitly bound by the 'self' argument. Maybe it has implications for multi-inheritance too, I've not tested it to that level.

SPOKE TOO SOON, ADDENDUM:

Adding bound= is still necessary to allow type inference to assume the minimal base form. It's not needed for my toy example because I don't do anything *with* the variable.

In this case, though, the next_line method needs help:

    def next_line(self: T) -> T:
	info = copy.copy(self)
        info.line += 1
        return info

Now, when we return a copy of ourselves, we are going to return something that is the same type as ourselves. If we remove the bound from the TypeVar here, "info.line" will become an error because mypy cannot infer the minimum form.

(It doesn't seem smart enough to notice that the class it is defined in does not inherit from anywhere, so it MUST be at least a QAPISourceInfo.)

So I am going to:

(1) Make __init__'s typing less dynamic, because it's not really needed.
(2) Leave the TypeVar in for the benefit of next_line().

--js

author
#87 lore raw

PATCH v5 25/36 qapi/source.py: delint with pylint

In reply to John Snow

Shush an error and leave a hint for future cleanups when we're allowed to use Python 3.7+.

---
 scripts/qapi/pylintrc  | 1 -
 scripts/qapi/source.py | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/qapi/pylintrc +0 -1
index 507f15537ab..d840b150313 100644
@@ -7,7 +7,6 @@ ignore-patterns=error.py,
                 gen.py,
                 parser.py,
                 schema.py,
-                source.py,
                 types.py,
                 visit.py,
 
scripts/qapi/source.py +3 -0
index 1cc6a5b82dc..ba991d798fe 100644
@@ -15,6 +15,9 @@
 
 
 class QAPISchemaPragma:
+    # Replace with @dataclass in Python 3.7+
+    # pylint: disable=too-few-public-methods
+
     def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
Signature · 1 line
2.26.2
author
#88 lore raw

PATCH v5 26/36 qapi/gen.py: Fix edge-case of _is_user_module

In reply to John Snow

The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.

---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +1 -1
index f2e2746fea5..1bad37fc06b 100644
@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
 
     @staticmethod
     def _is_user_module(name):
-        return name and not name.startswith('./')
+        return bool(name and not name.startswith('./'))
 
     @staticmethod
     def _is_builtin_module(name):
Signature · 1 line
2.26.2
Reviewed-by on v5 26/36
#89 lore raw

Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

Quoted 9 lines · John Snow

The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.

---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
#90 lore raw

Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

In reply to John Snow
Quoted 2 lines · John Snow

The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.

The edge case is impossible, as discussed in review of v2. I figure the type checker can't see that, so we need to help it some. Can we mention this in the commit message?

Quoted 13 lines
---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +0 -0
index f2e2746fea5..1bad37fc06b 100644

scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 246

Quoted 5 lines
scripts/qapi/gen.py +1 -1
@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
 
     @staticmethod
     def _is_user_module(name):
-        return name and not name.startswith('./')
+        return bool(name and not name.startswith('./'))

return not (name is None or name.startswith('./')

Looks slightly clearer to me.

Quoted 3 lines
scripts/qapi/gen.py +0 -0
     @staticmethod
     def _is_builtin_module(name):
author
#91 lore raw

Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

Quoted 9 lines · Markus Armbruster
Quoted 2 lines · John Snow

The edge case is that if the name is '', this expression returns a string instead of a bool, which violates our declared type.

The edge case is impossible, as discussed in review of v2. I figure the type checker can't see that, so we need to help it some. Can we mention this in the commit message?

Sure. It's quite possible we will be able to better model and constrain these types, but that will come after all of these patches.

Mypy just has no way of knowing that '' is forbidden. That constraint does not exist in the type system we have here at present.

You could, by the way, create a type called NonEmptyString and create a function that casts from str to NonEmptyString by means of an assertion and then annotating the return type.

Anyway, I'll update the commit message for now.

Quoted 25 lines
Quoted 13 lines
---
  scripts/qapi/gen.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/gen.py +0 -0
index f2e2746fea5..1bad37fc06b 100644

scripts/qapi/gen.py def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): · line 246

Quoted 5 lines
scripts/qapi/gen.py +1 -1
@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
 
      @staticmethod
      def _is_user_module(name):
-        return name and not name.startswith('./')
+        return bool(name and not name.startswith('./'))

return not (name is None or name.startswith('./')

Looks slightly clearer to me.

Quoted 3 lines
scripts/qapi/gen.py +0 -0
      @staticmethod
      def _is_builtin_module(name):
author
#92 lore raw

PATCH v5 27/36 qapi/gen.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
 1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +57 -47
index 1bad37fc06b..d0391cd8718 100644
@@ -17,7 +17,13 @@
 import errno
 import os
 import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
 
 from .common import (
     c_fname,
@@ -29,31 +35,31 @@
     mcgen,
 )
 from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         self.fname = fname
         self._preamble = ''
         self._body = ''
 
-    def preamble_add(self, text):
+    def preamble_add(self, text: str) -> None:
         self._preamble += text
 
-    def add(self, text):
+    def add(self, text: str) -> None:
         self._body += text
 
-    def get_content(self):
+    def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
-    def _top(self):
+    def _top(self) -> str:
         return ''
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return ''
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         # Include paths starting with ../ are used to reuse modules of the main
         # schema in specialised schemas. Don't overwrite the files that are
         # already generated for the main schema.
@@ -78,7 +84,7 @@ def write(self, output_dir):
         f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
     if before == after:
         return after   # suppress empty #if ... #endif
 
@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
 
 
 class QAPIGenCCode(QAPIGen):
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         super().__init__(fname)
-        self._start_if = None
+        self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-    def start_if(self, ifcond):
+    def start_if(self, ifcond: List[str]) -> None:
         assert self._start_if is None
         self._start_if = (ifcond, self._body, self._preamble)
 
-    def end_if(self):
+    def end_if(self) -> None:
         assert self._start_if
         self._wrap_ifcond()
         self._start_if = None
 
-    def _wrap_ifcond(self):
+    def _wrap_ifcond(self) -> None:
         self._body = _wrap_ifcond(self._start_if[0],
                                   self._start_if[1], self._body)
         self._preamble = _wrap_ifcond(self._start_if[0],
                                       self._start_if[2], self._preamble)
 
-    def get_content(self):
+    def get_content(self) -> str:
         assert self._start_if is None
         return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-    def __init__(self, fname, blurb, pydoc):
+    def __init__(self, fname: str, blurb: str, pydoc: str):
         super().__init__(fname)
         self._blurb = blurb
         self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
                                                   re.MULTILINE))
 
-    def _top(self):
+    def _top(self) -> str:
         return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -167,7 +171,7 @@ def _top(self):
 ''',
                      blurb=self._blurb, copyright=self._copyright)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -177,16 +181,15 @@ def _bottom(self):
 
 
 class QAPIGenH(QAPIGenC):
-
-    def _top(self):
+    def _top(self) -> str:
         return super()._top() + guardstart(self.fname)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:
     """
     A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 blurb: str,
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
         self._genh = QAPIGenH(self._prefix + self._what + '.h',
                               blurb, pydoc)
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         self._genc.write(output_dir)
         self._genh.write(output_dir)
 
 
 class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 user_blurb: str,
+                 builtin_blurb: Optional[str],
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._user_blurb = user_blurb
         self._builtin_blurb = builtin_blurb
         self._pydoc = pydoc
-        self._genc = None
-        self._genh = None
-        self._module = {}
-        self._main_module = None
+        self._genc: Optional[QAPIGenC] = None
+        self._genh: Optional[QAPIGenH] = None
+        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+        self._main_module: Optional[str] = None
 
     @staticmethod
-    def _is_user_module(name):
+    def _is_user_module(name: Optional[str]) -> bool:
         return bool(name and not name.startswith('./'))
 
     @staticmethod
-    def _is_builtin_module(name):
+    def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what, name):
+    def _module_dirname(self, what: str, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
 
-    def _module_basename(self, what, name):
+    def _module_basename(self, what: str, name: Optional[str]) -> str:
         ret = '' if self._is_builtin_module(name) else self._prefix
         if self._is_user_module(name):
             basename = os.path.basename(name)
@@ -266,27 +276,27 @@ def _module_basename(self, what, name):
             ret += re.sub(r'-', '-' + name + '-', what)
         return ret
 
-    def _module_filename(self, what, name):
+    def _module_filename(self, what: str, name: Optional[str]) -> str:
         return os.path.join(self._module_dirname(what, name),
                             self._module_basename(what, name))
 
-    def _add_module(self, name, blurb):
+    def _add_module(self, name: Optional[str], blurb: str) -> None:
         basename = self._module_filename(self._what, name)
         genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
         genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
         self._module[name] = (genc, genh)
         self._genc, self._genh = self._module[name]
 
-    def _add_user_module(self, name, blurb):
+    def _add_user_module(self, name: str, blurb: str) -> None:
         assert self._is_user_module(name)
         if self._main_module is None:
             self._main_module = name
         self._add_module(name, blurb)
 
-    def _add_system_module(self, name, blurb):
+    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
         self._add_module(name and './' + name, blurb)
 
-    def write(self, output_dir, opt_builtins=False):
+    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
         for name in self._module:
             if self._is_builtin_module(name) and not opt_builtins:
                 continue
@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
             genc.write(output_dir)
             genh.write(output_dir)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         pass
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: Optional[str]) -> None:
         if name is None:
             if self._builtin_blurb:
                 self._add_system_module(None, self._builtin_blurb)
@@ -314,7 +324,7 @@ def visit_module(self, name):
             self._add_user_module(name, self._user_blurb)
             self._begin_user_module(name)
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
         relname = os.path.relpath(self._module_filename(self._what, name),
                                   os.path.dirname(self._genh.fname))
         self._genh.preamble_add(mcgen('''
Signature · 1 line
2.26.2
#93 lore raw

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

Quoted 30 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
 1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +7 -1
index 1bad37fc06b..d0391cd8718 100644
@@ -17,7 +17,13 @@
 import errno
 import os
 import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
 
 from .common import (
     c_fname,

scripts/qapi/gen.py line 42

Quoted 10 lines
scripts/qapi/gen.py +2 -2
@@ -29,31 +35,31 @@
     mcgen,
 )
 from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):

I'd expect fname: str. Can you point me to the spot that passes None?

Quoted 39 lines
scripts/qapi/gen.py +7 -7
         self.fname = fname
         self._preamble = ''
         self._body = ''
 
-    def preamble_add(self, text):
+    def preamble_add(self, text: str) -> None:
         self._preamble += text
 
-    def add(self, text):
+    def add(self, text: str) -> None:
         self._body += text
 
-    def get_content(self):
+    def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
-    def _top(self):
+    def _top(self) -> str:
         return ''
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return ''
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         # Include paths starting with ../ are used to reuse modules of the main
         # schema in specialised schemas. Don't overwrite the files that are
         # already generated for the main schema.
@@ -78,7 +84,7 @@ def write(self, output_dir):
         f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
     if before == after:
         return after   # suppress empty #if ... #endif
 

scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 127

Quoted 6 lines
scripts/qapi/gen.py +1 -2
@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
 
 
 class QAPIGenCCode(QAPIGen):
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):

Likewise.

Quoted 22 lines
scripts/qapi/gen.py +4 -4
         super().__init__(fname)
-        self._start_if = None
+        self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-    def start_if(self, ifcond):
+    def start_if(self, ifcond: List[str]) -> None:
         assert self._start_if is None
         self._start_if = (ifcond, self._body, self._preamble)
 
-    def end_if(self):
+    def end_if(self) -> None:
         assert self._start_if
         self._wrap_ifcond()
         self._start_if = None
 
-    def _wrap_ifcond(self):
+    def _wrap_ifcond(self) -> None:
         self._body = _wrap_ifcond(self._start_if[0],
                                   self._start_if[1], self._body)
         self._preamble = _wrap_ifcond(self._start_if[0],
                                       self._start_if[2], self._preamble)
 

scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 152

Quoted 10 lines
scripts/qapi/gen.py +2 -3
-    def get_content(self):
+    def get_content(self) -> str:
         assert self._start_if is None
         return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-    def __init__(self, fname, blurb, pydoc):
+    def __init__(self, fname: str, blurb: str, pydoc: str):

Here it's just str.

Quoted 26 lines
scripts/qapi/gen.py +3 -4
         super().__init__(fname)
         self._blurb = blurb
         self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
                                                   re.MULTILINE))
 
-    def _top(self):
+    def _top(self) -> str:
         return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -167,7 +171,7 @@ def _top(self):
 ''',
                      blurb=self._blurb, copyright=self._copyright)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -177,16 +181,15 @@ def _bottom(self):
 
 
 class QAPIGenH(QAPIGenC):
-
-    def _top(self):
+    def _top(self) -> str:

scripts/qapi/gen.py def _bottom(self): · line 192

Quoted 10 lines
scripts/qapi/gen.py +2 -2
         return super()._top() + guardstart(self.fname)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:

Oh, the type hint for a *args is QAPIGenCCode, even though args is a tuple of excess positional arguments (which are all QAPIGenCCode).

Quoted 105 lines
scripts/qapi/gen.py +25 -18
     """
     A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 blurb: str,
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
         self._genh = QAPIGenH(self._prefix + self._what + '.h',
                               blurb, pydoc)
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         self._genc.write(output_dir)
         self._genh.write(output_dir)
 
 
 class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 user_blurb: str,
+                 builtin_blurb: Optional[str],
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._user_blurb = user_blurb
         self._builtin_blurb = builtin_blurb
         self._pydoc = pydoc
-        self._genc = None
-        self._genh = None
-        self._module = {}
-        self._main_module = None
+        self._genc: Optional[QAPIGenC] = None
+        self._genh: Optional[QAPIGenH] = None
+        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+        self._main_module: Optional[str] = None
 
     @staticmethod
-    def _is_user_module(name):
+    def _is_user_module(name: Optional[str]) -> bool:
         return bool(name and not name.startswith('./'))
 
     @staticmethod
-    def _is_builtin_module(name):
+    def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what, name):
+    def _module_dirname(self, what: str, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
 
-    def _module_basename(self, what, name):
+    def _module_basename(self, what: str, name: Optional[str]) -> str:
         ret = '' if self._is_builtin_module(name) else self._prefix
         if self._is_user_module(name):
             basename = os.path.basename(name)
@@ -266,27 +276,27 @@ def _module_basename(self, what, name):
             ret += re.sub(r'-', '-' + name + '-', what)
         return ret
 
-    def _module_filename(self, what, name):
+    def _module_filename(self, what: str, name: Optional[str]) -> str:
         return os.path.join(self._module_dirname(what, name),
                             self._module_basename(what, name))
 
-    def _add_module(self, name, blurb):
+    def _add_module(self, name: Optional[str], blurb: str) -> None:
         basename = self._module_filename(self._what, name)
         genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
         genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
         self._module[name] = (genc, genh)
         self._genc, self._genh = self._module[name]
 
-    def _add_user_module(self, name, blurb):
+    def _add_user_module(self, name: str, blurb: str) -> None:
         assert self._is_user_module(name)
         if self._main_module is None:
             self._main_module = name
         self._add_module(name, blurb)
 
-    def _add_system_module(self, name, blurb):
+    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
         self._add_module(name and './' + name, blurb)
 
-    def write(self, output_dir, opt_builtins=False):
+    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
         for name in self._module:
             if self._is_builtin_module(name) and not opt_builtins:
                 continue

scripts/qapi/gen.py def write(self, output_dir, opt_builtins=False): · line 307

Quoted 5 lines
scripts/qapi/gen.py +1 -1
@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
             genc.write(output_dir)
             genh.write(output_dir)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:

I figure the type hint None signals a simplifcation opportunity. No need to worry about it now.

Quoted 20 lines
scripts/qapi/gen.py +3 -3
         pass
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: Optional[str]) -> None:
         if name is None:
             if self._builtin_blurb:
                 self._add_system_module(None, self._builtin_blurb)
@@ -314,7 +324,7 @@ def visit_module(self, name):
             self._add_user_module(name, self._user_blurb)
             self._begin_user_module(name)
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
         relname = os.path.relpath(self._module_filename(self._what, name),
                                   os.path.dirname(self._genh.fname))
         self._genh.preamble_add(mcgen('''
author
#94 lore raw

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

Quoted 38 lines · Markus Armbruster
Quoted 36 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
  scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
  1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +8 -1
index 1bad37fc06b..d0391cd8718 100644
@@ -17,7 +17,13 @@
  import errno
  import os
  import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
 
  from .common import (
      c_fname,
@@ -29,31 +35,31 @@
      mcgen,
  )
  from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo
 
 
Quoted 7 lines

scripts/qapi/gen.py line 42

Quoted 4 lines
scripts/qapi/gen.py +1 -2
  class QAPIGen:
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):

I'd expect fname: str. Can you point me to the spot that passes None?

qapi/commands.py: self._regy = QAPIGenCCode(None)

Good time to mention again: I am disabling strict none checks for now in this conversion.

That means we can pass Optional[T] to functions expecting T and mypy will not complain. This should obviously be fixed long-term, but the cleanups involve things that are a higher class of finnicky.

If this alarms you, good! We'll fix it in time, but it doesn't break anything any worse than it already was.

If this check was disabled, you would be able to edit the Optional[str] to str and see what broke. But because I use this affordance for now, you would see no difference.

Quoted 76 lines
Quoted 39 lines
scripts/qapi/gen.py +7 -7
          self.fname = fname
          self._preamble = ''
          self._body = ''
 
-    def preamble_add(self, text):
+    def preamble_add(self, text: str) -> None:
          self._preamble += text
 
-    def add(self, text):
+    def add(self, text: str) -> None:
          self._body += text
 
-    def get_content(self):
+    def get_content(self) -> str:
          return self._top() + self._preamble + self._body + self._bottom()
 
-    def _top(self):
+    def _top(self) -> str:
          return ''
 
-    def _bottom(self):
+    def _bottom(self) -> str:
          return ''
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
          # Include paths starting with ../ are used to reuse modules of the main
          # schema in specialised schemas. Don't overwrite the files that are
          # already generated for the main schema.
@@ -78,7 +84,7 @@ def write(self, output_dir):
          f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
      if before == after:
          return after   # suppress empty #if ... #endif
 

scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 127

Quoted 6 lines
scripts/qapi/gen.py +1 -2
@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
 
 
  class QAPIGenCCode(QAPIGen):
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):

Likewise.

Quoted 28 lines
scripts/qapi/gen.py +5 -5
          super().__init__(fname)
-        self._start_if = None
+        self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-    def start_if(self, ifcond):
+    def start_if(self, ifcond: List[str]) -> None:
          assert self._start_if is None
          self._start_if = (ifcond, self._body, self._preamble)
 
-    def end_if(self):
+    def end_if(self) -> None:
          assert self._start_if
          self._wrap_ifcond()
          self._start_if = None
 
-    def _wrap_ifcond(self):
+    def _wrap_ifcond(self) -> None:
          self._body = _wrap_ifcond(self._start_if[0],
                                    self._start_if[1], self._body)
          self._preamble = _wrap_ifcond(self._start_if[0],
                                        self._start_if[2], self._preamble)
 
-    def get_content(self):
+    def get_content(self) -> str:
          assert self._start_if is None
          return super().get_content()
 
 
Quoted 7 lines

scripts/qapi/gen.py def build_params(arg_type: Optional[QAPISchemaObjectType], · line 152

Quoted 4 lines
scripts/qapi/gen.py +1 -2
  class QAPIGenC(QAPIGenCCode):
-
-    def __init__(self, fname, blurb, pydoc):
+    def __init__(self, fname: str, blurb: str, pydoc: str):

Here it's just str.

That's right.

Quoted 33 lines
Quoted 33 lines
scripts/qapi/gen.py +4 -5
          super().__init__(fname)
          self._blurb = blurb
          self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
                                                    re.MULTILINE))
 
-    def _top(self):
+    def _top(self) -> str:
          return mcgen('''
  /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -167,7 +171,7 @@ def _top(self):
  ''',
                       blurb=self._blurb, copyright=self._copyright)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
          return mcgen('''
 
  /* Dummy declaration to prevent empty .o file */
@@ -177,16 +181,15 @@ def _bottom(self):
 
 
  class QAPIGenH(QAPIGenC):
-
-    def _top(self):
+    def _top(self) -> str:
          return super()._top() + guardstart(self.fname)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
          return guardend(self.fname)
 
 
Quoted 7 lines

scripts/qapi/gen.py def _bottom(self): · line 192

Quoted 3 lines
scripts/qapi/gen.py +1 -1
  @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:

Oh, the type hint for a *args is QAPIGenCCode, even though args is a tuple of excess positional arguments (which are all QAPIGenCCode).

Yeah, it's the way type hints interact with the special '*' syntax. This is an Iterable of QAPIGenCCode.

**kwargs works the same way: you annotate the values, and the keys are assumed to be str.

Quoted 108 lines
Quoted 108 lines
scripts/qapi/gen.py +25 -18
      """
      A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
 
 
  class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 blurb: str,
+                 pydoc: str):
          self._prefix = prefix
          self._what = what
          self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
          self._genh = QAPIGenH(self._prefix + self._what + '.h',
                                blurb, pydoc)
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
          self._genc.write(output_dir)
          self._genh.write(output_dir)
 
 
  class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 user_blurb: str,
+                 builtin_blurb: Optional[str],
+                 pydoc: str):
          self._prefix = prefix
          self._what = what
          self._user_blurb = user_blurb
          self._builtin_blurb = builtin_blurb
          self._pydoc = pydoc
-        self._genc = None
-        self._genh = None
-        self._module = {}
-        self._main_module = None
+        self._genc: Optional[QAPIGenC] = None
+        self._genh: Optional[QAPIGenH] = None
+        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+        self._main_module: Optional[str] = None
 
      @staticmethod
-    def _is_user_module(name):
+    def _is_user_module(name: Optional[str]) -> bool:
          return bool(name and not name.startswith('./'))
 
      @staticmethod
-    def _is_builtin_module(name):
+    def _is_builtin_module(name: Optional[str]) -> bool:
          return not name
 
-    def _module_dirname(self, what, name):
+    def _module_dirname(self, what: str, name: Optional[str]) -> str:
          if self._is_user_module(name):
              return os.path.dirname(name)
          return ''
 
-    def _module_basename(self, what, name):
+    def _module_basename(self, what: str, name: Optional[str]) -> str:
          ret = '' if self._is_builtin_module(name) else self._prefix
          if self._is_user_module(name):
              basename = os.path.basename(name)
@@ -266,27 +276,27 @@ def _module_basename(self, what, name):
              ret += re.sub(r'-', '-' + name + '-', what)
          return ret
 
-    def _module_filename(self, what, name):
+    def _module_filename(self, what: str, name: Optional[str]) -> str:
          return os.path.join(self._module_dirname(what, name),
                              self._module_basename(what, name))
 
-    def _add_module(self, name, blurb):
+    def _add_module(self, name: Optional[str], blurb: str) -> None:
          basename = self._module_filename(self._what, name)
          genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
          genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
          self._module[name] = (genc, genh)
          self._genc, self._genh = self._module[name]
 
-    def _add_user_module(self, name, blurb):
+    def _add_user_module(self, name: str, blurb: str) -> None:
          assert self._is_user_module(name)
          if self._main_module is None:
              self._main_module = name
          self._add_module(name, blurb)
 
-    def _add_system_module(self, name, blurb):
+    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
          self._add_module(name and './' + name, blurb)
 
-    def write(self, output_dir, opt_builtins=False):
+    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
          for name in self._module:
              if self._is_builtin_module(name) and not opt_builtins:
                  continue
@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
              genc.write(output_dir)
              genh.write(output_dir)
 
Quoted 6 lines

scripts/qapi/gen.py def write(self, output_dir, opt_builtins=False): · line 307

Quoted 2 lines
scripts/qapi/gen.py +1 -1
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:

I figure the type hint None signals a simplifcation opportunity. No need to worry about it now.

Yes, the module name stuff in general has a smell to it. I didn't look deeper, but there's an opportunity for cleaning it up. I realize the type signature here has parity with _begin_user_module, but I noticed it was *never* called with anything that wasn't a literal None, so I added the stricter type.

Quoted 20 lines
Quoted 20 lines
scripts/qapi/gen.py +3 -3
          pass
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
          pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: Optional[str]) -> None:
          if name is None:
              if self._builtin_blurb:
                  self._add_system_module(None, self._builtin_blurb)
@@ -314,7 +324,7 @@ def visit_module(self, name):
              self._add_user_module(name, self._user_blurb)
              self._begin_user_module(name)
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
          relname = os.path.relpath(self._module_filename(self._what, name),
                                    os.path.dirname(self._genh.fname))
          self._genh.preamble_add(mcgen('''
#95 lore raw

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

In reply to John Snow
Quoted 30 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
 1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +7 -1
index 1bad37fc06b..d0391cd8718 100644
@@ -17,7 +17,13 @@
 import errno
 import os
 import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
 
 from .common import (
     c_fname,

scripts/qapi/gen.py line 38

Quoted 4 lines
scripts/qapi/gen.py +1 -0
@@ -29,31 +35,31 @@
     mcgen,
 )
 from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo

PATCH 03 has a similar cleanup. Are there more? Perhaps a separate patch doing just this kind of cleanup would make sense. Up to you.

[...]

author
#96 lore raw

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

Quoted 40 lines · Markus Armbruster
Quoted 30 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
  scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
  1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +7 -1
index 1bad37fc06b..d0391cd8718 100644
@@ -17,7 +17,13 @@
  import errno
  import os
  import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
 
  from .common import (
      c_fname,

scripts/qapi/gen.py line 38

Quoted 4 lines
scripts/qapi/gen.py +1 -0
@@ -29,31 +35,31 @@
      mcgen,
  )
  from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo

PATCH 03 has a similar cleanup. Are there more? Perhaps a separate patch doing just this kind of cleanup would make sense. Up to you.

Quoted 2 lines

[...]

This isn't a cleanup, I am just importing QAPISourceInfo to use for an annotation. It's relevant and required for this patch, and doesn't make sense on its own.

Patch 03 ... Oh, you mean identifying the correct location of QAPIError. Uh... nah? I think that was the only case of that one changing. Not worth pulling out or naming, I think.

#97 lore raw

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

Quoted 41 lines · John Snow
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
  scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
  1 file changed, 57 insertions(+), 47 deletions(-)
scripts/qapi/gen.py +7 -1
index 1bad37fc06b..d0391cd8718 100644
@@ -17,7 +17,13 @@
  import errno
  import os
  import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
    from .common import (
      c_fname,
Quoted 9 lines

scripts/qapi/gen.py line 38

Quoted 5 lines
scripts/qapi/gen.py +1 -0
@@ -29,31 +35,31 @@
      mcgen,
  )
  from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo

PATCH 03 has a similar cleanup. Are there more? Perhaps a separate patch doing just this kind of cleanup would make sense. Up to you. [...]

Quoted 3 lines

This isn't a cleanup, I am just importing QAPISourceInfo to use for an annotation. It's relevant and required for this patch, and doesn't make sense on its own.

I was mistaken. A case of patch-review-eye...

Quoted 3 lines

Patch 03 ... Oh, you mean identifying the correct location of QAPIError. Uh... nah? I think that was the only case of that one changing. Not worth pulling out or naming, I think.

Agree.

author
#98 lore raw

PATCH v5 28/36 qapi/gen.py: Enable checking with mypy

In reply to John Snow
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 1b8555dfa39..c6960ff2dbd 100644
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.gen]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.introspect]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
Signature · 1 line
2.26.2
author
#99 lore raw

PATCH v5 29/36 qapi/gen.py: Remove unused parameter

In reply to John Snow

module_basename doesn't use the 'what' argument, so remove it.

---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
index d0391cd8718..3624162bb77 100644
@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
     def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what: str, name: Optional[str]) -> str:
+    def _module_dirname(self, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
         return ret
 
     def _module_filename(self, what: str, name: Optional[str]) -> str:
-        return os.path.join(self._module_dirname(what, name),
+        return os.path.join(self._module_dirname(name),
                             self._module_basename(what, name))
 
     def _add_module(self, name: Optional[str], blurb: str) -> None:
Signature · 1 line
2.26.2
#100 lore raw

Re: [PATCH v5 29/36] qapi/gen.py: Remove unused parameter

Quoted 1 line · John Snow

module_basename doesn't use the 'what' argument, so remove it.

_module_dirname(), I suppose.

Quoted 30 lines
---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
index d0391cd8718..3624162bb77 100644
@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
     def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what: str, name: Optional[str]) -> str:
+    def _module_dirname(self, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
         return ret
 
     def _module_filename(self, what: str, name: Optional[str]) -> str:
-        return os.path.join(self._module_dirname(what, name),
+        return os.path.join(self._module_dirname(name),
                             self._module_basename(what, name))
 
     def _add_module(self, name: Optional[str], blurb: str) -> None:
author
#101 lore raw

Re: [PATCH v5 29/36] qapi/gen.py: Remove unused parameter

Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow

module_basename doesn't use the 'what' argument, so remove it.

_module_dirname(), I suppose.

My brain just translates stuff I didn't ask it to all the time. (Fixed.)

Quoted 30 lines
Quoted 30 lines
---
  scripts/qapi/gen.py | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -2
index d0391cd8718..3624162bb77 100644
@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
      def _is_builtin_module(name: Optional[str]) -> bool:
          return not name
 
-    def _module_dirname(self, what: str, name: Optional[str]) -> str:
+    def _module_dirname(self, name: Optional[str]) -> str:
          if self._is_user_module(name):
              return os.path.dirname(name)
          return ''
@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
          return ret
 
      def _module_filename(self, what: str, name: Optional[str]) -> str:
-        return os.path.join(self._module_dirname(what, name),
+        return os.path.join(self._module_dirname(name),
                              self._module_basename(what, name))
 
      def _add_module(self, name: Optional[str], blurb: str) -> None:
author
#102 lore raw

PATCH v5 30/36 qapi/gen.py: update write() to be more idiomatic

In reply to John Snow

Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.

---
 scripts/qapi/gen.py | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +11 -14
index 3624162bb77..579ee283297 100644
@@ -14,7 +14,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import (
@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
             return
         pathname = os.path.join(output_dir, self.fname)
         odir = os.path.dirname(pathname)
+
         if odir:
-            try:
-                os.makedirs(odir)
-            except os.error as e:
-                if e.errno != errno.EEXIST:
-                    raise
+            os.makedirs(odir, exist_ok=True)
+
+        # use os.open for O_CREAT to create and read a non-existant file
         fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-        f = open(fd, 'r+', encoding='utf-8')
-        text = self.get_content()
-        oldtext = f.read(len(text) + 1)
-        if text != oldtext:
-            f.seek(0)
-            f.truncate(0)
-            f.write(text)
-        f.close()
+        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+            text = self.get_content()
+            oldtext = fp.read(len(text) + 1)
+            if text != oldtext:
+                fp.seek(0)
+                fp.truncate(0)
+                fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
Signature · 1 line
2.26.2
Reviewed-by on v5 30/36
#103 lore raw

Re: [PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic

Quoted 28 lines · John Snow

Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.

---
 scripts/qapi/gen.py | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +0 -1
index 3624162bb77..579ee283297 100644
@@ -14,7 +14,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import (
@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
             return

scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 71

Quoted 10 lines
scripts/qapi/gen.py +2 -5
         pathname = os.path.join(output_dir, self.fname)
         odir = os.path.dirname(pathname)
+
         if odir:
-            try:
-                os.makedirs(odir)
-            except os.error as e:
-                if e.errno != errno.EEXIST:
-                    raise
+            os.makedirs(odir, exist_ok=True)

I wouldn't call this part "heavily subjective". When wrote the old code, exist_ok was still off limits (it's new in 3.2).

Quoted 11 lines
scripts/qapi/gen.py +2 -8
+
+        # use os.open for O_CREAT to create and read a non-existant file
         fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-        f = open(fd, 'r+', encoding='utf-8')
-        text = self.get_content()
-        oldtext = f.read(len(text) + 1)
-        if text != oldtext:
-            f.seek(0)
-            f.truncate(0)
-            f.write(text)
-        f.close()

scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 84

Quoted 10 lines
scripts/qapi/gen.py +7 -0
+        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+            text = self.get_content()
+            oldtext = fp.read(len(text) + 1)
+            if text != oldtext:
+                fp.seek(0)
+                fp.truncate(0)
+                fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
author
#104 lore raw

Re: [PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic

Quoted 35 lines · Markus Armbruster
Quoted 33 lines · John Snow

Make the file handling here just a tiny bit more idiomatic. (I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception, use fdopen() to wrap the file descriptor in a File-like object, and use a context manager for managing the file pointer.

---
  scripts/qapi/gen.py | 25 +++++++++++--------------
  1 file changed, 11 insertions(+), 14 deletions(-)
scripts/qapi/gen.py +1 -2
index 3624162bb77..579ee283297 100644
@@ -14,7 +14,6 @@
  # See the COPYING file in the top-level directory.
 
  from contextlib import contextmanager
-import errno
  import os
  import re
  from typing import (
@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
              return
          pathname = os.path.join(output_dir, self.fname)
          odir = os.path.dirname(pathname)
+
          if odir:
-            try:
Quoted 9 lines

scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 71

Quoted 5 lines
scripts/qapi/gen.py +1 -4
-                os.makedirs(odir)
-            except os.error as e:
-                if e.errno != errno.EEXIST:
-                    raise
+            os.makedirs(odir, exist_ok=True)

I wouldn't call this part "heavily subjective". When wrote the old code, exist_ok was still off limits (it's new in 3.2).

It's cool if you agree, I just realize that what people consider idiomatic is subjective unless it's enforcable by a tool. This isn't.

"I made this look more like if I wrote it, which caused Dopamine" is a bad commit message. (But more true.)

Quoted 20 lines
Quoted 20 lines
scripts/qapi/gen.py +9 -8
+
+        # use os.open for O_CREAT to create and read a non-existant file
          fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-        f = open(fd, 'r+', encoding='utf-8')
-        text = self.get_content()
-        oldtext = f.read(len(text) + 1)
-        if text != oldtext:
-            f.seek(0)
-            f.truncate(0)
-            f.write(text)
-        f.close()
+        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+            text = self.get_content()
+            oldtext = fp.read(len(text) + 1)
+            if text != oldtext:
+                fp.seek(0)
+                fp.truncate(0)
+                fp.write(text)
 
 
Quoted 4 lines

scripts/qapi/gen.py def write(self, output_dir: str) -> None: · line 84

Quoted 1 line
scripts/qapi/gen.py +0 -0
  def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:

Thanks, though :)

author
#105 lore raw

PATCH v5 31/36 qapi/gen.py: delint with pylint

In reply to John Snow

'fp' and 'fd' are self-evident in context, add them to the list of OK names.

_top and _bottom also need to stay standard methods because some users override the method and need to use self. Tell pylint to shush.

---
 scripts/qapi/gen.py   | 2 ++
 scripts/qapi/pylintrc | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)
scripts/qapi/gen.py +2 -0
index 579ee283297..29bdd140587 100644
@@ -53,9 +53,11 @@ def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
     def _top(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def _bottom(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def write(self, output_dir: str) -> None:
scripts/qapi/pylintrc +3 -2
index d840b150313..8badcb11cda 100644
@@ -4,7 +4,6 @@
 # The regex matches against base names, not paths.
 ignore-patterns=error.py,
                 expr.py,
-                gen.py,
                 parser.py,
                 schema.py,
                 types.py,
@@ -45,7 +44,9 @@ good-names=i,
            k,
            ex,
            Run,
-           _
+           _,
+           fp,  # fp = open(...)
+           fd,  # fd = os.open(...)
 
 [VARIABLES]
 
Signature · 1 line
2.26.2
author
#106 lore raw

PATCH v5 32/36 qapi/types.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 64 insertions(+), 27 deletions(-)
scripts/qapi/mypy.ini +0 -5
index c6960ff2dbd..83f19813553 100644
@@ -29,11 +29,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.visit]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
scripts/qapi/types.py +64 -22

207 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#107 lore raw

PATCH v5 33/36 qapi/types.py: remove one-letter variables

In reply to John Snow

"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++++++++++++++--------------
 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
index 8badcb11cda..b3c4cf46dbf 100644
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                types.py,
                 visit.py,
 
 
scripts/qapi/types.py +15 -14
index 766822feb3a..9d1e79d503d 100644
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
     .array = (const char *const[]) {
 ''',
                 c_name=c_name(name))
-    for m in members:
-        ret += gen_if(m.ifcond)
-        index = c_enum_const(name, m.name, prefix)
+    for member in members:
+        ret += gen_if(member.ifcond)
+        index = c_enum_const(name, member.name, prefix)
         ret += mcgen('''
         [%(index)s] = "%(name)s",
 ''',
-                     index=index, name=m.name)
-        ret += gen_endif(m.ifcond)
+                     index=index, name=member.name)
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
     },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
 ''',
                 c_name=c_name(name))
 
-    for m in enum_members:
-        ret += gen_if(m.ifcond)
+    for member in enum_members:
+        ret += gen_if(member.ifcond)
         ret += mcgen('''
     %(c_enum)s,
 ''',
-                     c_enum=c_enum_const(name, m.name, prefix))
-        ret += gen_endif(m.ifcond)
+                     c_enum=c_enum_const(name, member.name, prefix))
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
 } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
     objects_seen.add(name)
 
     ret = ''
-    if variants:
-        for v in variants.variants:
-            if isinstance(v.type, QAPISchemaObjectType):
-                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-                                  v.type.local_members, v.type.variants)
+    for variant in variants.variants if variants else ():
+        obj = variant.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret += gen_object(obj.name, obj.ifcond, obj.base,
+                          obj.local_members, obj.variants)
 
     ret += mcgen('''
 
Signature · 1 line
2.26.2
#108 lore raw

Re: [PATCH v5 33/36] qapi/types.py: remove one-letter variables

Quoted 30 lines · John Snow

"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++++++++++++++--------------
 2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
index 8badcb11cda..b3c4cf46dbf 100644
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                types.py,
                 visit.py,
 
 
scripts/qapi/types.py +0 -0
index 766822feb3a..9d1e79d503d 100644

scripts/qapi/types.py def gen_enum_lookup(name: str, · line 52

Quoted 7 lines
scripts/qapi/types.py +1 -3
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
     .array = (const char *const[]) {
 ''',
                 c_name=c_name(name))
-    for m in members:
-        ret += gen_if(m.ifcond)
-        index = c_enum_const(name, m.name, prefix)
+    for member in members:

Let's use @memb. It's more visually distinct from @members, and gen_struct_members() already uses it.

Quoted 31 lines
scripts/qapi/types.py +8 -6
+        ret += gen_if(member.ifcond)
+        index = c_enum_const(name, member.name, prefix)
         ret += mcgen('''
         [%(index)s] = "%(name)s",
 ''',
-                     index=index, name=m.name)
-        ret += gen_endif(m.ifcond)
+                     index=index, name=member.name)
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
     },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
 ''',
                 c_name=c_name(name))
 
-    for m in enum_members:
-        ret += gen_if(m.ifcond)
+    for member in enum_members:
+        ret += gen_if(member.ifcond)
         ret += mcgen('''
     %(c_enum)s,
 ''',
-                     c_enum=c_enum_const(name, m.name, prefix))
-        ret += gen_endif(m.ifcond)
+                     c_enum=c_enum_const(name, member.name, prefix))
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
 } %(c_name)s;

scripts/qapi/types.py def gen_object(name: str, ifcond: List[str], · line 151

Quoted 9 lines
scripts/qapi/types.py +1 -5
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
     objects_seen.add(name)
 
     ret = ''
-    if variants:
-        for v in variants.variants:
-            if isinstance(v.type, QAPISchemaObjectType):
-                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-                                  v.type.local_members, v.type.variants)
+    for variant in variants.variants if variants else ():

Let's use @var, for consistency with gen_variants().

Quoted 7 lines
scripts/qapi/types.py +5 -0
+        obj = variant.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret += gen_object(obj.name, obj.ifcond, obj.base,
+                          obj.local_members, obj.variants)
 
     ret += mcgen('''
author
#109 lore raw

Re: [PATCH v5 33/36] qapi/types.py: remove one-letter variables

Quoted 27 lines · Markus Armbruster
Quoted 25 lines · John Snow

"John, if pylint told you to jump off a bridge, would you?" Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

---
  scripts/qapi/pylintrc |  1 -
  scripts/qapi/types.py | 29 +++++++++++++++--------------
  2 files changed, 15 insertions(+), 15 deletions(-)
scripts/qapi/pylintrc +0 -1
index 8badcb11cda..b3c4cf46dbf 100644
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
                  expr.py,
                  parser.py,
                  schema.py,
-                types.py,
                  visit.py,
 
 
Quoted 16 lines

scripts/qapi/types.py def gen_enum_lookup(name: str, · line 52

Quoted 12 lines
scripts/qapi/types.py +1 -3
index 766822feb3a..9d1e79d503d 100644
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
      .array = (const char *const[]) {
  ''',
                  c_name=c_name(name))
-    for m in members:
-        ret += gen_if(m.ifcond)
-        index = c_enum_const(name, m.name, prefix)
+    for member in members:

Let's use @memb. It's more visually distinct from @members, and gen_struct_members() already uses it.

OK.

Quoted 33 lines
Quoted 33 lines
scripts/qapi/types.py +8 -6
+        ret += gen_if(member.ifcond)
+        index = c_enum_const(name, member.name, prefix)
          ret += mcgen('''
          [%(index)s] = "%(name)s",
  ''',
-                     index=index, name=m.name)
-        ret += gen_endif(m.ifcond)
+                     index=index, name=member.name)
+        ret += gen_endif(member.ifcond)
 
      ret += mcgen('''
      },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
  ''',
                  c_name=c_name(name))
 
-    for m in enum_members:
-        ret += gen_if(m.ifcond)
+    for member in enum_members:
+        ret += gen_if(member.ifcond)
          ret += mcgen('''
      %(c_enum)s,
  ''',
-                     c_enum=c_enum_const(name, m.name, prefix))
-        ret += gen_endif(m.ifcond)
+                     c_enum=c_enum_const(name, member.name, prefix))
+        ret += gen_endif(member.ifcond)
 
      ret += mcgen('''
  } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
      objects_seen.add(name)
 
Quoted 10 lines

scripts/qapi/types.py def gen_object(name: str, ifcond: List[str], · line 151

Quoted 7 lines
scripts/qapi/types.py +1 -5
      ret = ''
-    if variants:
-        for v in variants.variants:
-            if isinstance(v.type, QAPISchemaObjectType):
-                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-                                  v.type.local_members, v.type.variants)
+    for variant in variants.variants if variants else ():

Let's use @var, for consistency with gen_variants().

You're the boss.

(Said only when I don't actually still kinda sorta want to do it my way. Then you're the boss, but in the bad way. Like I am the petulant teenager working at a Burger King and I am having a disagreement with the Assistant Manager over the proper way to stock the ketchup packets.)

Quoted 7 lines
Quoted 7 lines
scripts/qapi/types.py +5 -0
+        obj = variant.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret += gen_object(obj.name, obj.ifcond, obj.base,
+                          obj.local_members, obj.variants)
 
      ret += mcgen('''
author
#110 lore raw

PATCH v5 34/36 qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

In reply to John Snow

This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.

---
 scripts/qapi/visit.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +7 -5
index 14f30c228b7..4f11fd325b8 100644
@@ -22,7 +22,7 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
         ret += gen_endif(memb.ifcond)
 
     if variants:
+        tag_member = variants.tag_member
+        assert isinstance(tag_member.type, QAPISchemaEnumType)
+
         ret += mcgen('''
     switch (obj->%(c_name)s) {
 ''',
-                     c_name=c_name(variants.tag_member.name))
+                     c_name=c_name(tag_member.name))
 
         for var in variants.variants:
-            case_str = c_enum_const(variants.tag_member.type.name,
-                                    var.name,
-                                    variants.tag_member.type.prefix)
+            case_str = c_enum_const(tag_member.type.name, var.name,
+                                    tag_member.type.prefix)
             ret += gen_if(var.ifcond)
             if var.type.name == 'q_empty':
                 # valid variant and nothing to do
Signature · 1 line
2.26.2
#111 lore raw

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

Quoted 24 lines · John Snow

This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.

---
 scripts/qapi/visit.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +1 -1
index 14f30c228b7..4f11fd325b8 100644
@@ -22,7 +22,7 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):

scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 88

Quoted 5 lines
scripts/qapi/visit.py +2 -0
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
         ret += gen_endif(memb.ifcond)
 
     if variants:
+        tag_member = variants.tag_member
+        assert isinstance(tag_member.type, QAPISchemaEnumType)

I'm curious: do you need the local variable to make the assertion stick?

Quoted 16 lines
scripts/qapi/visit.py +4 -4
+
         ret += mcgen('''
     switch (obj->%(c_name)s) {
 ''',
-                     c_name=c_name(variants.tag_member.name))
+                     c_name=c_name(tag_member.name))
 
         for var in variants.variants:
-            case_str = c_enum_const(variants.tag_member.type.name,
-                                    var.name,
-                                    variants.tag_member.type.prefix)
+            case_str = c_enum_const(tag_member.type.name, var.name,
+                                    tag_member.type.prefix)
             ret += gen_if(var.ifcond)
             if var.type.name == 'q_empty':
                 # valid variant and nothing to do
author
#112 lore raw

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

Quoted 28 lines · Markus Armbruster
Quoted 26 lines · John Snow

This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.

---
  scripts/qapi/visit.py | 12 +++++++-----
  1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +1 -1
index 14f30c228b7..4f11fd325b8 100644
@@ -22,7 +22,7 @@
      mcgen,
  )
  from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
 
  def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
          ret += gen_endif(memb.ifcond)
 
Quoted 6 lines

scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 88

Quoted 3 lines
scripts/qapi/visit.py +2 -0
      if variants:
+        tag_member = variants.tag_member
+        assert isinstance(tag_member.type, QAPISchemaEnumType)

I'm curious: do you need the local variable to make the assertion stick?

No, but it only sticks to the binding and not the data. i.e. assertions to downcast work on the *name*.

(This comes up somewhere in the schema.py patches where I make a change that looks completely pointless, but it makes mypy happy.)

I could have left it alone. I just saw a lot of repeated multi-dots and habitually created a temporary local for the purpose.

Quoted 16 lines
Quoted 16 lines
scripts/qapi/visit.py +4 -4
+
          ret += mcgen('''
      switch (obj->%(c_name)s) {
  ''',
-                     c_name=c_name(variants.tag_member.name))
+                     c_name=c_name(tag_member.name))
 
          for var in variants.variants:
-            case_str = c_enum_const(variants.tag_member.type.name,
-                                    var.name,
-                                    variants.tag_member.type.prefix)
+            case_str = c_enum_const(tag_member.type.name, var.name,
+                                    tag_member.type.prefix)
              ret += gen_if(var.ifcond)
              if var.type.name == 'q_empty':
                  # valid variant and nothing to do
#113 lore raw

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

Quoted 34 lines · John Snow
Quoted 23 lines · Markus Armbruster
Quoted 21 lines · John Snow

This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.

---
  scripts/qapi/visit.py | 12 +++++++-----
  1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +1 -1
index 14f30c228b7..4f11fd325b8 100644
@@ -22,7 +22,7 @@
      mcgen,
  )
  from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
Quoted 9 lines

scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 87

Quoted 6 lines
scripts/qapi/visit.py +2 -0
  def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
          ret += gen_endif(memb.ifcond)
        if variants:
+        tag_member = variants.tag_member
+        assert isinstance(tag_member.type, QAPISchemaEnumType)

I'm curious: do you need the local variable to make the assertion stick?

Quoted 2 lines

No, but it only sticks to the binding and not the data. i.e. assertions to downcast work on the *name*.

Would it stick to variants.tag_member, too?

I'm asking to learn, not to find a reason to make you change your patch.

Quoted 5 lines

(This comes up somewhere in the schema.py patches where I make a change that looks completely pointless, but it makes mypy happy.)

I could have left it alone. I just saw a lot of repeated multi-dots and habitually created a temporary local for the purpose.

Matter of taste. Long chains of dots make the code hard to read because they are so long. Temporary variable make it hard to read because you have to remember what they mean. Tradeoff. I come up with cases I find hard to decide all too often.

In case the local variable isn't needed for mypy: when you throw in something that isn't needed for the patch's stated purpose, it's best to mention it in the commit message, because not mentioning it is a review comment magnet :)

Put yourself in the reviewers shoes. Your lovingly crafted commit message puts him into a positive mood. He nods along while reading your obvious patch at a good pace. And then he runs smack into the unexpected unrelated part, and stops: oh, what's going on here? Back up some and read more slowly to make sure I understand.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
scripts/qapi/visit.py +4 -4
+
          ret += mcgen('''
      switch (obj->%(c_name)s) {
  ''',
-                     c_name=c_name(variants.tag_member.name))
+                     c_name=c_name(tag_member.name))
            for var in variants.variants:
-            case_str = c_enum_const(variants.tag_member.type.name,
-                                    var.name,
-                                    variants.tag_member.type.prefix)
+            case_str = c_enum_const(tag_member.type.name, var.name,
+                                    tag_member.type.prefix)
              ret += gen_if(var.ifcond)
              if var.type.name == 'q_empty':
                  # valid variant and nothing to do
author
#114 lore raw

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

Quoted 36 lines · Markus Armbruster
Quoted 34 lines · John Snow
Quoted 32 lines · Markus Armbruster
Quoted 23 lines · John Snow

This is true by design, but not presently able to be expressed in the type system. An assertion helps mypy understand our constraints.

---
   scripts/qapi/visit.py | 12 +++++++-----
   1 file changed, 7 insertions(+), 5 deletions(-)
scripts/qapi/visit.py +1 -1
index 14f30c228b7..4f11fd325b8 100644
@@ -22,7 +22,7 @@
       mcgen,
   )
   from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
   def gen_visit_decl(name, scalar=False):

scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 87

Quoted 4 lines
scripts/qapi/visit.py +2 -0
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
           ret += gen_endif(memb.ifcond)
         if variants:
+        tag_member = variants.tag_member
+        assert isinstance(tag_member.type, QAPISchemaEnumType)

I'm curious: do you need the local variable to make the assertion stick?

Quoted 5 lines
Quoted 2 lines

No, but it only sticks to the binding and not the data. i.e. assertions to downcast work on the *name*.

Would it stick to variants.tag_member, too?

As long as variants isn't re-bound in that scope, it should stick within that scope, yeah. (I think. Didn't test. It's my expectation.)

Quoted 18 lines

I'm asking to learn, not to find a reason to make you change your patch.

Quoted 5 lines

(This comes up somewhere in the schema.py patches where I make a change that looks completely pointless, but it makes mypy happy.)

I could have left it alone. I just saw a lot of repeated multi-dots and habitually created a temporary local for the purpose.

Matter of taste. Long chains of dots make the code hard to read because they are so long. Temporary variable make it hard to read because you have to remember what they mean. Tradeoff. I come up with cases I find hard to decide all too often.

In case the local variable isn't needed for mypy: when you throw in something that isn't needed for the patch's stated purpose, it's best to mention it in the commit message, because not mentioning it is a review comment magnet :)

Quoted 6 lines

Put yourself in the reviewers shoes. Your lovingly crafted commit message puts him into a positive mood. He nods along while reading your obvious patch at a good pace. And then he runs smack into the unexpected unrelated part, and stops: oh, what's going on here? Back up some and read more slowly to make sure I understand.

Yeah, understood. Well, part of it is knowing the review style of your reviewer too. It's been a while since we've personally swapped patches.

The actual process for much of this was: "There is an error! let me fix that error."

And then I do that, but I do so using idioms and patterns I'm familiar or comfortable with. And then post-hoc, not 100% of them were necessarily required, strictly, to fix the problem. Most of this series was written "one file at a time", and then split out post-fact into little per-warning changes.

I didn't even send part 1 until I finished typing the *entire* package, to make sure that things were as correct as possible from the very first commit.

I often don't really even notice that little changes aren't strictly necessary until they're challenged, it's usually not a conscious choice to try and sneak stuff in.

Still always trying to find a balance between "Easy to maintain and iterate" and "easy to review." Tough line for me.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
scripts/qapi/visit.py +4 -4
+
           ret += mcgen('''
       switch (obj->%(c_name)s) {
   ''',
-                     c_name=c_name(variants.tag_member.name))
+                     c_name=c_name(tag_member.name))
             for var in variants.variants:
-            case_str = c_enum_const(variants.tag_member.type.name,
-                                    var.name,
-                                    variants.tag_member.type.prefix)
+            case_str = c_enum_const(tag_member.type.name, var.name,
+                                    tag_member.type.prefix)
               ret += gen_if(var.ifcond)
               if var.type.name == 'q_empty':
                   # valid variant and nothing to do
#115 lore raw

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

John Snow <[email protected]> writes:

[...]

Quoted 2 lines

Still always trying to find a balance between "Easy to maintain and iterate" and "easy to review." Tough line for me.

I assure you it's a tough line for everyone.

[...]

author
#116 lore raw

PATCH v5 35/36 qapi/visit.py: remove unused parameters from gen_visit_object

In reply to John Snow

And this fixes the pylint report for this file, so make sure we check this in the future, too.

---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)
scripts/qapi/pylintrc +0 -1
index b3c4cf46dbf..b9e077a1642 100644
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                visit.py,
 
 
 [MESSAGES CONTROL]
scripts/qapi/visit.py +2 -2
index 4f11fd325b8..e54694e23db 100644
@@ -250,7 +250,7 @@ def gen_visit_alternate(name, variants):
     return ret
 
 
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
     return mcgen('''
 
 bool visit_type_%(c_name)s
@@ -343,7 +343,7 @@ def visit_object_type(self, name, info, ifcond, features,
             if not name.startswith('q_'):
                 # only explicit types need an allocating visit
                 self._genh.add(gen_visit_decl(name))
-                self._genc.add(gen_visit_object(name, base, members, variants))
+                self._genc.add(gen_visit_object(name))
 
     def visit_alternate_type(self, name, info, ifcond, features, variants):
         with ifcontext(ifcond, self._genh, self._genc):
Signature · 1 line
2.26.2
Reviewed-by on v5 35/36
#117 lore raw

Re: [PATCH v5 35/36] qapi/visit.py: remove unused parameters from gen_visit_object

Quoted 11 lines · John Snow

And this fixes the pylint report for this file, so make sure we check this in the future, too.

---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)
author
#118 lore raw

PATCH v5 36/36 qapi/visit.py: add type hint annotations

In reply to John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++----------
 2 files changed, 56 insertions(+), 22 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 83f19813553..74fc6c82153 100644
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
scripts/qapi/visit.py +56 -17

164 lines not on this page · raw message

Signature · 1 line
2.26.2
#119 lore raw

Re: [PATCH v5 36/36] qapi/visit.py: add type hint annotations

Quoted 48 lines · John Snow

Annotations do not change runtime behavior. This commit *only* adds annotations.

---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++----------
 2 files changed, 56 insertions(+), 22 deletions(-)
scripts/qapi/mypy.ini +0 -5
index 83f19813553..74fc6c82153 100644
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
scripts/qapi/visit.py +7 -1
index e54694e23db..14d4f0b261f 100644
@@ -13,6 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
     c_enum_const,
     c_name,
@@ -22,10 +24,20 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumType, QAPISchemaObjectType
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaEnumType,
+    QAPISchemaFeature,

scripts/qapi/visit.py line 40

Quoted 10 lines
scripts/qapi/visit.py +7 -1
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
 
 
-def gen_visit_decl(name, scalar=False):
+def gen_visit_decl(name: str, scalar: bool = False) -> str:

More abuse of review to learn: even through we initialize with False, we still need the type hint?

Quoted 3 lines
scripts/qapi/visit.py +0 -0
     c_type = c_name(name) + ' *'
     if not scalar:
         c_type += '*'

[...]

author
#120 lore raw

Re: [PATCH v5 36/36] qapi/visit.py: add type hint annotations

Quoted 2 lines · Markus Armbruster

More abuse of review to learn: even through we initialize with False, we still need the type hint?

Dunno.

jsnow@scv ~/s/q/scripts (python-qapi-cleanup-pt1)> mypy --config-file=qapi/mypy.ini qapi/; qapi/visit.py:40: error: Function is missing a type annotation for one or more arguments Found 1 error in 1 file (checked 14 source files)

...yup.

It also serves the purpose of constraining the type to also avoid unions with things that might be false-ish. Looks silly, but maybe future mypy installments will afford us more chances to use inference. It doesn't right now.

--js

#121 lore raw
In reply to John Snow
Quoted 14 lines · John Snow

Hi, this series adds static type hints to the QAPI module. This is part one!

Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6

- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)

In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.

Quoted 9 lines

Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.

Notes:

- After patch 07, isort -c should pass 100% on this and every future commit.

FIY, I went on to replicate/validate your testing, and I ran on patches 07 an later:

  isort --check-only --settings-path ./scripts/qapi/.isort.cfg --recursive ./scripts/qapi

With success.

Quoted 3 lines

- After patch 08, flake8 qapi/ should pass 100% on this and every future commit.

Here on patches 08 an later, I've run:

flake8 ./scripts/qapi

And starting on patch 24 ("qapi/source.py: add type hint annotations") it complained about:

/scripts/qapi/source.py:44:31: F821 undefined name 'T'

Quoted 3 lines

- After patch 09, pylint --rcfile=qapi/pylintrc qapi/ should pass 100% on this and every future commit.

Here I ran on patches 09 and later:

  pylint --rcfile=./scripts/qapi/pylintrc ./scripts/qapi/

And all succeeded.

Quoted 3 lines

- After patch 18, mypy --config-file=qapi/mypy.ini qapi/ should pass 100% on this and every future commit.

Here I ran on patches 18 and later:

  mypy --config-file=./scripts/qapi/mypy.ini ./scripts/qapi/

And all succeeded.

- Cleber.

author
#122 lore raw
Quoted 8 lines · Cleber Rosa

Here on patches 08 an later, I've run:

flake8 ./scripts/qapi

And starting on patch 24 ("qapi/source.py: add type hint annotations") it complained about:

   /scripts/qapi/source.py:44:31: F821 undefined name 'T'

I think that must be flake8 < 3.8.0. Try using >= 3.8.0.

(Yes, still working on getting proper pipenv/requirements/something up for these, sorry.)

#123 lore raw
Quoted 12 lines · John Snow
Quoted 8 lines · Cleber Rosa

Here on patches 08 an later, I've run:

flake8 ./scripts/qapi

And starting on patch 24 ("qapi/source.py: add type hint annotations") it complained about:

   /scripts/qapi/source.py:44:31: F821 undefined name 'T'

I think that must be flake8 < 3.8.0. Try using >= 3.8.0.

ACK, I was on 3.7.9. They're all passing now with 3.8.4.

- Cleber.

#124 lore raw
In reply to John Snow
Quoted 10 lines · John Snow

Hi, this series adds static type hints to the QAPI module. This is part one!

Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6

- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)

Quoted 7 lines

In general, this series tackles the cleanup of one individual QAPI module at a time. Once it passes pylint or mypy checks, those checks are enabled for that file.

Type hints are added in patches that add *only* type hints and change no other behavior. Any necessary changes to behavior to accommodate typing are split out into their own tiny patches.

Neat and pleasant to review. The care you put into this shows. Thanks!