Skip to contentSkip to conversation

move qapi under python/qemu/

PATCH In review John Snow 5 participants 21 messages Aug 20, 2024 – Sep 3, 2024 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
author
#1 lore raw

PATCH 1/8 python/qapi: correct re.Match type hints for 3.13

In reply to John Snow

typing.Match was removed in Python 3.13, so we need to use re.Match instead. However, Python 3.8 doesn't support using re.Match as a type hint directly, so we need a conditional for now.

The import is written oddly so that "Match" is explicitly re-exported for re-use by other modules. mypy will complain otherwise.

---
 scripts/qapi/common.py | 10 +++++++++-
 scripts/qapi/parser.py |  3 +--
 2 files changed, 10 insertions(+), 3 deletions(-)
scripts/qapi/common.py +9 -1
index 737b059e629..444b3acf53a 100644
@@ -12,16 +12,24 @@
 # See the COPYING file in the top-level directory.
 
 import re
+import sys
 from typing import (
     Any,
     Dict,
-    Match,
     Optional,
     Sequence,
     Union,
 )
 
 
+if sys.version_info < (3, 9):
+    # typing.Match was removed in 3.13,
+    # but it's still a necessity in 3.8.
+    from typing import \
+        Match as Match  # pylint: disable=useless-import-alias
+else:
+    Match = re.Match
+
 #: Magic string that gets removed along with all space to its right.
 EATSPACE = '\033EATSPACE.'
 POINTER_SUFFIX = ' *' + EATSPACE
scripts/qapi/parser.py +1 -2
index adc85b5b394..9a42b119131 100644
@@ -23,13 +23,12 @@
     Dict,
     List,
     Mapping,
-    Match,
     Optional,
     Set,
     Union,
 )
 
-from .common import must_match
+from .common import Match, must_match
 from .error import QAPISemError, QAPISourceError
 from .source import QAPISourceInfo
 
Signature · 1 line
2.45.0

John Snow

author
#0 lore raw

cover move qapi under python/qemu/

Move the QAPI generator module to python/qemu/qapi so that it's checked by the same standard linter/static analysis regime as the other python code.

This has the additional side-effect of marking me as a co-maintainer of the QAPI generator.

John Snow (8):
python/qapi: correct re.Match type hints for 3.13 python/qapi: change "FIXME" to "TODO" python/qapi: add pylint pragmas
python/qapi: remove outdated pragmas python/qapi: ignore missing docstrings in pylint
python: allow short names for variables on older pylint python/qapi: move scripts/qapi to python/qemu/qapi python/qapi: remove redundant linter configuration

 MAINTAINERS                                 |  2 +-
 docs/conf.py                                |  2 +-
 docs/sphinx/qapidoc.py                      |  6 +-
 meson.build                                 | 28 +++++-----
 {scripts => python/qemu}/qapi/__init__.py   |  0
 {scripts => python/qemu}/qapi/commands.py   |  4 +-
 {scripts => python/qemu}/qapi/common.py     | 12 +++-
 {scripts => python/qemu}/qapi/error.py      |  0
 {scripts => python/qemu}/qapi/events.py     |  4 +-
 {scripts => python/qemu}/qapi/expr.py       |  3 +
 {scripts => python/qemu}/qapi/gen.py        |  4 +-
 {scripts => python/qemu}/qapi/introspect.py |  3 +
 {scripts => python/qemu}/qapi/main.py       |  2 +
 {scripts => python/qemu}/qapi/parser.py     |  9 ++-
 {scripts => python/qemu}/qapi/schema.py     |  2 +-
 {scripts => python/qemu}/qapi/source.py     |  2 +
 {scripts => python/qemu}/qapi/types.py      |  2 +
 {scripts => python/qemu}/qapi/visit.py      |  3 +
 python/setup.cfg                            | 22 ++++----
 scripts/qapi-gen.py                         |  4 +-
 scripts/qapi/.flake8                        |  3 -
 scripts/qapi/.isort.cfg                     |  7 ---
 scripts/qapi/mypy.ini                       |  4 --
 scripts/qapi/pylintrc                       | 61 ---------------------
 tests/qapi-schema/meson.build               |  2 +-
 tests/qapi-schema/test-qapi.py              |  4 +-
 26 files changed, 78 insertions(+), 117 deletions(-)
 rename {scripts => python/qemu}/qapi/__init__.py (100%)
 rename {scripts => python/qemu}/qapi/commands.py (99%)
 rename {scripts => python/qemu}/qapi/common.py (96%)
 rename {scripts => python/qemu}/qapi/error.py (100%)
 rename {scripts => python/qemu}/qapi/events.py (98%)
 rename {scripts => python/qemu}/qapi/expr.py (99%)
 rename {scripts => python/qemu}/qapi/gen.py (99%)
 rename {scripts => python/qemu}/qapi/introspect.py (99%)
 rename {scripts => python/qemu}/qapi/main.py (98%)
 rename {scripts => python/qemu}/qapi/parser.py (99%)
 rename {scripts => python/qemu}/qapi/schema.py (99%)
 rename {scripts => python/qemu}/qapi/source.py (98%)
 rename {scripts => python/qemu}/qapi/types.py (99%)
 rename {scripts => python/qemu}/qapi/visit.py (99%)
 delete mode 100644 scripts/qapi/.flake8
 delete mode 100644 scripts/qapi/.isort.cfg
 delete mode 100644 scripts/qapi/mypy.ini
 delete mode 100644 scripts/qapi/pylintrc
Signature · 1 line
2.45.0
author
#3 lore raw

PATCH 2/8 python/qapi: change "FIXME" to "TODO"

qemu.git/python/setup.cfg disallows checking in any code with "XXX", "FIXME" or "TODO" in the comments. Soften the restriction to only prohibit "FIXME", and change the two occurrences of "FIXME" in qapi to read "TODO" instead.

---
 python/setup.cfg         | 5 +++++
 scripts/qapi/commands.py | 2 +-
 scripts/qapi/events.py   | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)
python/setup.cfg +5 -0
index 3b4e2cc5501..72b58c98c99 100644
@@ -169,6 +169,11 @@ ignore-signatures=yes
 # TODO: Remove after we opt in to Pylint 2.8.3. See commit msg.
 min-similarity-lines=6
 
+[pylint.miscellaneous]
+
+# forbid FIXME/XXX comments, allow TODO.
+notes=FIXME,
+      XXX,
 
 [isort]
 force_grid_wrap=4
scripts/qapi/commands.py +1 -1
index 79951a841f5..cffed6cd3ba 100644
@@ -385,7 +385,7 @@ def visit_command(self,
                       coroutine: bool) -> None:
         if not gen:
             return
-        # FIXME: If T is a user-defined type, the user is responsible
+        # TODO: If T is a user-defined type, the user is responsible
         # for making this work, i.e. to make T's condition the
         # conjunction of the T-returning commands' conditions.  If T
         # is a built-in type, this isn't possible: the
scripts/qapi/events.py +1 -1
index d1f639981a9..36dc0c50c78 100644
@@ -84,7 +84,7 @@ def gen_event_send(name: str,
                    boxed: bool,
                    event_enum_name: str,
                    event_emit: str) -> str:
-    # FIXME: Our declaration of local variables (and of 'errp' in the
+    # TODO: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
     # practice, we can rename our local variables with a leading _ prefix,
Signature · 1 line
2.45.0
author
#11 lore raw

PATCH 6/8 python: allow short names for variables on older pylint

In reply to John Snow

Pylint >= 3.0.0 disabled this feature, but older pylint does not: allow short names by default by using a regex to do so.

Incidentally, this removes the need for most of the allow list we had before, so remove most of that, too.

---
 python/setup.cfg | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
python/setup.cfg +5 -11
index 72b58c98c99..58dba90f815 100644
@@ -148,17 +148,11 @@ disable=consider-using-f-string,
 
 [pylint.basic]
 # Good variable names which should always be accepted, separated by a comma.
-good-names=i,
-           j,
-           k,
-           ex,
-           Run,
-           _,   # By convention: Unused variable
-           fh,  # fh = open(...)
-           fd,  # fd = os.open(...)
-           c,   # for c in string: ...
-           T,   # for TypeVars. See pylint#3401
-           SocketAddrT,  # Not sure why this is invalid.
+good-names=SocketAddrT,  # Not sure why this is invalid.
+
+# pylint < 3.0 warns by default on short variable names.
+# Disable this for older versions.
+good-names-rgxs=^[_a-z][_a-z0-9]?$
 
 [pylint.similarities]
 # Ignore imports when computing similarities.
Signature · 1 line
2.45.0
author
#10 lore raw

PATCH 5/8 python/qapi: ignore missing docstrings in pylint

In reply to John Snow

Maybe temporary, I am not sure. Instead of disabling docstring checking *globally* for all of our python files, just disable it for QAPI modules.

---
 scripts/qapi/commands.py   | 2 ++
 scripts/qapi/common.py     | 2 ++
 scripts/qapi/events.py     | 2 ++
 scripts/qapi/expr.py       | 2 ++
 scripts/qapi/gen.py        | 2 ++
 scripts/qapi/introspect.py | 2 ++
 scripts/qapi/main.py       | 2 ++
 scripts/qapi/parser.py     | 2 ++
 scripts/qapi/schema.py     | 2 +-
 scripts/qapi/source.py     | 2 ++
 scripts/qapi/types.py      | 2 ++
 scripts/qapi/visit.py      | 2 ++
 12 files changed, 23 insertions(+), 1 deletion(-)
scripts/qapi/commands.py +2 -0
index cffed6cd3ba..b01de93c965 100644
@@ -13,6 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
+# pylint: disable=missing-docstring
+
 from typing import (
     Dict,
     List,
scripts/qapi/common.py +2 -0
index 444b3acf53a..918a1ab728a 100644
@@ -11,6 +11,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+# pylint: disable=missing-docstring
+
 import re
 import sys
 from typing import (
scripts/qapi/events.py +2 -0
index 36dc0c50c78..9ab9ff4e695 100644
@@ -12,6 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
+# pylint: disable=missing-docstring
+
 from typing import List, Optional
 
 from .common import c_enum_const, c_name, mcgen
scripts/qapi/expr.py +2 -0
index f60e580dd36..c137dcf950a 100644
@@ -13,6 +13,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+# pylint: disable=missing-docstring
+
 """
 Normalize and validate (context-free) QAPI schema expression structures.
 
scripts/qapi/gen.py +2 -0
index ce94aee8e70..f869c751e53 100644
@@ -11,6 +11,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+# pylint: disable=missing-docstring
+
 from contextlib import contextmanager
 import os
 import re
scripts/qapi/introspect.py +2 -0
index 9d499f90b7c..44edc42d18b 100644
@@ -11,6 +11,8 @@
 See the COPYING file in the top-level directory.
 """
 
+# pylint: disable=missing-docstring
+
 from typing import (
     Any,
     Dict,
scripts/qapi/main.py +2 -0
index 316736b6a29..24ffa15aa2c 100644
@@ -7,6 +7,8 @@
 This is the main entry point for generating C code from the QAPI schema.
 """
 
+# pylint: disable=missing-docstring
+
 import argparse
 import sys
 from typing import Optional
scripts/qapi/parser.py +2 -0
index 9bb039fe8d3..9113c9d1506 100644
@@ -14,6 +14,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+# pylint: disable=missing-docstring
+
 from collections import OrderedDict
 import os
 import re
scripts/qapi/schema.py +1 -1
index d65c35f6ee6..a06b3e30ffd 100644
@@ -12,7 +12,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-# pylint: disable=too-many-lines
+# pylint: disable=too-many-lines, missing-docstring
 
 # TODO catching name collisions in generated code would be nice
 
scripts/qapi/source.py +2 -0
index 7b379fdc925..ad7f4e1a0e5 100644
@@ -9,6 +9,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+# pylint: disable=missing-docstring
+
 import copy
 from typing import List, Optional, TypeVar
 
scripts/qapi/types.py +2 -0
index 0dd0b00ada3..abba5983870 100644
@@ -13,6 +13,8 @@
 # See the COPYING file in the top-level directory.
 """
 
+# pylint: disable=missing-docstring
+
 from typing import List, Optional
 
 from .common import c_enum_const, c_name, mcgen
scripts/qapi/visit.py +2 -0
index 20ce6be9978..851f3b01e97 100644
@@ -13,6 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
+# pylint: disable=missing-docstring
+
 from typing import List, Optional
 
 from .common import (
Signature · 1 line
2.45.0
author
#20 lore raw

PATCH 8/8 python/qapi: remove redundant linter configuration

In reply to John Snow

Now that the qemu.qapi module is checked by the standard python tests, we don't need separate configuration for it anymore.

---
 python/qemu/qapi/.flake8    |  3 --
 python/qemu/qapi/.isort.cfg |  7 -----
 python/qemu/qapi/mypy.ini   |  4 ---
 python/qemu/qapi/pylintrc   | 61 -------------------------------------
 4 files changed, 75 deletions(-)
 delete mode 100644 python/qemu/qapi/.flake8
 delete mode 100644 python/qemu/qapi/.isort.cfg
 delete mode 100644 python/qemu/qapi/mypy.ini
 delete mode 100644 python/qemu/qapi/pylintrc
python/qemu/qapi/.flake8 deleted +0 -3
deleted file mode 100644
index a873ff67309..00000000000
@@ -1,3 +0,0 @@
-[flake8]
-# Prefer pylint's bare-except checks to flake8's
-extend-ignore = E722
python/qemu/qapi/.isort.cfg deleted +0 -7
deleted file mode 100644
index 643caa1fbd6..00000000000
@@ -1,7 +0,0 @@
-[settings]
-force_grid_wrap=4
-force_sort_within_sections=True
-include_trailing_comma=True
-line_length=72
-lines_after_imports=2
-multi_line_output=3
python/qemu/qapi/mypy.ini deleted +0 -4
deleted file mode 100644
index 8109470a031..00000000000
@@ -1,4 +0,0 @@
-[mypy]
-strict = True
-disallow_untyped_calls = False
-python_version = 3.8
python/qemu/qapi/pylintrc deleted +0 -61

61 lines not on this page · raw message

Signature · 1 line
2.45.0
author
#9 lore raw

PATCH 4/8 python/qapi: remove outdated pragmas

In reply to John Snow

These pragmas are no longer neccessary under our current linter/static analysis versions; they can be removed.

---
 scripts/qapi/gen.py | 2 --
 1 file changed, 2 deletions(-)
scripts/qapi/gen.py +0 -2
index 6a8abe00415..ce94aee8e70 100644
@@ -62,11 +62,9 @@ def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
     def _top(self) -> str:
-        # pylint: disable=no-self-use
         return ''
 
     def _bottom(self) -> str:
-        # pylint: disable=no-self-use
         return ''
 
     def write(self, output_dir: str) -> None:
Signature · 1 line
2.45.0
author
#8 lore raw

PATCH 3/8 python/qapi: add pylint pragmas

In reply to John Snow

We are preparing to move the QAPI generator code into qemu.git/python/qemu/qapi.

The qemu.git/python pylint configuration is stricter than the current qapi generator configuration. These additional pragmas bridge the gap without requiring us to loosen the requirements in the python directory.

---
 scripts/qapi/expr.py       | 1 +
 scripts/qapi/introspect.py | 1 +
 scripts/qapi/parser.py     | 4 ++++
 scripts/qapi/visit.py      | 1 +
 4 files changed, 7 insertions(+)
scripts/qapi/expr.py +1 -0
index cae0a083591..f60e580dd36 100644
@@ -604,6 +604,7 @@ def check_exprs(exprs: List[QAPIExpression]) -> List[QAPIExpression]:
     :raise QAPISemError: When any expression fails validation.
     :return: The same list of expressions (now modified).
     """
+    # pylint: disable=too-many-branches
     for expr in exprs:
         info = expr.info
         doc = expr.doc
scripts/qapi/introspect.py +1 -0
index ac14b20f308..9d499f90b7c 100644
@@ -106,6 +106,7 @@ def _tree_to_qlit(obj: JSONValue,
     :param dict_value: True when the value being processed belongs to a
                        dict key; which suppresses the output indent.
     """
+    # pylint: disable=too-many-branches
 
     def indent(level: int) -> str:
         return level * 4 * ' '
scripts/qapi/parser.py +4 -0
index 9a42b119131..9bb039fe8d3 100644
@@ -118,6 +118,8 @@ def _parse(self) -> None:
 
         :return: None.  Results are stored in ``.exprs`` and ``.docs``.
         """
+        # pylint: disable=too-many-branches
+
         cur_doc = None
 
         # May raise OSError; allow the caller to handle it.
@@ -290,6 +292,7 @@ def accept(self, skip_comment: bool = True) -> None:
 
             ``.tok`` and ``.val`` will both be None at EOF.
         """
+        # pylint: disable=too-many-branches
         while True:
             self.tok = self.src[self.cursor]
             self.pos = self.cursor
@@ -478,6 +481,7 @@ def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:
             doc.append_line(line)
 
     def get_doc(self) -> 'QAPIDoc':
+        # pylint: disable=too-many-statements,too-many-branches
         if self.val != '##':
             raise QAPIParseError(
                 self, "junk after '##' at start of documentation comment")
scripts/qapi/visit.py +1 -0
index 12f92e429f6..20ce6be9978 100644
@@ -65,6 +65,7 @@ def gen_visit_object_members(name: str,
                              base: Optional[QAPISchemaObjectType],
                              members: List[QAPISchemaObjectTypeMember],
                              branches: Optional[QAPISchemaBranches]) -> str:
+    # pylint: disable=too-many-branches
     ret = mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
Signature · 1 line
2.45.0
author
#12 lore raw

PATCH 7/8 python/qapi: move scripts/qapi to python/qemu/qapi

In reply to John Snow

This is being done for the sake of unifying the linting and static type analysis configurations between scripts/qapi and python/qemu/*.

With this change, the qapi module will now be checked by mypy, flake8, pylint, isort etc under all python versions from 3.8 through 3.13 under a variety of different dependency configurations in the GitLab testing pipelines.

The tests can be run locally, as always:

Quoted 4 lines

cd qemu.git/python
make check-minreqs
make check-tox
make check-dev

"check-minreqs" is the must-pass GitLab test. "check-tox" is the optional allowed-to-fail GitLab test.

---
 MAINTAINERS                                 |  2 +-
 docs/conf.py                                |  2 +-
 docs/sphinx/qapidoc.py                      |  6 ++---
 meson.build                                 | 28 ++++++++++-----------
 {scripts => python/qemu}/qapi/.flake8       |  0
 {scripts => python/qemu}/qapi/.isort.cfg    |  0
 {scripts => python/qemu}/qapi/__init__.py   |  0
 {scripts => python/qemu}/qapi/commands.py   |  0
 {scripts => python/qemu}/qapi/common.py     |  0
 {scripts => python/qemu}/qapi/error.py      |  0
 {scripts => python/qemu}/qapi/events.py     |  0
 {scripts => python/qemu}/qapi/expr.py       |  0
 {scripts => python/qemu}/qapi/gen.py        |  0
 {scripts => python/qemu}/qapi/introspect.py |  0
 {scripts => python/qemu}/qapi/main.py       |  0
 {scripts => python/qemu}/qapi/mypy.ini      |  0
 {scripts => python/qemu}/qapi/parser.py     |  0
 {scripts => python/qemu}/qapi/pylintrc      |  0
 {scripts => python/qemu}/qapi/schema.py     |  0
 {scripts => python/qemu}/qapi/source.py     |  0
 {scripts => python/qemu}/qapi/types.py      |  0
 {scripts => python/qemu}/qapi/visit.py      |  0
 python/setup.cfg                            |  1 +
 scripts/qapi-gen.py                         |  4 ++-
 tests/qapi-schema/meson.build               |  2 +-
 tests/qapi-schema/test-qapi.py              |  4 +--
 26 files changed, 26 insertions(+), 23 deletions(-)
 rename {scripts => python/qemu}/qapi/.flake8 (100%)
 rename {scripts => python/qemu}/qapi/.isort.cfg (100%)
 rename {scripts => python/qemu}/qapi/__init__.py (100%)
 rename {scripts => python/qemu}/qapi/commands.py (100%)
 rename {scripts => python/qemu}/qapi/common.py (100%)
 rename {scripts => python/qemu}/qapi/error.py (100%)
 rename {scripts => python/qemu}/qapi/events.py (100%)
 rename {scripts => python/qemu}/qapi/expr.py (100%)
 rename {scripts => python/qemu}/qapi/gen.py (100%)
 rename {scripts => python/qemu}/qapi/introspect.py (100%)
 rename {scripts => python/qemu}/qapi/main.py (100%)
 rename {scripts => python/qemu}/qapi/mypy.ini (100%)
 rename {scripts => python/qemu}/qapi/parser.py (100%)
 rename {scripts => python/qemu}/qapi/pylintrc (100%)
 rename {scripts => python/qemu}/qapi/schema.py (100%)
 rename {scripts => python/qemu}/qapi/source.py (100%)
 rename {scripts => python/qemu}/qapi/types.py (100%)
 rename {scripts => python/qemu}/qapi/visit.py (100%)
MAINTAINERS +1 -1
index 3584d6a6c6d..1912940631d 100644
@@ -3214,7 +3214,7 @@ F: tests/unit/test-qapi-*.c
 F: tests/unit/test-qmp-*.c
 F: tests/unit/test-visitor-serialization.c
 F: scripts/qapi-gen.py
-F: scripts/qapi/*
+F: python/qemu/qapi/*
 F: docs/sphinx/qapidoc.py
 F: docs/devel/qapi*
 T: git https://repo.or.cz/qemu/armbru.git qapi-next
docs/conf.py +1 -1
index 876f6768815..6600db976b3 100644
@@ -46,7 +46,7 @@
 # Our extensions are in docs/sphinx; the qapidoc extension requires
 # the QAPI modules from scripts/.
 sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
-sys.path.insert(0, os.path.join(qemu_docdir, "../scripts"))
+sys.path.insert(0, os.path.join(qemu_docdir, "../python"))
 
 
 # -- General configuration ------------------------------------------------
docs/sphinx/qapidoc.py +3 -3
index 738b2450fb1..777fd1ac836 100644
@@ -33,9 +33,9 @@
 from docutils import nodes
 from docutils.parsers.rst import Directive, directives
 from docutils.statemachine import ViewList
-from qapi.error import QAPIError, QAPISemError
-from qapi.gen import QAPISchemaVisitor
-from qapi.schema import QAPISchema
+from qemu.qapi.error import QAPIError, QAPISemError
+from qemu.qapi.gen import QAPISchemaVisitor
+from qemu.qapi.schema import QAPISchema
 
 from sphinx import addnodes
 from sphinx.directives.code import CodeBlock
meson.build +14 -14
index fbda17c987e..f96c9bebe0c 100644
@@ -3274,20 +3274,20 @@ genh += configure_file(output: 'config-host.h', configuration: config_host_data)
 hxtool = find_program('scripts/hxtool')
 shaderinclude = find_program('scripts/shaderinclude.py')
 qapi_gen = find_program('scripts/qapi-gen.py')
-qapi_gen_depends = [ meson.current_source_dir() / 'scripts/qapi/__init__.py',
-                     meson.current_source_dir() / 'scripts/qapi/commands.py',
-                     meson.current_source_dir() / 'scripts/qapi/common.py',
-                     meson.current_source_dir() / 'scripts/qapi/error.py',
-                     meson.current_source_dir() / 'scripts/qapi/events.py',
-                     meson.current_source_dir() / 'scripts/qapi/expr.py',
-                     meson.current_source_dir() / 'scripts/qapi/gen.py',
-                     meson.current_source_dir() / 'scripts/qapi/introspect.py',
-                     meson.current_source_dir() / 'scripts/qapi/main.py',
-                     meson.current_source_dir() / 'scripts/qapi/parser.py',
-                     meson.current_source_dir() / 'scripts/qapi/schema.py',
-                     meson.current_source_dir() / 'scripts/qapi/source.py',
-                     meson.current_source_dir() / 'scripts/qapi/types.py',
-                     meson.current_source_dir() / 'scripts/qapi/visit.py',
+qapi_gen_depends = [ meson.current_source_dir() / 'python/qemu/qapi/__init__.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/commands.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/common.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/error.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/events.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/expr.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/gen.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/introspect.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/main.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/parser.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/schema.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/source.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/types.py',
+                     meson.current_source_dir() / 'python/qemu/qapi/visit.py',
                      meson.current_source_dir() / 'scripts/qapi-gen.py'
 ]
 
python/qemu/qapi/.flake8 renamed from scripts/qapi/.flake8 +0 -0
similarity index 100%
rename from scripts/qapi/.flake8
rename to python/qemu/qapi/.flake8
python/qemu/qapi/.isort.cfg renamed from scripts/qapi/.isort.cfg +0 -0
similarity index 100%
rename from scripts/qapi/.isort.cfg
rename to python/qemu/qapi/.isort.cfg
python/qemu/qapi/__init__.py renamed from scripts/qapi/__init__.py +0 -0
similarity index 100%
rename from scripts/qapi/__init__.py
rename to python/qemu/qapi/__init__.py
python/qemu/qapi/commands.py renamed from scripts/qapi/commands.py +0 -0
similarity index 100%
rename from scripts/qapi/commands.py
rename to python/qemu/qapi/commands.py
python/qemu/qapi/common.py renamed from scripts/qapi/common.py +0 -0
similarity index 100%
rename from scripts/qapi/common.py
rename to python/qemu/qapi/common.py
python/qemu/qapi/error.py renamed from scripts/qapi/error.py +0 -0
similarity index 100%
rename from scripts/qapi/error.py
rename to python/qemu/qapi/error.py
python/qemu/qapi/events.py renamed from scripts/qapi/events.py +0 -0
similarity index 100%
rename from scripts/qapi/events.py
rename to python/qemu/qapi/events.py
python/qemu/qapi/expr.py renamed from scripts/qapi/expr.py +0 -0
similarity index 100%
rename from scripts/qapi/expr.py
rename to python/qemu/qapi/expr.py
python/qemu/qapi/gen.py renamed from scripts/qapi/gen.py +0 -0
similarity index 100%
rename from scripts/qapi/gen.py
rename to python/qemu/qapi/gen.py
python/qemu/qapi/introspect.py renamed from scripts/qapi/introspect.py +0 -0
similarity index 100%
rename from scripts/qapi/introspect.py
rename to python/qemu/qapi/introspect.py
python/qemu/qapi/main.py renamed from scripts/qapi/main.py +0 -0
similarity index 100%
rename from scripts/qapi/main.py
rename to python/qemu/qapi/main.py
python/qemu/qapi/mypy.ini renamed from scripts/qapi/mypy.ini +0 -0
similarity index 100%
rename from scripts/qapi/mypy.ini
rename to python/qemu/qapi/mypy.ini
python/qemu/qapi/parser.py renamed from scripts/qapi/parser.py +0 -0
similarity index 100%
rename from scripts/qapi/parser.py
rename to python/qemu/qapi/parser.py
python/qemu/qapi/pylintrc renamed from scripts/qapi/pylintrc +0 -0
similarity index 100%
rename from scripts/qapi/pylintrc
rename to python/qemu/qapi/pylintrc
python/qemu/qapi/schema.py renamed from scripts/qapi/schema.py +0 -0
similarity index 100%
rename from scripts/qapi/schema.py
rename to python/qemu/qapi/schema.py
python/qemu/qapi/source.py renamed from scripts/qapi/source.py +0 -0
similarity index 100%
rename from scripts/qapi/source.py
rename to python/qemu/qapi/source.py
python/qemu/qapi/types.py renamed from scripts/qapi/types.py +0 -0
similarity index 100%
rename from scripts/qapi/types.py
rename to python/qemu/qapi/types.py
python/qemu/qapi/visit.py renamed from scripts/qapi/visit.py +0 -0
similarity index 100%
rename from scripts/qapi/visit.py
rename to python/qemu/qapi/visit.py
python/setup.cfg +1 -0
index 58dba90f815..d1582f74ddd 100644
@@ -28,6 +28,7 @@ packages =
     qemu.qmp
     qemu.machine
     qemu.utils
+    qemu.qapi
 
 [options.package_data]
 * = py.typed
scripts/qapi-gen.py +3 -1
index f3518d29a54..42912c91716 100644
@@ -11,9 +11,11 @@
 execution environment.
 """
 
+import os
 import sys
 
-from qapi import main
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
+from qemu.qapi import main
 
 if __name__ == '__main__':
     sys.exit(main.main())
tests/qapi-schema/meson.build +1 -1
index 0f479d93170..080444d0cd4 100644
@@ -1,5 +1,5 @@
 test_env = environment()
-test_env.set('PYTHONPATH', meson.project_source_root() / 'scripts')
+test_env.set('PYTHONPATH', meson.project_source_root() / 'python')
 test_env.set('PYTHONIOENCODING', 'utf-8')
 
 schemas = [
tests/qapi-schema/test-qapi.py +2 -2
index 7e3f9f4aa1f..dcc0d949971 100755
@@ -18,8 +18,8 @@
 import sys
 from io import StringIO
 
-from qapi.error import QAPIError
-from qapi.schema import QAPISchema, QAPISchemaVisitor
+from qemu.qapi.error import QAPIError
+from qemu.qapi.schema import QAPISchema, QAPISchemaVisitor
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
Signature · 1 line
2.45.0
Reviewed-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] python/qapi: correct re.Match type hints for 3.13

In reply to John Snow
Quoted 4 lines · John Snow

typing.Match was removed in Python 3.13, so we need to use re.Match instead. However, Python 3.8 doesn't support using re.Match as a type hint directly, so we need a conditional for now.

Quoted 8 lines

The import is written oddly so that "Match" is explicitly re-exported for re-use by other modules. mypy will complain otherwise.

---
  scripts/qapi/common.py | 10 +++++++++-
  scripts/qapi/parser.py |  3 +--
  2 files changed, 10 insertions(+), 3 deletions(-)
#4 lore raw

Re: [PATCH 2/8] python/qapi: change "FIXME" to "TODO"

In reply to John Snow
Quoted 6 lines · John Snow

qemu.git/python/setup.cfg disallows checking in any code with "XXX", "FIXME" or "TODO" in the comments. Soften the restriction to only prohibit "FIXME", and change the two occurrences of "FIXME" in qapi to read "TODO" instead.

I don't like forbidding FIXME comments. It's as futile as forbidding known bugs. All it can accomplish is making people use other, and likely less clear ways to document them.

Perhaps projects exist that use FIXME comments only for known bugs in uncommitted code. To me, that feels *nuts*. I commit all kinds of crap in my tree. I don't need silly "make check" failures while I develop, the non-silly ones cause enough friction already.

In fact, we're quite happy to use FIXME comments even in merged code:

    $ git-grep FIXME | wc -l
    494

I can't see why python/ should be different.

Quoted 40 lines
---
 python/setup.cfg         | 5 +++++
 scripts/qapi/commands.py | 2 +-
 scripts/qapi/events.py   | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)
python/setup.cfg +5 -0
index 3b4e2cc5501..72b58c98c99 100644
@@ -169,6 +169,11 @@ ignore-signatures=yes
 # TODO: Remove after we opt in to Pylint 2.8.3. See commit msg.
 min-similarity-lines=6
 
+[pylint.miscellaneous]
+
+# forbid FIXME/XXX comments, allow TODO.
+notes=FIXME,
+      XXX,
 
 [isort]
 force_grid_wrap=4
scripts/qapi/commands.py +1 -1
index 79951a841f5..cffed6cd3ba 100644
@@ -385,7 +385,7 @@ def visit_command(self,
                       coroutine: bool) -> None:
         if not gen:
             return
-        # FIXME: If T is a user-defined type, the user is responsible
+        # TODO: If T is a user-defined type, the user is responsible
         # for making this work, i.e. to make T's condition the
         # conjunction of the T-returning commands' conditions.  If T
         # is a built-in type, this isn't possible: the
scripts/qapi/events.py +0 -0
index d1f639981a9..36dc0c50c78 100644

scripts/qapi/events.py def gen_event_send(name: str, · line 90

Quoted 8 lines
scripts/qapi/events.py +1 -1
@@ -84,7 +84,7 @@ def gen_event_send(name: str,
                    boxed: bool,
                    event_enum_name: str,
                    event_emit: str) -> str:
-    # FIXME: Our declaration of local variables (and of 'errp' in the
+    # TODO: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
     # practice, we can rename our local variables with a leading _ prefix,

Starting a comment with TODO tells me there's work to do.

Starting it with FIXME tells me there's a bug to fix. That's more specific. Replacing FIXME by TODO loses information.

#13 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

In reply to John Snow
Quoted 15 lines · John Snow

This is being done for the sake of unifying the linting and static type analysis configurations between scripts/qapi and python/qemu/*.

With this change, the qapi module will now be checked by mypy, flake8, pylint, isort etc under all python versions from 3.8 through 3.13 under a variety of different dependency configurations in the GitLab testing pipelines.

The tests can be run locally, as always:

Quoted 4 lines

cd qemu.git/python
make check-minreqs
make check-tox
make check-dev

Quoted 4 lines

"check-minreqs" is the must-pass GitLab test. "check-tox" is the optional allowed-to-fail GitLab test.

I don't understand why we have to keep Python code in its own directory just to get it checked. We wouldn't do that say for Rust code, would we? Anyway, if it's the price of checking, I'll pay[*].

[...]

Quoted 7 lines
scripts/qapi-gen.py +0 -0
index f3518d29a54..42912c91716 100644
@@ -11,9 +11,11 @@
 execution environment.
 """

scripts/qapi-gen.py line 21

Quoted 10 lines
scripts/qapi-gen.py +3 -1
 
+import os
 import sys
 
-from qapi import main
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
+from qemu.qapi import main
 
 if __name__ == '__main__':
     sys.exit(main.main())

Suggest to use the opportunity to rename to just qapi-gen (no .py) and chmod +x, possibly in a separate patch.

[...]

[*] Grudgingly.

#14 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

Quoted 17 lines · Markus Armbruster
Quoted 15 lines · John Snow

This is being done for the sake of unifying the linting and static type analysis configurations between scripts/qapi and python/qemu/*.

With this change, the qapi module will now be checked by mypy, flake8, pylint, isort etc under all python versions from 3.8 through 3.13 under a variety of different dependency configurations in the GitLab testing pipelines.

The tests can be run locally, as always:

Quoted 4 lines

cd qemu.git/python
make check-minreqs
make check-tox
make check-dev

Quoted 8 lines
Quoted 4 lines

"check-minreqs" is the must-pass GitLab test. "check-tox" is the optional allowed-to-fail GitLab test.

I don't understand why we have to keep Python code in its own directory just to get it checked. We wouldn't do that say for Rust code, would we? Anyway, if it's the price of checking, I'll pay[*].

The 'check-tox' target is defined in python/Makefile, and is written to only check code below that location, which is a somewhat arbitrary choice.

Having this in "make" at all is a bit outdated. Ideally all the targets in python/Makefile should be converted into meson targets and/or tests, in a "python" suite.

IOW, we should make 'check-tox' a target in meson.build at the top level, and have it check any .py file in the source tree, with an exclude list for files we know are not "clean" yet, so we don't have to move stuff around as we clean up individual files.

Quoted 24 lines

[...]

Quoted 7 lines
scripts/qapi-gen.py +0 -0
index f3518d29a54..42912c91716 100644
@@ -11,9 +11,11 @@
 execution environment.
 """

scripts/qapi-gen.py line 21

Quoted 10 lines
scripts/qapi-gen.py +3 -1
 
+import os
 import sys
 
-from qapi import main
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
+from qemu.qapi import main
 
 if __name__ == '__main__':
     sys.exit(main.main())

Suggest to use the opportunity to rename to just qapi-gen (no .py) and chmod +x, possibly in a separate patch.

Quoted 5 lines

[...]

[*] Grudgingly.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#15 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

Quoted 27 lines · Daniel P. Berrangé
Quoted 17 lines · Markus Armbruster
Quoted 15 lines · John Snow

This is being done for the sake of unifying the linting and static type analysis configurations between scripts/qapi and python/qemu/*.

With this change, the qapi module will now be checked by mypy, flake8, pylint, isort etc under all python versions from 3.8 through 3.13 under a variety of different dependency configurations in the GitLab testing pipelines.

The tests can be run locally, as always:

Quoted 4 lines

cd qemu.git/python
make check-minreqs
make check-tox
make check-dev

Quoted 8 lines
Quoted 4 lines

"check-minreqs" is the must-pass GitLab test. "check-tox" is the optional allowed-to-fail GitLab test.

I don't understand why we have to keep Python code in its own directory just to get it checked. We wouldn't do that say for Rust code, would we? Anyway, if it's the price of checking, I'll pay[*].

Quoted 8 lines

The 'check-tox' target is defined in python/Makefile, and is written to only check code below that location, which is a somewhat arbitrary choice.

Having this in "make" at all is a bit outdated. Ideally all the targets in python/Makefile should be converted into meson targets and/or tests, in a "python" suite.

Yes, ideally, but there are still gaps between the python tooling ecosystem and our own conventions in the QEMU source tree. Saying "everything under python/ should adhere to python ecosystem conventions" was just an extremely convenient way to make python tooling not throw a hissy fit over stuff in that directory. The "make" targets in that directory are really just simple script invocations that we don't really need "make" for at all, it's just so I don't have to re-explain how to invoke the tests as they stand. I don't think it's "outdated" in that sense, it's just a nice little syntactic shorthand using a tool people are familiar with - not too far out from what we do for the VM tests.

One of the barriers to implementing this in "make check" et al is setting up the package installation needed for e.g. mypy and the optional feature packages (like urwid, fusepy, etc) in a way that is unobtrusive to the build system. The build-time configure venv was a step in that direction, but the work remains unfinished.

I am in the process (right now!) of setting up mkvenv to install the in-tree python packages to the configure venv - one of the barriers *here* is that this is a little bit slow with older setuptools, and due to very tumultuous changes in Python packaging in the last several years (fallout from PEP-517, PEP-518 and PEP-660) is that it is taking me a good long while to ensure it's rock solid across all of our supported platforms while supporting isolated offline builds. (Python stuff was really, really, really not designed to cope very well with that restriction, I have learned in the last few years.)

Another reason for the decision to treat the python/ subdirectory as a "mini-repository" was purely for sake of ease with splitting out subpackages that became useful beyond the confines of our castle walls:
qemu.qmp in particular. By structuring everything here like an upstream package, it becomes pretty trivial to fork things out into standalone packages, because they're already structured identically to how standalone packages would be.

TLDR: "It makes my life easier to follow python ecosystem norms in this directory, but I still want to integrate it more formally to the QEMU build system and am working on that."

Quoted 7 lines

IOW, we should make 'check-tox' a target in meson.build at the top level, and have it check any .py file in the source tree, with an exclude list for files we know are not "clean" yet, so we don't have to move stuff around as we clean up individual files.

Maybe someday.

Right now, I consider anything under python/ to be "maintained" for the python ecosystem and anything outside of this to be "good luck!".

I do not volunteer to apply the same level of effort I do for python/ to *every last python script in the tree*. The standards I apply in python/ are vastly more strict than those that could apply to everything else; I don't think I could handle the workload of polishing every last python thing in the tree up to the same standard. qemu.qmp, qemu.machine and qapi are just vastly more important than the majority of one-off scripts that automate niche tasks I have no domain expertise in.

For any python scripts with import dependencies on other in-tree modules, it is useful to structure those imported modules as normal packages so that mypy, pylint, etc can function with minimal fuss and without needing lots of prone-to-breakage environment hacks and custom launcher scripts to make them work. Anything that *doesn't* need to import anything else from the tree is relatively easy to check where it lives, I just don't do that at the moment. It's possible we could make a separate test that applies a much lower bar of type checking and linting that really only catches *hard* errors and applies to everything in the tree automatically, but I don't think I want to lower the bar for the stuff already in python/.

That said, I *do* want to move the python tests into our usual build testing infrastructure, there's just some more work to do with mkvenv to bridge the gap, which I'm working on currently.

Quoted 43 lines
Quoted 12 lines

[...]

Quoted 8 lines
scripts/qapi-gen.py +0 -0
index f3518d29a54..42912c91716 100644
@@ -11,9 +11,11 @@
 execution environment.
 """
 
Quoted 5 lines

scripts/qapi-gen.py

Quoted 5 lines
scripts/qapi-gen.py +2 -1
+import os
 import sys
 
-from qapi import main
+sys.path.append(os.path.join(os.path.dirname(__file__), '..',

'python'))

Quoted 5 lines

scripts/qapi-gen.py line 21

Quoted 4 lines
scripts/qapi-gen.py +1 -0
+from qemu.qapi import main
 
 if __name__ == '__main__':
     sys.exit(main.main())
Quoted 8 lines

Suggest to use the opportunity to rename to just qapi-gen (no .py) and chmod +x, possibly in a separate patch.

[...]

[*] Grudgingly.

With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|

author
#16 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

In reply to Markus Armbruster
Quoted 17 lines · Markus Armbruster
Quoted 15 lines · John Snow

This is being done for the sake of unifying the linting and static type analysis configurations between scripts/qapi and python/qemu/*.

With this change, the qapi module will now be checked by mypy, flake8, pylint, isort etc under all python versions from 3.8 through 3.13 under a variety of different dependency configurations in the GitLab testing pipelines.

The tests can be run locally, as always:

Quoted 4 lines

cd qemu.git/python
make check-minreqs
make check-tox
make check-dev

Quoted 9 lines
Quoted 4 lines

"check-minreqs" is the must-pass GitLab test. "check-tox" is the optional allowed-to-fail GitLab test.

I don't understand why we have to keep Python code in its own directory just to get it checked. We wouldn't do that say for Rust code, would we? Anyway, if it's the price of checking, I'll pay[*].

Gave Dan a related answer. For you, my explanation is:

- It's nice to have just one configuration for static analysis in just one place
- It's nice to have that configuration follow python ecosystem norms
- It's nice to use standard python management tools to configure and test the supported versions of static analysis tools, again kept in one place.
- Just moving the folder costs virtually nothing.
- Moving it here makes it easier for me to test the eventual integration with make check in one place.
- I like being able to say that anything under python/ is fiercely guarded by high standards (and the gitlab pipelines) and everything else is not. I consider this to be organizationally simple and easy to communicate. i.e., I find it attractive to say that "python is maintained, scripts are YMMV." I am *not* willing to maintain everything under scripts/ with the same level of effort I apply to python/. I think it's important to allow people to commit low-development-cost scripts ("contrib quality") that they can run from time to time and not everything needs to be held to a crystalline perfect standard, but some stuff does.

If I'm being honest, I also just don't want to develop more testing infrastructure and scaffolding to start picking up scattershot python modules from elsewhere in the tree. I'd rather bring qapi into the fold and then continue working on integrating python/ static analysis tests to the make check suite instead. I've spent enough time already writing and carrying around my little ad-hoc static analysis scripts for qapi during the strict typing conversion, and ensuring static analysis passes with totally arbitrary versions of whatever tools the user happens to have installed sounds like a colossal pain. I already have a system set up to do all of the environment prep work so that it Just Works :tm: and is tested across a large matrix of tooling versions to ensure it continues to work both locally (for developer ease) and in the gitlab pipeline (for rigorous testing) with both forms of test readily accessible in the local developer
environment: I'd deeply appreciate just being able to let that system do what I designed it to do.

This series is 99% "converge on a static analysis configuration standard" and 1% "move it into place so it starts being checked regularly." I think that's worth a simple "git mv", honestly. Do we each lose some control over our preferred standard of formatting? Yes, but we gain consistency and ease of testing.

As for rust: I dunno! I imagine there are similar benefits there for modeling things as standards compliant packages, too. I'm not doing rust tooling right now, so I can't say.

Quoted 24 lines

[...]

Quoted 7 lines
scripts/qapi-gen.py +0 -0
index f3518d29a54..42912c91716 100644
@@ -11,9 +11,11 @@
 execution environment.
 """

scripts/qapi-gen.py line 21

Quoted 10 lines
scripts/qapi-gen.py +3 -1
 
+import os
 import sys
 
-from qapi import main
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
+from qemu.qapi import main
 
 if __name__ == '__main__':
     sys.exit(main.main())

Suggest to use the opportunity to rename to just qapi-gen (no .py) and chmod +x, possibly in a separate patch.

Quoted 5 lines

[...]

[*] Grudgingly.

Why the resistance? Is there some harm I've overlooked? This seems fairly benign to me.

author
#5 lore raw

Re: [PATCH 2/8] python/qapi: change "FIXME" to "TODO"

In reply to Markus Armbruster
Quoted 75 lines · Markus Armbruster
Quoted 6 lines · John Snow

qemu.git/python/setup.cfg disallows checking in any code with "XXX", "FIXME" or "TODO" in the comments. Soften the restriction to only prohibit "FIXME", and change the two occurrences of "FIXME" in qapi to read "TODO" instead.

I don't like forbidding FIXME comments. It's as futile as forbidding known bugs. All it can accomplish is making people use other, and likely less clear ways to document them.

Perhaps projects exist that use FIXME comments only for known bugs in uncommitted code. To me, that feels *nuts*. I commit all kinds of crap in my tree. I don't need silly "make check" failures while I develop, the non-silly ones cause enough friction already.

In fact, we're quite happy to use FIXME comments even in merged code:

    $ git-grep FIXME | wc -l
    494

I can't see why python/ should be different.

Quoted 40 lines
---
 python/setup.cfg         | 5 +++++
 scripts/qapi/commands.py | 2 +-
 scripts/qapi/events.py   | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)
python/setup.cfg +5 -0
index 3b4e2cc5501..72b58c98c99 100644
@@ -169,6 +169,11 @@ ignore-signatures=yes
 # TODO: Remove after we opt in to Pylint 2.8.3. See commit msg.
 min-similarity-lines=6
 
+[pylint.miscellaneous]
+
+# forbid FIXME/XXX comments, allow TODO.
+notes=FIXME,
+      XXX,
 
 [isort]
 force_grid_wrap=4
scripts/qapi/commands.py +1 -1
index 79951a841f5..cffed6cd3ba 100644
@@ -385,7 +385,7 @@ def visit_command(self,
                       coroutine: bool) -> None:
         if not gen:
             return
-        # FIXME: If T is a user-defined type, the user is responsible
+        # TODO: If T is a user-defined type, the user is responsible
         # for making this work, i.e. to make T's condition the
         # conjunction of the T-returning commands' conditions.  If T
         # is a built-in type, this isn't possible: the
scripts/qapi/events.py +0 -0
index d1f639981a9..36dc0c50c78 100644

scripts/qapi/events.py def gen_event_send(name: str, · line 90

Quoted 8 lines
scripts/qapi/events.py +1 -1
@@ -84,7 +84,7 @@ def gen_event_send(name: str,
                    boxed: bool,
                    event_enum_name: str,
                    event_emit: str) -> str:
-    # FIXME: Our declaration of local variables (and of 'errp' in the
+    # TODO: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
     # practice, we can rename our local variables with a leading _

prefix,

Quoted 5 lines

Starting a comment with TODO tells me there's work to do.

Starting it with FIXME tells me there's a bug to fix. That's more specific. Replacing FIXME by TODO loses information.

meh. I do use the "prohibit fixme" personally because I've the memory of a goldfish and I like setting up bombs for myself when I run tests, but willing to cede if it gets me what I want otherwise. I could be coerced to using "XXX" as my WIP testing bomb. Or maybe literally just adding "WIP" as a new bomb. Is that a fair trade?

There are likely other standards differences between the two subtrees, potentially things like documentation string length and so on -- I invite you to take a look at the setup.cfg file and tweak things to your liking and run "make check-minreqs" to see what barks, if anything.

After you run that command, you can type "source .dev-venv/bin/activate.sh" (or .fish or whatever) and then "pylint --generate-rcfile | less" to get a sample config and see all of the buttons, knobs and levers you could pull. You can leave the environment when you're done with "deactivate".

Mentioning this only because there have been times in the past that my formatting hasn't been to your liking, but there are avenues to programmatically enforce it to make my qapi patches nicer for your tastes in the future.

--js

#6 lore raw

Re: [PATCH 2/8] python/qapi: change "FIXME" to "TODO"

Quoted 78 lines · John Snow
Quoted 76 lines · Markus Armbruster
Quoted 6 lines · John Snow

qemu.git/python/setup.cfg disallows checking in any code with "XXX", "FIXME" or "TODO" in the comments. Soften the restriction to only prohibit "FIXME", and change the two occurrences of "FIXME" in qapi to read "TODO" instead.

I don't like forbidding FIXME comments. It's as futile as forbidding known bugs. All it can accomplish is making people use other, and likely less clear ways to document them.

Perhaps projects exist that use FIXME comments only for known bugs in uncommitted code. To me, that feels *nuts*. I commit all kinds of crap in my tree. I don't need silly "make check" failures while I develop, the non-silly ones cause enough friction already.

In fact, we're quite happy to use FIXME comments even in merged code:

    $ git-grep FIXME | wc -l
    494

I can't see why python/ should be different.

Quoted 40 lines
---
 python/setup.cfg         | 5 +++++
 scripts/qapi/commands.py | 2 +-
 scripts/qapi/events.py   | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)
python/setup.cfg +5 -0
index 3b4e2cc5501..72b58c98c99 100644
@@ -169,6 +169,11 @@ ignore-signatures=yes
 # TODO: Remove after we opt in to Pylint 2.8.3. See commit msg.
 min-similarity-lines=6
 
+[pylint.miscellaneous]
+
+# forbid FIXME/XXX comments, allow TODO.
+notes=FIXME,
+      XXX,
 
 [isort]
 force_grid_wrap=4
scripts/qapi/commands.py +1 -1
index 79951a841f5..cffed6cd3ba 100644
@@ -385,7 +385,7 @@ def visit_command(self,
                       coroutine: bool) -> None:
         if not gen:
             return
-        # FIXME: If T is a user-defined type, the user is responsible
+        # TODO: If T is a user-defined type, the user is responsible
         # for making this work, i.e. to make T's condition the
         # conjunction of the T-returning commands' conditions.  If T
         # is a built-in type, this isn't possible: the
scripts/qapi/events.py +0 -0
index d1f639981a9..36dc0c50c78 100644

scripts/qapi/events.py def gen_event_send(name: str, · line 90

Quoted 8 lines
scripts/qapi/events.py +1 -1
@@ -84,7 +84,7 @@ def gen_event_send(name: str,
                    boxed: bool,
                    event_enum_name: str,
                    event_emit: str) -> str:
-    # FIXME: Our declaration of local variables (and of 'errp' in the
+    # TODO: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
     # practice, we can rename our local variables with a leading prefix,

Starting a comment with TODO tells me there's work to do.

Quoted 8 lines
Quoted 2 lines

Starting it with FIXME tells me there's a bug to fix. That's more specific. Replacing FIXME by TODO loses information.

meh. I do use the "prohibit fixme" personally because I've the memory of a goldfish and I like setting up bombs for myself when I run tests, but willing to cede if it gets me what I want otherwise. I could be coerced to using "XXX" as my WIP testing bomb. Or maybe literally just adding "WIP" as a new bomb. Is that a fair trade?

I understand you'd like to have some kind of stylized comment that automated testing will reject, to serve as a "do not post before resolving this issue" reminder. Fair?

If yes, whatever floats your boat and doesn't interfere with existing practice.

    $ git-grep -w FIXME | wc -l
    493
    $ git-grep -w TODO | wc -l
    1249
    $ git-grep -w XXX | wc -l
    78288
    $ git-grep -w WIP
    Binary file pc-bios/skiboot.lid matches
Quoted 10 lines

There are likely other standards differences between the two subtrees, potentially things like documentation string length and so on -- I invite you to take a look at the setup.cfg file and tweak things to your liking and run "make check-minreqs" to see what barks, if anything.

After you run that command, you can type "source .dev-venv/bin/activate.sh" (or .fish or whatever) and then "pylint --generate-rcfile | less" to get a sample config and see all of the buttons, knobs and levers you could pull. You can leave the environment when you're done with "deactivate".

Quoted 4 lines

Mentioning this only because there have been times in the past that my formatting hasn't been to your liking, but there are avenues to programmatically enforce it to make my qapi patches nicer for your tastes in the future.

Thanks!

#17 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

In reply to John Snow
Quoted 11 lines · John Snow

Gave Dan a related answer. For you, my explanation is:

- It's nice to have just one configuration for static analysis in just one place
- It's nice to have that configuration follow python ecosystem norms
- It's nice to use standard python management tools to configure and test the supported versions of static analysis tools, again kept in one place.
- Just moving the folder costs virtually nothing.
- Moving it here makes it easier for me to test the eventual integration with make check in one place.
- I like being able to say that anything under python/ is fiercely

Quoted 8 lines

guarded by high standards (and the gitlab pipelines) and everything else is not. I consider this to be organizationally simple and easy to communicate. i.e., I find it attractive to say that "python is maintained, scripts are YMMV." I am *not* willing to maintain everything under scripts/ with the same level of effort I apply to python/. I think it's important to allow people to commit low-development-cost scripts ("contrib quality") that they can run from time to time and not everything needs to be held to a crystalline perfect standard, but some stuff does.

FYI, I was NOT suggesting that you maintain anything under scripts/.

Rather I'm saying that if we want to apply python code standards, we should (ultimately) apply them to all python code in the tree, and that *ALL* maintainers and contributors should comply.

Consider our C standards - we don't apply them selectively to a subset of the tree - we expect all maintainers to comply. I'd like us to have the same be true of Python.

The only real issue we have with python standards is bringing existing code upto par, before we can enable the checks.

Currently we have no easy way for other maintainers to enable their python code be checked, without moving their code under python/ which is undesirable IMHO.

If we move the python coding standards to meson.build, such that apply to all of the source tree, and then exclude everything except python/, we make it easier for other maintainers to bring stuff upto par. All need do at that point is remove the exclusion rule for files incrementally as they fix them.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

Quoted 22 lines · Daniel P. Berrangé
Quoted 11 lines · John Snow

Gave Dan a related answer. For you, my explanation is:

- It's nice to have just one configuration for static analysis in just one place
- It's nice to have that configuration follow python ecosystem norms
- It's nice to use standard python management tools to configure and test the supported versions of static analysis tools, again kept in one place.
- Just moving the folder costs virtually nothing.
- Moving it here makes it easier for me to test the eventual integration with make check in one place.
- I like being able to say that anything under python/ is fiercely

Quoted 8 lines

guarded by high standards (and the gitlab pipelines) and everything else is not. I consider this to be organizationally simple and easy to communicate. i.e., I find it attractive to say that "python is maintained, scripts are YMMV." I am *not* willing to maintain everything under scripts/ with the same level of effort I apply to python/. I think it's important to allow people to commit low-development-cost scripts ("contrib quality") that they can run from time to time and not everything needs to be held to a crystalline perfect standard, but some stuff does.

Quoted 5 lines

FYI, I was NOT suggesting that you maintain anything under scripts/.

Rather I'm saying that if we want to apply python code standards, we should (ultimately) apply them to all python code in the tree, and that *ALL* maintainers and contributors should comply.

To be clear up front: this is more of a tangential musing than anything else, and I'm not making a concrete proposal wrt this patchset.

scripts/, like contrib/, is a bit of an odd directory structure from my point of view. I don't think something happening to be a script ought to affect where we put it in our source tree -- scripts/ is really more like "tooling used at build time and by developers". If we had a user-facing utility that happened to be written in python or in shell, that ought to go in tools/ I think, for instance. To the extent that our standards are lower in scripts/ it should be because we know that the audience and usage pattern for those utilities is limited and so it's not necessarily worth the effort to bring them up to the standards we'd apply to user-facing code. (Similarly we're a bit sloppier in C code in tests/ than we are in C code that goes into QEMU proper.)

IMHO half of contrib/ ought to be in tools/, and for contrib/ I'm particularly not a fan of having a bit of the directory tree that's labelled as "put stuff here that we don't care about". Either we care about it and it should go in the appropriate place in the tree for what it is, or else we don't care about it and it should live out-of-tree...

-- PMM

author
#7 lore raw

Re: [PATCH 2/8] python/qapi: change "FIXME" to "TODO"

In reply to Markus Armbruster
Quoted 91 lines · Markus Armbruster
Quoted 1 line · John Snow

On Fri, Aug 30, 2024 at 7:09 AM Markus Armbruster <[email protected]>

Quoted 64 lines · wrote:
Quoted 64 lines
Quoted 6 lines · John Snow

qemu.git/python/setup.cfg disallows checking in any code with "XXX", "FIXME" or "TODO" in the comments. Soften the restriction to only prohibit "FIXME", and change the two occurrences of "FIXME" in qapi to read "TODO" instead.

I don't like forbidding FIXME comments. It's as futile as forbidding known bugs. All it can accomplish is making people use other, and likely less clear ways to document them.

Perhaps projects exist that use FIXME comments only for known bugs in uncommitted code. To me, that feels *nuts*. I commit all kinds of crap in my tree. I don't need silly "make check" failures while I develop, the non-silly ones cause enough friction already.

In fact, we're quite happy to use FIXME comments even in merged code:

    $ git-grep FIXME | wc -l
    494

I can't see why python/ should be different.

Quoted 38 lines
---
 python/setup.cfg         | 5 +++++
 scripts/qapi/commands.py | 2 +-
 scripts/qapi/events.py   | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)
python/setup.cfg +5 -0
index 3b4e2cc5501..72b58c98c99 100644
@@ -169,6 +169,11 @@ ignore-signatures=yes
 # TODO: Remove after we opt in to Pylint 2.8.3. See commit msg.
 min-similarity-lines=6
 
+[pylint.miscellaneous]
+
+# forbid FIXME/XXX comments, allow TODO.
+notes=FIXME,
+      XXX,
 
 [isort]
 force_grid_wrap=4
scripts/qapi/commands.py +1 -1
index 79951a841f5..cffed6cd3ba 100644
@@ -385,7 +385,7 @@ def visit_command(self,
                       coroutine: bool) -> None:
         if not gen:
             return
-        # FIXME: If T is a user-defined type, the user is responsible
+        # TODO: If T is a user-defined type, the user is responsible
         # for making this work, i.e. to make T's condition the
         # conjunction of the T-returning commands' conditions.  If T
         # is a built-in type, this isn't possible: the
scripts/qapi/events.py +0 -1
index d1f639981a9..36dc0c50c78 100644
--- a/scripts/qapi/events.py
Quoted 8 lines
Quoted 8 lines

scripts/qapi/events.py def gen_event_send(name: str, · line 88

Quoted 8 lines

+++ b/scripts/qapi/events.py

scripts/qapi/events.py +1 -1
@@ -84,7 +84,7 @@ def gen_event_send(name: str,
                    boxed: bool,
                    event_enum_name: str,
                    event_emit: str) -> str:
-    # FIXME: Our declaration of local variables (and of 'errp' in the
+    # TODO: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the

event's

Quoted 1 line
Quoted 1 line

scripts/qapi/events.py def gen_event_send(name: str, · line 89

Quoted 1 line
scripts/qapi/events.py +0 -0
     # data type passed in as parameters.  If this collision ever

hits in

Quoted 1 line
Quoted 1 line

scripts/qapi/events.py def gen_event_send(name: str, · line 90

Quoted 1 line
scripts/qapi/events.py +0 -0
     # practice, we can rename our local variables with a leading

prefix,

Quoted 7 lines
Quoted 5 lines

Starting a comment with TODO tells me there's work to do.

Starting it with FIXME tells me there's a bug to fix. That's more specific. Replacing FIXME by TODO loses information.

meh. I do use the "prohibit fixme" personally because I've the memory of

a

Quoted 2 lines

goldfish and I like setting up bombs for myself when I run tests, but willing to cede if it gets me what I want otherwise. I could be coerced

Quoted 9 lines

to

Quoted 1 line

using "XXX" as my WIP testing bomb. Or maybe literally just adding "WIP"

as

Quoted 1 line

a new bomb. Is that a fair trade?

I understand you'd like to have some kind of stylized comment that automated testing will reject, to serve as a "do not post before resolving this issue" reminder. Fair?

Yep, that's it exactly.

Quoted 12 lines

If yes, whatever floats your boat and doesn't interfere with existing practice.

    $ git-grep -w FIXME | wc -l
    493
    $ git-grep -w TODO | wc -l
    1249
    $ git-grep -w XXX | wc -l
    78288
    $ git-grep -w WIP
    Binary file pc-bios/skiboot.lid matches

"WIP" it is. I'll adjust the conf and nix this patch.

Quoted 11 lines
Quoted 6 lines

There are likely other standards differences between the two subtrees, potentially things like documentation string length and so on -- I invite you to take a look at the setup.cfg file and tweak things to your liking and run "make check-minreqs" to see what barks, if anything.

After you run that command, you can type "source

.dev-venv/bin/activate.sh"

Quoted 1 line

(or .fish or whatever) and then "pylint --generate-rcfile | less" to get

a

Quoted 1 line

sample config and see all of the buttons, knobs and levers you could

pull.

Quoted 9 lines
Quoted 6 lines

You can leave the environment when you're done with "deactivate".

Mentioning this only because there have been times in the past that my formatting hasn't been to your liking, but there are avenues to programmatically enforce it to make my qapi patches nicer for your tastes in the future.

Thanks!

No problem! I know our tastes don't always match but I figure if we can agree on a config file, it'll make things nicer for the both of us in the future.

We can discuss at KVM Forum too any of the little nits or preferences you'd like to see automated and enforced to unify the Python style we use in tree. I'd also like to discuss the usage of black formatting to automate away the formatting guesswork. It'd be a lot of churn, but it would drastically simplify things. (e.g. "make autoformat" in the python dir, plus we can look into emacs/vim/vscode config for the repo)

I'll handle the config/maintenance/tooling but you can let me know your preferences.

(This series should be a good example of which things I have configured for python/ that we didn't tackle for qapi. It isn't very much. Some things you prefer may now be *unhandled*, I think docstring length being confined to 72(?) or less might be one of them, I don't recall others at the moment.)

author
#19 lore raw

Re: [PATCH 7/8] python/qapi: move scripts/qapi to python/qemu/qapi

In reply to Daniel P. Berrangé
Quoted 24 lines · Daniel P. Berrangé
Quoted 3 lines · John Snow

Gave Dan a related answer. For you, my explanation is:

- It's nice to have just one configuration for static analysis in just

one

Quoted 9 lines

place
- It's nice to have that configuration follow python ecosystem norms
- It's nice to use standard python management tools to configure and test the supported versions of static analysis tools, again kept in one place.
- Just moving the folder costs virtually nothing.
- Moving it here makes it easier for me to test the eventual integration with make check in one place.
- I like being able to say that anything under python/ is fiercely guarded by high standards (and the gitlab pipelines) and everything else

is

Quoted 1 line

not. I consider this to be organizationally simple and easy to

communicate.

Quoted 2 lines

i.e., I find it attractive to say that "python is maintained, scripts are YMMV." I am *not* willing to maintain everything under scripts/ with

the

Quoted 1 line

same level of effort I apply to python/. I think it's important to

allow

Quoted 1 line

people to commit low-development-cost scripts ("contrib quality") that

they

Quoted 9 lines
Quoted 2 lines

can run from time to time and not everything needs to be held to a crystalline perfect standard, but some stuff does.

FYI, I was NOT suggesting that you maintain anything under scripts/.

Rather I'm saying that if we want to apply python code standards, we should (ultimately) apply them to all python code in the tree, and that *ALL* maintainers and contributors should comply.

Right, it's just that the level of care to bring everything up to that standard is currently more effort than I'd like to spend.

Ideally everything WOULD be on the same standard, but...

Quoted 7 lines

Consider our C standards - we don't apply them selectively to a subset of the tree - we expect all maintainers to comply. I'd like us to have the same be true of Python.

The only real issue we have with python standards is bringing existing code upto par, before we can enable the checks.

Yeah, exactly. It took me long enough to do this with qmp, machine, qom and qapi. It'll take aeons for iotests.

It's just not on my radar right now as a priority.

More tractable: protect everything used for the build to that high standard, worry about the rest later.

Quoted 4 lines

Currently we have no easy way for other maintainers to enable their python code be checked, without moving their code under python/ which is undesirable IMHO.

It's possible I can extend a "lower standard" to everything outside of python/ to help enforce the basics, without mandatory typing.

(I did add an iotest checker to python/ tests that enforces a lower standard to those tests. I intend to remove iotests 297 once I get a top-level "make check-python" instituted.)

I don't think one unified standard is something we can do in the near term. Mechanically I think it's easy, but in practice pushing all the linting patches through has been like pulling teeth and I've lost appetite for pursuing it beyond whatever is *super duper important*.

In retrospect, 3.6 was too early to try adding static typing. I think I won't bother for anything else until we have 3.9 as a minimum. After the pushback last time, I doubt I'll make the push any time soon unless something really urgent comes up.

Quoted 6 lines

If we move the python coding standards to meson.build, such that apply to all of the source tree, and then exclude everything except python/, we make it easier for other maintainers to bring stuff upto par. All need do at that point is remove the exclusion rule for files incrementally as they fix them.

Not against this in the future, I just think there are some higher priority items to take care of first:

* Begin protecting qapi with the existing python static analysis regime
* Add a "make check-python" target to the top level makefile that can set up the environment it needs on-demand and handles what python/'s "make check-minreqs" currently does (incl iotest 297)
* Drop python/qemu/qmp from the tree in favor of the standalone package. There are mkvenv changes I'm currently developing here to make this happen. It's a little involved due to the wide spread of setuptools versions on platforms we support.

Once I drill through these, I'm more than happy to try to support/maintain everything else to a lower baseline of care. Maybe this winter, if you'd be willing to volunteer some review time for the push.

Quoted 12 lines

With regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|