Skip to contentSkip to conversation

qapi: add formal "intro" section

PATCH Superseded John Snow 2 participants 31 messages Jun 3 – Jun 16 EDAC-core

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

v3 cover qapi: add formal "intro" section

Hiya, this is a series that explores a potential syntax for a designated "Intro" section. Markus knows why I want this, but for everyone else: a designated "Introduction" section is useful for the desired "inliner" feature for the new QAPI doc system. Commits explain a bit more. This is prep work and doesn't really change anything tangibly except source code syntax for the QAPI docs.

It is designed so that this conversion can happen incrementally with no actual difference to the rendered manuals, so each QAPI module can be converted one at a time for easier review and merging in an arbitrary order.

This series demonstrates conversion of just four modules; if I'm given a thumbs up, I will convert the rest of QAPI, one module (or maintainer stanza) per patch like how I handled adding cross-references.

**

V3: changed intro section syntax, made various fixes and changes; I'm going to be honest Markus I did not keep a strict account this time ;_;

V2: punt intro sections to the subsequent line and adjust the parser accordingly. Left as an RFC patch that undoes some things this series does instead of properly factoring into the series.

John Snow (16):
python: temporarily restrict max mypy version tests/qapi: generate output in source order qapi/docs: remove unused QAPIDoc subsection members qapi/docs: add has_features property qapi/docs: make remaining subsection members "private" qapi/docs: fix comment phrasing
qapi/docs: add "Intro" section
qapi/parser: move _insert_near_kind() method qapi/docs: adjust stub member insertion algorithm qapi/docs: remove implicit Plain section qapi/docs: add rendering for INTRO sections qapi/docs: add "Intro" section parsing
qapi: convert intro sections for accelerator.json
qapi: convert intro sections for acpi-hest.json
qapi: convert intro sections for acpi.json
qapi: convert intro sections for audio.json

 docs/devel/qapi-code-gen.rst   |  16 ++--
 docs/sphinx/qapidoc.py         |  59 +++++++-----
 qapi/accelerator.json          |  18 ++--
 qapi/acpi-hest.json            |   4 +-
 qapi/acpi.json                 |  18 ++--
 qapi/audio.json                |  88 +++++++-----------
 python/setup.cfg               |   3 +-
 scripts/qapi/parser.py         | 162 +++++++++++++++++++--------------
 tests/qapi-schema/doc-good.out |  86 ++++++++---------
 tests/qapi-schema/test-qapi.py |  14 +--
 10 files changed, 236 insertions(+), 232 deletions(-)
Signature · 1 line
2.54.0
author
#1 lore raw

PATCH v3 1/16 python: temporarily restrict max mypy version

The newest versions of mypy do not support targeting Python 3.9, which we still support. I want to address that soon, but in the meantime it's nice if the tests pass.

---
 python/setup.cfg | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
python/setup.cfg +2 -1
index 80975933971..ad4121a58d8 100644
@@ -40,7 +40,8 @@ devel =
     flake8 >= 5.0.4
     fusepy >= 2.0.4
     isort >= 5.6.0
-    mypy >= 1.4.0
+    # Max ver is temporary until we drop 3.9
+    mypy >= 1.4.0, <2.0.0
     pylint >= 2.17.3
     pylint != 3.2.4; python_version<"3.9"
     pytest >= 6.0.2
Signature · 1 line
2.54.0
author
#2 lore raw

PATCH v3 2/16 tests/qapi: generate output in source order

In reply to John Snow

Rewrite the test doc generator to produce output in source order instead of arbitrarily by section name.

This patch removes our last use of the "body" field, which has an effect on how the sections of each test documention block are printed. We now print the name of the section followed by the section text for all sections except Members and Features, which are printed as "Member=%s" or "Feature=%s" followed by the section text, respectively.

This patch is motivated by a desire to move the QAPIDoc API away from named fields for specific sections in a bid to force all users to simply iterate through all_sections in order, instead - and to remove the named subsections.

---
 tests/qapi-schema/doc-good.out | 82 +++++++++++++++++-----------------
 tests/qapi-schema/test-qapi.py | 14 +++---
 2 files changed, 48 insertions(+), 48 deletions(-)
tests/qapi-schema/doc-good.out +41 -41
index 04a55072646..b9829e2f841 100644
@@ -54,15 +54,15 @@ event EVT_BOXED Object
     boxed=True
     feature feat3
 doc freeform
-    body=
+    Plain
 *******
 Section
 *******
 doc freeform
-    body=
+    Plain
 Just text, no heading.
 doc freeform
-    body=
+    Plain
 Subsection
 ==========
 
@@ -106,84 +106,84 @@ Examples:
 - *verbatim*
 - {braces}
 doc symbol=Enum
-    body=
+    Plain
 
-    arg=one
+    Member=one
 The _one_ {and only}, description on the same line
-    arg=two
+    Member=two
 
-    feature=enum-feat
+    Feature=enum-feat
 Also _one_ {and only}
-    feature=enum-member-feat
+    Feature=enum-member-feat
 a member feature
-    section=Plain
+    Plain
 @two is undocumented
 doc symbol=Base
-    body=
+    Plain
 
-    arg=base1
+    Member=base1
  description starts on a new line,
  minimally indented
 doc symbol=Variant1
-    body=
+    Plain
 A paragraph
 
 Another paragraph
 
 @var1 is undocumented
-    arg=var1
+    Member=var1
 
-    feature=variant1-feat
+    Feature=variant1-feat
 a feature
-    feature=member-feat
+    Feature=member-feat
 a member feature
 doc symbol=Variant2
-    body=
+    Plain
 
 doc symbol=Object
-    body=
+    Plain
 
-    feature=union-feat1
+    Feature=union-feat1
 a feature
 doc symbol=Alternate
-    body=
+    Plain
 
-    arg=i
+    Member=i
 description starts on the same line
     remainder indented the same
     @b is undocumented
-    arg=b
+    Member=b
 
-    feature=alt-feat
+    Feature=alt-feat
 a feature
 doc freeform
-    body=
+    Plain
 Another subsection
 ==================
 doc symbol=cmd
-    body=
+    Plain
 
-    arg=arg1
+    Member=arg1
     description starts on a new line,
     indented
-    arg=arg2
+    Member=arg2
 description starts on the same line
     remainder indented differently
-    arg=arg3
+    Member=arg3
 
-    feature=cmd-feat1
+    Feature=cmd-feat1
 a feature
-    feature=cmd-feat2
+    Feature=cmd-feat2
 another feature
-    section=Plain
+    Plain
 .. note:: @arg3 is undocumented
-    section=Returns
+    Returns
 @Object
-    section=Errors
+    Errors
 some
-    section=Todo
+    Todo
 frobnicate
-    section=Plain
+    Plain
 .. admonition:: Notes
 
  - Lorem ipsum dolor sit amet
@@ -207,23 +207,23 @@ Examples::
 
 Note::
     Ceci n'est pas une note
-    section=Since
+    Since
 2.10
 doc symbol=cmd-boxed
-    body=
+    Plain
 If you're bored enough to read this, go see a video of boxed cats
-    feature=cmd-feat1
+    Feature=cmd-feat1
 a feature
-    feature=cmd-feat2
+    Feature=cmd-feat2
 another feature
-    section=Plain
+    Plain
 .. qmp-example::
 
    -> "this example"
 
    <- ... has no title ...
 doc symbol=EVT_BOXED
-    body=
+    Plain
 
-    feature=feat3
+    Feature=feat3
 a feature
tests/qapi-schema/test-qapi.py +7 -7
index cf7fb8a6df5..5beb96e894f 100755
@@ -19,6 +19,7 @@
 from io import StringIO
 
 from qapi.error import QAPIError
+from qapi.parser import QAPIDoc
 from qapi.schema import QAPISchema, QAPISchemaVisitor
 
 
@@ -116,13 +117,12 @@ def test_frontend(fname):
             print('doc symbol=%s' % doc.symbol)
         else:
             print('doc freeform')
-        print('    body=\n%s' % doc.body.text)
-        for arg, section in doc.args.items():
-            print('    arg=%s\n%s' % (arg, section.text))
-        for feat, section in doc.features.items():
-            print('    feature=%s\n%s' % (feat, section.text))
-        for section in doc.sections:
-            print('    section=%s\n%s' % (section.kind, section.text))
+        for section in doc.all_sections:
+            if isinstance(section, QAPIDoc.ArgSection):
+                print('    %s=%s' % (section.kind, section.name))
+            else:
+                print('    %s' % section.kind)
+            print(section.text)
 
 
 def open_test_result(dir_name, file_name, update):
Signature · 1 line
2.54.0
author
#3 lore raw

PATCH v3 3/16 qapi/docs: remove unused QAPIDoc subsection members

In reply to John Snow

All users of the QAPIDoc object should be iterating over all_sections and not grabbing arbitrary subsections, if possible. Remove the 'body' and 'sections' members, as they are no longer used.

---
 scripts/qapi/parser.py | 6 ------
 1 file changed, 6 deletions(-)
scripts/qapi/parser.py +0 -6
index c3cf33904ef..b33edbba74f 100644
@@ -732,8 +732,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.all_sections: List[QAPIDoc.Section] = [
             QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
         ]
-        # the body section
-        self.body: Optional[QAPIDoc.Section] = self.all_sections[0]
         # dicts mapping parameter/feature names to their description
         self.args: Dict[str, QAPIDoc.ArgSection] = {}
         self.features: Dict[str, QAPIDoc.ArgSection] = {}
@@ -742,8 +740,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.errors: Optional[QAPIDoc.Section] = None
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
-        # sections other than .body, .args, .features
-        self.sections: List[QAPIDoc.Section] = []
 
     def end(self) -> None:
         for section in self.all_sections:
@@ -766,7 +762,6 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
 
         # start new section
         section = self.Section(info, kind)
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def new_tagged_section(
@@ -790,7 +785,6 @@ def new_tagged_section(
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
             self.since = section
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def _new_description(
Signature · 1 line
2.54.0
#4 lore raw

Re: [PATCH v3 03/16] qapi/docs: remove unused QAPIDoc subsection members

Uh, "qapi/docs:"? Shouldn't this be "qapi/parser:"?

Quoted 48 lines · John Snow

All users of the QAPIDoc object should be iterating over all_sections and not grabbing arbitrary subsections, if possible. Remove the 'body' and 'sections' members, as they are no longer used.

---
 scripts/qapi/parser.py | 6 ------
 1 file changed, 6 deletions(-)
scripts/qapi/parser.py +0 -6
index c3cf33904ef..b33edbba74f 100644
@@ -732,8 +732,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.all_sections: List[QAPIDoc.Section] = [
             QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
         ]
-        # the body section
-        self.body: Optional[QAPIDoc.Section] = self.all_sections[0]
         # dicts mapping parameter/feature names to their description
         self.args: Dict[str, QAPIDoc.ArgSection] = {}
         self.features: Dict[str, QAPIDoc.ArgSection] = {}
@@ -742,8 +740,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.errors: Optional[QAPIDoc.Section] = None
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
-        # sections other than .body, .args, .features
-        self.sections: List[QAPIDoc.Section] = []
 
     def end(self) -> None:
         for section in self.all_sections:
@@ -766,7 +762,6 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
 
         # start new section
         section = self.Section(info, kind)
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def new_tagged_section(
@@ -790,7 +785,6 @@ def new_tagged_section(
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
             self.since = section
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def _new_description(
author
#5 lore raw

Re: [PATCH v3 03/16] qapi/docs: remove unused QAPIDoc subsection members

Quoted 3 lines · Markus Armbruster

Uh, "qapi/docs:"? Shouldn't this be "qapi/parser:"?

Uh, sure. It's the parser of docs ... forgive me

Quoted 51 lines
Quoted 48 lines · John Snow

All users of the QAPIDoc object should be iterating over all_sections and not grabbing arbitrary subsections, if possible. Remove the 'body' and 'sections' members, as they are no longer used.

---
 scripts/qapi/parser.py | 6 ------
 1 file changed, 6 deletions(-)
scripts/qapi/parser.py +0 -6
index c3cf33904ef..b33edbba74f 100644
@@ -732,8 +732,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.all_sections: List[QAPIDoc.Section] = [
             QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
         ]
-        # the body section
-        self.body: Optional[QAPIDoc.Section] = self.all_sections[0]
         # dicts mapping parameter/feature names to their description
         self.args: Dict[str, QAPIDoc.ArgSection] = {}
         self.features: Dict[str, QAPIDoc.ArgSection] = {}
@@ -742,8 +740,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.errors: Optional[QAPIDoc.Section] = None
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
-        # sections other than .body, .args, .features
-        self.sections: List[QAPIDoc.Section] = []
 
     def end(self) -> None:
         for section in self.all_sections:
@@ -766,7 +762,6 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
 
         # start new section
         section = self.Section(info, kind)
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def new_tagged_section(
@@ -790,7 +785,6 @@ def new_tagged_section(
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
             self.since = section
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def _new_description(
#6 lore raw

Re: [PATCH v3 03/16] qapi/docs: remove unused QAPIDoc subsection members

Quoted 6 lines · John Snow
Quoted 3 lines · Markus Armbruster

Uh, "qapi/docs:"? Shouldn't this be "qapi/parser:"?

Uh, sure. It's the parser of docs ... forgive me

I see. qapi/docs makes me expect a QAPI documentation patch. I go with qapi/parser to avoid such confusion here and in later patches.

Aside: at some point, I tried separating the doc parser from the code parser, so they can live in separate source files, but their entanglement resisted successfully.

author
#7 lore raw

PATCH v3 4/16 qapi/docs: add has_features property

In reply to John Snow

Begin preparing to remove public access to the .features member by introducing a semantic "has features" property instead.

---
 scripts/qapi/parser.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/qapi/parser.py +6 -2
index b33edbba74f..dcc58170bd5 100644
@@ -555,7 +555,7 @@ def get_doc(self) -> 'QAPIDoc':
                     break
                 # Non-blank line, first of a section
                 if line == 'Features:':
-                    if doc.features:
+                    if doc.has_features:
                         raise QAPIParseError(
                             self, "duplicated 'Features:' line")
                     self.accept(False)
@@ -570,7 +570,7 @@ def get_doc(self) -> 'QAPIDoc':
                         if text:
                             doc.append_line(text)
                         line = self.get_doc_indented(doc)
-                    if not doc.features:
+                    if not doc.has_features:
                         raise QAPIParseError(
                             self, 'feature descriptions expected')
                     no_more_args = True
@@ -741,6 +741,10 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
 
+    @property
+    def has_features(self) -> bool:
+        return bool(self.features)
+
     def end(self) -> None:
         for section in self.all_sections:
             section.text = section.text.strip('\n')
Signature · 1 line
2.54.0
author
#8 lore raw

PATCH v3 5/16 qapi/docs: make remaining subsection members "private"

In reply to John Snow

These fields are used to provide error checking and internal logistics and should not be used by a user of the library to directly access documentation sections, so make them private.

The "since" field alone is left public, as the qapidoc generator does use this field to pull that section out of the regular flow of the document.

---
 scripts/qapi/parser.py | 48 +++++++++++++++++++++---------------------
 1 file changed, 24 insertions(+), 24 deletions(-)
scripts/qapi/parser.py +24 -24
index dcc58170bd5..da4756a7424 100644
@@ -733,17 +733,17 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
             QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
         ]
         # dicts mapping parameter/feature names to their description
-        self.args: Dict[str, QAPIDoc.ArgSection] = {}
-        self.features: Dict[str, QAPIDoc.ArgSection] = {}
+        self._args: Dict[str, QAPIDoc.ArgSection] = {}
+        self._features: Dict[str, QAPIDoc.ArgSection] = {}
         # a command's "Returns" and "Errors" section
-        self.returns: Optional[QAPIDoc.Section] = None
-        self.errors: Optional[QAPIDoc.Section] = None
+        self._returns: Optional[QAPIDoc.Section] = None
+        self._errors: Optional[QAPIDoc.Section] = None
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
 
     @property
     def has_features(self) -> bool:
-        return bool(self.features)
+        return bool(self._features)
 
     def end(self) -> None:
         for section in self.all_sections:
@@ -775,15 +775,15 @@ def new_tagged_section(
     ) -> None:
         section = self.Section(info, kind)
         if kind == QAPIDoc.Kind.RETURNS:
-            if self.returns:
+            if self._returns:
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
-            self.returns = section
+            self._returns = section
         elif kind == QAPIDoc.Kind.ERRORS:
-            if self.errors:
+            if self._errors:
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
-            self.errors = section
+            self._errors = section
         elif kind == QAPIDoc.Kind.SINCE:
             if self.since:
                 raise QAPISemError(
@@ -807,16 +807,16 @@ def _new_description(
         desc[name] = section
 
     def new_argument(self, info: QAPISourceInfo, name: str) -> None:
-        self._new_description(info, name, QAPIDoc.Kind.MEMBER, self.args)
+        self._new_description(info, name, QAPIDoc.Kind.MEMBER, self._args)
 
     def new_feature(self, info: QAPISourceInfo, name: str) -> None:
-        self._new_description(info, name, QAPIDoc.Kind.FEATURE, self.features)
+        self._new_description(info, name, QAPIDoc.Kind.FEATURE, self._features)
 
     def append_line(self, line: str) -> None:
         self.all_sections[-1].append_line(line)
 
     def connect_member(self, member: 'QAPISchemaMember') -> None:
-        if member.name not in self.args:
+        if member.name not in self._args:
             assert member.info
             if self.symbol not in member.info.pragma.documentation_exceptions:
                 raise QAPISemError(member.info,
@@ -827,7 +827,7 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
 
             section = QAPIDoc.ArgSection(
                 self.info, QAPIDoc.Kind.MEMBER, member.name)
-            self.args[member.name] = section
+            self._args[member.name] = section
 
             # Determine where to insert stub doc - it should go at the
             # end of the members section(s), if any. Note that index 0
@@ -839,14 +839,14 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
                     index += 1
             self.all_sections.insert(index, section)
 
-        self.args[member.name].connect(member)
+        self._args[member.name].connect(member)
 
     def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
-        if feature.name not in self.features:
+        if feature.name not in self._features:
             raise QAPISemError(feature.info,
                                "feature '%s' lacks documentation"
                                % feature.name)
-        self.features[feature.name].connect(feature)
+        self._features[feature.name].connect(feature)
 
     def ensure_returns(self, info: QAPISourceInfo) -> None:
 
@@ -887,18 +887,18 @@ def _insert_near_kind(
 
     def check_expr(self, expr: QAPIExpression) -> None:
         if 'command' in expr:
-            if self.returns and 'returns' not in expr:
+            if self._returns and 'returns' not in expr:
                 raise QAPISemError(
-                    self.returns.info,
+                    self._returns.info,
                     "'Returns' section, but command doesn't return anything")
         else:
-            if self.returns:
+            if self._returns:
                 raise QAPISemError(
-                    self.returns.info,
+                    self._returns.info,
                     "'Returns' section is only valid for commands")
-            if self.errors:
+            if self._errors:
                 raise QAPISemError(
-                    self.errors.info,
+                    self._errors.info,
                     "'Errors' section is only valid for commands")
 
     def check(self) -> None:
@@ -918,5 +918,5 @@ def check_args_section(
                         "do" if len(bogus) > 1 else "does"
                     ))
 
-        check_args_section(self.args, 'member')
-        check_args_section(self.features, 'feature')
+        check_args_section(self._args, 'member')
+        check_args_section(self._features, 'feature')
Signature · 1 line
2.54.0
author
#9 lore raw

PATCH v3 6/16 qapi/docs: fix comment phrasing

In reply to John Snow

Plaintext sections can be one or more paragraphs, the original comment was a mistake.

---
 scripts/qapi/parser.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/parser.py +1 -1
index da4756a7424..d8068bb1bf6 100644
@@ -631,7 +631,7 @@ def get_doc(self) -> 'QAPIDoc':
                     line = self.get_doc_indented(doc)
                     no_more_args = True
                 else:
-                    # plain paragraph
+                    # plain paragraph(s)
                     doc.ensure_untagged_section(self.info)
                     doc.append_line(line)
                     line = self.get_doc_paragraph(doc)
Signature · 1 line
2.54.0
author
#10 lore raw

PATCH v3 7/16 qapi/docs: add "Intro" section

In reply to John Snow

This patch adds an "Intro" doc section, intended to eventually replace the "Plain" doc section alongside a forthcoming "Details" section.

For now, this section is not actually instantiated or used, but subsequent patches will slowly convert the leading introductory sections of QAPIDoc documentation to use this new section.

A main motivation of this series of changes is to more explicitly delineate the "Introductory" documentation for each QAPI definition for the sake of the inliner. When inlining members, examples, and details from multiple QAPIDoc sections, we will want to omit the "Introductory" text from inlined definitions while keeping notes, caution boxes, examples, and so on. This new section facilitates that.

---
 docs/sphinx/qapidoc.py | 2 +-
 scripts/qapi/parser.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
docs/sphinx/qapidoc.py +1 -1
index c2f09bac16c..1f7c15b7075 100644
@@ -368,7 +368,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
         for i, section in enumerate(sections):
             section.text = self.reformat_arobase(section.text)
 
-            if section.kind == QAPIDoc.Kind.PLAIN:
+            if section.kind.name in ("PLAIN", "INTRO"):
                 self.visit_paragraph(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
scripts/qapi/parser.py +2 -1
index d8068bb1bf6..97e7dacb0fd 100644
@@ -681,6 +681,7 @@ class Kind(enum.Enum):
         ERRORS = 4
         SINCE = 5
         TODO = 6
+        INTRO = 7
 
         @staticmethod
         def from_string(kind: str) -> 'QAPIDoc.Kind':
@@ -748,7 +749,7 @@ def has_features(self) -> bool:
     def end(self) -> None:
         for section in self.all_sections:
             section.text = section.text.strip('\n')
-            if section.kind != QAPIDoc.Kind.PLAIN and section.text == '':
+            if not (section.kind.name in ("INTRO", "PLAIN") or section.text):
                 raise QAPISemError(
                     section.info, "text required after '%s:'" % section.kind)
 
Signature · 1 line
2.54.0
#11 lore raw

Re: [PATCH v3 07/16] qapi/docs: add "Intro" section

Quoted 25 lines · John Snow

This patch adds an "Intro" doc section, intended to eventually replace the "Plain" doc section alongside a forthcoming "Details" section.

For now, this section is not actually instantiated or used, but subsequent patches will slowly convert the leading introductory sections of QAPIDoc documentation to use this new section.

A main motivation of this series of changes is to more explicitly delineate the "Introductory" documentation for each QAPI definition for the sake of the inliner. When inlining members, examples, and details from multiple QAPIDoc sections, we will want to omit the "Introductory" text from inlined definitions while keeping notes, caution boxes, examples, and so on. This new section facilitates that.

---
 docs/sphinx/qapidoc.py | 2 +-
 scripts/qapi/parser.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
docs/sphinx/qapidoc.py +0 -0
index c2f09bac16c..1f7c15b7075 100644

docs/sphinx/qapidoc.py def visit_sections(self, ent: QAPISchemaDefinition) -> None: · line 371

Quoted 5 lines
docs/sphinx/qapidoc.py +1 -1
@@ -368,7 +368,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
         for i, section in enumerate(sections):
             section.text = self.reformat_arobase(section.text)
 
-            if section.kind == QAPIDoc.Kind.PLAIN:
+            if section.kind.name in ("PLAIN", "INTRO"):

Is the change from QAPIDoc.Kind.FOO to "FOO" just for brevity, or is something going on I don't understand?

Quoted 23 lines
docs/sphinx/qapidoc.py +0 -0
                 self.visit_paragraph(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
scripts/qapi/parser.py +2 -1
index d8068bb1bf6..97e7dacb0fd 100644
@@ -681,6 +681,7 @@ class Kind(enum.Enum):
         ERRORS = 4
         SINCE = 5
         TODO = 6
+        INTRO = 7
 
         @staticmethod
         def from_string(kind: str) -> 'QAPIDoc.Kind':
@@ -748,7 +749,7 @@ def has_features(self) -> bool:
     def end(self) -> None:
         for section in self.all_sections:
             section.text = section.text.strip('\n')
-            if section.kind != QAPIDoc.Kind.PLAIN and section.text == '':
+            if not (section.kind.name in ("INTRO", "PLAIN") or section.text):
                 raise QAPISemError(
                     section.info, "text required after '%s:'" % section.kind)
author
#12 lore raw

Re: [PATCH v3 07/16] qapi/docs: add "Intro" section

Quoted 31 lines · Markus Armbruster
Quoted 28 lines · John Snow

This patch adds an "Intro" doc section, intended to eventually replace the "Plain" doc section alongside a forthcoming "Details" section.

For now, this section is not actually instantiated or used, but subsequent patches will slowly convert the leading introductory sections of QAPIDoc documentation to use this new section.

A main motivation of this series of changes is to more explicitly delineate the "Introductory" documentation for each QAPI definition for the sake of the inliner. When inlining members, examples, and details from multiple QAPIDoc sections, we will want to omit the "Introductory" text from inlined definitions while keeping notes, caution boxes, examples, and so on. This new section facilitates that.

---
 docs/sphinx/qapidoc.py | 2 +-
 scripts/qapi/parser.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
docs/sphinx/qapidoc.py +0 -0
index c2f09bac16c..1f7c15b7075 100644
@@ -368,7 +368,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
         for i, section in enumerate(sections):
             section.text = self.reformat_arobase(section.text)
 
Quoted 5 lines

docs/sphinx/qapidoc.py def visit_sections(self, ent: QAPISchemaDefinition) -> None: · line 371

Quoted 2 lines
docs/sphinx/qapidoc.py +1 -1
-            if section.kind == QAPIDoc.Kind.PLAIN:
+            if section.kind.name in ("PLAIN", "INTRO"):

Is the change from QAPIDoc.Kind.FOO to "FOO" just for brevity, or is something going on I don't understand?

Just brevity, honestly. I'm using the .name attribute and checking against strings. Otherwise it didn't fit on a single line. Inconsistent style, but I guess I chose what I disliked least.

Quoted 25 lines
Quoted 24 lines
docs/sphinx/qapidoc.py +0 -0
                 self.visit_paragraph(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
scripts/qapi/parser.py +2 -1
index d8068bb1bf6..97e7dacb0fd 100644
@@ -681,6 +681,7 @@ class Kind(enum.Enum):
         ERRORS = 4
         SINCE = 5
         TODO = 6
+        INTRO = 7
 
         @staticmethod
         def from_string(kind: str) -> 'QAPIDoc.Kind':
@@ -748,7 +749,7 @@ def has_features(self) -> bool:
     def end(self) -> None:
         for section in self.all_sections:
             section.text = section.text.strip('\n')
-            if section.kind != QAPIDoc.Kind.PLAIN and section.text == '':
+            if not (section.kind.name in ("INTRO", "PLAIN") or section.text):
                 raise QAPISemError(
                     section.info, "text required after '%s:'" % section.kind)
author
#13 lore raw

PATCH v3 8/16 qapi/parser: move _insert_near_kind() method

In reply to John Snow

Move this function out from underneath ensure_returns and make it available for general purpose use as an object method instead. This is purely a scope-level patch with no functional changes.

---
 scripts/qapi/parser.py | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)
scripts/qapi/parser.py +17 -16
index 97e7dacb0fd..261f8ba9f8b 100644
@@ -816,6 +816,22 @@ def new_feature(self, info: QAPISourceInfo, name: str) -> None:
     def append_line(self, line: str) -> None:
         self.all_sections[-1].append_line(line)
 
+    def _insert_near_kind(
+        self,
+        kind: 'QAPIDoc.Kind',
+        new_sect: 'QAPIDoc.Section',
+        after: bool = False,
+    ) -> bool:
+        """Insert or append a new doc section at a specific point."""
+        for idx, sect in enumerate(reversed(self.all_sections)):
+            if sect.kind == kind:
+                pos = len(self.all_sections) - idx - 1
+                if after:
+                    pos += 1
+                self.all_sections.insert(pos, new_sect)
+                return True
+        return False
+
     def connect_member(self, member: 'QAPISchemaMember') -> None:
         if member.name not in self._args:
             assert member.info
@@ -850,28 +866,13 @@ def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
         self._features[feature.name].connect(feature)
 
     def ensure_returns(self, info: QAPISourceInfo) -> None:
-
-        def _insert_near_kind(
-            kind: QAPIDoc.Kind,
-            new_sect: QAPIDoc.Section,
-            after: bool = False,
-        ) -> bool:
-            for idx, sect in enumerate(reversed(self.all_sections)):
-                if sect.kind == kind:
-                    pos = len(self.all_sections) - idx - 1
-                    if after:
-                        pos += 1
-                    self.all_sections.insert(pos, new_sect)
-                    return True
-            return False
-
         if any(s.kind == QAPIDoc.Kind.RETURNS for s in self.all_sections):
             return
 
         # Stub "Returns" section for undocumented returns value
         stub = QAPIDoc.Section(info, QAPIDoc.Kind.RETURNS)
 
-        if any(_insert_near_kind(kind, stub, after) for kind, after in (
+        if any(self._insert_near_kind(kind, stub, after) for kind, after in (
                 # 1. If arguments, right after those.
                 (QAPIDoc.Kind.MEMBER, True),
                 # 2. Elif errors, right *before* those.
Signature · 1 line
2.54.0
author
#14 lore raw

PATCH v3 9/16 qapi/docs: adjust stub member insertion algorithm

In reply to John Snow

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections. To accommodate this, the insertion algorithm that places stub and dummy members must be adjusted to cope with not only the finished state, but temporary intermediate states while the series is merged.

This algorithm can handle zero-or-more PLAIN *or* INTRO sections at the beginning of a QAPIDoc object, in contrast to the previous algorithm which assumed and relied upon there being always one PLAIN section at the beginning of every QAPIDoc section list.

In other words: (PLAIN | INTRO)* <EverythingElse>

This does not impact what the parser itself will actually produce. As of this patch, the parser will still always generate QAPIDoc section lists that start with precisely one PLAIN section (whether or not it is empty), followed by the remaining sections. Those remaining sections may or may not include additional PLAIN sections, but never two such sections contiguously as the parser will always treat that layout as one PLAIN section consisting of multiple paragraph(s).

In other other words: This insertion algorithm is more lenient than the parser, but this is on purpose for flexibility mid-stream as we convert QAPI to using explicit introductory sections. The allowed order of sections will eventually become strictly enforced in the parser, which will in turn allow dramatic simplifications to the insertion algorithm. This only exists as transitory code until we are able to enforce that order.

Fear not: the intermediate rest output before and after this patch are byte identical, so failing all else, we at least know it doesn't make anything worse.

Lastly, because we have three places in the code that need to insert stub/dummy sections, we take the opportunity to consolidate this code to handle all three cases with one function. This winds up necessitating the qapidoc.py generator actually modify the section list to insert a "dummy" member that acts as a placeholder for "The members of ..." text. While it looks like a code smell to modify the caller's argument, it is ultimately safe because the QAPI Schema object is re-parsed and re-constructed in memory for each individual process that needs to operate on it. In other words, the Sphinx document generator already does have "its own copy" of the section lists, so it is "safe" to modify here without regards to other consumers of the QAPIDoc objects. It only *looks* like it smells bad. Ultimately, this code will also be removed once the inliner is merged, so it is only a temporary aesthetic issue regardless.

That's my story and I'm sticking to it.

---
 docs/sphinx/qapidoc.py | 42 ++++++++++++++++--------------
 scripts/qapi/parser.py | 58 +++++++++++++++++++++++++++++-------------
 2 files changed, 64 insertions(+), 36 deletions(-)
docs/sphinx/qapidoc.py +23 -19
index 1f7c15b7075..16ad15fe94f 100644
@@ -349,30 +349,38 @@ def _get_target(
                 )
 
     def visit_sections(self, ent: QAPISchemaDefinition) -> None:
+        # Generate a placeholder right after the member section(s) which
+        # may be used to generate documentation for "The members of..."
+        # pointers in the rendered document.
+        #
+        # This is a temporary hack until the inliner is merged. Note
+        # that although we modify the caller's section list, the
+        # Sphinx document generator has its own copy of the parsed
+        # schema in memory, so this action does not interfere with
+        # other users of the QAPISchema or QAPIDoc objects outside of
+        # the document generator. Fishy, but not harmful.
+        if ent.doc:
+            ent.doc.append_member_stub(
+                QAPIDoc.ArgSection(
+                    ent.doc.info, QAPIDoc.Kind.MEMBER, "q_dummy"
+                )
+            )
+
         sections = ent.doc.all_sections if ent.doc else []
 
-        # Determine the index location at which we should generate
-        # documentation for "The members of ..." pointers. This should
-        # go at the end of the members section(s) if any. Note that
-        # index 0 is assumed to be a plain intro section, even if it is
-        # empty; and that a members section if present will always
-        # immediately follow the opening PLAIN section.
-        gen_index = 1
-        if len(sections) > 1:
-            while sections[gen_index].kind == QAPIDoc.Kind.MEMBER:
-                gen_index += 1
-                if gen_index >= len(sections):
-                    break
-
         # Add sections in source order:
-        for i, section in enumerate(sections):
+        for section in sections:
             section.text = self.reformat_arobase(section.text)
 
             if section.kind.name in ("PLAIN", "INTRO"):
                 self.visit_paragraph(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
-                self.visit_member(section)
+                if section.name == "q_dummy":
+                    # Generate "The members of ..." entries if necessary
+                    self._insert_member_pointer(ent)
+                else:
+                    self.visit_member(section)
             elif section.kind == QAPIDoc.Kind.FEATURE:
                 assert isinstance(section, QAPIDoc.ArgSection)
                 self.visit_feature(section)
@@ -386,10 +394,6 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
             else:
                 assert False
 
-            # Generate "The members of ..." entries if necessary:
-            if i == gen_index - 1:
-                self._insert_member_pointer(ent)
-
         self.ensure_blank_line()
 
     # Transmogrification core methods
scripts/qapi/parser.py +41 -17

77 lines not on this page · raw message

Signature · 1 line
2.54.0
#15 lore raw

Re: [PATCH v3 09/16] qapi/docs: adjust stub member insertion algorithm

Quick first pass...

Quoted 6 lines · John Snow

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections. To accommodate this, the insertion algorithm that places stub and dummy members must be adjusted to cope with not only the finished state, but temporary intermediate states while the series is merged.

Perhaps:

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections.

This will affect the code that inserts "Not documented" descriptions for undocumented members ("stub sections") and the dummy section that marks the spot for "The members of ..." references.

Adjust the algorithm to cope with not only the finished state, but temporary intermediate states while the series is merged.

Quoted 23 lines

This algorithm can handle zero-or-more PLAIN *or* INTRO sections at the beginning of a QAPIDoc object, in contrast to the previous algorithm which assumed and relied upon there being always one PLAIN section at the beginning of every QAPIDoc section list.

In other words: (PLAIN | INTRO)* <EverythingElse>

This does not impact what the parser itself will actually produce. As of this patch, the parser will still always generate QAPIDoc section lists that start with precisely one PLAIN section (whether or not it is empty), followed by the remaining sections. Those remaining sections may or may not include additional PLAIN sections, but never two such sections contiguously as the parser will always treat that layout as one PLAIN section consisting of multiple paragraph(s).

In other other words: This insertion algorithm is more lenient than the parser, but this is on purpose for flexibility mid-stream as we convert QAPI to using explicit introductory sections. The allowed order of sections will eventually become strictly enforced in the parser, which will in turn allow dramatic simplifications to the insertion algorithm. This only exists as transitory code until we are able to enforce that order.

Quoted 1 line

Fear not: the intermediate rest output before and after this patch

"ReST output"?

Quoted 17 lines

are byte identical, so failing all else, we at least know it doesn't make anything worse.

Lastly, because we have three places in the code that need to insert stub/dummy sections, we take the opportunity to consolidate this code to handle all three cases with one function. This winds up necessitating the qapidoc.py generator actually modify the section list to insert a "dummy" member that acts as a placeholder for "The members of ..." text. While it looks like a code smell to modify the caller's argument, it is ultimately safe because the QAPI Schema object is re-parsed and re-constructed in memory for each individual process that needs to operate on it. In other words, the Sphinx document generator already does have "its own copy" of the section lists, so it is "safe" to modify here without regards to other consumers of the QAPIDoc objects. It only *looks* like it smells bad. Ultimately, this code will also be removed once the inliner is merged, so it is only a temporary aesthetic issue regardless.

Such trickery, even when safe, risks making attentive readers go "WAT?!?" I find it tolerable only because we plan to replace it fairly soon.

The code finding the spot to insert stub/dummy sections is more complicated than it has any right to be, both before and after your patch. It reconstructs information the doc parser has, but doesn't pass on in usable form. Passing that info on would likely be simpler and cleaner. However, you tell me your inliner patches also simplify things, and they already exist. So let's go with those.

Quoted 3 lines

That's my story and I'm sticking to it.

author
#16 lore raw

Re: [PATCH v3 09/16] qapi/docs: adjust stub member insertion algorithm

Quoted 18 lines · Markus Armbruster

Quick first pass...

Quoted 6 lines · John Snow

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections. To accommodate this, the insertion algorithm that places stub and dummy members must be adjusted to cope with not only the finished state, but temporary intermediate states while the series is merged.

Perhaps:

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections.

Quoted 6 lines

This will affect the code that inserts "Not documented" descriptions for undocumented members ("stub sections") and the dummy section that marks the spot for "The members of ..." references.

Adjust the algorithm to cope with not only the finished state, but temporary intermediate states while the series is merged.

Sure.

Quoted 24 lines
Quoted 24 lines

This algorithm can handle zero-or-more PLAIN *or* INTRO sections at the beginning of a QAPIDoc object, in contrast to the previous algorithm which assumed and relied upon there being always one PLAIN section at the beginning of every QAPIDoc section list.

In other words: (PLAIN | INTRO)* <EverythingElse>

This does not impact what the parser itself will actually produce. As of this patch, the parser will still always generate QAPIDoc section lists that start with precisely one PLAIN section (whether or not it is empty), followed by the remaining sections. Those remaining sections may or may not include additional PLAIN sections, but never two such sections contiguously as the parser will always treat that layout as one PLAIN section consisting of multiple paragraph(s).

In other other words: This insertion algorithm is more lenient than the parser, but this is on purpose for flexibility mid-stream as we convert QAPI to using explicit introductory sections. The allowed order of sections will eventually become strictly enforced in the parser, which will in turn allow dramatic simplifications to the insertion algorithm. This only exists as transitory code until we are able to enforce that order.

Quoted 3 lines
Quoted 1 line

Fear not: the intermediate rest output before and after this patch

"ReST output"?

Sure

Quoted 23 lines
Quoted 4 lines

are byte identical, so failing all else, we at least know it doesn't make anything worse.

Quoted 14 lines

Lastly, because we have three places in the code that need to insert stub/dummy sections, we take the opportunity to consolidate this code to handle all three cases with one function. This winds up necessitating the qapidoc.py generator actually modify the section list to insert a "dummy" member that acts as a placeholder for "The members of ..." text. While it looks like a code smell to modify the caller's argument, it is ultimately safe because the QAPI Schema object is re-parsed and re-constructed in memory for each individual process that needs to operate on it. In other words, the Sphinx document generator already does have "its own copy" of the section lists, so it is "safe" to modify here without regards to other consumers of the QAPIDoc objects. It only *looks* like it smells bad. Ultimately, this code will also be removed once the inliner is merged, so it is only a temporary aesthetic issue regardless.

Such trickery, even when safe, risks making attentive readers go "WAT?!?" I find it tolerable only because we plan to replace it fairly soon.

Quoted 6 lines

The code finding the spot to insert stub/dummy sections is more complicated than it has any right to be, both before and after your patch. It reconstructs information the doc parser has, but doesn't pass on in usable form. Passing that info on would likely be simpler and cleaner. However, you tell me your inliner patches also simplify things, and they already exist. So let's go with those.

Right, the basic idea is this:

- Differentiate INTRO with new syntax
- Convert "PLAIN" into "DETAILS"
- Enforce strict section ordering: INTRO only at the beginning, DETAILS only at or quite near to the end

Once that is cemented, the insertion algorithm can become much simpler and dumber; e.g. using a map of regions where we can append to the end of a region without having to iterate through to find our sweet spot.

Since that's the plan for the inliner anyway, some temporary ugliness here in order to ensure that we do not regress even one byte seems acceptable.

Additionally, once the inliner is merged, we won't need the "dummy" sections at all, so those go away entirely, and all of the ugliness of that particular part of the hack with it - i.e. no more modifying the caller's argument to insert a bookmark.

So, it's a little gross, but it's just a band-aid to get to where we want to be; and it isn't as gross as it looks.

With your suggested commentary changes, do I have your blessing to push forward? In the interim, please review the rest of the series so I can send you a new version.

--js

Quoted 5 lines
Quoted 4 lines

That's my story and I'm sticking to it.

#17 lore raw

Re: [PATCH v3 09/16] qapi/docs: adjust stub member insertion algorithm

Quoted 108 lines · John Snow
Quoted 18 lines · Markus Armbruster

Quick first pass...

Quoted 6 lines · John Snow

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections. To accommodate this, the insertion algorithm that places stub and dummy members must be adjusted to cope with not only the finished state, but temporary intermediate states while the series is merged.

Perhaps:

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections.

Quoted 6 lines

This will affect the code that inserts "Not documented" descriptions for undocumented members ("stub sections") and the dummy section that marks the spot for "The members of ..." references.

Adjust the algorithm to cope with not only the finished state, but temporary intermediate states while the series is merged.

Sure.

Quoted 24 lines
Quoted 24 lines

This algorithm can handle zero-or-more PLAIN *or* INTRO sections at the beginning of a QAPIDoc object, in contrast to the previous algorithm which assumed and relied upon there being always one PLAIN section at the beginning of every QAPIDoc section list.

In other words: (PLAIN | INTRO)* <EverythingElse>

This does not impact what the parser itself will actually produce. As of this patch, the parser will still always generate QAPIDoc section lists that start with precisely one PLAIN section (whether or not it is empty), followed by the remaining sections. Those remaining sections may or may not include additional PLAIN sections, but never two such sections contiguously as the parser will always treat that layout as one PLAIN section consisting of multiple paragraph(s).

In other other words: This insertion algorithm is more lenient than the parser, but this is on purpose for flexibility mid-stream as we convert QAPI to using explicit introductory sections. The allowed order of sections will eventually become strictly enforced in the parser, which will in turn allow dramatic simplifications to the insertion algorithm. This only exists as transitory code until we are able to enforce that order.

Quoted 3 lines
Quoted 1 line

Fear not: the intermediate rest output before and after this patch

"ReST output"?

Sure

Quoted 23 lines
Quoted 4 lines

are byte identical, so failing all else, we at least know it doesn't make anything worse.

Quoted 14 lines

Lastly, because we have three places in the code that need to insert stub/dummy sections, we take the opportunity to consolidate this code to handle all three cases with one function. This winds up necessitating the qapidoc.py generator actually modify the section list to insert a "dummy" member that acts as a placeholder for "The members of ..." text. While it looks like a code smell to modify the caller's argument, it is ultimately safe because the QAPI Schema object is re-parsed and re-constructed in memory for each individual process that needs to operate on it. In other words, the Sphinx document generator already does have "its own copy" of the section lists, so it is "safe" to modify here without regards to other consumers of the QAPIDoc objects. It only *looks* like it smells bad. Ultimately, this code will also be removed once the inliner is merged, so it is only a temporary aesthetic issue regardless.

Such trickery, even when safe, risks making attentive readers go "WAT?!?" I find it tolerable only because we plan to replace it fairly soon.

Quoted 6 lines

The code finding the spot to insert stub/dummy sections is more complicated than it has any right to be, both before and after your patch. It reconstructs information the doc parser has, but doesn't pass on in usable form. Passing that info on would likely be simpler and cleaner. However, you tell me your inliner patches also simplify things, and they already exist. So let's go with those.

Right, the basic idea is this:

- Differentiate INTRO with new syntax
- Convert "PLAIN" into "DETAILS"
- Enforce strict section ordering: INTRO only at the beginning, DETAILS only at or quite near to the end

Once that is cemented, the insertion algorithm can become much simpler and dumber; e.g. using a map of regions where we can append to the end of a region without having to iterate through to find our sweet spot.

Since that's the plan for the inliner anyway, some temporary ugliness here in order to ensure that we do not regress even one byte seems acceptable.

Additionally, once the inliner is merged, we won't need the "dummy" sections at all, so those go away entirely, and all of the ugliness of that particular part of the hack with it - i.e. no more modifying the caller's argument to insert a bookmark.

Quoted 6 lines

So, it's a little gross, but it's just a band-aid to get to where we want to be; and it isn't as gross as it looks.

With your suggested commentary changes, do I have your blessing to push forward? In the interim, please review the rest of the series so I can send you a new version.

Yes, you do.

In addition to the commit message changes, I'd like to have a comment in the code.

Perhaps as separate patch before your series:

scripts/qapi/parser.py +6 -0
index c3cf33904e..844afc6e0e 100644
@@ -852,6 +852,12 @@ def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
 
     def ensure_returns(self, info: QAPISourceInfo) -> None:
 
+        # This is more complicated than it ought to be.  The doc
+        # parser should already know where a generated RETURNS section
+        # should go.  It currently doesn't, mostly because it accepts
+        # tagged sections in any order.
+        # TODO Tighten doc syntax and simplify.
+
         def _insert_near_kind(
             kind: QAPIDoc.Kind,
             new_sect: QAPIDoc.Section,
Signature · 7 lines
2.54.0

We could adjust the comment within your series to also cover the new
uses of _insert_near_kind(), but I don't think that's necessary.  It
should serve as a reminder as is.

If that works for you, I'll post the patch.
Quoted 8 lines

--js

Quoted 5 lines
Quoted 4 lines

That's my story and I'm sticking to it.

author
#18 lore raw

Re: [PATCH v3 09/16] qapi/docs: adjust stub member insertion algorithm

Quoted 118 lines · Markus Armbruster
Quoted 108 lines · John Snow
Quoted 18 lines · Markus Armbruster

Quick first pass...

Quoted 6 lines · John Snow

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections. To accommodate this, the insertion algorithm that places stub and dummy members must be adjusted to cope with not only the finished state, but temporary intermediate states while the series is merged.

Perhaps:

A forthcoming patch removes the implicit PLAIN section that always starts a QAPIDoc section list. Further future changes begin converting "PLAIN" sections to "INTRO" sections.

Quoted 6 lines

This will affect the code that inserts "Not documented" descriptions for undocumented members ("stub sections") and the dummy section that marks the spot for "The members of ..." references.

Adjust the algorithm to cope with not only the finished state, but temporary intermediate states while the series is merged.

Sure.

Quoted 24 lines
Quoted 24 lines

This algorithm can handle zero-or-more PLAIN *or* INTRO sections at the beginning of a QAPIDoc object, in contrast to the previous algorithm which assumed and relied upon there being always one PLAIN section at the beginning of every QAPIDoc section list.

In other words: (PLAIN | INTRO)* <EverythingElse>

This does not impact what the parser itself will actually produce. As of this patch, the parser will still always generate QAPIDoc section lists that start with precisely one PLAIN section (whether or not it is empty), followed by the remaining sections. Those remaining sections may or may not include additional PLAIN sections, but never two such sections contiguously as the parser will always treat that layout as one PLAIN section consisting of multiple paragraph(s).

In other other words: This insertion algorithm is more lenient than the parser, but this is on purpose for flexibility mid-stream as we convert QAPI to using explicit introductory sections. The allowed order of sections will eventually become strictly enforced in the parser, which will in turn allow dramatic simplifications to the insertion algorithm. This only exists as transitory code until we are able to enforce that order.

Quoted 3 lines
Quoted 1 line

Fear not: the intermediate rest output before and after this patch

"ReST output"?

Sure

Quoted 23 lines
Quoted 4 lines

are byte identical, so failing all else, we at least know it doesn't make anything worse.

Quoted 14 lines

Lastly, because we have three places in the code that need to insert stub/dummy sections, we take the opportunity to consolidate this code to handle all three cases with one function. This winds up necessitating the qapidoc.py generator actually modify the section list to insert a "dummy" member that acts as a placeholder for "The members of ..." text. While it looks like a code smell to modify the caller's argument, it is ultimately safe because the QAPI Schema object is re-parsed and re-constructed in memory for each individual process that needs to operate on it. In other words, the Sphinx document generator already does have "its own copy" of the section lists, so it is "safe" to modify here without regards to other consumers of the QAPIDoc objects. It only *looks* like it smells bad. Ultimately, this code will also be removed once the inliner is merged, so it is only a temporary aesthetic issue regardless.

Such trickery, even when safe, risks making attentive readers go "WAT?!?" I find it tolerable only because we plan to replace it fairly soon.

Quoted 6 lines

The code finding the spot to insert stub/dummy sections is more complicated than it has any right to be, both before and after your patch. It reconstructs information the doc parser has, but doesn't pass on in usable form. Passing that info on would likely be simpler and cleaner. However, you tell me your inliner patches also simplify things, and they already exist. So let's go with those.

Right, the basic idea is this:

- Differentiate INTRO with new syntax
- Convert "PLAIN" into "DETAILS"
- Enforce strict section ordering: INTRO only at the beginning, DETAILS only at or quite near to the end

Once that is cemented, the insertion algorithm can become much simpler and dumber; e.g. using a map of regions where we can append to the end of a region without having to iterate through to find our sweet spot.

Since that's the plan for the inliner anyway, some temporary ugliness here in order to ensure that we do not regress even one byte seems acceptable.

Additionally, once the inliner is merged, we won't need the "dummy" sections at all, so those go away entirely, and all of the ugliness of that particular part of the hack with it - i.e. no more modifying the caller's argument to insert a bookmark.

Quoted 6 lines

So, it's a little gross, but it's just a band-aid to get to where we want to be; and it isn't as gross as it looks.

With your suggested commentary changes, do I have your blessing to push forward? In the interim, please review the rest of the series so I can send you a new version.

Quoted 6 lines

Yes, you do.

In addition to the commit message changes, I'd like to have a comment in the code.

Perhaps as separate patch before your series:

It's easy enough to add it as a separate patch right before this one instead of in a separate series. Added in with no rebasing issues, so I'm going to do it that way just to avoid any potential merging issues.

Quoted 36 lines
scripts/qapi/parser.py +6 -1
index c3cf33904e..844afc6e0e 100644
@@ -852,6 +852,12 @@ def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
 
     def ensure_returns(self, info: QAPISourceInfo) -> None:
 
+        # This is more complicated than it ought to be.  The doc
+        # parser should already know where a generated RETURNS section
+        # should go.  It currently doesn't, mostly because it accepts
+        # tagged sections in any order.
+        # TODO Tighten doc syntax and simplify.
+
         def _insert_near_kind(
             kind: QAPIDoc.Kind,
             new_sect: QAPIDoc.Section,
--

2.54.0

We could adjust the comment within your series to also cover the new uses of _insert_near_kind(), but I don't think that's necessary. It should serve as a reminder as is.

If that works for you, I'll post the patch.

Quoted 8 lines

--js

Quoted 5 lines
Quoted 4 lines

That's my story and I'm sticking to it.

author
#19 lore raw

PATCH v3 10/16 qapi/docs: remove implicit Plain section

In reply to John Snow

Prior to this patch, we always create an empty Plain section. Removing this allows us to gradually phase out the "Plain" section in favor of "Intro" and "Details" sections while keeping "Plain" around for the interim churn during the series - meaning that we don't actually know at __init__ time which type of section we'll have first.

---
 scripts/qapi/parser.py         |  4 +---
 tests/qapi-schema/doc-good.out | 14 --------------
 2 files changed, 1 insertion(+), 17 deletions(-)
scripts/qapi/parser.py +1 -3
index 406edf6aa9d..4beec4f9eda 100644
@@ -730,9 +730,7 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         # definition doc's symbol, None for free-form doc
         self.symbol: Optional[str] = symbol
         # the sections in textual order
-        self.all_sections: List[QAPIDoc.Section] = [
-            QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
-        ]
+        self.all_sections: List[QAPIDoc.Section] = []
         # dicts mapping parameter/feature names to their description
         self._args: Dict[str, QAPIDoc.ArgSection] = {}
         self._features: Dict[str, QAPIDoc.ArgSection] = {}
tests/qapi-schema/doc-good.out +0 -14
index b9829e2f841..16f44221771 100644
@@ -106,8 +106,6 @@ Examples:
 - *verbatim*
 - {braces}
 doc symbol=Enum
-    Plain
-
     Member=one
 The _one_ {and only}, description on the same line
     Member=two
@@ -119,8 +117,6 @@ a member feature
     Plain
 @two is undocumented
 doc symbol=Base
-    Plain
-
     Member=base1
  description starts on a new line,
  minimally indented
@@ -138,16 +134,10 @@ a feature
     Feature=member-feat
 a member feature
 doc symbol=Variant2
-    Plain
-
 doc symbol=Object
-    Plain
-
     Feature=union-feat1
 a feature
 doc symbol=Alternate
-    Plain
-
     Member=i
 description starts on the same line
     remainder indented the same
@@ -161,8 +151,6 @@ doc freeform
 Another subsection
 ==================
 doc symbol=cmd
-    Plain
-
     Member=arg1
     description starts on a new line,
     indented
@@ -223,7 +211,5 @@ another feature
 
    <- ... has no title ...
 doc symbol=EVT_BOXED
-    Plain
-
     Feature=feat3
 a feature
Signature · 1 line
2.54.0
author
#20 lore raw

PATCH v3 11/16 qapi/docs: add rendering for INTRO sections

In reply to John Snow

Amend the qapidoc generator to handle and render INTRO sections.

The only real difference here from other sections is that we need to dedent the text so it renders correctly. Members and Features are also indented, but do not require a dedent() because they are always used in tandem with an rST construct that forms the start of a new indented block; there is coincidental harmony.

Plaintext sections, however, do not start their own block and thus need to be dedented to prevent accidentally rendering them as a blockquote or a syntax error.

This dedent transformation on the text does not reflow the text, so source line information remains accurate, and the "blame" chain of custody for sphinx rST parsing error messages continues to be correct even through this transformation.

---
 docs/sphinx/qapidoc.py | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
docs/sphinx/qapidoc.py +12 -3
index 16ad15fe94f..317dc44b1b8 100644
@@ -35,6 +35,7 @@
 from pathlib import Path
 import re
 import sys
+import textwrap
 from typing import TYPE_CHECKING
 
 from docutils import nodes
@@ -150,8 +151,15 @@ def add_lines(
         self,
         content: str,
         info: QAPISourceInfo,
+        dedent: bool = False,
     ) -> None:
         lines = content.splitlines(True)
+
+        if dedent:
+            lines = textwrap.dedent(content).splitlines(True)
+        else:
+            lines = content.splitlines(True)
+
         for i, line in enumerate(lines):
             self.add_line_raw(line, info.fname, info.line + i)
 
@@ -223,13 +231,14 @@ def reformat_arobase(text: str) -> str:
 
     # Transmogrification helpers
 
-    def visit_paragraph(self, section: QAPIDoc.Section) -> None:
+    def visit_plaintext(self, section: QAPIDoc.Section) -> None:
         # Squelch empty paragraphs.
         if not section.text:
             return
 
+        dedent = bool(section.kind == QAPIDoc.Kind.INTRO)
         self.ensure_blank_line()
-        self.add_lines(section.text, section.info)
+        self.add_lines(section.text, section.info, dedent)
         self.ensure_blank_line()
 
     def visit_member(self, section: QAPIDoc.ArgSection) -> None:
@@ -373,7 +382,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
             section.text = self.reformat_arobase(section.text)
 
             if section.kind.name in ("PLAIN", "INTRO"):
-                self.visit_paragraph(section)
+                self.visit_plaintext(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
                 if section.name == "q_dummy":
Signature · 1 line
2.54.0
#21 lore raw

Re: [PATCH v3 11/16] qapi/docs: add rendering for INTRO sections

Quoted 51 lines · John Snow

Amend the qapidoc generator to handle and render INTRO sections.

The only real difference here from other sections is that we need to dedent the text so it renders correctly. Members and Features are also indented, but do not require a dedent() because they are always used in tandem with an rST construct that forms the start of a new indented block; there is coincidental harmony.

Plaintext sections, however, do not start their own block and thus need to be dedented to prevent accidentally rendering them as a blockquote or a syntax error.

This dedent transformation on the text does not reflow the text, so source line information remains accurate, and the "blame" chain of custody for sphinx rST parsing error messages continues to be correct even through this transformation.

---
 docs/sphinx/qapidoc.py | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
docs/sphinx/qapidoc.py +8 -0
index 16ad15fe94f..317dc44b1b8 100644
@@ -35,6 +35,7 @@
 from pathlib import Path
 import re
 import sys
+import textwrap
 from typing import TYPE_CHECKING
 
 from docutils import nodes
@@ -150,8 +151,15 @@ def add_lines(
         self,
         content: str,
         info: QAPISourceInfo,
+        dedent: bool = False,
     ) -> None:
         lines = content.splitlines(True)
+
+        if dedent:
+            lines = textwrap.dedent(content).splitlines(True)
+        else:
+            lines = content.splitlines(True)
+
         for i, line in enumerate(lines):
             self.add_line_raw(line, info.fname, info.line + i)
 

docs/sphinx/qapidoc.py def reformat_arobase(text: str) -> str: · line 239

Quoted 10 lines
docs/sphinx/qapidoc.py +2 -1
@@ -223,13 +231,14 @@ def reformat_arobase(text: str) -> str:
 
     # Transmogrification helpers
 
-    def visit_paragraph(self, section: QAPIDoc.Section) -> None:
+    def visit_plaintext(self, section: QAPIDoc.Section) -> None:
         # Squelch empty paragraphs.
         if not section.text:
             return
 
+        dedent = bool(section.kind == QAPIDoc.Kind.INTRO)

Could use a comment explaining why INTRO needs to be dedented.

Quoted 15 lines
docs/sphinx/qapidoc.py +2 -2
         self.ensure_blank_line()
-        self.add_lines(section.text, section.info)
+        self.add_lines(section.text, section.info, dedent)
         self.ensure_blank_line()
 
     def visit_member(self, section: QAPIDoc.ArgSection) -> None:
@@ -373,7 +382,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
             section.text = self.reformat_arobase(section.text)
 
             if section.kind.name in ("PLAIN", "INTRO"):
-                self.visit_paragraph(section)
+                self.visit_plaintext(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
                 if section.name == "q_dummy":
author
#22 lore raw

Re: [PATCH v3 11/16] qapi/docs: add rendering for INTRO sections

Quoted 63 lines · Markus Armbruster
Quoted 60 lines · John Snow

Amend the qapidoc generator to handle and render INTRO sections.

The only real difference here from other sections is that we need to dedent the text so it renders correctly. Members and Features are also indented, but do not require a dedent() because they are always used in tandem with an rST construct that forms the start of a new indented block; there is coincidental harmony.

Plaintext sections, however, do not start their own block and thus need to be dedented to prevent accidentally rendering them as a blockquote or a syntax error.

This dedent transformation on the text does not reflow the text, so source line information remains accurate, and the "blame" chain of custody for sphinx rST parsing error messages continues to be correct even through this transformation.

---
 docs/sphinx/qapidoc.py | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
docs/sphinx/qapidoc.py +9 -1
index 16ad15fe94f..317dc44b1b8 100644
@@ -35,6 +35,7 @@
 from pathlib import Path
 import re
 import sys
+import textwrap
 from typing import TYPE_CHECKING
 
 from docutils import nodes
@@ -150,8 +151,15 @@ def add_lines(
         self,
         content: str,
         info: QAPISourceInfo,
+        dedent: bool = False,
     ) -> None:
         lines = content.splitlines(True)
+
+        if dedent:
+            lines = textwrap.dedent(content).splitlines(True)
+        else:
+            lines = content.splitlines(True)
+
         for i, line in enumerate(lines):
             self.add_line_raw(line, info.fname, info.line + i)
 
@@ -223,13 +231,14 @@ def reformat_arobase(text: str) -> str:
 
     # Transmogrification helpers
 
-    def visit_paragraph(self, section: QAPIDoc.Section) -> None:
+    def visit_plaintext(self, section: QAPIDoc.Section) -> None:
         # Squelch empty paragraphs.
         if not section.text:
             return
 
Quoted 3 lines

docs/sphinx/qapidoc.py def reformat_arobase(text: str) -> str: · line 239

Quoted 1 line
docs/sphinx/qapidoc.py +1 -0
+        dedent = bool(section.kind == QAPIDoc.Kind.INTRO)

Could use a comment explaining why INTRO needs to be dedented.

# Please see commit message

(Joking, I added a comment.)

Quoted 17 lines
Quoted 16 lines
docs/sphinx/qapidoc.py +2 -2
         self.ensure_blank_line()
-        self.add_lines(section.text, section.info)
+        self.add_lines(section.text, section.info, dedent)
         self.ensure_blank_line()
 
     def visit_member(self, section: QAPIDoc.ArgSection) -> None:
@@ -373,7 +382,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
             section.text = self.reformat_arobase(section.text)
 
             if section.kind.name in ("PLAIN", "INTRO"):
-                self.visit_paragraph(section)
+                self.visit_plaintext(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
                 if section.name == "q_dummy":
author
#23 lore raw

PATCH v3 12/16 qapi/docs: add "Intro" section parsing

In reply to John Snow

Add parsing for explicit Intro section syntax.

A side effect of this patch is that we will (currently) always create an empty Intro section, similar to how we used to have an empty Plain section. The tests are adjusted accordingly, rendered document output does not change at all.

---
 docs/devel/qapi-code-gen.rst   | 16 +++++++---------
 scripts/qapi/parser.py         |  4 ++--
 tests/qapi-schema/doc-good.out | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+), 11 deletions(-)
docs/devel/qapi-code-gen.rst +7 -9
index 3a632b4a648..b1cc5b5f0db 100644
@@ -984,11 +984,11 @@ definition it documents.
 When documentation is required (see pragma_ 'doc-required'), every
 definition must have documentation.
 
-Definition documentation starts with a line naming the definition,
-followed by an optional overview, a description of each argument (for
-commands and events), member (for structs and unions), branch (for
-alternates), or value (for enums), a description of each feature (if
-any), and finally optional tagged sections.
+Definition documentation starts with a description naming the
+definition with an optional indented overview, a description of each
+argument (for commands and events), member (for structs and unions),
+branch (for alternates), or value (for enums), a description of each
+feature (if any), and finally optional tagged sections.
 
 Descriptions start with '\@name:'.  The description text must be
 indented like this::
@@ -1093,8 +1093,7 @@ Examples of complete definition documentation::
 
  ##
  # @BlockStats:
- #
- # Statistics of a virtual block device or a block backing device.
+ #     Statistics of a virtual block device or a block backing device.
  #
  # @device: If the stats are for a virtual block device, the name
  #     corresponding to the virtual block device.
@@ -1111,8 +1110,7 @@ Examples of complete definition documentation::
 
  ##
  # @query-blockstats:
- #
- # Query the @BlockStats for all virtual block devices.
+ #     Query the @BlockStats for all virtual block devices.
  #
  # @query-nodes: If true, the command will query all the block nodes
  #     ... explain, explain ...
scripts/qapi/parser.py +2 -2
index 4beec4f9eda..2afbf6a9c81 100644
@@ -542,8 +542,8 @@ def get_doc(self) -> 'QAPIDoc':
             if not symbol:
                 raise QAPIParseError(self, "name required after '@'")
             doc = QAPIDoc(info, symbol)
-            self.accept(False)
-            line = self.get_doc_line()
+            doc.all_sections.append(QAPIDoc.Section(info, QAPIDoc.Kind.INTRO))
+            line = self.get_doc_indented(doc)
             no_more_args = False
 
             while line is not None:
tests/qapi-schema/doc-good.out +18 -0
index 16f44221771..371dd25ffc7 100644
@@ -106,6 +106,8 @@ Examples:
 - *verbatim*
 - {braces}
 doc symbol=Enum
+    Intro
+
     Member=one
 The _one_ {and only}, description on the same line
     Member=two
@@ -117,10 +119,14 @@ a member feature
     Plain
 @two is undocumented
 doc symbol=Base
+    Intro
+
     Member=base1
  description starts on a new line,
  minimally indented
 doc symbol=Variant1
+    Intro
+
     Plain
 A paragraph
 
@@ -134,10 +140,16 @@ a feature
     Feature=member-feat
 a member feature
 doc symbol=Variant2
+    Intro
+
 doc symbol=Object
+    Intro
+
     Feature=union-feat1
 a feature
 doc symbol=Alternate
+    Intro
+
     Member=i
 description starts on the same line
     remainder indented the same
@@ -151,6 +163,8 @@ doc freeform
 Another subsection
 ==================
 doc symbol=cmd
+    Intro
+
     Member=arg1
     description starts on a new line,
     indented
@@ -198,6 +212,8 @@ Note::
     Since
 2.10
 doc symbol=cmd-boxed
+    Intro
+
     Plain
 If you're bored enough to read this, go see a video of boxed cats
     Feature=cmd-feat1
@@ -211,5 +227,7 @@ another feature
 
    <- ... has no title ...
 doc symbol=EVT_BOXED
+    Intro
+
     Feature=feat3
 a feature
Signature · 1 line
2.54.0
#24 lore raw

Re: [PATCH v3 12/16] qapi/docs: add "Intro" section parsing

Quoted 25 lines · John Snow

Add parsing for explicit Intro section syntax.

A side effect of this patch is that we will (currently) always create an empty Intro section, similar to how we used to have an empty Plain section. The tests are adjusted accordingly, rendered document output does not change at all.

---
 docs/devel/qapi-code-gen.rst   | 16 +++++++---------
 scripts/qapi/parser.py         |  4 ++--
 tests/qapi-schema/doc-good.out | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+), 11 deletions(-)
docs/devel/qapi-code-gen.rst +0 -3
index 3a632b4a648..b1cc5b5f0db 100644
@@ -984,11 +984,11 @@ definition it documents.
 When documentation is required (see pragma_ 'doc-required'), every
 definition must have documentation.
 
-Definition documentation starts with a line naming the definition,
-followed by an optional overview, a description of each argument (for
-commands and events), member (for structs and unions), branch (for

docs/devel/qapi-code-gen.rst definition it documents. · line 994

Quoted 10 lines
docs/devel/qapi-code-gen.rst +5 -2
-alternates), or value (for enums), a description of each feature (if
-any), and finally optional tagged sections.
+Definition documentation starts with a description naming the
+definition with an optional indented overview, a description of each
+argument (for commands and events), member (for structs and unions),
+branch (for alternates), or value (for enums), a description of each
+feature (if any), and finally optional tagged sections.
 
 Descriptions start with '\@name:'.  The description text must be
 indented like this::

This works if we convert the entire schema in the same series. I guess that's the plan. Is it?

Quoted 20 lines
docs/devel/qapi-code-gen.rst +2 -4
@@ -1093,8 +1093,7 @@ Examples of complete definition documentation::
 
  ##
  # @BlockStats:
- #
- # Statistics of a virtual block device or a block backing device.
+ #     Statistics of a virtual block device or a block backing device.
  #
  # @device: If the stats are for a virtual block device, the name
  #     corresponding to the virtual block device.
@@ -1111,8 +1110,7 @@ Examples of complete definition documentation::
 
  ##
  # @query-blockstats:
- #
- # Query the @BlockStats for all virtual block devices.
+ #     Query the @BlockStats for all virtual block devices.
  #
  # @query-nodes: If true, the command will query all the block nodes
  #     ... explain, explain ...

[...]

author
#25 lore raw

Re: [PATCH v3 12/16] qapi/docs: add "Intro" section parsing

Quoted 36 lines · Markus Armbruster
Quoted 33 lines · John Snow

Add parsing for explicit Intro section syntax.

A side effect of this patch is that we will (currently) always create an empty Intro section, similar to how we used to have an empty Plain section. The tests are adjusted accordingly, rendered document output does not change at all.

---
 docs/devel/qapi-code-gen.rst   | 16 +++++++---------
 scripts/qapi/parser.py         |  4 ++--
 tests/qapi-schema/doc-good.out | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+), 11 deletions(-)
docs/devel/qapi-code-gen.rst +5 -5
index 3a632b4a648..b1cc5b5f0db 100644
@@ -984,11 +984,11 @@ definition it documents.
 When documentation is required (see pragma_ 'doc-required'), every
 definition must have documentation.
 
-Definition documentation starts with a line naming the definition,
-followed by an optional overview, a description of each argument (for
-commands and events), member (for structs and unions), branch (for
-alternates), or value (for enums), a description of each feature (if
-any), and finally optional tagged sections.
+Definition documentation starts with a description naming the
+definition with an optional indented overview, a description of each
+argument (for commands and events), member (for structs and unions),
+branch (for alternates), or value (for enums), a description of each
+feature (if any), and finally optional tagged sections.
 
Quoted 5 lines

docs/devel/qapi-code-gen.rst definition it documents. · line 994

Quoted 2 lines
docs/devel/qapi-code-gen.rst +0 -0
 Descriptions start with '\@name:'.  The description text must be
 indented like this::

This works if we convert the entire schema in the same series. I guess that's the plan. Is it?

I still plan to do it piecemeal: I will probably send everything out in a big series, but you will be able to cherry-pick and stage those patches individually as you feel they are ready. This way, modules that need further revisions don't hold up the entire series and v2, v3, etc won't email bomb all of creation just to fix an indent in a single module somewhere.

I think this documentation still covers it just fine: "an optional indented overview". Technically the things we haven't converted just omit that indented overview. Those overviews just remain as "plaintext".

It's not perfect documentation, but the expected window where we have two competing systems should be quite small in the grand scheme, so I don't think it's worth writing docs to cover the technicalities of the transition.

--js

Quoted 24 lines
Quoted 21 lines
docs/devel/qapi-code-gen.rst +2 -4
@@ -1093,8 +1093,7 @@ Examples of complete definition documentation::
 
  ##
  # @BlockStats:
- #
- # Statistics of a virtual block device or a block backing device.
+ #     Statistics of a virtual block device or a block backing device.
  #
  # @device: If the stats are for a virtual block device, the name
  #     corresponding to the virtual block device.
@@ -1111,8 +1110,7 @@ Examples of complete definition documentation::
 
  ##
  # @query-blockstats:
- #
- # Query the @BlockStats for all virtual block devices.
+ #     Query the @BlockStats for all virtual block devices.
  #
  # @query-nodes: If true, the command will query all the block nodes
  #     ... explain, explain ...

[...]

#26 lore raw

Re: [PATCH v3 12/16] qapi/docs: add "Intro" section parsing

Quoted 43 lines · John Snow
Quoted 36 lines · Markus Armbruster
Quoted 33 lines · John Snow

Add parsing for explicit Intro section syntax.

A side effect of this patch is that we will (currently) always create an empty Intro section, similar to how we used to have an empty Plain section. The tests are adjusted accordingly, rendered document output does not change at all.

---
 docs/devel/qapi-code-gen.rst   | 16 +++++++---------
 scripts/qapi/parser.py         |  4 ++--
 tests/qapi-schema/doc-good.out | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+), 11 deletions(-)
docs/devel/qapi-code-gen.rst +5 -5
index 3a632b4a648..b1cc5b5f0db 100644
@@ -984,11 +984,11 @@ definition it documents.
 When documentation is required (see pragma_ 'doc-required'), every
 definition must have documentation.
 
-Definition documentation starts with a line naming the definition,
-followed by an optional overview, a description of each argument (for
-commands and events), member (for structs and unions), branch (for
-alternates), or value (for enums), a description of each feature (if
-any), and finally optional tagged sections.
+Definition documentation starts with a description naming the
+definition with an optional indented overview, a description of each
+argument (for commands and events), member (for structs and unions),
+branch (for alternates), or value (for enums), a description of each
+feature (if any), and finally optional tagged sections.
 
Quoted 5 lines

docs/devel/qapi-code-gen.rst definition it documents. · line 994

Quoted 2 lines
docs/devel/qapi-code-gen.rst +0 -0
 Descriptions start with '\@name:'.  The description text must be
 indented like this::

This works if we convert the entire schema in the same series. I guess that's the plan. Is it?

Quoted 6 lines

I still plan to do it piecemeal: I will probably send everything out in a big series, but you will be able to cherry-pick and stage those patches individually as you feel they are ready. This way, modules that need further revisions don't hold up the entire series and v2, v3, etc won't email bomb all of creation just to fix an indent in a single module somewhere.

We'll see.

The important bit is separating "split the initial part into intro (to be discarded by the inliner) and non-intro (to be kept)" from purely mechanical "make the entire initial part syntactically intro".

The transition to intro syntax will likely be bothersome for contributors, and I feel kind of bad about it. But I also feel we do need to make intro syntactically obvious. We could try to ease the pain by silently accepting old-style patches (not using intro syntax) for a while, then have another bulk conversion.

Quoted 4 lines

I think this documentation still covers it just fine: "an optional indented overview". Technically the things we haven't converted just omit that indented overview. Those overviews just remain as "plaintext".

You're right. "Optional" carries even more weight here than I thought.

Quoted 4 lines

It's not perfect documentation, but the expected window where we have two competing systems should be quite small in the grand scheme, so I don't think it's worth writing docs to cover the technicalities of the transition.

Okay.

[...]

author
#27 lore raw

PATCH v3 13/16 qapi: convert intro sections for accelerator.json

In reply to John Snow
---
 qapi/accelerator.json | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
qapi/accelerator.json +6 -12
index b59df378664..efc0fa81418 100644
@@ -13,8 +13,7 @@
 
 ##
 # @KvmInfo:
-#
-# Information about support for KVM acceleration
+#     Information about support for KVM acceleration
 #
 # @enabled: true if KVM acceleration is active
 #
@@ -26,8 +25,7 @@
 
 ##
 # @query-kvm:
-#
-# Return information about KVM acceleration
+#     Return information about KVM acceleration
 #
 # Features:
 #
@@ -46,8 +44,7 @@
 
 ##
 # @x-accel-stats:
-#
-# Query accelerator statistics
+#     Query accelerator statistics
 #
 # Features:
 #
@@ -63,8 +60,7 @@
 
 ##
 # @Accelerator:
-#
-# Information about support for MSHV acceleration
+#     Information about support for MSHV acceleration
 #
 # @hvf: Apple Hypervisor.framework
 #
@@ -88,8 +84,7 @@
 
 ##
 # @AcceleratorInfo:
-#
-# Information about support for various accelerators
+#     Information about support for various accelerators
 #
 # @enabled: the accelerator that is in use
 #
@@ -102,8 +97,7 @@
 
 ##
 # @query-accelerators:
-#
-# Return information about accelerators
+#     Return information about accelerators
 #
 # Returns: @AcceleratorInfo
 #
Signature · 1 line
2.54.0
author
#28 lore raw

PATCH v3 14/16 qapi: convert intro sections for acpi-hest.json

In reply to John Snow
---
 qapi/acpi-hest.json | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
qapi/acpi-hest.json +2 -2
index a01f1dee095..8db804df197 100644
@@ -13,8 +13,8 @@
 
 ##
 # @inject-ghes-v2-error:
-#
-# Inject an error with additional ACPI 6.1 GHESv2 error information
+#     Inject an error with additional ACPI 6.1 GHESv2 error
+#     information
 #
 # @cper: contains a base64 encoded string with raw data for a single
 #     CPER record with Generic Error Status Block, Generic Error Data
Signature · 1 line
2.54.0
author
#29 lore raw

PATCH v3 15/16 qapi: convert intro sections for acpi.json

In reply to John Snow
---
 qapi/acpi.json | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
qapi/acpi.json +7 -11
index 906b3687a55..82e82d7cf89 100644
@@ -13,8 +13,7 @@
 
 ##
 # @AcpiTableOptions:
-#
-# Specify an ACPI table on the command line to load.
+#     Specify an ACPI table on the command line to load.
 #
 # At most one of @file and @data can be specified.  The list of files
 # specified by any one of them is loaded and concatenated in order.
@@ -81,10 +80,9 @@
 
 ##
 # @ACPIOSTInfo:
-#
-# OSPM Status Indication for a device.  For description of possible
-# values of @source and @status fields see "_OST (OSPM Status
-# Indication)" chapter of ACPI5.0 spec.
+#     OSPM Status Indication for a device.  For description
+#     of possible values of @source and @status fields see "_OST (OSPM
+#     Status Indication)" chapter of ACPI5.0 spec.
 #
 # @device: device ID associated with slot
 #
@@ -107,9 +105,8 @@
 
 ##
 # @query-acpi-ospm-status:
-#
-# Return a list of `ACPIOSTInfo` for devices that support status
-# reporting via ACPI _OST method.
+#     Return a list of `ACPIOSTInfo` for devices
+#     that support status reporting via ACPI _OST method.
 #
 # Since: 2.1
 #
@@ -126,8 +123,7 @@
 
 ##
 # @ACPI_DEVICE_OST:
-#
-# Emitted when guest executes ACPI _OST method.
+#     Emitted when guest executes ACPI _OST method.
 #
 # @info: OSPM Status Indication
 #
Signature · 1 line
2.54.0
author
#30 lore raw

PATCH v3 16/16 qapi: convert intro sections for audio.json

In reply to John Snow
---
 qapi/audio.json | 88 ++++++++++++++++++-------------------------------
 1 file changed, 32 insertions(+), 56 deletions(-)
qapi/audio.json +32 -56
index 2df87b97101..72190f20993 100644
@@ -14,9 +14,8 @@
 
 ##
 # @AudiodevPerDirectionOptions:
-#
-# General audio backend options that are used for both playback and
-# recording.
+#     General audio backend options that are used for both playback
+#     and recording.
 #
 # @mixing-engine: use QEMU's mixing engine to mix all streams inside
 #     QEMU and convert audio formats when not supported by the
@@ -53,8 +52,7 @@
 
 ##
 # @AudiodevGenericOptions:
-#
-# Generic driver-specific options.
+#     Generic driver-specific options.
 #
 # @in: options of the capture stream
 #
@@ -69,8 +67,7 @@
 
 ##
 # @AudiodevDBusOptions:
-#
-# Options of the D-Bus audio backend.
+#     Options of the D-Bus audio backend.
 #
 # @in: options of the capture stream
 #
@@ -89,9 +86,8 @@
 
 ##
 # @AudiodevAlsaPerDirectionOptions:
-#
-# Options of the ALSA backend that are used for both playback and
-# recording.
+#     Options of the ALSA backend that are used for both playback and
+#     recording.
 #
 # @dev: the name of the ALSA device to use (default 'default')
 #
@@ -111,8 +107,7 @@
 
 ##
 # @AudiodevAlsaOptions:
-#
-# Options of the ALSA audio backend.
+#     Options of the ALSA audio backend.
 #
 # @in: options of the capture stream
 #
@@ -130,8 +125,7 @@
 
 ##
 # @AudiodevSndioOptions:
-#
-# Options of the sndio audio backend.
+#     Options of the sndio audio backend.
 #
 # @in: options of the capture stream
 #
@@ -152,9 +146,8 @@
 
 ##
 # @AudiodevCoreaudioPerDirectionOptions:
-#
-# Options of the Core Audio backend that are used for both playback
-# and recording.
+#     Options of the Core Audio backend that are used for both
+#     playback and recording.
 #
 # @buffer-count: number of buffers
 #
@@ -167,8 +160,7 @@
 
 ##
 # @AudiodevCoreaudioOptions:
-#
-# Options of the coreaudio audio backend.
+#     Options of the coreaudio audio backend.
 #
 # @in: options of the capture stream
 #
@@ -183,8 +175,7 @@
 
 ##
 # @AudiodevDsoundOptions:
-#
-# Options of the DirectSound audio backend.
+#     Options of the DirectSound audio backend.
 #
 # @in: options of the capture stream
 #
@@ -203,9 +194,8 @@
 
 ##
 # @AudiodevJackPerDirectionOptions:
-#
-# Options of the JACK backend that are used for both playback and
-# recording.
+#     Options of the JACK backend that are used for both playback and
+#     recording.
 #
 # @server-name: select from among several possible concurrent server
 #     instances (default: environment variable $JACK_DEFAULT_SERVER if
@@ -237,8 +227,7 @@
 
 ##
 # @AudiodevJackOptions:
-#
-# Options of the JACK audio backend.
+#     Options of the JACK audio backend.
 #
 # @in: options of the capture stream
 #
@@ -253,9 +242,8 @@
 
 ##
 # @AudiodevOssPerDirectionOptions:
-#
-# Options of the OSS backend that are used for both playback and
-# recording.
+#     Options of the OSS backend that are used for both playback and
+#     recording.
 #
 # @dev: file name of the OSS device (default '/dev/dsp')
 #
@@ -275,8 +263,7 @@
 
 ##
 # @AudiodevOssOptions:
-#
-# Options of the OSS audio backend.
+#     Options of the OSS audio backend.
 #
 # @in: options of the capture stream
 #
@@ -305,9 +292,8 @@
 
 ##
 # @AudiodevPaPerDirectionOptions:
-#
-# Options of the Pulseaudio backend that are used for both playback
-# and recording.
+#     Options of the Pulseaudio backend that are used for both
+#     playback and recording.
 #
 # @name: name of the sink/source to use
 #
@@ -330,8 +316,7 @@
 
 ##
 # @AudiodevPaOptions:
-#
-# Options of the PulseAudio audio backend.
+#     Options of the PulseAudio audio backend.
 #
 # @in: options of the capture stream
 #
@@ -349,9 +334,8 @@
 
 ##
 # @AudiodevPipewirePerDirectionOptions:
-#
-# Options of the PipeWire backend that are used for both playback and
-# recording.
+#     Options of the PipeWire backend that are used for both playback
+#     and recording.
 #
 # @name: name of the sink/source to use
 #
@@ -374,8 +358,7 @@
 
 ##
 # @AudiodevPipewireOptions:
-#
-# Options of the PipeWire audio backend.
+#     Options of the PipeWire audio backend.
 #
 # @in: options of the capture stream
 #
@@ -390,9 +373,8 @@
 
 ##
 # @AudiodevSdlPerDirectionOptions:
-#
-# Options of the SDL audio backend that are used for both playback and
-# recording.
+#     Options of the SDL audio backend that are used for both playback
+#     and recording.
 #
 # @buffer-count: number of buffers (default 4)
 #
@@ -405,8 +387,7 @@
 
 ##
 # @AudiodevSdlOptions:
-#
-# Options of the SDL audio backend.
+#     Options of the SDL audio backend.
 #
 # @in: options of the recording stream
 #
@@ -421,8 +402,7 @@
 
 ##
 # @AudiodevWavOptions:
-#
-# Options of the wav audio backend.
+#     Options of the wav audio backend.
 #
 # @in: options of the capture stream
 #
@@ -440,8 +420,7 @@
 
 ##
 # @AudioFormat:
-#
-# An enumeration of possible audio formats.
+#     An enumeration of possible audio formats.
 #
 # @u8: unsigned 8 bit integer
 #
@@ -464,8 +443,7 @@
 
 ##
 # @AudiodevDriver:
-#
-# An enumeration of possible audio backend drivers.
+#     An enumeration of possible audio backend drivers.
 #
 # @jack: JACK audio backend (since 5.1)
 #
@@ -488,8 +466,7 @@
 
 ##
 # @Audiodev:
-#
-# Options of an audio backend.
+#     Options of an audio backend.
 #
 # @id: identifier of the backend
 #
@@ -534,8 +511,7 @@
 
 ##
 # @query-audiodevs:
-#
-# Return information about audiodev configuration
+#     Return information about audiodev configuration
 #
 # Since: 8.0
 ##
Signature · 1 line
2.54.0