Skip to contentSkip to conversation

docs: add basic sphinx-domain rST generator to qapidoc

PATCH In review John Snow 2 participants 26 messages Jan 14, 2025 – Jan 16, 2025

lore
Share & tools
lore

John Snow

author
#0 lore raw

v2 cover docs: add basic sphinx-domain rST generator to qapidoc

based-on: https://patchew.org/QEMU/[email protected]/

Hi!

This series is a very, very barebones implementation for the new QAPI doc generator. It does not have many features that I presented on at KVM Forum; the point of this patch set is instead to present a stripped down basis for ongoing work so we can discuss on-list with full context of the code available to do so.

The documentation this series generates is *not suitable* for replacing the current document generator, it has a few glaring omissions - on purpose - those features have been factored out intentionally so they can be reviewed with fuller context and more careful review.

What this series does:

- Adds the new "Transmogrifier" rST generator to qapidoc.py, which generates an in-memory rST document using qapi-domain directives. - Adds a test document that showcases this new transmogrifier. (located at build/docs/manual/qapi/index.html; also visible at the bottom of the left column TOC on generated docs.)

What this series currently *regresses* /if/ it were to replace the current doc generator:

- "ifcond" data for anything other than top-level entities is not considered or rendered. This means "if" statements for features and members are entirely absent. This includes "special features" that have been hoisted into the preamble as well, such as :unstable: or :deprecated:. This issue is largely unaddressed in my WIP so far, I need to solve this in future patch discussion.

- *branches* are themselves not considered at all; they're skipped entirely for now. They will be included alongside the inliner later.

- "inherited" members (Markus hates that I use this term, I'm sorry, I know it isn't 1:1 with OO inheritance I just don't have a nicer word that my brain remembers, forgive me) are skipped; "The members of ..." messages are not rendered either. This is addressed by the inliner.

- Undocumented members are not auto-generated.

Other things unhandled, but are not regressions per se:

- The inliner is not present at all. This series renders only documentation exactly as it is exists in the source files.

- Undocumented return types are not autogenerated.

- Pseudofeatures (Things like allow-oob) are not generated as documented features.

- Documentation culling: all entities are documented whether or not they're relevant to the wire format.

Everything else *should* be no worse off than it was, and in many cases better.

My current goal is to tentatively agree that these patches look good (save for some minor nits like terminology/comments/docs/commit nits), then to send an updated version of this series with additional commits that start addressing the regressions and adding the new features like the inliner.

Once the new generator as a whole looks good, I will shift focus back to review on the qapi-domain extension itself. The reason for doing the review in reverse order is because the design of the generator informs the design of the domain, but reviewers are currently more familiar with the generator - so we're doing the review backwards. Treat it as a magical black box for now that somehow magically transforms rST into pretty docs.

V2:
- Mostly adjusted commit messages and comments per list feedback
- Changed "tag" to "kind" and adjusted code, comments and commits accordingly
- removed assertion around special features + conditionals, for now

John Snow (23):
docs/qapidoc: support header-less freeform sections qapi/parser: adjust info location for doc body section docs/qapidoc: remove example section support
qapi: expand tags to all doc sections
qapi/schema: add __repr__ to QAPIDoc.Section docs/qapidoc: add transmogrifier stub
docs/qapidoc: add transmogrifier class stub docs/qapidoc: add visit_module() method
qapi/source: allow multi-line QAPISourceInfo advancing docs/qapidoc: add visit_freeform() method docs/qapidoc: add preamble() method
docs/qapidoc: add visit_paragraph() method docs/qapidoc: add visit_errors() method
docs/qapidoc: add format_type() method
docs/qapidoc: add add_field() and generate_field() helper methods docs/qapidoc: add visit_feature() method
docs/qapidoc: prepare to record entity being transmogrified docs/qapidoc: add visit_returns() method
docs/qapidoc: add visit_member() method
docs/qapidoc: add visit_sections() method docs/qapidoc: add visit_entity()
docs/qapidoc: implement transmogrify() method docs/qapidoc: add transmogrifier test document

 docs/index.rst                 |   1 +
 docs/qapi/index.rst            |  53 +++++
 docs/sphinx/qapidoc.py         | 418 ++++++++++++++++++++++++++++++---
 scripts/qapi/parser.py         | 118 +++++++---
 scripts/qapi/source.py         |   4 +-
 tests/qapi-schema/doc-good.out |  10 +-
 tests/qapi-schema/test-qapi.py |   2 +-
 7 files changed, 541 insertions(+), 65 deletions(-)
 create mode 100644 docs/qapi/index.rst
Signature · 1 line
2.47.1
author
#1 lore raw

PATCH v2 1/23 docs/qapidoc: support header-less freeform sections

The code as written can't handle if a header isn't found and will crash, because node will be uninitialized. If we don't have a section title, create a generic block to insert text into instead.

(This patch also removes a lingering pylint warning in the QAPIDoc implementation that prevents getting a clean baseline to use for forthcoming additions.)

I am not attempting to *fully* clean up the existing QAPIDoc implementation in pylint because I intend to delete it anyway; this patch merely accomplishes a baseline under a specific pylint configuration:

PYTHONPATH=../../scripts/ pylint --disable=fixme,too-many-lines,\
consider-using-f-string,missing-docstring,unused-argument,\
too-many-arguments,too-many-positional-arguments,\
too-many-public-methods \
qapidoc.py

(under at least pylint 3.3.1; more robust tamping down of the environment needed to consistently perform checks will happen later - hopefully soon, sorry for the inconvenience.)

This at least ensures there aren't regressions outside of these general warnings in the new qapidoc.py code to be committed.

---
 docs/sphinx/qapidoc.py | 2 ++
 1 file changed, 2 insertions(+)
docs/sphinx/qapidoc.py +2 -0
index 5f96b46270b..5a4d7388b29 100644
@@ -421,6 +421,8 @@ def freeform(self, doc):
             node = self._start_new_heading(heading, len(leader))
             if text == '':
                 return
+        else:
+            node = nodes.container()
 
         self._parse_text_into_node(text, node)
         self._cur_doc = None
Signature · 1 line
2.47.1
author
#2 lore raw

PATCH v2 2/23 qapi/parser: adjust info location for doc body section

In reply to John Snow

Instead of using the info object for the doc block as a whole (which always points to the very first line of the block), update the info pointer for each call to ensure_untagged_section when the existing section is otherwise empty. This way, Sphinx error information will match precisely to where the text actually starts.

For example, this patch will move the info pointer for the "Hello!" untagged section ...

Quoted 3 lines

## <-- from here ...
# Hello! <-- ... to here.
##

---
 scripts/qapi/parser.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/qapi/parser.py +5 -1
index adc85b5b394..36cb64a677a 100644
@@ -687,7 +687,11 @@ def end(self) -> None:
     def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
         if self.all_sections and not self.all_sections[-1].tag:
             # extend current section
-            self.all_sections[-1].text += '\n'
+            section = self.all_sections[-1]
+            if not section.text:
+                # Section is empty so far; update info to start *here*.
+                section.info = info
+            section.text += '\n'
             return
         # start new section
         section = self.Section(info)
Signature · 1 line
2.47.1
author
#3 lore raw

PATCH v2 3/23 docs/qapidoc: remove example section support

In reply to John Snow

Since commit 3c5f6114 (qapi: remove "Example" doc section), Example sections no longer exist, so this support in qapidoc is now dead code.

---
 docs/sphinx/qapidoc.py | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)
docs/sphinx/qapidoc.py +1 -8
index 5a4d7388b29..61997fd21af 100644
@@ -254,10 +254,6 @@ def _nodes_for_features(self, doc):
         section += dlnode
         return [section]
 
-    def _nodes_for_example(self, exampletext):
-        """Return list of doctree nodes for a code example snippet"""
-        return [nodes.literal_block(exampletext, exampletext)]
-
     def _nodes_for_sections(self, doc):
         """Return list of doctree nodes for additional sections"""
         nodelist = []
@@ -275,10 +271,7 @@ def _nodes_for_sections(self, doc):
                 continue
 
             snode = self._make_section(section.tag)
-            if section.tag.startswith('Example'):
-                snode += self._nodes_for_example(dedent(section.text))
-            else:
-                self._parse_text_into_node(dedent(section.text), snode)
+            self._parse_text_into_node(dedent(section.text), snode)
             nodelist.append(snode)
         return nodelist
 
Signature · 1 line
2.47.1
author
#4 lore raw

PATCH v2 4/23 qapi: expand tags to all doc sections

In reply to John Snow

This patch adds an explicit section "kind" to all QAPIDoc sections. Members/Features are now explicitly marked as such, with the name now being stored in a dedicated "name" field (which qapidoc.py was not actually using anyway.)

The qapi-schema tests are updated to account for the new section names; mostly "TODO" becomes "Todo" and None becomes "Plain".

---
 docs/sphinx/qapidoc.py         |   7 ++-
 scripts/qapi/parser.py         | 109 ++++++++++++++++++++++++---------
 tests/qapi-schema/doc-good.out |  10 +--
 tests/qapi-schema/test-qapi.py |   2 +-
 4 files changed, 90 insertions(+), 38 deletions(-)
docs/sphinx/qapidoc.py +4 -3
index 61997fd21af..d622398f1da 100644
@@ -35,6 +35,7 @@
 from docutils.statemachine import ViewList
 from qapi.error import QAPIError, QAPISemError
 from qapi.gen import QAPISchemaVisitor
+from qapi.parser import QAPIDoc
 from qapi.schema import QAPISchema
 
 from sphinx import addnodes
@@ -258,11 +259,11 @@ def _nodes_for_sections(self, doc):
         """Return list of doctree nodes for additional sections"""
         nodelist = []
         for section in doc.sections:
-            if section.tag and section.tag == 'TODO':
+            if section.kind == QAPIDoc.Kind.TODO:
                 # Hide TODO: sections
                 continue
 
-            if not section.tag:
+            if section.kind == QAPIDoc.Kind.PLAIN:
                 # Sphinx cannot handle sectionless titles;
                 # Instead, just append the results to the prior section.
                 container = nodes.container()
@@ -270,7 +271,7 @@ def _nodes_for_sections(self, doc):
                 nodelist += container.children
                 continue
 
-            snode = self._make_section(section.tag)
+            snode = self._make_section(section.kind.name.title())
             self._parse_text_into_node(dedent(section.text), snode)
             nodelist.append(snode)
         return nodelist
scripts/qapi/parser.py +80 -29

195 lines not on this page · raw message

tests/qapi-schema/doc-good.out +5 -5
index ec277be91e9..2d33a305ee7 100644
@@ -110,7 +110,7 @@ The _one_ {and only}, description on the same line
 Also _one_ {and only}
     feature=enum-member-feat
 a member feature
-    section=None
+    section=Plain
 @two is undocumented
 doc symbol=Base
     body=
@@ -168,15 +168,15 @@ description starts on the same line
 a feature
     feature=cmd-feat2
 another feature
-    section=None
+    section=Plain
 .. note:: @arg3 is undocumented
     section=Returns
 @Object
     section=Errors
 some
-    section=TODO
+    section=Todo
 frobnicate
-    section=None
+    section=Plain
 .. admonition:: Notes
 
  - Lorem ipsum dolor sit amet
@@ -209,7 +209,7 @@ If you're bored enough to read this, go see a video of boxed cats
 a feature
     feature=cmd-feat2
 another feature
-    section=None
+    section=Plain
 .. qmp-example::
 
    -> "this example"
tests/qapi-schema/test-qapi.py +1 -1
index 7e3f9f4aa1f..bca924309be 100755
@@ -131,7 +131,7 @@ def test_frontend(fname):
         for feat, section in doc.features.items():
             print('    feature=%s\n%s' % (feat, section.text))
         for section in doc.sections:
-            print('    section=%s\n%s' % (section.tag, section.text))
+            print('    section=%s\n%s' % (section.kind, section.text))
 
 
 def open_test_result(dir_name, file_name, update):
Signature · 1 line
2.47.1
author
#5 lore raw

PATCH v2 5/23 qapi/schema: add __repr__ to QAPIDoc.Section

In reply to John Snow

Makes debugging far more pleasant when you can just print(section) and get something reasonable to display.

---
 scripts/qapi/parser.py | 3 +++
 1 file changed, 3 insertions(+)
scripts/qapi/parser.py +3 -0
index c3004aa70c6..cb259c42a6d 100644
@@ -673,6 +673,9 @@ def __init__(
             # section text without tag
             self.text = ''
 
+        def __repr__(self) -> str:
+            return f"<QAPIDoc.Section tag={self.tag!r} text={self.text!r}>"
+
         def append_line(self, line: str) -> None:
             self.text += line + '\n'
 
Signature · 1 line
2.47.1
#6 lore raw

Re: [PATCH v2 05/23] qapi/schema: add __repr__ to QAPIDoc.Section

Quoted 13 lines · John Snow

Makes debugging far more pleasant when you can just print(section) and get something reasonable to display.

---
 scripts/qapi/parser.py | 3 +++
 1 file changed, 3 insertions(+)
scripts/qapi/parser.py +0 -0
index c3004aa70c6..cb259c42a6d 100644

scripts/qapi/parser.py def __init__( · line 677

Quoted 5 lines
scripts/qapi/parser.py +2 -0
@@ -673,6 +673,9 @@ def __init__(
             # section text without tag
             self.text = ''
 
+        def __repr__(self) -> str:
+            return f"<QAPIDoc.Section tag={self.tag!r} text={self.text!r}>"

pylint points out

    scripts/qapi/parser.py:677:43: E1101: Instance of 'Section' has no 'tag' member (no-member)

Previous patch renamed it to @kind.

Quoted 3 lines
scripts/qapi/parser.py +1 -0
+
         def append_line(self, line: str) -> None:
             self.text += line + '\n'
author
#7 lore raw

Re: [PATCH v2 05/23] qapi/schema: add __repr__ to QAPIDoc.Section

Quoted 22 lines · Markus Armbruster
Quoted 13 lines · John Snow

Makes debugging far more pleasant when you can just print(section) and get something reasonable to display.

---
 scripts/qapi/parser.py | 3 +++
 1 file changed, 3 insertions(+)
scripts/qapi/parser.py +0 -0
index c3004aa70c6..cb259c42a6d 100644

scripts/qapi/parser.py def __init__( · line 677

Quoted 5 lines
scripts/qapi/parser.py +2 -0
@@ -673,6 +673,9 @@ def __init__(
             # section text without tag
             self.text = ''
 
+        def __repr__(self) -> str:
+            return f"<QAPIDoc.Section tag={self.tag!r}

text={self.text!r}>"

Quoted 12 lines

pylint points out

    scripts/qapi/parser.py:677:43: E1101: Instance of 'Section' has no

'tag' member (no-member)

Previous patch renamed it to @kind.

Quoted 3 lines
scripts/qapi/parser.py +1 -0
+
         def append_line(self, line: str) -> None:
             self.text += line + '\n'

Oops. Was only linting qapidoc.py and not the parser.py stuff. Will fix for v3.

author
#8 lore raw

PATCH v2 6/23 docs/qapidoc: add transmogrifier stub

In reply to John Snow

This commit adds a stubbed option to the qapi-doc directive that opts-in to the new rST generator; the implementation of which will follow in subsequent commits.

Once all QAPI documents have been converted, this option and the old qapidoc implementation can be dropped.

Note that moving code outside of the try...except block has no impact because the code moved outside of that block does not ever raise a QAPIError.

---
 docs/sphinx/qapidoc.py | 41 ++++++++++++++++++++++++++++-------------
 1 file changed, 28 insertions(+), 13 deletions(-)
docs/sphinx/qapidoc.py +28 -13
index d622398f1da..dc72f3fd3f3 100644
@@ -452,9 +452,9 @@ def _parse_text_into_node(self, doctext, node):
         rstlist.append("", self._cur_doc.info.fname, self._cur_doc.info.line)
         self._sphinx_directive.do_parse(rstlist, node)
 
-    def get_document_nodes(self):
-        """Return the list of docutils nodes which make up the document"""
-        return self._top_node.children
+    def get_document_node(self):
+        """Return the root docutils node which makes up the document"""
+        return self._top_node
 
 
 # Turn the black formatter on for the rest of the file.
@@ -503,7 +503,10 @@ class QAPIDocDirective(NestedDirective):
 
     required_argument = 1
     optional_arguments = 1
-    option_spec = {"qapifile": directives.unchanged_required}
+    option_spec = {
+        "qapifile": directives.unchanged_required,
+        "transmogrify": directives.flag,
+    }
     has_content = False
 
     def new_serialno(self):
@@ -511,10 +514,24 @@ def new_serialno(self):
         env = self.state.document.settings.env
         return "qapidoc-%d" % env.new_serialno("qapidoc")
 
+    def transmogrify(self, schema) -> nodes.Element:
+        raise NotImplementedError
+
+    def legacy(self, schema) -> nodes.Element:
+        vis = QAPISchemaGenRSTVisitor(self)
+        vis.visit_begin(schema)
+        for doc in schema.docs:
+            if doc.symbol:
+                vis.symbol(doc, schema.lookup_entity(doc.symbol))
+            else:
+                vis.freeform(doc)
+        return vis.get_document_node()
+
     def run(self):
         env = self.state.document.settings.env
         qapifile = env.config.qapidoc_srctree + "/" + self.arguments[0]
         qapidir = os.path.dirname(qapifile)
+        transmogrify = "transmogrify" in self.options
 
         try:
             schema = QAPISchema(qapifile)
@@ -522,20 +539,18 @@ def run(self):
             # First tell Sphinx about all the schema files that the
             # output documentation depends on (including 'qapifile' itself)
             schema.visit(QAPISchemaGenDepVisitor(env, qapidir))
-
-            vis = QAPISchemaGenRSTVisitor(self)
-            vis.visit_begin(schema)
-            for doc in schema.docs:
-                if doc.symbol:
-                    vis.symbol(doc, schema.lookup_entity(doc.symbol))
-                else:
-                    vis.freeform(doc)
-            return vis.get_document_nodes()
         except QAPIError as err:
             # Launder QAPI parse errors into Sphinx extension errors
             # so they are displayed nicely to the user
             raise ExtensionError(str(err)) from err
 
+        if transmogrify:
+            contentnode = self.transmogrify(schema)
+        else:
+            contentnode = self.legacy(schema)
+
+        return contentnode.children
+
 
 class QMPExample(CodeBlock, NestedDirective):
     """
Signature · 1 line
2.47.1
author
#9 lore raw

PATCH v2 7/23 docs/qapidoc: add transmogrifier class stub

In reply to John Snow

Add the beginnings of the Transmogrifier class by adding the rST conversion helpers that will be used to build the virtual rST document.

This version of the class does not actually "do anything" yet; each individual feature is added one-at-a-time in the forthcoming commits.

---
 docs/sphinx/qapidoc.py | 66 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 65 insertions(+), 1 deletion(-)
docs/sphinx/qapidoc.py +65 -1
index dc72f3fd3f3..6593c3f28cd 100644
@@ -24,6 +24,7 @@
 https://www.sphinx-doc.org/en/master/development/index.html
 """
 
+from contextlib import contextmanager
 import os
 import re
 import sys
@@ -32,11 +33,12 @@
 
 from docutils import nodes
 from docutils.parsers.rst import Directive, directives
-from docutils.statemachine import ViewList
+from docutils.statemachine import StringList, ViewList
 from qapi.error import QAPIError, QAPISemError
 from qapi.gen import QAPISchemaVisitor
 from qapi.parser import QAPIDoc
 from qapi.schema import QAPISchema
+from qapi.source import QAPISourceInfo
 
 from sphinx import addnodes
 from sphinx.directives.code import CodeBlock
@@ -61,6 +63,68 @@ def dedent(text: str) -> str:
     return lines[0] + textwrap.dedent("".join(lines[1:]))
 
 
+class Transmogrifier:
+    def __init__(self, schema):
+        self._result = StringList()
+        self.indent = 0
+
+    # General-purpose rST generation functions
+
+    def get_indent(self) -> str:
+        return "   " * self.indent
+
+    @contextmanager
+    def indented(self):
+        self.indent += 1
+        try:
+            yield
+        finally:
+            self.indent -= 1
+
+    def add_line_raw(self, line: str, source: str, *lineno: int) -> None:
+        """Append one line of generated reST to the output."""
+
+        # NB: Sphinx uses zero-indexed lines; subtract one.
+        lineno = tuple((n - 1 for n in lineno))
+
+        if line.strip():
+            # not a blank line
+            self._result.append(
+                self.get_indent() + line.rstrip("\n"), source, *lineno
+            )
+        else:
+            self._result.append("", source, *lineno)
+
+    def add_line(self, content: str, info: QAPISourceInfo) -> None:
+        # NB: We *require* an info object; this works out OK because we
+        # don't document built-in objects that don't have
+        # one. Everything else should.
+        assert info
+        self.add_line_raw(content, info.fname, info.line)
+
+    def add_lines(
+        self,
+        content: str,
+        info: QAPISourceInfo,
+    ) -> None:
+        assert info
+        lines = content.splitlines(True)
+        for i, line in enumerate(lines):
+            self.add_line_raw(line, info.fname, info.line + i)
+
+    def ensure_blank_line(self) -> None:
+        # Empty document -- no blank line required.
+        if not self._result:
+            return
+
+        # Last line isn't blank, add one.
+        if self._result[-1].strip():  # pylint: disable=no-member
+            fname, line = self._result.info(-1)
+            # New blank line is credited to one-after the current last line.
+            # +2: correct for zero/one index, then increment by one.
+            self.add_line_raw("", fname, line + 2)
+
+
 # Disable black auto-formatter until re-enabled:
 # fmt: off
 
Signature · 1 line
2.47.1
author
#10 lore raw

PATCH v2 8/23 docs/qapidoc: add visit_module() method

In reply to John Snow

This method annotates the start of a new module, crediting the source location to the first line of the module file.

---
 docs/sphinx/qapidoc.py | 9 +++++++++
 1 file changed, 9 insertions(+)
docs/sphinx/qapidoc.py +9 -0
index 6593c3f28cd..658eae3e386 100644
@@ -26,6 +26,7 @@
 
 from contextlib import contextmanager
 import os
+from pathlib import Path
 import re
 import sys
 import textwrap
@@ -124,6 +125,14 @@ def ensure_blank_line(self) -> None:
             # +2: correct for zero/one index, then increment by one.
             self.add_line_raw("", fname, line + 2)
 
+    # Transmogrification core methods
+
+    def visit_module(self, path: str) -> None:
+        name = Path(path).stem
+        # module directives are credited to the first line of a module file.
+        self.add_line_raw(f".. qapi:module:: {name}", path, 1)
+        self.ensure_blank_line()
+
 
 # Disable black auto-formatter until re-enabled:
 # fmt: off
Signature · 1 line
2.47.1
author
#11 lore raw

PATCH v2 9/23 qapi/source: allow multi-line QAPISourceInfo advancing

In reply to John Snow

This is for the sake of the new rST generator (the "transmogrifier") so we can advance multiple lines on occasion while keeping the generated<-->source mappings accurate.

next_line now simply takes an optional n parameter which chooses the number of lines to advance.

RFC: Here's the exorbitant detail on why I want this:

This is used mainly when converting section syntax in free-form documentation to more traditional rST section header syntax, which does not always line up 1:1 for line counts.

For example:

```
 ##
 # = Section     <-- Info is pointing here, "L1"
 #
 # Lorem Ipsum
 ##
```

would be transformed to rST as:

```
=======        <-- L1
Section        <-- L1
=======        <-- L1
               <-- L2
Lorem Ipsum    <-- L3
```

After consuming the single "Section" line from the source, we want to advance the source pointer to the next non-empty line which requires jumping by more than one line.

---
 scripts/qapi/source.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/source.py +2 -2
index 7b379fdc925..ffdc3f482ac 100644
@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self: T) -> T:
+    def next_line(self: T, n: int = 1) -> T:
         info = copy.copy(self)
-        info.line += 1
+        info.line += n
         return info
 
     def loc(self) -> str:
Signature · 1 line
2.47.1
author
#12 lore raw

PATCH v2 10/23 docs/qapidoc: add visit_freeform() method

In reply to John Snow
---
 docs/sphinx/qapidoc.py | 47 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
docs/sphinx/qapidoc.py +47 -0
index 658eae3e386..c42cc3705aa 100644
@@ -133,6 +133,53 @@ def visit_module(self, path: str) -> None:
         self.add_line_raw(f".. qapi:module:: {name}", path, 1)
         self.ensure_blank_line()
 
+    def visit_freeform(self, doc) -> None:
+        # TODO: Once the old qapidoc transformer is deprecated, freeform
+        # sections can be updated to pure rST, and this transformed removed.
+        #
+        # For now, translate our micro-format into rST. Code adapted
+        # from Peter Maydell's freeform().
+
+        assert len(doc.all_sections) == 1, doc.all_sections
+        body = doc.all_sections[0]
+        text = body.text
+        info = doc.info
+
+        if re.match(r"=+ ", text):
+            # Section/subsection heading (if present, will always be the
+            # first line of the block)
+            (heading, _, text) = text.partition("\n")
+            (leader, _, heading) = heading.partition(" ")
+            level = len(leader) + 1  # Implicit +1 for heading in .rST stub
+
+            # https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections
+            markers = {
+                1: "#",
+                2: "*",
+                3: "=",
+                4: "-",
+                5: "^",
+                6: '"',
+            }
+            overline = level <= 2
+            marker = markers[level]
+
+            self.ensure_blank_line()
+            # This credits all 2 or 3 lines to the single source line.
+            if overline:
+                self.add_line(marker * len(heading), info)
+            self.add_line(heading, info)
+            self.add_line(marker * len(heading), info)
+            self.ensure_blank_line()
+
+            # Eat blank line(s) and advance info
+            trimmed = text.lstrip("\n")
+            text = trimmed
+            info = info.next_line(len(text) - len(trimmed) + 1)
+
+        self.add_lines(text, info)
+        self.ensure_blank_line()
+
 
 # Disable black auto-formatter until re-enabled:
 # fmt: off
Signature · 1 line
2.47.1
author
#13 lore raw

PATCH v2 11/23 docs/qapidoc: add preamble() method

In reply to John Snow

This method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.

---
 docs/sphinx/qapidoc.py | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)
docs/sphinx/qapidoc.py +31 -1
index c42cc3705aa..97868e5c375 100644
@@ -38,7 +38,7 @@
 from qapi.error import QAPIError, QAPISemError
 from qapi.gen import QAPISchemaVisitor
 from qapi.parser import QAPIDoc
-from qapi.schema import QAPISchema
+from qapi.schema import QAPISchema, QAPISchemaEntity
 from qapi.source import QAPISourceInfo
 
 from sphinx import addnodes
@@ -125,6 +125,36 @@ def ensure_blank_line(self) -> None:
             # +2: correct for zero/one index, then increment by one.
             self.add_line_raw("", fname, line + 2)
 
+    # Transmogrification helpers
+
+    def preamble(self, ent: QAPISchemaEntity) -> None:
+        """
+        Generate option lines for qapi entity directives.
+        """
+        if ent.doc and ent.doc.since:
+            assert ent.doc.since.kind == QAPIDoc.Kind.SINCE
+            # Generated from the entity's docblock; info location is exact.
+            self.add_line(f":since: {ent.doc.since.text}", ent.doc.since.info)
+
+        if ent.ifcond.is_present():
+            doc = ent.ifcond.docgen()
+            # Generated from entity definition; info location is approximate.
+            self.add_line(f":ifcond: {doc}", ent.info)
+
+        # Hoist special features such as :deprecated: and :unstable:
+        # into the options block for the entity. If, in the future, new
+        # special features are added, qapi-domain will chirp about
+        # unrecognized options and fail until they are handled in
+        # qapi-domain.
+        for feat in ent.features:
+            if feat.is_special():
+                # FIXME: handle ifcond if present. How to display that
+                # information is TBD.
+                # Generated from entity def; info location is approximate.
+                self.add_line(f":{feat.name}:", feat.info)
+
+        self.ensure_blank_line()
+
     # Transmogrification core methods
 
     def visit_module(self, path: str) -> None:
Signature · 1 line
2.47.1
author
#14 lore raw

PATCH v2 12/23 docs/qapidoc: add visit_paragraph() method

In reply to John Snow

This transforms "formerly known as untagged sections" into our pure intermediate rST format. These sections are already pure rST, so this method doesn't do a whole lot except ensure appropriate newlines.

---
 docs/sphinx/qapidoc.py | 9 +++++++++
 1 file changed, 9 insertions(+)
docs/sphinx/qapidoc.py +9 -0
index 97868e5c375..4b4cd6359e0 100644
@@ -127,6 +127,15 @@ def ensure_blank_line(self) -> None:
 
     # Transmogrification helpers
 
+    def visit_paragraph(self, section: QAPIDoc.Section) -> None:
+        # Squelch empty paragraphs.
+        if not section.text:
+            return
+
+        self.ensure_blank_line()
+        self.add_lines(section.text, section.info)
+        self.ensure_blank_line()
+
     def preamble(self, ent: QAPISchemaEntity) -> None:
         """
         Generate option lines for qapi entity directives.
Signature · 1 line
2.47.1
author
#15 lore raw

PATCH v2 13/23 docs/qapidoc: add visit_errors() method

In reply to John Snow

Notably, this method does not currently address the formatting issues present with the "errors" section in QAPIDoc and just vomits the text verbatim into the rST doc, with somewhat inconsistent results.

To be addressed in a future revision.

---
 docs/sphinx/qapidoc.py | 6 ++++++
 1 file changed, 6 insertions(+)
docs/sphinx/qapidoc.py +6 -0
index 4b4cd6359e0..64d13565c60 100644
@@ -136,6 +136,12 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None:
         self.add_lines(section.text, section.info)
         self.ensure_blank_line()
 
+    def visit_errors(self, section: QAPIDoc.Section) -> None:
+        # FIXME: the formatting for errors may be inconsistent and may
+        # or may not require different newline placement to ensure
+        # proper rendering as a nested list.
+        self.add_lines(f":error:\n{section.text}", section.info)
+
     def preamble(self, ent: QAPISchemaEntity) -> None:
         """
         Generate option lines for qapi entity directives.
Signature · 1 line
2.47.1
author
#16 lore raw

PATCH v2 14/23 docs/qapidoc: add format_type() method

In reply to John Snow

This method is responsible for generating a type name for a given member with the correct annotations for the QAPI domain. Features and enums do not *have* types, so they return None. Everything else returns the type name with a "?" suffix if that type is optional, and ensconced in [brackets] if it's an array type.

---
 docs/sphinx/qapidoc.py | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
docs/sphinx/qapidoc.py +28 -2
index 64d13565c60..b1eaac6e215 100644
@@ -30,7 +30,7 @@
 import re
 import sys
 import textwrap
-from typing import List
+from typing import List, Optional
 
 from docutils import nodes
 from docutils.parsers.rst import Directive, directives
@@ -38,7 +38,14 @@
 from qapi.error import QAPIError, QAPISemError
 from qapi.gen import QAPISchemaVisitor
 from qapi.parser import QAPIDoc
-from qapi.schema import QAPISchema, QAPISchemaEntity
+from qapi.schema import (
+    QAPISchema,
+    QAPISchemaArrayType,
+    QAPISchemaEntity,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectTypeMember,
+)
 from qapi.source import QAPISourceInfo
 
 from sphinx import addnodes
@@ -125,6 +132,25 @@ def ensure_blank_line(self) -> None:
             # +2: correct for zero/one index, then increment by one.
             self.add_line_raw("", fname, line + 2)
 
+    def format_type(self, ent) -> Optional[str]:
+        if isinstance(ent, (QAPISchemaEnumMember, QAPISchemaFeature)):
+            return None
+
+        qapi_type = ent
+        optional = False
+        if isinstance(ent, QAPISchemaObjectTypeMember):
+            qapi_type = ent.type
+            optional = ent.optional
+
+        if isinstance(qapi_type, QAPISchemaArrayType):
+            ret = f"[{qapi_type.element_type.doc_type()}]"
+        else:
+            ret = qapi_type.doc_type()
+        if optional:
+            ret += "?"
+
+        return ret
+
     # Transmogrification helpers
 
     def visit_paragraph(self, section: QAPIDoc.Section) -> None:
Signature · 1 line
2.47.1
author
#17 lore raw

PATCH v2 15/23 docs/qapidoc: add add_field() and generate_field() helper methods

In reply to John Snow

These are simple rST generation methods that assist in getting the types and formatting correct for a field list entry. add_field() is a more raw, direct call while generate_field() is intended to be used for generating the correct field from a member object.

---
 docs/sphinx/qapidoc.py | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
docs/sphinx/qapidoc.py +25 -0
index b1eaac6e215..6c0e7b6004c 100644
@@ -44,6 +44,7 @@
     QAPISchemaEntity,
     QAPISchemaEnumMember,
     QAPISchemaFeature,
+    QAPISchemaMember,
     QAPISchemaObjectTypeMember,
 )
 from qapi.source import QAPISourceInfo
@@ -132,6 +133,20 @@ def ensure_blank_line(self) -> None:
             # +2: correct for zero/one index, then increment by one.
             self.add_line_raw("", fname, line + 2)
 
+    def add_field(
+        self,
+        kind: str,
+        name: str,
+        body: str,
+        info: QAPISourceInfo,
+        typ: Optional[str] = None,
+    ) -> None:
+        if typ:
+            text = f":{kind} {typ} {name}: {body}"
+        else:
+            text = f":{kind} {name}: {body}"
+        self.add_lines(text, info)
+
     def format_type(self, ent) -> Optional[str]:
         if isinstance(ent, (QAPISchemaEnumMember, QAPISchemaFeature)):
             return None
@@ -151,6 +166,16 @@ def format_type(self, ent) -> Optional[str]:
 
         return ret
 
+    def generate_field(
+        self,
+        kind: str,
+        member: QAPISchemaMember,
+        body: str,
+        info: QAPISourceInfo,
+    ) -> None:
+        typ = self.format_type(member)
+        self.add_field(kind, member.name, body, info, typ)
+
     # Transmogrification helpers
 
     def visit_paragraph(self, section: QAPIDoc.Section) -> None:
Signature · 1 line
2.47.1
author
#18 lore raw

PATCH v2 16/23 docs/qapidoc: add visit_feature() method

In reply to John Snow

This adds a simple ":feat name: lorem ipsum ..." line to the generated rST document, so at the moment it's only for "top level" features.

---
 docs/sphinx/qapidoc.py | 9 +++++++++
 1 file changed, 9 insertions(+)
docs/sphinx/qapidoc.py +9 -0
index 6c0e7b6004c..e9af072c5a6 100644
@@ -187,6 +187,15 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None:
         self.add_lines(section.text, section.info)
         self.ensure_blank_line()
 
+    def visit_feature(self, section: QAPIDoc.ArgSection) -> None:
+        # FIXME - ifcond for features is not handled at all yet!
+        # Proposal: decorate the right-hand column with some graphical
+        # element to indicate conditional availability?
+        assert section.text  # Guaranteed by parser.py
+        assert section.member
+
+        self.generate_field("feat", section.member, section.text, section.info)
+
     def visit_errors(self, section: QAPIDoc.Section) -> None:
         # FIXME: the formatting for errors may be inconsistent and may
         # or may not require different newline placement to ensure
Signature · 1 line
2.47.1
author
#19 lore raw

PATCH v2 17/23 docs/qapidoc: prepare to record entity being transmogrified

In reply to John Snow

Prepare to keep a record of which entity we're working on documenting for the purposes of being able to change certain generative features conditionally and create stronger assertions.

If you find yourself asking: "Wait, but where does the current entity actually get recorded?!", you're right! That part comes with the visit_entity() implementation, which gets added later.

---
 docs/sphinx/qapidoc.py | 6 ++++++
 1 file changed, 6 insertions(+)
docs/sphinx/qapidoc.py +6 -0
index e9af072c5a6..a4dbfa3b3dd 100644
@@ -74,9 +74,15 @@ def dedent(text: str) -> str:
 
 class Transmogrifier:
     def __init__(self, schema):
+        self._curr_ent = None
         self._result = StringList()
         self.indent = 0
 
+    @property
+    def entity(self) -> QAPISchemaEntity:
+        assert self._curr_ent is not None
+        return self._curr_ent
+
     # General-purpose rST generation functions
 
     def get_indent(self) -> str:
Signature · 1 line
2.47.1
author
#20 lore raw

PATCH v2 18/23 docs/qapidoc: add visit_returns() method

In reply to John Snow

Generates :returns: fields for explicit returns statements. Note that this does not presently handle undocumented returns, which is handled in a later commit.

---
 docs/sphinx/qapidoc.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
docs/sphinx/qapidoc.py +14 -0
index a4dbfa3b3dd..cbc62eb5084 100644
@@ -41,6 +41,7 @@
 from qapi.schema import (
     QAPISchema,
     QAPISchemaArrayType,
+    QAPISchemaCommand,
     QAPISchemaEntity,
     QAPISchemaEnumMember,
     QAPISchemaFeature,
@@ -202,6 +203,19 @@ def visit_feature(self, section: QAPIDoc.ArgSection) -> None:
 
         self.generate_field("feat", section.member, section.text, section.info)
 
+    def visit_returns(self, section: QAPIDoc.Section) -> None:
+        assert isinstance(self.entity, QAPISchemaCommand)
+        rtype = self.entity.ret_type
+        # q_empty can produce None, but we won't be documenting anything
+        # without an explicit return statement in the doc block, and we
+        # should not have any such explicit statements when there is no
+        # return value.
+        assert rtype
+
+        typ = self.format_type(rtype)
+        assert section.text  # We don't expect empty returns sections.
+        self.add_field("returns", typ, section.text, section.info)
+
     def visit_errors(self, section: QAPIDoc.Section) -> None:
         # FIXME: the formatting for errors may be inconsistent and may
         # or may not require different newline placement to ensure
Signature · 1 line
2.47.1
author
#21 lore raw

PATCH v2 19/23 docs/qapidoc: add visit_member() method

In reply to John Snow

This method is used for generating the "members" of a wide variety of things, including structs, unions, enums, alternates, etc. The field name it uses to do so is dependent on the type of entity the "member" belongs to.

---
 docs/sphinx/qapidoc.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
docs/sphinx/qapidoc.py +27 -0
index cbc62eb5084..5f00615ae32 100644
@@ -74,6 +74,16 @@ def dedent(text: str) -> str:
 
 
 class Transmogrifier:
+    # Field names used for different entity types:
+    field_types = {
+        "enum": "value",
+        "struct": "memb",
+        "union": "memb",
+        "event": "memb",
+        "command": "arg",
+        "alternate": "choice",
+    }
+
     def __init__(self, schema):
         self._curr_ent = None
         self._result = StringList()
@@ -84,6 +94,10 @@ def entity(self) -> QAPISchemaEntity:
         assert self._curr_ent is not None
         return self._curr_ent
 
+    @property
+    def member_field_type(self) -> str:
+        return self.field_types[self.entity.meta]
+
     # General-purpose rST generation functions
 
     def get_indent(self) -> str:
@@ -194,6 +208,19 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None:
         self.add_lines(section.text, section.info)
         self.ensure_blank_line()
 
+    def visit_member(self, section: QAPIDoc.ArgSection) -> None:
+        # TODO: ifcond for members
+        # TODO?: features for members (documented at entity-level,
+        # but sometimes defined per-member. Should we add such
+        # information to member descriptions when we can?)
+        assert section.text
+        self.generate_field(
+            self.member_field_type,
+            section.member,
+            section.text,
+            section.info,
+        )
+
     def visit_feature(self, section: QAPIDoc.ArgSection) -> None:
         # FIXME - ifcond for features is not handled at all yet!
         # Proposal: decorate the right-hand column with some graphical
Signature · 1 line
2.47.1
author
#22 lore raw

PATCH v2 20/23 docs/qapidoc: add visit_sections() method

In reply to John Snow

Implement the actual main dispatch method that processes and handles the list of doc sections for a given QAPI entity.

---
 docs/sphinx/qapidoc.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
docs/sphinx/qapidoc.py +23 -0
index 5f00615ae32..73076a7d6ae 100644
@@ -277,6 +277,29 @@ def preamble(self, ent: QAPISchemaEntity) -> None:
 
         self.ensure_blank_line()
 
+    def visit_sections(self, ent: QAPISchemaEntity) -> None:
+        sections = ent.doc.all_sections if ent.doc else []
+
+        # Add sections *in the order they are documented*:
+        for section in sections:
+            if section.kind == QAPIDoc.Kind.PLAIN:
+                self.visit_paragraph(section)
+            elif section.kind == QAPIDoc.Kind.MEMBER:
+                self.visit_member(section)
+            elif section.kind == QAPIDoc.Kind.FEATURE:
+                self.visit_feature(section)
+            elif section.kind in (QAPIDoc.Kind.SINCE, QAPIDoc.Kind.TODO):
+                # Since is handled in preamble, TODO is skipped intentionally.
+                pass
+            elif section.kind == QAPIDoc.Kind.RETURNS:
+                self.visit_returns(section)
+            elif section.kind == QAPIDoc.Kind.ERRORS:
+                self.visit_errors(section)
+            else:
+                assert False
+
+        self.ensure_blank_line()
+
     # Transmogrification core methods
 
     def visit_module(self, path: str) -> None:
Signature · 1 line
2.47.1
author
#23 lore raw

PATCH v2 21/23 docs/qapidoc: add visit_entity()

In reply to John Snow

Finally, the core entry method for a qapi entity.

---
 docs/sphinx/qapidoc.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)
docs/sphinx/qapidoc.py +13 -0
index 73076a7d6ae..a2fc8d25ff7 100644
@@ -355,6 +355,19 @@ def visit_freeform(self, doc) -> None:
         self.add_lines(text, info)
         self.ensure_blank_line()
 
+    def visit_entity(self, ent):
+        assert ent is not None
+
+        try:
+            self._curr_ent = ent
+            # This line gets credited to the start of the /definition/.
+            self.add_line(f".. qapi:{ent.meta}:: {ent.name}", ent.info)
+            with self.indented():
+                self.preamble(ent)
+                self.visit_sections(ent)
+        finally:
+            self._curr_ent = None
+
 
 # Disable black auto-formatter until re-enabled:
 # fmt: off
Signature · 1 line
2.47.1
author
#24 lore raw

PATCH v2 22/23 docs/qapidoc: implement transmogrify() method

In reply to John Snow

This is the true top-level processor for the new transmogrifier; responsible both for generating the intermediate rST and then running the nested parse on that generated document to produce the final docutils tree that is then - very finally - postprocessed by sphinx for final rendering to HTML &c.

---
 docs/sphinx/qapidoc.py | 47 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 46 insertions(+), 1 deletion(-)
docs/sphinx/qapidoc.py +46 -1
index a2fc8d25ff7..5d946a46637 100644
@@ -2,6 +2,7 @@
 #
 # QEMU qapidoc QAPI file parsing extension
 #
+# Copyright (c) 2024 Red Hat
 # Copyright (c) 2020 Linaro
 #
 # This work is licensed under the terms of the GNU GPLv2 or later.
@@ -53,12 +54,15 @@
 from sphinx import addnodes
 from sphinx.directives.code import CodeBlock
 from sphinx.errors import ExtensionError
+from sphinx.util import logging
 from sphinx.util.docutils import switch_source_input
 from sphinx.util.nodes import nested_parse_with_titles
 
 
 __version__ = "1.0"
 
+logger = logging.getLogger(__name__)
+
 
 def dedent(text: str) -> str:
     # Adjust indentation to make description text parse as paragraph.
@@ -89,6 +93,10 @@ def __init__(self, schema):
         self._result = StringList()
         self.indent = 0
 
+    @property
+    def result(self) -> StringList:
+        return self._result
+
     @property
     def entity(self) -> QAPISchemaEntity:
         assert self._curr_ent is not None
@@ -823,7 +831,43 @@ def new_serialno(self):
         return "qapidoc-%d" % env.new_serialno("qapidoc")
 
     def transmogrify(self, schema) -> nodes.Element:
-        raise NotImplementedError
+        logger.info("Transmogrifying QAPI to rST ...")
+        vis = Transmogrifier(schema)
+        modules = set()
+
+        for doc in schema.docs:
+            module_source = doc.info.fname
+            if module_source not in modules:
+                vis.visit_module(module_source)
+                modules.add(module_source)
+
+            if doc.symbol:
+                ent = schema.lookup_entity(doc.symbol)
+                assert ent
+                vis.visit_entity(ent)
+            else:
+                vis.visit_freeform(doc)
+
+        logger.info("Transmogrification complete.")
+
+        contentnode = nodes.section()
+        content = vis.result
+        titles_allowed = True
+
+        logger.info("Transmogrifier running nested parse ...")
+        with switch_source_input(self.state, content):
+            if titles_allowed:
+                node: nodes.Element = nodes.section()
+                node.document = self.state.document
+                nested_parse_with_titles(self.state, content, contentnode)
+            else:
+                node = nodes.paragraph()
+                node.document = self.state.document
+                self.state.nested_parse(content, 0, contentnode)
+        logger.info("Transmogrifier's nested parse completed.")
+        sys.stdout.flush()
+
+        return contentnode
 
     def legacy(self, schema) -> nodes.Element:
         vis = QAPISchemaGenRSTVisitor(self)
@@ -957,6 +1001,7 @@ def run(self) -> List[nodes.Node]:
 
 def setup(app):
     """Register qapi-doc directive with Sphinx"""
+    app.setup_extension("qapi-domain")
     app.add_config_value("qapidoc_srctree", None, "env")
     app.add_directive("qapi-doc", QAPIDocDirective)
     app.add_directive("qmp-example", QMPExample)
Signature · 1 line
2.47.1
author
#25 lore raw

PATCH v2 23/23 docs/qapidoc: add transmogrifier test document

In reply to John Snow
---
 docs/index.rst      |  1 +
 docs/qapi/index.rst | 53 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)
 create mode 100644 docs/qapi/index.rst
docs/index.rst +1 -0
index 0b9ee9901d9..11c18c598a8 100644
@@ -18,3 +18,4 @@ Welcome to QEMU's documentation!
    interop/index
    specs/index
    devel/index
+   qapi/index
docs/qapi/index.rst new file +53 -0

53 lines not on this page · raw message

Signature · 1 line
2.47.1