docs: add basic sphinx-domain rST generator to qapidoc
cover docs: add basic sphinx-domain rST generator to qapidoc
based-on: https://patchew.org/QEMU/[email protected]/
Hi!
This series is a very, very barebones implementation for the new QAPI doc generator. It does not have many features that I presented on at KVM Forum; the point of this patch set is instead to present a stripped down basis for ongoing work so we can discuss on-list with full context of the code available to do so.
The documentation this series generates is *not suitable* for replacing the current document generator, it has a few glaring omissions - on purpose - those features have been factored out intentionally so they can be reviewed with fuller context and more careful review.
What this series does:
- Adds the new "Transmogrifier" rST generator to qapidoc.py, which generates an in-memory rST document using qapi-domain directives.
- Adds a test document that showcases this new transmogrifier.
What this series very notably does not do (yet):
- "ifcond" data for anything other than top-level entities is not considered or rendered. This means "if" statements for features and members are entirely absent.
- The inliner is not present at all. This series renders only documentation exactly as it is exists in the source files.
- *branches* are themselves not considered at all; they're skipped entirely for now. They will be included alongside the inliner in either a subsequent series or a followup to this series.
- Undocumented members and return statements are not autogenerated.
- Pseudofeatures (Things like allow-oob) are not generated as documented features.
- Documentation culling: all entities are documented whether or not they're relevant to the wire format.
My goal in doing it this way is to save the "fancy" features for later so we can focus on reviewing and tightening up the core functionality of the transmogrifier. Once we're on steadier ground, I will re-add the fanciful features while adjusting the qapi-domain mechanisms. Once everything looks "roughly right, give or take some minor nits", I will switch back to the qapi-domain series itself for review before we merge everything together.
John Snow (23):
docs/qapidoc: support header-less freeform sections qapi/parser: adjust info location for doc body section docs/qapidoc: remove example section support
qapi: expand tags to all doc sections
qapi/schema: add __repr__ to QAPIDoc.Section docs/qapidoc: add transmogrifier stub
docs/qapidoc: add transmogrifier class stub docs/qapidoc: add visit_module() method
qapi/source: allow multi-line QAPISourceInfo advancing docs/qapidoc: add visit_freeform() method docs/qapidoc: add preamble() method
docs/qapidoc: add visit_paragraph() method docs/qapidoc: add visit_errors() method
docs/qapidoc: add format_type() method
docs/qapidoc: add add_field() and generate_field() helper methods docs/qapidoc: add visit_feature() method
docs/qapidoc: record current documented entity in transmogrifier docs/qapidoc: add visit_returns() method
docs/qapidoc: add visit_member() method
docs/qapidoc: add visit_sections() method docs/qapidoc: add visit_entity()
docs/qapidoc: implement transmogrify() method docs/qapidoc: add transmogrifier test document
docs/index.rst | 1 + docs/qapi/index.rst | 53 ++++++ docs/sphinx/qapidoc.py | 419 ++++++++++++++++++++++++++++++++++++++--- scripts/qapi/parser.py | 97 +++++++--- scripts/qapi/source.py | 4 +- 5 files changed, 524 insertions(+), 50 deletions(-) create mode 100644 docs/qapi/index.rst
Signature · 1 line
2.47.0
PATCH 1/23 docs/qapidoc: support header-less freeform sections
The code as written can't handle if a header isn't found, because node will be uninitialized. If we don't have a section title, create a generic block to insert text into instead.
This patch removes a lingering pylint warning in the QAPIDoc implementation that prevents getting a clean baseline to use for forthcoming additions.
I am not attempting to *fully* clean up the existing QAPIDoc implementation in pylint because I intend to delete it anyway; this patch merely accomplishes a baseline under a specific pylint configuration:
PYTHONPATH=../../scripts/ pylint --disable=fixme,too-many-lines,\
consider-using-f-string,missing-docstring,unused-argument,\
too-many-arguments,too-many-positional-arguments,\
too-many-public-methods \
qapidoc.pyThis at least ensures there aren't regressions outside of these general warnings in the new qapidoc.py code to be committed.
--- docs/sphinx/qapidoc.py | 2 ++ 1 file changed, 2 insertions(+)
docs/sphinx/qapidoc.py +2 -0
@@ -421,6 +421,8 @@ def freeform(self, doc): node = self._start_new_heading(heading, len(leader)) if text == '': return + else: + node = nodes.container() self._parse_text_into_node(text, node) self._cur_doc = None
Signature · 1 line
2.47.0
Re: [PATCH 01/23] docs/qapidoc: support header-less freeform sections
Quoted 2 lines · John Snow
The code as written can't handle if a header isn't found, because
nodewill be uninitialized.
Yes, we initialize @node only if we have a heading.
Made me wonder what happens when we don't. So I deleted the = from the "# = Subsection" line in doc-good.json, and got:
Exception occurred:
File "/work/armbru/qemu/docs/sphinx/qapidoc.py", line 425, in freeform
self._parse_text_into_node(text, node)
^^^^
UnboundLocalError: cannot access local variable 'node' where it is not associated with a value
So you're fixing a crash bug, but that's perhaps less than clear from the commit message.
Quoted 4 lines
If we don't have a section title, create a
generic block to insert text into instead.This patch removes a lingering pylint warning in the QAPIDoc implementation
Can you show me the warning? My pylint doesn't...
Quoted 8 lines
that prevents getting a clean baseline to use for forthcoming additions.
I am not attempting to *fully* clean up the existing QAPIDoc implementation in pylint because I intend to delete it anyway; this patch merely accomplishes a baseline under a specific pylint configuration:
Quoted 5 lines
PYTHONPATH=../../scripts/ pylint --disable=fixme,too-many-lines,\
consider-using-f-string,missing-docstring,unused-argument,\
too-many-arguments,too-many-positional-arguments,\
too-many-public-methods \
qapidoc.pyWhat version of pylint? Mine chokes on too-many-positional-arguments.
Quoted 13 lines
This at least ensures there aren't regressions outside of these general warnings in the new
qapidoc.pycode to be committed.Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 2 ++ 1 file changed, 2 insertions(+)docs/sphinx/qapidoc.py +0
-0
docs/sphinx/qapidoc.py def freeform(self, doc): · line 428
Quoted 8 lines
docs/sphinx/qapidoc.py +2
-0@@ -421,6 +421,8 @@ def freeform(self, doc): node = self._start_new_heading(heading, len(leader)) if text == '': return + else: + node = nodes.container() self._parse_text_into_node(text, node) self._cur_doc = None
Plausible enough (and I acked a similar fix previously, commit 2664f3176a8), but I'm a Sphinx ignoramus :)
Re: [PATCH 01/23] docs/qapidoc: support header-less freeform sections
Quoted 37 lines · Markus Armbruster
Quoted 2 lines · John Snow
The code as written can't handle if a header isn't found, because
nodewill be uninitialized.Yes, we initialize @node only if we have a heading.
Made me wonder what happens when we don't. So I deleted the = from the "# = Subsection" line in doc-good.json, and got:
Exception occurred:
File "/work/armbru/qemu/docs/sphinx/qapidoc.py", line 425, in
freeform
self._parse_text_into_node(text, node)
^^^^
UnboundLocalError: cannot access local variable 'node' where it is not
associated with a valueSo you're fixing a crash bug, but that's perhaps less than clear from the commit message.
Quoted 4 lines
If we don't have a section title, create a
generic block to insert text into instead.This patch removes a lingering pylint warning in the QAPIDoc
implementation
Can you show me the warning? My pylint doesn't...
Quoted 8 lines
that prevents getting a clean baseline to use for forthcoming additions.
I am not attempting to *fully* clean up the existing QAPIDoc implementation in pylint because I intend to delete it anyway; this patch merely accomplishes a baseline under a specific pylint configuration:
Quoted 8 lines
Quoted 5 lines
PYTHONPATH=../../scripts/ pylint --disable=fixme,too-many-lines,\ consider-using-f-string,missing-docstring,unused-argument,\ too-many-arguments,too-many-positional-arguments,\ too-many-public-methods \ qapidoc.pyWhat version of pylint? Mine chokes on too-many-positional-arguments.
3.3.1 here; if yours doesn't have that warning, there's no need to disable it. just remove that flag from the CLI.
(I promise I do want to get this rigorously checked and automated, I'm sorry it's taken so long to achieve.)
Quoted 27 lines
Quoted 14 lines
This at least ensures there aren't regressions outside of these general warnings in the new
qapidoc.pycode to be committed.Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 2 ++ 1 file changed, 2 insertions(+)docs/sphinx/qapidoc.py +0
-0docs/sphinx/qapidoc.py def freeform(self, doc): · line 428
Quoted 8 lines
docs/sphinx/qapidoc.py +2
-0@@ -421,6 +421,8 @@ def freeform(self, doc): node = self._start_new_heading(heading, len(leader)) if text == '': return + else: + node = nodes.container() self._parse_text_into_node(text, node) self._cur_doc = NonePlausible enough (and I acked a similar fix previously, commit
2664f3176a8), but I'm a Sphinx ignoramus :)
Re: [PATCH 01/23] docs/qapidoc: support header-less freeform sections
Quoted 44 lines · John Snow
Quoted 42 lines · Markus Armbruster
Quoted 2 lines · John Snow
The code as written can't handle if a header isn't found, because
nodewill be uninitialized.Yes, we initialize @node only if we have a heading.
Made me wonder what happens when we don't. So I deleted the = from the "# = Subsection" line in doc-good.json, and got:
Exception occurred:
File "/work/armbru/qemu/docs/sphinx/qapidoc.py", line 425, in
freeform
self._parse_text_into_node(text, node)
^^^^
UnboundLocalError: cannot access local variable 'node' where it is not
associated with a valueSo you're fixing a crash bug, but that's perhaps less than clear from the commit message.
Quoted 4 lines
If we don't have a section title, create a
generic block to insert text into instead.This patch removes a lingering pylint warning in the QAPIDoc implementation
Can you show me the warning? My pylint doesn't...
Quoted 13 lines
that prevents getting a clean baseline to use for forthcoming additions.
I am not attempting to *fully* clean up the existing QAPIDoc implementation in pylint because I intend to delete it anyway; this patch merely accomplishes a baseline under a specific pylint configuration:
PYTHONPATH=../../scripts/ pylint --disable=fixme,too-many-lines,\ consider-using-f-string,missing-docstring,unused-argument,\ too-many-arguments,too-many-positional-arguments,\ too-many-public-methods \ qapidoc.py
Quoted 4 lines
Quoted 1 line
What version of pylint? Mine chokes on too-many-positional-arguments.
3.3.1 here; if yours doesn't have that warning, there's no need to disable it. just remove that flag from the CLI.
I've since upgraded to 3.3.3, which doesn't choke.
Quoted 2 lines
(I promise I do want to get this rigorously checked and automated, I'm sorry it's taken so long to achieve.)
[...]
PATCH 2/23 qapi/parser: adjust info location for doc body section
In reply to John SnowInstead of using the info object for the doc block as a whole (which always points to the very first line of the block), update the info pointer for each call to ensure_untagged_section when the existing section is otherwise empty. This way, Sphinx error information will match precisely to where the text actually starts.
For example, this patch will move the info pointer for the "Hello!" untagged section ...
Quoted 3 lines
## <-- from here ...
# Hello! <-- ... to here.
##
--- scripts/qapi/parser.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/qapi/parser.py +5 -1
@@ -687,7 +687,11 @@ def end(self) -> None: def ensure_untagged_section(self, info: QAPISourceInfo) -> None: if self.all_sections and not self.all_sections[-1].tag: # extend current section - self.all_sections[-1].text += '\n' + section = self.all_sections[-1] + if not section.text: + # Section is empty so far; update info to start *here*. + section.info = info + section.text += '\n' return # start new section section = self.Section(info)
Signature · 1 line
2.47.0
PATCH 3/23 docs/qapidoc: remove example section support
In reply to John SnowSince 3c5f6114 we no longer actually generate Example sections, so this support in qapidoc is now dead code.
--- docs/sphinx/qapidoc.py | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
docs/sphinx/qapidoc.py +1 -8
@@ -254,10 +254,6 @@ def _nodes_for_features(self, doc): section += dlnode return [section] - def _nodes_for_example(self, exampletext): - """Return list of doctree nodes for a code example snippet""" - return [nodes.literal_block(exampletext, exampletext)] - def _nodes_for_sections(self, doc): """Return list of doctree nodes for additional sections""" nodelist = []
@@ -275,10 +271,7 @@ def _nodes_for_sections(self, doc): continue snode = self._make_section(section.tag) - if section.tag.startswith('Example'): - snode += self._nodes_for_example(dedent(section.text)) - else: - self._parse_text_into_node(dedent(section.text), snode) + self._parse_text_into_node(dedent(section.text), snode) nodelist.append(snode) return nodelist
Signature · 1 line
2.47.0
Re: [PATCH 03/23] docs/qapidoc: remove example section support
Quoted 1 line · John Snow
Since
3c5f6114we no longer actually generate Example sections, so this
Suggest s/generate/support/
Quoted 1 line
support in qapidoc is now dead code.
Oversight, I guess.
Quoted 2 lines
Signed-off-by John Snow <[email protected]>
Re: [PATCH 03/23] docs/qapidoc: remove example section support
Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow
Since
3c5f6114we no longer actually generate Example sections, so thisSuggest s/generate/support/
I meant to say: because the parser doesn't support them, it no longer creates such section objects, so there's no need to handle them.
Quoted 4 lines
Quoted 1 line
support in qapidoc is now dead code.
Oversight, I guess.
O:-)
Quoted 6 lines
Quoted 2 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Markus Armbruster <[email protected]>
Re: [PATCH 03/23] docs/qapidoc: remove example section support
Quoted 11 lines · John Snow
Quoted 6 lines · Markus Armbruster
Quoted 1 line · John Snow
Since
3c5f6114we no longer actually generate Example sections, so thisSuggest s/generate/support/
I meant to say: because the parser doesn't support them, it no longer creates such section objects, so there's no need to handle them.
"Since commit 3c5f6114d9f (qapi: remove "Example" doc section), Example sections no longer exist, so ..."?
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
support in qapidoc is now dead code.
Oversight, I guess.
O:-)
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Markus Armbruster <[email protected]>
PATCH 4/23 qapi: expand tags to all doc sections
In reply to John SnowThis patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which qapidoc.py was not actually using anyway.)
WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
--- docs/sphinx/qapidoc.py | 7 ++-- scripts/qapi/parser.py | 88 ++++++++++++++++++++++++++++++++---------- 2 files changed, 72 insertions(+), 23 deletions(-)
docs/sphinx/qapidoc.py +4 -3
@@ -35,6 +35,7 @@ from docutils.statemachine import ViewList from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor +from qapi.parser import QAPIDoc from qapi.schema import QAPISchema from sphinx import addnodes
@@ -258,11 +259,11 @@ def _nodes_for_sections(self, doc): """Return list of doctree nodes for additional sections""" nodelist = [] for section in doc.sections: - if section.tag and section.tag == 'TODO': + if section.tag == QAPIDoc.Tag.TODO: # Hide TODO: sections continue - if not section.tag: + if section.tag == QAPIDoc.Tag.UNTAGGED: # Sphinx cannot handle sectionless titles; # Instead, just append the results to the prior section. container = nodes.container()
@@ -270,7 +271,7 @@ def _nodes_for_sections(self, doc): nodelist += container.children continue - snode = self._make_section(section.tag) + snode = self._make_section(section.tag.name.title()) self._parse_text_into_node(dedent(section.text), snode) nodelist.append(snode) return nodelist
scripts/qapi/parser.py +68 -20
171 lines not on this page · raw message
Signature · 1 line
2.47.0
Re: [PATCH 04/23] qapi: expand tags to all doc sections
Quoted 18 lines · John Snow
This patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which
qapidoc.pywas not actually using anyway.)WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
Quoted 1 line
Signed-off-by John Snow <[email protected]>
Doesn't pass 'make check' for me. Diff appended. It shows the error messages get worse.
tests/qapi-schema/doc-duplicated-return.err +1 -1
@@ -1 +1 @@ -doc-duplicated-return.json:8: duplicated 'Returns' section +doc-duplicated-return.json:8: duplicated 'Tag.RETURNS' section
tests/qapi-schema/doc-duplicated-since.err +1 -1
@@ -1 +1 @@ -doc-duplicated-since.json:8: duplicated 'Since' section +doc-duplicated-since.json:8: duplicated 'Tag.SINCE' section
tests/qapi-schema/doc-empty-section.err +1 -1
@@ -1 +1 @@ -doc-empty-section.json:6: text required after 'Errors:' +doc-empty-section.json:6: text required after 'Tag.ERRORS:'
tests/qapi-schema/doc-good.out +8 -8
@@ -110,7 +110,7 @@ The _one_ {and only}, description on the same line Also _one_ {and only} feature=enum-member-feat a member feature - section=None + section=Tag.UNTAGGED @two is undocumented doc symbol=Base body=
@@ -168,15 +168,15 @@ description starts on the same line a feature feature=cmd-feat2 another feature - section=None + section=Tag.UNTAGGED .. note:: @arg3 is undocumented - section=Returns + section=Tag.RETURNS @Object - section=Errors + section=Tag.ERRORS some - section=TODO + section=Tag.TODO frobnicate - section=None + section=Tag.UNTAGGED .. admonition:: Notes - Lorem ipsum dolor sit amet
@@ -200,7 +200,7 @@ Examples:: Note:: Ceci n'est pas une note - section=Since + section=Tag.SINCE 2.10 doc symbol=cmd-boxed body=
@@ -209,7 +209,7 @@ If you're bored enough to read this, go see a video of boxed cats a feature feature=cmd-feat2 another feature - section=None + section=Tag.UNTAGGED .. qmp-example:: -> "this example"
Re: [PATCH 04/23] qapi: expand tags to all doc sections
Quoted 20 lines · Markus Armbruster
Quoted 18 lines · John Snow
This patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which
qapidoc.pywas not actually using anyway.)WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
Quoted 5 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>Doesn't pass 'make check' for me. Diff appended. It shows the error messages get worse.
Whoops! My per-patch tester wasn't running make check, only linters and build testing. I'll fix this, but please keep reviewing in the meantime.
Apologies for the oversight.
Quoted 81 lines
tests/qapi-schema/doc-duplicated-return.err +1
-1@@ -1 +1 @@ -doc-duplicated-return.json:8: duplicated 'Returns' section +doc-duplicated-return.json:8: duplicated 'Tag.RETURNS' sectiontests/qapi-schema/doc-duplicated-since.err +1
-1@@ -1 +1 @@ -doc-duplicated-since.json:8: duplicated 'Since' section +doc-duplicated-since.json:8: duplicated 'Tag.SINCE' sectiontests/qapi-schema/doc-empty-section.err +1
-1@@ -1 +1 @@ -doc-empty-section.json:6: text required after 'Errors:' +doc-empty-section.json:6: text required after 'Tag.ERRORS:'tests/qapi-schema/doc-good.out +7
-7@@ -110,7 +110,7 @@ The _one_ {and only}, description on the same line Also _one_ {and only} feature=enum-member-feat a member feature - section=None + section=Tag.UNTAGGED @two is undocumented doc symbol=Base body=@@ -168,15 +168,15 @@ description starts on the same line a feature feature=cmd-feat2 another feature - section=None + section=Tag.UNTAGGED .. note:: @arg3 is undocumented - section=Returns + section=Tag.RETURNS @Object - section=Errors + section=Tag.ERRORS some - section=TODO + section=Tag.TODO frobnicate - section=None + section=Tag.UNTAGGED .. admonition:: Notes - Lorem ipsum dolor sit amet@@ -200,7 +200,7 @@ Examples:: Note:: Ceci n'est pas une note - section=Since + section=Tag.SINCE 2.10 doc symbol=cmd-boxed body=@@ -209,7 +209,7 @@ If you're bored enough to read this, go see a video ofboxed cats
a feature
feature=cmd-feat2
another featuretests/qapi-schema/doc-good.out +1
-1- section=None + section=Tag.UNTAGGED .. qmp-example:: -> "this example"
Re: [PATCH 04/23] qapi: expand tags to all doc sections
Quoted 24 lines · John Snow
Quoted 22 lines · Markus Armbruster
Quoted 19 lines · John Snow
This patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which
qapidoc.pywas not actually using anyway.)WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
Signed-off-by John Snow <[email protected]>
Quoted 7 lines
Quoted 2 lines
Doesn't pass 'make check' for me. Diff appended. It shows the error messages get worse.
Whoops! My per-patch tester wasn't running make check, only linters and build testing. I'll fix this, but please keep reviewing in the meantime.
Apologies for the oversight.
No worries :)
Re: [PATCH 04/23] qapi: expand tags to all doc sections
In reply to John SnowQuoted 18 lines · John Snow
This patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which
qapidoc.pywas not actually using anyway.)WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
Quoted 1 line
Signed-off-by John Snow <[email protected]>
A free-form doc comment consists of just one untagged section, actually. I don't remember whether anything relies on "just one".
The term "tagged" is rooted in doc comment syntax. docs/devel/qapi-code-gen.rst section "Definition documentation":
Definition documentation starts with a line naming the definition, followed by an optional overview, a description of each argument (for commands and events), member (for structs and unions), branch (for alternates), or value (for enums), a description of each feature (if any), and finally optional tagged sections.
Sadly, this isn't fully accurate anymore.
Descriptions start with '\@name:'. The description text must be indented [...]
A tagged section begins with a paragraph that starts with one of the following words: "Since:", "Returns:", "Errors:", "TODO:". It ends with the start of a new section.
The second and subsequent lines of tagged sections must be indented [...]
Nothing about untagged sections. These are sections that aren't descriptions or tagged. Example:
# @Returns: Lorem ipsum dolor sit amet, consectetur adipiscing elit,
# sed do eiusmod tempor incididunt ut labore et dolore magna
# aliqua.
#
# Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris # nisi ut aliquip ex ea commodo consequat.
#
# Duis aute irure dolor in reprehenderit in voluptate velit esse # cillum dolore eu fugiat nulla pariatur.
##
Here, the tagged "Returns" section ends after "aliqua." Why? Because "Ut enim" isn't indented. The untagged section ends after "pariatur."
We parse a definition doc comment as a sequence of sections.
The first one is the overview.
Member / argument descriptions, if any, are next.
Then we may have any number of tagged or untagged sections. If I remember correctly, you'd like to banish them. Let's pretend they can't exist here.
Then we may have a "Features:" line followed by feature descriptions.
Finally, we may have any number of tagged or untagged sections.
Each of these sections is represented as an instance of type Section, and the entire definition doc as an instance of type QAPIDoc.
Section has a member @tag of type str.
For tagged sections, it's the tag, i.e "Since", "Returns", ... Obvious enough.
For overview and other untagged sections, it's None. Still obvious.
For descriptions, it's the name of the thing being described. Less than obvious. Note that descriptions are actually instances of ArgSection, a subtype of Section, which prevents confusion with tagged sections.
QAPIDoc has the overview in member @body, member / argument descriptions in @args, feature descriptions in @features, and the remaining sections in @sections.
I'm in favor of cleaning this up some.
I think we can keep the Section name.
Moving the name of the thing being described from @tag to @name is good. What value to put into @tag then? Whatever suits you.
Perhaps we should rename @tag to avoid undue ties to tagged sections. @kind would work for me.
Value None for untagged sections is fine with me. If a string suits you better, that's fine, too. "untagged", "plain", I don't know, propose something.
@body, @args, and so forth aren't exactly great names. If they truly annoy or confuse you, feel free to propose better ones.
Re: [PATCH 04/23] qapi: expand tags to all doc sections
Quoted 20 lines · Markus Armbruster
Quoted 18 lines · John Snow
This patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which
qapidoc.pywas not actually using anyway.)WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
Quoted 5 lines
Quoted 1 line
Signed-off-by John Snow <[email protected]>A free-form doc comment consists of just one untagged section, actually. I don't remember whether anything relies on "just one".
Sure, yes. Sorry, I keep thinking of documentation as containing "any number of sections" but keep eliding the fact that our parser implementation currently will never actually create multiple adjacent "untagged" sections.
I don't even change this anywhere even in my offline WIP, so it's just me being over-general.
(I don't think it winds up being relevant or mattering to anything in this series or my larger project beyond some word choices.)
Quoted 41 lines
The term "tagged" is rooted in doc comment syntax.
docs/devel/qapi-code-gen.rstsection "Definition documentation":Definition documentation starts with a line naming the definition, followed by an optional overview, a description of each argument (for commands and events), member (for structs and unions), branch (for alternates), or value (for enums), a description of each feature (if any), and finally optional tagged sections.
Sadly, this isn't fully accurate anymore.
Descriptions start with '\@name:'. The description text must be indented [...]
A tagged section begins with a paragraph that starts with one of the following words: "Since:", "Returns:", "Errors:", "TODO:". It ends
with
the start of a new section.The second and subsequent lines of tagged sections must be indented [...]
Nothing about untagged sections. These are sections that aren't descriptions or tagged. Example:
# @Returns: Lorem ipsum dolor sit amet, consectetur adipiscing elit,
# sed do eiusmod tempor incididunt ut labore et dolore magna
# aliqua.
#
# Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris # nisi ut aliquip ex ea commodo consequat.
#
# Duis aute irure dolor in reprehenderit in voluptate velit esse # cillum dolore eu fugiat nulla pariatur.
##Here, the tagged "Returns" section ends after "aliqua." Why? Because "Ut enim" isn't indented. The untagged section ends after "pariatur."
We parse a definition doc comment as a sequence of sections.
Quoted 8 lines
The first one is the overview.
Member / argument descriptions, if any, are next.
Then we may have any number of tagged or untagged sections. If I remember correctly, you'd like to banish them. Let's pretend they can't exist here.
I think you're referring to my desire to banish "untagged" sections from appearing *between* "tagged" sections. Yes, that's still a desire; though I make no movement on it in this series as sent to list, and this change is entirely unrelated to that desire.
(It's more related to being able to distinguish features from members, and later, distinguishing intro/details. This patch still serves a purpose even without the inliner or the complexities it brings, but serves both needs.)
((Reminder: the reason for this desire is because "tagged sections" are rendered in html as a two-column list, and free paragraphs appearing between list entries looks bad in the rendered documentation, because it means ending the table, starting paragraph(s), then starting a new table. If free text is meant to be associated with a specific member/feature/section-group, it should be marked up (in SOME way) so that the renderer can achieve that grouping visually.
(There will be a standalone patch that implements this restriction and we can debate this there, I'm only giving you context here.)))
Quoted 9 lines
Then we may have a "Features:" line followed by feature descriptions.
Finally, we may have any number of tagged or untagged sections.
Each of these sections is represented as an instance of type Section, and the entire definition doc as an instance of type QAPIDoc.
Section has a member @tag of type str.
Quoted 9 lines
For tagged sections, it's the tag, i.e "Since", "Returns", ... Obvious enough.
For overview and other untagged sections, it's None. Still obvious.
For descriptions, it's the name of the thing being described. Less than obvious. Note that descriptions are actually instances of ArgSection, a subtype of Section, which prevents confusion with tagged sections.
Note that this patch changes this as well; it becomes "member" or "feature" as appropriate and the name is moved into a dedicated name field that belongs to the ArgSection class.
(Turns out legacy qapidoc doesn't use this stored name at all anyway, it fetches the name via the linked feature/member instead.)
Quoted 11 lines
QAPIDoc has the overview in member @body, member / argument descriptions in @args, feature descriptions in @features, and the remaining sections in @sections.
I'm in favor of cleaning this up some.
I think we can keep the Section name.
Moving the name of the thing being described from @tag to @name is good. What value to put into @tag then? Whatever suits you.
What suits me is "member" and "feature". :)
Quoted 3 lines
Perhaps we should rename @tag to avoid undue ties to tagged sections. @kind would work for me.
Sold!
Quoted 4 lines
Value None for untagged sections is fine with me. If a string suits you better, that's fine, too. "untagged", "plain", I don't know, propose something.
For static typing reasons, an explicit tag is preferred to distinguish from it being "optional".
I could cope with any of:
"plain",
"text",
"plaintext",
"paragraphs",
"unstructured",
"free"
... keeping in mind that I do intend to "split" this tag/kind into "intro" and "details" later. i.e. this is a temporary tag/kind label.
I think I like "text" the most because it says the least. What about you?
Quoted 3 lines
@body, @args, and so forth aren't exactly great names. If they truly annoy or confuse you, feel free to propose better ones.
I believe they can be removed entirely once the old qapidoc is sunset, leaving only .sections[] behind.
This removes the temptation to pick out sections "out of order".
We only need the list of sections in their source order to generate the appropriate rST.
(Note: the inliner actually does need to filter sections somewhat to do its inlining magic, but we'll talk about that later. All you need to know right now is that my WIP does not utilize any field except .sections[], so the others can in fact be dropped as redundant once we make the switch. This patch helps enable the paradigm of "everything you need to render a section is contained within the Section object itself" which lends itself well to the new transmogrifier, the goal of always processing/rendering in source order, and facilitating the mechanics of the inliner.)
...
In case I got too rambly, my action items for this patch are:
- fix the test (already done)
- rename tag to kind
- rename "untagged" to "text", possibly changing it again pending your feedback.
--js
Quoted 1 line
Re: [PATCH 04/23] qapi: expand tags to all doc sections
Quoted 24 lines · John Snow
Quoted 22 lines · Markus Armbruster
Quoted 19 lines · John Snow
This patch adds an explicit section tag to all QAPIDoc sections. Members/Features are now explicitly tagged as such, with the name now being stored in a dedicated "name" field (which
qapidoc.pywas not actually using anyway.)WIP: Yeah, the difference between "tagged" and "untagged" sections is now pretty poorly named, and explicitly giving "untagged" sections an "UNTAGGED" tag is ... well, worse. but mechanically, this accomplishes what I need for the series.
Please suggest better naming conventions, keeping in mind that I currently have plans for a future patch that splits the "UNTAGGED" tag into "INTRO" and "DETAILS" tags. But, we still need a meta-name for the category of sections that are "formerly known as untagged" but cannot be called "freeform" because that name is used for the category of docblocks that are not attached to an entity (but happens to be comprised entirely of "formerly known as untagged" sections.)
Signed-off-by John Snow <[email protected]>
Quoted 8 lines
Quoted 3 lines
A free-form doc comment consists of just one untagged section, actually. I don't remember whether anything relies on "just one".
Sure, yes. Sorry, I keep thinking of documentation as containing "any number of sections" but keep eliding the fact that our parser implementation currently will never actually create multiple adjacent "untagged" sections.
Yes, the parsers grows an untagged section until another section starts. An untagged section can consist of multiple paragraphs, just like tagged sections.
Quoted 51 lines
I don't even change this anywhere even in my offline WIP, so it's just me being over-general.
(I don't think it winds up being relevant or mattering to anything in this series or my larger project beyond some word choices.)
Quoted 45 lines
The term "tagged" is rooted in doc comment syntax.
docs/devel/qapi-code-gen.rstsection "Definition documentation":Definition documentation starts with a line naming the definition, followed by an optional overview, a description of each argument (for commands and events), member (for structs and unions), branch (for alternates), or value (for enums), a description of each feature (if any), and finally optional tagged sections.
Sadly, this isn't fully accurate anymore.
Descriptions start with '\@name:'. The description text must be indented [...]
A tagged section begins with a paragraph that starts with one of the following words: "Since:", "Returns:", "Errors:", "TODO:". It ends
with
the start of a new section.The second and subsequent lines of tagged sections must be indented [...]
Nothing about untagged sections. These are sections that aren't descriptions or tagged. Example:
# @Returns: Lorem ipsum dolor sit amet, consectetur adipiscing elit,
# sed do eiusmod tempor incididunt ut labore et dolore magna
# aliqua.
#
# Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris # nisi ut aliquip ex ea commodo consequat.
#
# Duis aute irure dolor in reprehenderit in voluptate velit esse # cillum dolore eu fugiat nulla pariatur.
##Here, the tagged "Returns" section ends after "aliqua." Why? Because "Ut enim" isn't indented. The untagged section ends after "pariatur."
We parse a definition doc comment as a sequence of sections.
The first one is the overview.
Member / argument descriptions, if any, are next.
Quoted 7 lines
Quoted 4 lines
Then we may have any number of tagged or untagged sections. If I remember correctly, you'd like to banish them. Let's pretend they can't exist here.
I think you're referring to my desire to banish "untagged" sections from appearing *between* "tagged" sections.
Editing accident, sorry! You interpreted it correctly.
Quoted 53 lines
Yes, that's still a desire; though I
make no movement on it in this series as sent to list, and this change is entirely unrelated to that desire.(It's more related to being able to distinguish features from members, and later, distinguishing intro/details. This patch still serves a purpose even without the inliner or the complexities it brings, but serves both needs.)
((Reminder: the reason for this desire is because "tagged sections" are rendered in html as a two-column list, and free paragraphs appearing between list entries looks bad in the rendered documentation, because it means ending the table, starting paragraph(s), then starting a new table. If free text is meant to be associated with a specific member/feature/section-group, it should be marked up (in SOME way) so that the renderer can achieve that grouping visually.
(There will be a standalone patch that implements this restriction and we can debate this there, I'm only giving you context here.)))
Quoted 9 lines
Then we may have a "Features:" line followed by feature descriptions.
Finally, we may have any number of tagged or untagged sections.
Each of these sections is represented as an instance of type Section, and the entire definition doc as an instance of type QAPIDoc.
Section has a member @tag of type str.
Quoted 9 lines
For tagged sections, it's the tag, i.e "Since", "Returns", ... Obvious enough.
For overview and other untagged sections, it's None. Still obvious.
For descriptions, it's the name of the thing being described. Less than obvious. Note that descriptions are actually instances of ArgSection, a subtype of Section, which prevents confusion with tagged sections.
Note that this patch changes this as well; it becomes "member" or "feature" as appropriate and the name is moved into a dedicated name field that belongs to the ArgSection class.
(Turns out legacy qapidoc doesn't use this stored name at all anyway, it fetches the name via the linked feature/member instead.)
Quoted 6 lines
QAPIDoc has the overview in member @body, member / argument descriptions in @args, feature descriptions in @features, and the remaining sections in @sections.
I'm in favor of cleaning this up some.
Quoted 7 lines
Quoted 5 lines
I think we can keep the Section name.
Moving the name of the thing being described from @tag to @name is good. What value to put into @tag then? Whatever suits you.
What suits me is "member" and "feature". :)
Okay.
Doesn't entirely clean up the terminology mess. According to docs/devel/qapi-code-gen.rst, struct and union types have members, alternate types have alternatives, enum types have values, commands have arguments, and events have event-specific data, which is a mouthful, so we often say arguments. Using one of them ("member") to refer to the generalization of them all is suboptimal, but it's no worse than before.
ArgSection is even more general: it's features, too. Again, no worse than before.
I'm *not* asking you to clean this up. I'm just pointing out we could use fresh naming ideas here.
Quoted 24 lines
Quoted 3 lines
Perhaps we should rename @tag to avoid undue ties to tagged sections. @kind would work for me.
Sold!
Quoted 4 lines
Value None for untagged sections is fine with me. If a string suits you better, that's fine, too. "untagged", "plain", I don't know, propose something.
For static typing reasons, an explicit tag is preferred to distinguish from it being "optional".
I could cope with any of:
"plain",
"text",
"plaintext",
"paragraphs",
"unstructured",
"free"
Quoted 4 lines
... keeping in mind that I do intend to "split" this tag/kind into "intro" and "details" later. i.e. this is a temporary tag/kind label.
I think I like "text" the most because it says the least. What about you?
Point, but the other kinds of section are text, too. "plain"?
Quoted 8 lines
Quoted 3 lines
@body, @args, and so forth aren't exactly great names. If they truly annoy or confuse you, feel free to propose better ones.
I believe they can be removed entirely once the old qapidoc is sunset, leaving only .sections[] behind.
This removes the temptation to pick out sections "out of order".
I've long wanted strict in-order processing, to avoid surprising reordering of input in the output.
Quoted 12 lines
We only need the list of sections in their source order to generate the appropriate rST.
(Note: the inliner actually does need to filter sections somewhat to do its inlining magic, but we'll talk about that later. All you need to know right now is that my WIP does not utilize any field except .sections[], so the others can in fact be dropped as redundant once we make the switch. This patch helps enable the paradigm of "everything you need to render a section is contained within the Section object itself" which lends itself well to the new transmogrifier, the goal of always processing/rendering in source order, and facilitating the mechanics of the inliner.)
Quoted 8 lines
...
In case I got too rambly, my action items for this patch are:
- fix the test (already done)
- rename tag to kind
- rename "untagged" to "text", possibly changing it again pending your feedback.
Sounds good!
PATCH 5/23 qapi/schema: add __repr__ to QAPIDoc.Section
In reply to John SnowMakes debugging far more pleasant when you can just print(section) and get something reasonable to display.
--- scripts/qapi/parser.py | 3 +++ 1 file changed, 3 insertions(+)
scripts/qapi/parser.py +3 -0
@@ -670,6 +670,9 @@ def __init__( # section text without tag self.text = '' + def __repr__(self) -> str: + return f"<QAPIDoc.Section tag={self.tag!r} text={self.text!r}>" + def append_line(self, line: str) -> None: self.text += line + '\n'
Signature · 1 line
2.47.0
PATCH 6/23 docs/qapidoc: add transmogrifier stub
In reply to John SnowThis commit adds a stubbed option to the qapi-doc directive that opts-in to the new rST generator; the implementation of which will follow in subsequent commits.
Once all QAPI documents have been converted, this option and the old qapidoc implementation can be dropped.
Note that moving code outside of the try...except block has no impact because the code moved outside of that block does not ever raise a QAPIError.
--- docs/sphinx/qapidoc.py | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
docs/sphinx/qapidoc.py +28 -13
@@ -452,9 +452,9 @@ def _parse_text_into_node(self, doctext, node): rstlist.append("", self._cur_doc.info.fname, self._cur_doc.info.line) self._sphinx_directive.do_parse(rstlist, node) - def get_document_nodes(self): - """Return the list of docutils nodes which make up the document""" - return self._top_node.children + def get_document_node(self): + """Return the root docutils node which makes up the document""" + return self._top_node # Turn the black formatter on for the rest of the file.
@@ -503,7 +503,10 @@ class QAPIDocDirective(NestedDirective): required_argument = 1 optional_arguments = 1 - option_spec = {"qapifile": directives.unchanged_required} + option_spec = { + "qapifile": directives.unchanged_required, + "transmogrify": directives.flag, + } has_content = False def new_serialno(self):
@@ -511,10 +514,24 @@ def new_serialno(self): env = self.state.document.settings.env return "qapidoc-%d" % env.new_serialno("qapidoc") + def transmogrify(self, schema) -> nodes.Element: + raise NotImplementedError + + def legacy(self, schema) -> nodes.Element: + vis = QAPISchemaGenRSTVisitor(self) + vis.visit_begin(schema) + for doc in schema.docs: + if doc.symbol: + vis.symbol(doc, schema.lookup_entity(doc.symbol)) + else: + vis.freeform(doc) + return vis.get_document_node() + def run(self): env = self.state.document.settings.env qapifile = env.config.qapidoc_srctree + "/" + self.arguments[0] qapidir = os.path.dirname(qapifile) + transmogrify = "transmogrify" in self.options try: schema = QAPISchema(qapifile)
@@ -522,20 +539,18 @@ def run(self): # First tell Sphinx about all the schema files that the # output documentation depends on (including 'qapifile' itself) schema.visit(QAPISchemaGenDepVisitor(env, qapidir)) - - vis = QAPISchemaGenRSTVisitor(self) - vis.visit_begin(schema) - for doc in schema.docs: - if doc.symbol: - vis.symbol(doc, schema.lookup_entity(doc.symbol)) - else: - vis.freeform(doc) - return vis.get_document_nodes() except QAPIError as err: # Launder QAPI parse errors into Sphinx extension errors # so they are displayed nicely to the user raise ExtensionError(str(err)) from err + if transmogrify: + contentnode = self.transmogrify(schema) + else: + contentnode = self.legacy(schema) + + return contentnode.children + class QMPExample(CodeBlock, NestedDirective): """
Signature · 1 line
2.47.0
PATCH 7/23 docs/qapidoc: add transmogrifier class stub
In reply to John SnowAdd the beginnings of the Transmogrifier class by adding the rST conversion helpers that will be used to build the virtual rST document.
This version of the class does not actually "do anything" yet; each individual feature is added one-at-a-time in the forthcoming commits.
--- docs/sphinx/qapidoc.py | 66 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)
docs/sphinx/qapidoc.py +65 -1
@@ -24,6 +24,7 @@ https://www.sphinx-doc.org/en/master/development/index.html """ +from contextlib import contextmanager import os import re import sys
@@ -32,11 +33,12 @@ from docutils import nodes from docutils.parsers.rst import Directive, directives -from docutils.statemachine import ViewList +from docutils.statemachine import StringList, ViewList from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc from qapi.schema import QAPISchema +from qapi.source import QAPISourceInfo from sphinx import addnodes from sphinx.directives.code import CodeBlock
@@ -61,6 +63,68 @@ def dedent(text: str) -> str: return lines[0] + textwrap.dedent("".join(lines[1:])) +class Transmogrifier: + def __init__(self, schema): + self._result = StringList() + self.indent = 0 + + # General-purpose rST generation functions + + def get_indent(self) -> str: + return " " * self.indent + + @contextmanager + def indented(self): + self.indent += 1 + try: + yield + finally: + self.indent -= 1 + + def add_line_raw(self, line: str, source: str, *lineno: int) -> None: + """Append one line of generated reST to the output.""" + + # NB: Sphinx uses zero-indexed lines; subtract one. + lineno = tuple((n - 1 for n in lineno)) + + if line.strip(): + # not a blank line + self._result.append( + self.get_indent() + line.rstrip("\n"), source, *lineno + ) + else: + self._result.append("", source, *lineno) + + def add_line(self, content: str, info: QAPISourceInfo) -> None: + # NB: We *require* an info object; this works out OK because we + # don't document built-in objects that don't have + # one. Everything else should. + assert info + self.add_line_raw(content, info.fname, info.line) + + def add_lines( + self, + content: str, + info: QAPISourceInfo, + ) -> None: + assert info + lines = content.splitlines(True) + for i, line in enumerate(lines): + self.add_line_raw(line, info.fname, info.line + i) + + def ensure_blank_line(self) -> None: + # Empty document -- no blank line required. + if not self._result: + return + + # Last line isn't blank, add one. + if self._result[-1].strip(): # pylint: disable=no-member + fname, line = self._result.info(-1) + # New blank line is credited to one-after the current last line. + # +2: correct for zero/one index, then increment by one. + self.add_line_raw("", fname, line + 2) + + # Disable black auto-formatter until re-enabled: # fmt: off
Signature · 1 line
2.47.0
PATCH 8/23 docs/qapidoc: add visit_module() method
In reply to John SnowThis method annotates the start of a new module, crediting the source location to the first line of the module file.
--- docs/sphinx/qapidoc.py | 9 +++++++++ 1 file changed, 9 insertions(+)
docs/sphinx/qapidoc.py +9 -0
@@ -26,6 +26,7 @@ from contextlib import contextmanager import os +from pathlib import Path import re import sys import textwrap
@@ -124,6 +125,14 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + # Transmogrification core methods + + def visit_module(self, path: str) -> None: + name = Path(path).stem + # module directives are credited to the first line of a module file. + self.add_line_raw(f".. qapi:module:: {name}", path, 1) + self.ensure_blank_line() + # Disable black auto-formatter until re-enabled: # fmt: off
Signature · 1 line
2.47.0
PATCH 9/23 qapi/source: allow multi-line QAPISourceInfo advancing
In reply to John SnowThis is for the sake of the new rST generator (the "transmogrifier") so we can advance multiple lines on occasion while keeping the generated<-->source mappings accurate.
next_line now simply takes an optional n parameter which chooses the number of lines to advance.
RFC: Here's the exorbitant detail on why I want this:
This is used mainly when converting section syntax in free-form documentation to more traditional rST section header syntax, which does not always line up 1:1 for line counts.
For example:
``` ## # = Section <-- Info is pointing here, "L1" # # Lorem Ipsum ## ```
would be transformed to rST as:
```
======= <-- L1
Section <-- L1
======= <-- L1
<-- L2
Lorem Ipsum <-- L3
```After consuming the single "Section" line from the source, we want to advance the source pointer to the next non-empty line which requires jumping by more than one line.
--- scripts/qapi/source.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/qapi/source.py +2 -2
@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self: T) -> T: + def next_line(self: T, n: int = 1) -> T: info = copy.copy(self) - info.line += 1 + info.line += n return info def loc(self) -> str:
Signature · 1 line
2.47.0
Re: [PATCH 09/23] qapi/source: allow multi-line QAPISourceInfo advancing
Quoted 26 lines · John Snow
This is for the sake of the new rST generator (the "transmogrifier") so we can advance multiple lines on occasion while keeping the generated<-->source mappings accurate.
next_linenow simply takes an optional n parameter which chooses the number of lines to advance.RFC: Here's the exorbitant detail on why I want this:
This is used mainly when converting section syntax in free-form documentation to more traditional rST section header syntax, which does not always line up 1:1 for line counts.
For example:
``` ## # = Section <-- Info is pointing here, "L1" # # Lorem Ipsum ## ```would be transformed to rST as:
Quoted 7 lines
```
======= <-- L1
Section <-- L1
======= <-- L1
<-- L2
Lorem Ipsum <-- L3
```I can't help to wonder... Could we simply use rST markup instead?
"Later", "maybe later", or even "please ask me later" would be perfectly acceptable answers.
Quoted 15 lines
After consuming the single "Section" line from the source, we want to advance the source pointer to the next non-empty line which requires jumping by more than one line.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/source.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/qapi/source.py +0
-0@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta
scripts/qapi/source.py def set_defn(self, meta: str, name: str) -> None: · line 55
Quoted 10 lines
scripts/qapi/source.py +2
-2self.defn_name = name - def next_line(self: T) -> T: + def next_line(self: T, n: int = 1) -> T: info = copy.copy(self) - info.line += 1 + info.line += n return info def loc(self) -> str:
Assuming we need this:
Re: [PATCH 09/23] qapi/source: allow multi-line QAPISourceInfo advancing
Quoted 36 lines · Markus Armbruster
Quoted 26 lines · John Snow
This is for the sake of the new rST generator (the "transmogrifier") so we can advance multiple lines on occasion while keeping the generated<-->source mappings accurate.
next_linenow simply takes an optional n parameter which chooses the number of lines to advance.RFC: Here's the exorbitant detail on why I want this:
This is used mainly when converting section syntax in free-form documentation to more traditional rST section header syntax, which does not always line up 1:1 for line counts.
For example:
``` ## # = Section <-- Info is pointing here, "L1" # # Lorem Ipsum ## ```would be transformed to rST as:
Quoted 7 lines
``` ======= <-- L1 Section <-- L1 ======= <-- L1 <-- L2 Lorem Ipsum <-- L3 ```
Quoted 5 lines
I can't help to wonder... Could we simply use rST markup instead?
"Later", "maybe later", or even "please ask me later" would be perfectly acceptable answers.
Yeah, I'd be happy with that, I just didn't want to add more complexity to the pile so I went for what I felt was "simplest":
- Leave source syntax alone
- Copy and modify the existing freeform doc parser
- Quickly allow for multi-line advancing where it appeared to be important.
Modifying freeform syntax to be purely rST that isn't modified or rewritten at all has benefits:
- No need to mangle or multiplex source line source information
- Less code
- More straightforward
I'm quite happy to do it later, is there some kind of "ticket" system you'd tolerate using for tracking nits for cleanup? I *will* forget if we don't listify and track them, I'm sorry (but wise enough) to admit. I just don't want to get sidetracked on little side-quests right now. (Quite prone to this...)
Quoted 25 lines
Quoted 25 lines
After consuming the single "Section" line from the source, we want to advance the source pointer to the next non-empty line which requires jumping by more than one line.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/source.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/qapi/source.py +2
-2@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self: T) -> T: + def next_line(self: T, n: int = 1) -> T: info = copy.copy(self) - info.line += 1 + info.line += n return info
Quoted 4 lines
scripts/qapi/source.py def set_defn(self, meta: str, name: str) -> None: · line 55
Quoted 1 line
scripts/qapi/source.py +0
-0def loc(self) -> str:Assuming we need this:
Reviewed-by Markus Armbruster <[email protected]>
Thanks! We can always drop stuff later if we wind up not needing it, it's just a means to an end.
--js
Re: [PATCH 09/23] qapi/source: allow multi-line QAPISourceInfo advancing
Quoted 38 lines · John Snow
Quoted 36 lines · Markus Armbruster
Quoted 33 lines · John Snow
This is for the sake of the new rST generator (the "transmogrifier") so we can advance multiple lines on occasion while keeping the generated<-->source mappings accurate.
next_linenow simply takes an optional n parameter which chooses the number of lines to advance.RFC: Here's the exorbitant detail on why I want this:
This is used mainly when converting section syntax in free-form documentation to more traditional rST section header syntax, which does not always line up 1:1 for line counts.
For example:
``` ## # = Section <-- Info is pointing here, "L1" # # Lorem Ipsum ## ```would be transformed to rST as:
``` ======= <-- L1 Section <-- L1 ======= <-- L1 <-- L2 Lorem Ipsum <-- L3 ```
Quoted 8 lines
Quoted 5 lines
I can't help to wonder... Could we simply use rST markup instead?
"Later", "maybe later", or even "please ask me later" would be perfectly acceptable answers.
Yeah, I'd be happy with that, I just didn't want to add more complexity to the pile so I went for what I felt was "simplest":
Avoiding mission creep is good.
Quoted 11 lines
- Leave source syntax alone
- Copy and modify the existing freeform doc parser
- Quickly allow for multi-line advancing where it appeared to be important.Modifying freeform syntax to be purely rST that isn't modified or rewritten at all has benefits:
- No need to mangle or multiplex source line source information
- Less code
- More straightforward
Quoted 5 lines
I'm quite happy to do it later, is there some kind of "ticket" system you'd tolerate using for tracking nits for cleanup? I *will* forget if we don't listify and track them, I'm sorry (but wise enough) to admit. I just don't want to get sidetracked on little side-quests right now. (Quite prone to this...)
TODO comment in code this would obviously kill? Not exactly a ticket system...
scripts/qapi/TODO? Still not a ticket system...
Other ideas?
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
After consuming the single "Section" line from the source, we want to advance the source pointer to the next non-empty line which requires jumping by more than one line.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/source.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/qapi/source.py +2
-2@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self: T) -> T: + def next_line(self: T, n: int = 1) -> T: info = copy.copy(self) - info.line += 1 + info.line += n return info
Quoted 7 lines
Quoted 4 lines
scripts/qapi/source.py def set_defn(self, meta: str, name: str) -> None: · line 55
Quoted 1 line
scripts/qapi/source.py +0
-0def loc(self) -> str:Assuming we need this:
Reviewed-by Markus Armbruster <[email protected]>Thanks! We can always drop stuff later if we wind up not needing it, it's just a means to an end.
Yes, and this one isn't exactly a complexity bomb :)
Re: [PATCH 09/23] qapi/source: allow multi-line QAPISourceInfo advancing
Quoted 71 lines · Markus Armbruster
Quoted 1 line · John Snow
On Fri, Dec 20, 2024 at 8:22 AM Markus Armbruster <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 1 line · John Snow
This is for the sake of the new rST generator (the "transmogrifier")
so
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
we can advance multiple lines on occasion while keeping the generated<-->source mappings accurate.
next_linenow simply takes an optional n parameter which chooses the number of lines to advance.RFC: Here's the exorbitant detail on why I want this:
This is used mainly when converting section syntax in free-form documentation to more traditional rST section header syntax, which does not always line up 1:1 for line counts.
For example:
``` ## # = Section <-- Info is pointing here, "L1" # # Lorem Ipsum ## ```would be transformed to rST as:
``` ======= <-- L1 Section <-- L1 ======= <-- L1 <-- L2 Lorem Ipsum <-- L3 ```Quoted 7 lines
Quoted 5 lines
I can't help to wonder... Could we simply use rST markup instead?
"Later", "maybe later", or even "please ask me later" would be perfectly acceptable answers.
Yeah, I'd be happy with that, I just didn't want to add more complexity
to
Quoted 1 line
the pile so I went for what I felt was "simplest":
Avoiding mission creep is good.
Quoted 3 lines
- Leave source syntax alone
- Copy and modify the existing freeform doc parser
- Quickly allow for multi-line advancing where it appeared to beimportant.
Quoted 2 lines
Modifying freeform syntax to be purely rST that isn't modified or
rewritten
Quoted 7 lines
at all has benefits:
- No need to mangle or multiplex source line source information
- Less code
- More straightforwardI'm quite happy to do it later, is there some kind of "ticket" system
you'd
Quoted 2 lines
tolerate using for tracking nits for cleanup? I *will* forget if we don't listify and track them, I'm sorry (but wise enough) to admit. I just
Quoted 9 lines
don't
Quoted 2 lines
want to get sidetracked on little side-quests right now. (Quite prone to this...)
TODO comment in code this would obviously kill? Not exactly a ticket system...
scripts/qapi/TODO? Still not a ticket system...
If a TODO is fine (and you don't mind pinging me in the future), then the comment I left in the visit_freeform() function (it's in another patch) explaining that the custom parser can be dropped after we sunset the old qapidoc is likely sufficient if I just add a "TODO".
Sound good?
Quoted 36 lines
Other ideas?
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
After consuming the single "Section" line from the source, we want to advance the source pointer to the next non-empty line which requires jumping by more than one line.
Signed-off-by John Snow <[email protected]>--- scripts/qapi/source.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/qapi/source.py +2
-2@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None: self.defn_meta = meta self.defn_name = name - def next_line(self: T) -> T: + def next_line(self: T, n: int = 1) -> T: info = copy.copy(self) - info.line += 1 + info.line += n return infoQuoted 7 lines
Quoted 4 lines
scripts/qapi/source.py def set_defn(self, meta: str, name: str) -> None: · line 55
Quoted 1 line
scripts/qapi/source.py +0
-0def loc(self) -> str:Assuming we need this:
Reviewed-by Markus Armbruster <[email protected]>Thanks! We can always drop stuff later if we wind up not needing it, it's just a means to an end.
Yes, and this one isn't exactly a complexity bomb :)
Re: [PATCH 09/23] qapi/source: allow multi-line QAPISourceInfo advancing
Quoted 3 lines · John Snow
Quoted 1 line · Markus Armbruster
John Snow <[email protected]> writes:
[...]
Quoted 19 lines
Quoted 13 lines
Quoted 7 lines
Modifying freeform syntax to be purely rST that isn't modified or rewritten at all has benefits:
- No need to mangle or multiplex source line source information
- Less code
- More straightforwardQuoted 5 lines
I'm quite happy to do it later, is there some kind of "ticket" system you'd tolerate using for tracking nits for cleanup? I *will* forget if we don't listify and track them, I'm sorry (but wise enough) to admit. I just don't want to get sidetracked on little side-quests right now. (Quite prone to this...)
Quoted 5 lines
TODO comment in code this would obviously kill? Not exactly a ticket system...
scripts/qapi/TODO? Still not a ticket system...
Quoted 6 lines
If a TODO is fine (and you don't mind pinging me in the future), then the comment I left in the
visit_freeform()function (it's in another patch) explaining that the custom parser can be dropped after we sunset the old qapidoc is likely sufficient if I just add a "TODO".Sound good?
Yes.
[...]
PATCH 10/23 docs/qapidoc: add visit_freeform() method
In reply to John Snow--- docs/sphinx/qapidoc.py | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
docs/sphinx/qapidoc.py +47 -0
@@ -133,6 +133,53 @@ def visit_module(self, path: str) -> None: self.add_line_raw(f".. qapi:module:: {name}", path, 1) self.ensure_blank_line() + def visit_freeform(self, doc) -> None: + # Once the old qapidoc transformer is deprecated, + # freeform sections can be transformed into pure rST. + # + # For now, translate our micro-format into rST. + # Code adapted from Peter Maydell's freeform(). + + assert len(doc.all_sections) == 1, doc.all_sections + body = doc.all_sections[0] + text = body.text + info = doc.info + + if re.match(r"=+ ", text): + # Section/subsection heading (if present, will always be the + # first line of the block) + (heading, _, text) = text.partition("\n") + (leader, _, heading) = heading.partition(" ") + level = len(leader) + 1 # Implicit +1 for heading in .rST stub + + # https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections + markers = { + 1: "#", + 2: "*", + 3: "=", + 4: "-", + 5: "^", + 6: '"', + } + overline = level <= 2 + marker = markers[level] + + self.ensure_blank_line() + # This credits all 2 or 3 lines to the single source line. + if overline: + self.add_line(marker * len(heading), info) + self.add_line(heading, info) + self.add_line(marker * len(heading), info) + self.ensure_blank_line() + + # Eat blank line(s) and advance info + trimmed = text.lstrip("\n") + text = trimmed + info = info.next_line(len(text) - len(trimmed) + 1) + + self.add_lines(text, info) + self.ensure_blank_line() + # Disable black auto-formatter until re-enabled: # fmt: off
Signature · 1 line
2.47.0
Re: [PATCH 10/23] docs/qapidoc: add visit_freeform() method
Quoted 46 lines · John Snow
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)docs/sphinx/qapidoc.py +33
-0@@ -133,6 +133,53 @@ def visit_module(self, path: str) -> None: self.add_line_raw(f".. qapi:module:: {name}", path, 1) self.ensure_blank_line() + def visit_freeform(self, doc) -> None: + # Once the old qapidoc transformer is deprecated, + # freeform sections can be transformed into pure rST. + # + # For now, translate our micro-format into rST. + # Code adapted from Peter Maydell's freeform(). + + assert len(doc.all_sections) == 1, doc.all_sections + body = doc.all_sections[0] + text = body.text + info = doc.info + + if re.match(r"=+ ", text): + # Section/subsection heading (if present, will always be the + # first line of the block) + (heading, _, text) = text.partition("\n") + (leader, _, heading) = heading.partition(" ") + level = len(leader) + 1 # Implicit +1 for heading in .rST stub + + # https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections + markers = { + 1: "#", + 2: "*", + 3: "=", + 4: "-", + 5: "^", + 6: '"', + } + overline = level <= 2 + marker = markers[level] + + self.ensure_blank_line() + # This credits all 2 or 3 lines to the single source line.
docs/sphinx/qapidoc.py def visit_module(self, path: str) -> None: · line 178
Quoted 10 lines
docs/sphinx/qapidoc.py +10
-0+ if overline: + self.add_line(marker * len(heading), info) + self.add_line(heading, info) + self.add_line(marker * len(heading), info) + self.ensure_blank_line() + + # Eat blank line(s) and advance info + trimmed = text.lstrip("\n") + text = trimmed + info = info.next_line(len(text) - len(trimmed) + 1)
We could instead eat newlines one at a time, calling .next_line() without an argument for each newline eaten. Less efficient, but won't matter here. Possibly easier to understand.
Quoted 7 lines
docs/sphinx/qapidoc.py +4
-0+ + self.add_lines(text, info) + self.ensure_blank_line() + # Disable black auto-formatter until re-enabled: # fmt: off
Re: [PATCH 10/23] docs/qapidoc: add visit_freeform() method
Quoted 56 lines · Markus Armbruster
Quoted 21 lines · John Snow
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)docs/sphinx/qapidoc.py +8
-0@@ -133,6 +133,53 @@ def visit_module(self, path: str) -> None: self.add_line_raw(f".. qapi:module:: {name}", path, 1) self.ensure_blank_line() + def visit_freeform(self, doc) -> None: + # Once the old qapidoc transformer is deprecated, + # freeform sections can be transformed into pure rST. + # + # For now, translate our micro-format into rST. + # Code adapted from Peter Maydell's freeform(). + + assert len(doc.all_sections) == 1, doc.all_sectionsdocs/sphinx/qapidoc.py def visit_module(self, path: str) -> None: · line 153
Quoted 10 lines
docs/sphinx/qapidoc.py +10
-0+ body = doc.all_sections[0] + text = body.text + info = doc.info + + if re.match(r"=+ ", text): + # Section/subsection heading (if present, will always be the + # first line of the block) + (heading, _, text) = text.partition("\n") + (leader, _, heading) = heading.partition(" ") + level = len(leader) + 1 # Implicit +1 for heading in .rSTstub
docs/sphinx/qapidoc.py def visit_module(self, path: str) -> None: · line 155
Quoted 2 lines
docs/sphinx/qapidoc.py +2
-0+ + #https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections
Quoted 19 lines
docs/sphinx/qapidoc.py +19
-0+ markers = { + 1: "#", + 2: "*", + 3: "=", + 4: "-", + 5: "^", + 6: '"', + } + overline = level <= 2 + marker = markers[level] + + self.ensure_blank_line() + # This credits all 2 or 3 lines to the single source line. + if overline: + self.add_line(marker * len(heading), info) + self.add_line(heading, info) + self.add_line(marker * len(heading), info) + self.ensure_blank_line() +
Quoted 9 lines
docs/sphinx/qapidoc.py def visit_module(self, path: str) -> None: · line 178
Quoted 4 lines
docs/sphinx/qapidoc.py +4
-0+ # Eat blank line(s) and advance info + trimmed = text.lstrip("\n") + text = trimmed + info = info.next_line(len(text) - len(trimmed) + 1)We could instead eat newlines one at a time, calling .next_line() without an argument for each newline eaten. Less efficient, but won't matter here. Possibly easier to understand.
Eh, I'm gonna delete this later anyway. Will send v2 without touching this, but I can touch it for next time if it still bothers you.
--js
PATCH 11/23 docs/qapidoc: add preamble() method
In reply to John SnowThis method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)
docs/sphinx/qapidoc.py +32 -1
@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes
@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location is exact. + self.add_line(f":since: {ent.doc.since.text}", ent.doc.since.info) + + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location is approximate. + self.add_line(f":ifcond: {doc}", ent.info) + + # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future, new + # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcond property. + # (Hello, intrepid developer in the future who changed that!) + # ((With luck, you are not me.)) + assert not feat.ifcond.is_present() + # Generated from entity def; info location is approximate. + self.add_line(f":{feat.name}:", feat.info) + + self.ensure_blank_line() + # Transmogrification core methods def visit_module(self, path: str) -> None:
Signature · 1 line
2.47.0
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 42 lines · John Snow
This method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +17
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location is exact. + self.add_line(f":since: {ent.doc.since.text}", ent.doc.since.info) + + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location is approximate. + self.add_line(f":ifcond: {doc}", ent.info) +
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 10 lines
docs/sphinx/qapidoc.py +10
-0+ # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future, new + # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcond property. + # (Hello, intrepid developer in the future who changed that!) + # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()
Nope :)
The attempt to add a conditional special feature now fails with
Sphinx parallel build error:
AssertionError
If you want to outlaw conditional special features, reject them cleanly in schema.py, document the restriction in docs/devel/qapi-code-gen.rst, and explain why in the commit message. Recommend a separate commit, to make it stand out in git-log.
Quoted 8 lines
docs/sphinx/qapidoc.py +5
-0+ # Generated from entity def; info location is approximate. + self.add_line(f":{feat.name}:", feat.info) + + self.ensure_blank_line() + # Transmogrification core methods def visit_module(self, path: str) -> None:
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 64 lines · Markus Armbruster
Quoted 25 lines · John Snow
This method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +1
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2)docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 136
Quoted 10 lines
docs/sphinx/qapidoc.py +9
-0+ # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location isexact.
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 137
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ self.add_line(f":since: {ent.doc.since.text}",ent.doc.since.info)
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 4 lines
docs/sphinx/qapidoc.py +4
-0+ + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location isapproximate.
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 9 lines
docs/sphinx/qapidoc.py +9
-0+ self.add_line(f":ifcond: {doc}", ent.info) + + # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future, new + # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcondproperty.
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 151
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ # (Hello, intrepid developer in the future who changedthat!)
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 2 lines
docs/sphinx/qapidoc.py +2
-0+ # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()Nope :)
The attempt to add a conditional special feature now fails with
Quoted 8 lines
Sphinx parallel build error:
AssertionErrorIf you want to outlaw conditional special features, reject them cleanly in
schema.py, document the restriction indocs/devel/qapi-code-gen.rst, and explain why in the commit message. Recommend a separate commit, to make it stand out in git-log.
Do you advocate this? I wasn't sure what it *meant* for a special feature to be conditional; I couldn't conceive of what it meant to have an ifcond for "deprecated" or "unstable", for instance. It sounds like it isn't well defined, but we happen to not expressly forbid it.
I guard against it here because, similarly, I have no idea how to handle the case where it's true.
I didn't realize we technically allow it, though ... would you like me to move to expressly forbid it in the parser? (Failing that, I have no idea how to display this information otherwise, so I'd need you to sketch something out for me; so my inclination is to forbid it as you suggest. Future developers can always lift the restriction once they have some use-case in mind and a plan for how to display that information.)
--js
Quoted 12 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 154
Quoted 2 lines
docs/sphinx/qapidoc.py +1
-0+ # Generated from entity def; info location isapproximate.
Quoted 7 lines
docs/sphinx/qapidoc.py +4
-0+ self.add_line(f":{feat.name}:", feat.info) + + self.ensure_blank_line() + # Transmogrification core methods def visit_module(self, path: str) -> None:
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 69 lines · John Snow
Quoted 59 lines · Markus Armbruster
Quoted 42 lines · John Snow
This method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +17
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location is exact. + self.add_line(f":since: {ent.doc.since.text}", ent.doc.since.info) + + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location is approximate. + self.add_line(f":ifcond: {doc}", ent.info) +docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 10 lines
docs/sphinx/qapidoc.py +10
-0+ # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future, new + # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcond property. + # (Hello, intrepid developer in the future who changed that!) + # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()Nope :)
The attempt to add a conditional special feature now fails with
Quoted 7 lines
Sphinx parallel build error:
AssertionErrorIf you want to outlaw conditional special features, reject them cleanly in
schema.py, document the restriction indocs/devel/qapi-code-gen.rst, and explain why in the commit message. Recommend a separate commit, to make it stand out in git-log.
Quoted 4 lines
Do you advocate this? I wasn't sure what it *meant* for a special feature to be conditional; I couldn't conceive of what it meant to have an ifcond for "deprecated" or "unstable", for instance. It sounds like it isn't well defined, but we happen to not expressly forbid it.
Semantics are clear enough to me.
"Conditional special feature" combines "conditional feature" (which is a special case of conditional thing) with special feature (which is a special case of feature).
The QAPI schema language supports compile-time conditionals for certain things. Generated code behaves as if the thing didn't exist unless its condition is true.
QAPI schema features are strings exposed to clients in introspection.
Combine the two: a conditional feature is exposed if and only if its condition is true.
Existing uses: dynamic-auto-read-only if CONFIG_POSIX, fdset if CONFIG_BLKIO_VHOST_VDPA_FD.
A special feature is a feature that has special meaning to the generator, i.e. we generate different code in places when it's present.
Combine: we enable different code for a conditional special feature only when its condition is true.
No existing uses so far.
Implementation is straightforward, too.
Any code we generate for a conditional thing is guarded by #if ... #endif.
For features, we generate suitable data into qapi-introspect.c.
For special features, we generate a little additional code here and there; look for .is_special() to find it.
Bug: this additional code lacks #if ... #endif. Simple oversight, should be easy enough to fix.
Quoted 9 lines
I guard against it here because, similarly, I have no idea how to handle the case where it's true.
I didn't realize we technically allow it, though ... would you like me to move to expressly forbid it in the parser? (Failing that, I have no idea how to display this information otherwise, so I'd need you to sketch something out for me; so my inclination is to forbid it as you suggest. Future developers can always lift the restriction once they have some use-case in mind and a plan for how to display that information.)
I think we should first make a reasonable effort at figuring out how to handle conditional special features. If we fail, we can add the restriction instead.
How do you handle features in general, and special features in particular?
How do you handle conditionals in general?
How do you combine feature and conditional?
How could you combine special feature and conditonal?
[...]
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 106 lines · Markus Armbruster
Quoted 1 line · John Snow
On Fri, Dec 20, 2024 at 9:15 AM Markus Armbruster <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 1 line · John Snow
This method adds the options/preamble to each definition block.
Notably,
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
:since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +1
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2)Quoted 9 lines
Quoted 9 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 9 lines
docs/sphinx/qapidoc.py +9
-0+ # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location isexact.
Quoted 1 line
Quoted 1 line
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 137
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ self.add_line(f":since: {ent.doc.since.text}",ent.doc.since.info)
Quoted 4 lines
Quoted 4 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 4 lines
docs/sphinx/qapidoc.py +4
-0+ + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location isapproximate.
Quoted 4 lines
Quoted 4 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 4 lines
docs/sphinx/qapidoc.py +4
-0+ self.add_line(f":ifcond: {doc}", ent.info) + + # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future,new
Quoted 5 lines
Quoted 5 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 5 lines
docs/sphinx/qapidoc.py +5
-0+ # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcondproperty.
Quoted 1 line
Quoted 1 line
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 151
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ # (Hello, intrepid developer in the future whochanged that!)
Quoted 10 lines
Quoted 10 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 2 lines
docs/sphinx/qapidoc.py +2
-0+ # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()Nope :)
The attempt to add a conditional special feature now fails with
Sphinx parallel build error:
AssertionErrorQuoted 8 lines
Quoted 4 lines
If you want to outlaw conditional special features, reject them cleanly in
schema.py, document the restriction indocs/devel/qapi-code-gen.rst, and explain why in the commit message. Recommend a separate commit, to make it stand out in git-log.Do you advocate this? I wasn't sure what it *meant* for a special feature to be conditional; I couldn't conceive of what it meant to have an ifcond for "deprecated" or "unstable", for instance. It sounds like it isn't
well
Quoted 1 line
defined, but we happen to not expressly forbid it.
Semantics are clear enough to me.
"Conditional special feature" combines "conditional feature" (which is a special case of conditional thing) with special feature (which is a special case of feature).
The QAPI schema language supports compile-time conditionals for certain things. Generated code behaves as if the thing didn't exist unless its condition is true.
QAPI schema features are strings exposed to clients in introspection.
Combine the two: a conditional feature is exposed if and only if its condition is true.
Existing uses: dynamic-auto-read-only if
CONFIG_POSIX, fdset ifCONFIG_BLKIO_VHOST_VDPA_FD.A special feature is a feature that has special meaning to the generator, i.e. we generate different code in places when it's present.
Quoted 7 lines
Combine: we enable different code for a conditional special feature only when its condition is true.
No existing uses so far.
Implementation is straightforward, too.
Mechanically, it's well defined... but that isn't my concern.
What I asked you was: "what would it mean for deprecated to be conditional?"
Without understanding that, I have no chance to design a system that handles that information accordingly for the documentation.
Quoted 21 lines
Any code we generate for a conditional thing is guarded by #if ... #endif.
For features, we generate suitable data into
qapi-introspect.c.For special features, we generate a little additional code here and there; look for .is_special() to find it.
Bug: this additional code lacks #if ... #endif. Simple oversight, should be easy enough to fix.
Quoted 9 lines
I guard against it here because, similarly, I have no idea how to handle the case where it's true.
I didn't realize we technically allow it, though ... would you like me to move to expressly forbid it in the parser? (Failing that, I have no idea how to display this information otherwise, so I'd need you to sketch something out for me; so my inclination is to forbid it as you suggest. Future developers can always lift the restriction once they have some use-case in mind and a plan for how to display that information.)
Quoted 4 lines
I think we should first make a reasonable effort at figuring out how to handle conditional special features. If we fail, we can add the restriction instead.
I don't think I agree; I don't think it's worth spending time defining a feature we don't use in our minds to then design around a hypothetical future use.
Easier to admit we don't use it and save defining the semantics for the future developer who stumbles across a reason for needing it.
Quoted 3 lines
How do you handle features in general, and special features in particular?
Features: they go in the :feat: field list.
Special features: they go in the :feat: field list, and also receive a special option flag to the directive to allow for special rendering of the "signature bar" (the header line for a given entity definition in the rendered documentation.)
e.g.
@deprecated will generate two things:
1) the definition block will get a :deprecated: option, which enables the html renderer to (potentially) do special things like color the entry differently, add a "warning pip"/eyecatch, etc. It's scoped to the whole definition.
2) A :feat deprecated: line which is added to the features "field list" as per normal for features.
Quoted 2 lines
How do you handle conditionals in general?
In this series, I don't!
...except for top-level conditionals, which are currently passed as-is as an argument to the :ifcond: directive option. The format of that argument and what the HTML renderer actually does with it is currently TBD.
(In current prototypes I just print the conditional string in the signature bar.)
Quoted 2 lines
How do you combine feature and conditional?
Notably TBD. Unhandled both in this series as-is *and* in my larger WIP.
Quoted 2 lines
How could you combine special feature and conditonal?
Bewilderingly uncertain.
Mechanically, it *could* be handled the same way :ifcond: directive options are.
e.g.
```
.. qapi:command:: x-beware-of-the-leopard
:deprecated: {ifcond stuff here}
:unstable: {ifcond stuff here}
lorem ipsum dolor sit amet
:feat deprecated: blah blah (if: ...)
:feat unstable: blah blah (if: ...)
```Semantically and in the rendered output, I have absolutely no clue whatsoever; and doubt I could figure it out without a use case in front of me to design around ... so I'd prefer to put up a traffic barrier for now.
i.o.w. the design issue arises specifically because special features are semanticaly special and are "hoisted" to be block-level in the transmogrified rST doc, and so syntax needs to be designed, implemented and tested if they are to be conditional.
Yet, we have no such cases to facilitate the design, implementation and testing.
So, I'd like to prohibit until such time as we have such a case.
Quoted 3 lines
[...]
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 107 lines · John Snow
Quoted 97 lines · Markus Armbruster
Quoted 69 lines · John Snow
Quoted 59 lines · Markus Armbruster
Quoted 42 lines · John Snow
This method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +17
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location is exact. + self.add_line(f":since: {ent.doc.since.text}", ent.doc.since.info) + + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location is approximate. + self.add_line(f":ifcond: {doc}", ent.info) +docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 10 lines
docs/sphinx/qapidoc.py +10
-0+ # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future, new + # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcond property. + # (Hello, intrepid developer in the future who changed that!) + # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()Nope :)
The attempt to add a conditional special feature now fails with
Quoted 7 lines
Sphinx parallel build error:
AssertionErrorIf you want to outlaw conditional special features, reject them cleanly in
schema.py, document the restriction indocs/devel/qapi-code-gen.rst, and explain why in the commit message. Recommend a separate commit, to make it stand out in git-log.Quoted 4 lines
Do you advocate this? I wasn't sure what it *meant* for a special feature to be conditional; I couldn't conceive of what it meant to have an ifcond for "deprecated" or "unstable", for instance. It sounds like it isn't well defined, but we happen to not expressly forbid it.
Semantics are clear enough to me.
"Conditional special feature" combines "conditional feature" (which is a special case of conditional thing) with special feature (which is a special case of feature).
The QAPI schema language supports compile-time conditionals for certain things. Generated code behaves as if the thing didn't exist unless its condition is true.
QAPI schema features are strings exposed to clients in introspection.
Combine the two: a conditional feature is exposed if and only if its condition is true.
Existing uses: dynamic-auto-read-only if
CONFIG_POSIX, fdset ifCONFIG_BLKIO_VHOST_VDPA_FD.A special feature is a feature that has special meaning to the generator, i.e. we generate different code in places when it's present.
Quoted 7 lines
Combine: we enable different code for a conditional special feature only when its condition is true.
No existing uses so far.
Implementation is straightforward, too.
Quoted 6 lines
Mechanically, it's well defined... but that isn't my concern.
What I asked you was: "what would it mean for deprecated to be conditional?"
Without understanding that, I have no chance to design a system that handles that information accordingly for the documentation.
Special feature deprecated means the use of the thing that has it is deprecated with this particular qemu-system-FOO.
Why "this particular"? Because it wasn't deprecated in some prior version of QEMU. Features (special or not) are always, always about the specific qemu-system-FOO binary you're talking to via QMP.
Conditionals are compile-time schema configuration. Either the condition is true or not at compile time. If it's true, the thing that has it is there, else it isn't.
Now combine the two. If the condition of special feature 'deprecate' is true, the thing that has it is deprecated with this particular qemu-system-FOO. Else it isn't.
Quoted 25 lines
Quoted 21 lines
Any code we generate for a conditional thing is guarded by #if ... #endif.
For features, we generate suitable data into
qapi-introspect.c.For special features, we generate a little additional code here and there; look for .is_special() to find it.
Bug: this additional code lacks #if ... #endif. Simple oversight, should be easy enough to fix.
Quoted 9 lines
I guard against it here because, similarly, I have no idea how to handle the case where it's true.
I didn't realize we technically allow it, though ... would you like me to move to expressly forbid it in the parser? (Failing that, I have no idea how to display this information otherwise, so I'd need you to sketch something out for me; so my inclination is to forbid it as you suggest. Future developers can always lift the restriction once they have some use-case in mind and a plan for how to display that information.)
Quoted 3 lines
I think we should first make a reasonable effort at figuring out how to handle conditional special features. If we fail, we can add the restriction instead.
Quoted 6 lines
I don't think I agree; I don't think it's worth spending time defining a feature we don't use in our minds to then design around a hypothetical future use.
Easier to admit we don't use it and save defining the semantics for the future developer who stumbles across a reason for needing it.
Semantics of "apply conditional to feature" don't need defining, because they are simply the same as "apply conditional to any other thing." When you have a definition for the general case that works for all the special cases, then outlawing one special case *increases* complexity. That's why I'm reluctant.
I figure your actual problem more concrete than "what does it even mean?", namely how to present conditionals in generated documentation. In that context, features *may* differ from any other thing!
Aside: the (mostly generated) QMP introspection is about this particular qemu-system-FOO, like all of QMP, whereas the (mostly generated) QMP documentation is about any qemu-system-FOO that could be built from this particular working tree. May lead to doc generation problems that have no match elsewhere.
Quoted 32 lines
Quoted 3 lines
How do you handle features in general, and special features in particular?
Features: they go in the :feat: field list.
Special features: they go in the :feat: field list, and also receive a special option flag to the directive to allow for special rendering of the "signature bar" (the header line for a given entity definition in the rendered documentation.)e.g.
@deprecated will generate two things:
1) the definition block will get a :deprecated: option, which enables the html renderer to (potentially) do special things like color the entry differently, add a "warning pip"/eyecatch, etc. It's scoped to the whole definition.
2) A :feat deprecated: line which is added to the features "field list" as per normal for features.
Quoted 2 lines
How do you handle conditionals in general?
In this series, I don't!
...except for top-level conditionals, which are currently passed as-is as an argument to the :ifcond: directive option. The format of that argument and what the HTML renderer actually does with it is currently TBD.
Quoted 8 lines
(In current prototypes I just print the conditional string in the signature bar.)
Quoted 2 lines
How do you combine feature and conditional?
Notably TBD. Unhandled both in this series as-is *and* in my larger WIP.
How do you combine definition and conditional? You answered that right
above: largely TBD.
How do you combine enum value / object type member / command argument / event argument and conditional? Also answered right above: not yet.
What does that mean for the doc generator code? Does it simply ignore conditionals there?
Quoted 21 lines
Quoted 2 lines
How could you combine special feature and conditonal?
Bewilderingly uncertain.
Mechanically, it *could* be handled the same way :ifcond: directive options are.
e.g.
``` .. qapi:command:: x-beware-of-the-leopard :deprecated: {ifcond stuff here} :unstable: {ifcond stuff here} lorem ipsum dolor sit amet :feat deprecated: blah blah (if: ...) :feat unstable: blah blah (if: ...) ```
Quoted 8 lines
Semantically and in the rendered output, I have absolutely no clue whatsoever; and doubt I could figure it out without a use case in front of me to design around ... so I'd prefer to put up a traffic barrier for now.
i.o.w. the design issue arises specifically because special features are semanticaly special and are "hoisted" to be block-level in the transmogrified rST doc, and so syntax needs to be designed, implemented and tested if they are to be conditional.
You choose to make them special in the documentation. They haven't been special there so far. I'm not telling you not to, I'm just pointing out this is a problem we elect to have :)
Quoted 4 lines
Yet, we have no such cases to facilitate the design, implementation and testing.
So, I'd like to prohibit until such time as we have such a case.
I believe what you need isn't so much an explanation of semantics, it's use cases. And that's fair!
Let me offer two.
1. Imagine we're trying to develop something big & complex enough to make keeping it out of tree until done impractical. We feel even an in-tree branch would be impractical. Instead, we commit it to master, default off, configure --enable-complex-thing to get it. Not exactly something we do all the time, but not outlandish, either.
Further imagine that the big & complex thing will involve some new QMP commands replacing existing ones. As usual, we want to mark the ones being replaced deprecated, so we can remove them after a grace period.
But simply deprecating them in the schema would be premature! The big & complex thing may fail, and if it does, we rip it out. If it succeeds, we enable it unconditionally.
We can express what we're doing by making feature deprecated conditional on CONFIG_COMPLEX_THING, which is defined by --enable-complex-thing.
That way, the complex thing doesn't affect the QMP interface unless we enable it. Good. And if we enable it, we do get the deprecation, and can test management applications cope with it.
2. Imagine we turned allow-preconfig into a special feature. Further imagine some whole-stack feature requires a certain command to have allow-preconfig, and you got tasked with implementing it. Which you duly did, to everybody's satisfaction. It is now days to the release, rc4 has sailed, and some dude comes out of the woodwork to report the command crashes in preconfig state. Weird! It worked *fine* in your meticulous testing. After some back and forth, you discover that the reporter builds with --enable-exotic-crap, which you didn't test, because you weren't even aware it exists. You enable it, and now the command crashes for you, too. Fixing this after rc4 is out of the question, too much churn. You could, however, make feature allow-preconfig conditional on not CONFIG_EXOTIC_CRAP.
Quoted 1 line
Quoted 1 line
[...]
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 326 lines · Markus Armbruster
Quoted 1 line · John Snow
On Thu, Jan 9, 2025, 5:34 AM Markus Armbruster <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 1 line · John Snow
On Fri, Dec 20, 2024 at 9:15 AM Markus Armbruster <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line · John Snow
This method adds the options/preamble to each definition block.
Notably,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
:since: and :ifcond: are added, as are any "special features" such
as
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
:deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +1
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfoQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 125
Quoted 3 lines
docs/sphinx/qapidoc.py +0
-0from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment byone.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 11 lines
docs/sphinx/qapidoc.py +9
-0self.add_line_raw("", fname, line + 2) + # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info locationis exact.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 137
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ self.add_line(f":since: {ent.doc.since.text}",ent.doc.since.info)
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 4 lines
docs/sphinx/qapidoc.py +4
-0+ + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location isapproximate.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 3 lines
docs/sphinx/qapidoc.py +3
-0+ self.add_line(f":ifcond: {doc}", ent.info) + + # Hoist special features such as :deprecated: and:unstable:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 145
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ # into the options block for the entity. If, in thefuture, new
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None:
Quoted 5 lines
docs/sphinx/qapidoc.py +5
-0+ # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have anifcond property.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 151
Quoted 1 line
docs/sphinx/qapidoc.py +1
-0+ # (Hello, intrepid developer in the future whochanged that!)
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 2 lines
docs/sphinx/qapidoc.py +2
-0+ # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()Nope :)
The attempt to add a conditional special feature now fails with
Sphinx parallel build error:
AssertionErrorIf you want to outlaw conditional special features, reject them
cleanly
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in
schema.py, document the restriction in
docs/devel/qapi-code-gen.rst,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and explain why in the commit message. Recommend a separate commit,
to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
make it stand out in git-log.
Do you advocate this? I wasn't sure what it *meant* for a special
feature
Quoted 1 line
Quoted 1 line
Quoted 1 line
to be conditional; I couldn't conceive of what it meant to have an
ifcond
Quoted 1 line
Quoted 1 line
Quoted 1 line
for "deprecated" or "unstable", for instance. It sounds like it isn't
well
Quoted 26 lines
Quoted 26 lines
Quoted 1 line
defined, but we happen to not expressly forbid it.
Semantics are clear enough to me.
"Conditional special feature" combines "conditional feature" (which is a special case of conditional thing) with special feature (which is a special case of feature).
The QAPI schema language supports compile-time conditionals for certain things. Generated code behaves as if the thing didn't exist unless its condition is true.
QAPI schema features are strings exposed to clients in introspection.
Combine the two: a conditional feature is exposed if and only if its condition is true.
Existing uses: dynamic-auto-read-only if
CONFIG_POSIX, fdset ifCONFIG_BLKIO_VHOST_VDPA_FD.A special feature is a feature that has special meaning to the generator, i.e. we generate different code in places when it's present.
Combine: we enable different code for a conditional special feature only when its condition is true.
Quoted 8 lines
Quoted 4 lines
No existing uses so far.
Implementation is straightforward, too.
Mechanically, it's well defined... but that isn't my concern.
What I asked you was: "what would it mean for deprecated to be
conditional?"
Quoted 3 lines
Without understanding that, I have no chance to design a system that handles that information accordingly for the documentation.
Special feature deprecated means the use of the thing that has it is deprecated with this particular qemu-system-FOO.
Why "this particular"? Because it wasn't deprecated in some prior version of QEMU. Features (special or not) are always, always about the specific qemu-system-FOO binary you're talking to via QMP.
Conditionals are compile-time schema configuration. Either the condition is true or not at compile time. If it's true, the thing that has it is there, else it isn't.
Now combine the two. If the condition of special feature 'deprecate' is true, the thing that has it is deprecated with this particular qemu-system-FOO. Else it isn't.
Quoted 5 lines
Quoted 5 lines
Any code we generate for a conditional thing is guarded by #if ... #endif.
For features, we generate suitable data into
qapi-introspect.c.Quoted 7 lines
Quoted 7 lines
For special features, we generate a little additional code here and there; look for .is_special() to find it.
Bug: this additional code lacks #if ... #endif. Simple oversight, should be easy enough to fix.
Quoted 1 line
I guard against it here because, similarly, I have no idea how to
handle
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the case where it's true.
I didn't realize we technically allow it, though ... would you like
me to
Quoted 1 line
Quoted 1 line
Quoted 1 line
move to expressly forbid it in the parser? (Failing that, I have no
idea
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
how to display this information otherwise, so I'd need you to sketch something out for me; so my inclination is to forbid it as you
suggest.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Future developers can always lift the restriction once they have some use-case in mind and a plan for how to display that information.)
I think we should first make a reasonable effort at figuring out how to handle conditional special features. If we fail, we can add the restriction instead.
Quoted 6 lines
I don't think I agree; I don't think it's worth spending time defining a feature we don't use in our minds to then design around a hypothetical future use.
Easier to admit we don't use it and save defining the semantics for the future developer who stumbles across a reason for needing it.
Semantics of "apply conditional to feature" don't need defining, because they are simply the same as "apply conditional to any other thing." When you have a definition for the general case that works for all the special cases, then outlawing one special case *increases* complexity. That's why I'm reluctant.
I figure your actual problem more concrete than "what does it even mean?", namely how to present conditionals in generated documentation. In that context, features *may* differ from any other thing!
Aside: the (mostly generated) QMP introspection is about this particular qemu-system-FOO, like all of QMP, whereas the (mostly generated) QMP documentation is about any qemu-system-FOO that could be built from this particular working tree. May lead to doc generation problems that have no match elsewhere.
Quoted 7 lines
Quoted 3 lines
How do you handle features in general, and special features in particular?
Features: they go in the :feat: field list.
Special features: they go in the :feat: field list, and also receive a special option flag to the directive to allow for special rendering ofthe
Quoted 5 lines
"signature bar" (the header line for a given entity definition in the rendered documentation.)
e.g.
Quoted 8 lines
@deprecated will generate two things:
1) the definition block will get a :deprecated: option, which enables the html renderer to (potentially) do special things like color the entry differently, add a "warning pip"/eyecatch, etc. It's scoped to the whole definition.
2) A :feat deprecated: line which is added to the features "field list"
as
Quoted 6 lines
per normal for features.
Quoted 2 lines
How do you handle conditionals in general?
Quoted 7 lines
In this series, I don't!
...except for top-level conditionals, which are currently passed as-is as an argument to the :ifcond: directive option. The format of that argument and what the HTML renderer actually does with it is currently TBD.
(In current prototypes I just print the conditional string in the
signature
Quoted 7 lines
bar.)
Quoted 2 lines
How do you combine feature and conditional?
Notably TBD. Unhandled both in this series as-is *and* in my larger WIP.
How do you combine definition and conditional? You answered that right
above: largely TBD.How do you combine enum value / object type member / command argument / event argument and conditional? Also answered right above: not yet.
What does that mean for the doc generator code? Does it simply ignore conditionals there?
Quoted 6 lines
Quoted 2 lines
How could you combine special feature and conditonal?
Bewilderingly uncertain.
Mechanically, it *could* be handled the same way :ifcond: directive
options
Quoted 15 lines
are.
e.g.
``` .. qapi:command:: x-beware-of-the-leopard :deprecated: {ifcond stuff here} :unstable: {ifcond stuff here} lorem ipsum dolor sit amet :feat deprecated: blah blah (if: ...) :feat unstable: blah blah (if: ...) ```Quoted 2 lines
Semantically and in the rendered output, I have absolutely no clue whatsoever; and doubt I could figure it out without a use case in front
of
Quoted 1 line
me to design around ... so I'd prefer to put up a traffic barrier for
now.
Quoted 4 lines
i.o.w. the design issue arises specifically because special features are semanticaly special and are "hoisted" to be block-level in the transmogrified rST doc, and so syntax needs to be designed, implemented
and
Quoted 1 line
tested if they are to be conditional.
You choose to make them special in the documentation. They haven't been special there so far. I'm not telling you not to, I'm just pointing out this is a problem we elect to have :)
Quoted 4 lines
Yet, we have no such cases to facilitate the design, implementation and testing.
So, I'd like to prohibit until such time as we have such a case.
I believe what you need isn't so much an explanation of semantics, it's use cases. And that's fair!
Let me offer two.
1. Imagine we're trying to develop something big & complex enough to make keeping it out of tree until done impractical. We feel even an in-tree branch would be impractical. Instead, we commit it to master, default off, configure --enable-complex-thing to get it. Not exactly something we do all the time, but not outlandish, either.
Further imagine that the big & complex thing will involve some new QMP commands replacing existing ones. As usual, we want to mark the ones being replaced deprecated, so we can remove them after a grace period.
But simply deprecating them in the schema would be premature! The big & complex thing may fail, and if it does, we rip it out. If it succeeds, we enable it unconditionally.
We can express what we're doing by making feature deprecated conditional on
CONFIG_COMPLEX_THING, which is defined by --enable-complex-thing.That way, the complex thing doesn't affect the QMP interface unless we enable it. Good. And if we enable it, we do get the deprecation, and can test management applications cope with it.
Quoted 17 lines
2. Imagine we turned allow-preconfig into a special feature. Further imagine some whole-stack feature requires a certain command to have allow-preconfig, and you got tasked with implementing it. Which you duly did, to everybody's satisfaction. It is now days to the release, rc4 has sailed, and some dude comes out of the woodwork to report the command crashes in preconfig state. Weird! It worked *fine* in your meticulous testing. After some back and forth, you discover that the reporter builds with --enable-exotic-crap, which you didn't test, because you weren't even aware it exists. You enable it, and now the command crashes for you, too. Fixing this after rc4 is out of the question, too much churn. You could, however, make feature allow-preconfig conditional on not
CONFIG_EXOTIC_CRAP.Quoted 1 line
Quoted 1 line
[...]
Hm, alright. I think I'll just stub out conditionals with a TODO and circle back to how I'll handle them; I need to do a pass on ifcond handling in general, so I guess I'll get to it then. I still think it's a little weird, but you don't, and you're the QAPI guy ;)
--js
Re: [PATCH 11/23] docs/qapidoc: add preamble() method
Quoted 81 lines · John Snow
Quoted 79 lines · Markus Armbruster
Quoted 77 lines · John Snow
Quoted 75 lines · Markus Armbruster
Quoted 73 lines · John Snow
Quoted 59 lines · Markus Armbruster
Quoted 42 lines · John Snow
This method adds the options/preamble to each definition block. Notably, :since: and :ifcond: are added, as are any "special features" such as :deprecated: and :unstable:.
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)docs/sphinx/qapidoc.py +17
-1@@ -38,7 +38,7 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema +from qapi.schema import QAPISchema, QAPISchemaEntity from qapi.source import QAPISourceInfo from sphinx import addnodes@@ -125,6 +125,37 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + # Transmogrification helpers + + def preamble(self, ent: QAPISchemaEntity) -> None: + """ + Generate option lines for qapi entity directives. + """ + if ent.doc and ent.doc.since: + assert ent.doc.since.tag == QAPIDoc.Tag.SINCE + # Generated from the entity's docblock; info location is exact. + self.add_line(f":since: {ent.doc.since.text}", ent.doc.since.info) + + if ent.ifcond.is_present(): + doc = ent.ifcond.docgen() + # Generated from entity definition; info location is approximate. + self.add_line(f":ifcond: {doc}", ent.info) +docs/sphinx/qapidoc.py def ensure_blank_line(self) -> None: · line 153
Quoted 10 lines
docs/sphinx/qapidoc.py +10
-0+ # Hoist special features such as :deprecated: and :unstable: + # into the options block for the entity. If, in the future, new + # special features are added, qapi-domain will chirp about + # unrecognized options and fail. + for feat in ent.features: + if feat.is_special(): + # We don't expect special features to have an ifcond property. + # (Hello, intrepid developer in the future who changed that!) + # ((With luck, you are not me.)) + assert not feat.ifcond.is_present()Nope :)
The attempt to add a conditional special feature now fails with
Quoted 7 lines
Sphinx parallel build error:
AssertionErrorIf you want to outlaw conditional special features, reject them cleanly in
schema.py, document the restriction indocs/devel/qapi-code-gen.rst, and explain why in the commit message. Recommend a separate commit, to make it stand out in git-log.Do you advocate this? I wasn't sure what it *meant* for a special feature to be conditional; I couldn't conceive of what it meant to have an ifcond for "deprecated" or "unstable", for instance. It sounds like it isn't well defined, but we happen to not expressly forbid it.
[...]
Quoted 42 lines
Quoted 42 lines
I believe what you need isn't so much an explanation of semantics, it's use cases. And that's fair!
Let me offer two.
1. Imagine we're trying to develop something big & complex enough to make keeping it out of tree until done impractical. We feel even an in-tree branch would be impractical. Instead, we commit it to master, default off, configure --enable-complex-thing to get it. Not exactly something we do all the time, but not outlandish, either.
Further imagine that the big & complex thing will involve some new QMP commands replacing existing ones. As usual, we want to mark the ones being replaced deprecated, so we can remove them after a grace period.
But simply deprecating them in the schema would be premature! The big & complex thing may fail, and if it does, we rip it out. If it succeeds, we enable it unconditionally.
We can express what we're doing by making feature deprecated conditional on
CONFIG_COMPLEX_THING, which is defined by --enable-complex-thing.That way, the complex thing doesn't affect the QMP interface unless we enable it. Good. And if we enable it, we do get the deprecation, and can test management applications cope with it.
2. Imagine we turned allow-preconfig into a special feature. Further imagine some whole-stack feature requires a certain command to have allow-preconfig, and you got tasked with implementing it. Which you duly did, to everybody's satisfaction. It is now days to the release, rc4 has sailed, and some dude comes out of the woodwork to report the command crashes in preconfig state. Weird! It worked *fine* in your meticulous testing. After some back and forth, you discover that the reporter builds with --enable-exotic-crap, which you didn't test, because you weren't even aware it exists. You enable it, and now the command crashes for you, too. Fixing this after rc4 is out of the question, too much churn. You could, however, make feature allow-preconfig conditional on not
CONFIG_EXOTIC_CRAP.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
[...]
Hm, alright. I think I'll just stub out conditionals with a TODO and circle back to how I'll handle them; I need to do a pass on ifcond handling in general, so I guess I'll get to it then.
Makes sense (although I'd use FIXME rather than TODO).
If resolving them turns out to be hard, we can talk about adding restrictions to help you over the hump.
Quoted 2 lines
I still think it's a little weird,
but you don't, and you're the QAPI guy ;)
I admit constructing believable use cases took a bit of thought. Calling them a little weird can't offend me :)
PATCH 12/23 docs/qapidoc: add visit_paragraph() method
In reply to John SnowThis transforms "formerly known as untagged sections" into our pure intermediate rST format. These sections are already pure rST, so this method doesn't do a whole lot except ensure appropriate newlines.
--- docs/sphinx/qapidoc.py | 9 +++++++++ 1 file changed, 9 insertions(+)
docs/sphinx/qapidoc.py +9 -0
@@ -127,6 +127,15 @@ def ensure_blank_line(self) -> None: # Transmogrification helpers + def visit_paragraph(self, section: QAPIDoc.Section) -> None: + # Squelch empty paragraphs. + if not section.text: + return + + self.ensure_blank_line() + self.add_lines(section.text, section.info) + self.ensure_blank_line() + def preamble(self, ent: QAPISchemaEntity) -> None: """ Generate option lines for qapi entity directives.
Signature · 1 line
2.47.0
PATCH 13/23 docs/qapidoc: add visit_errors() method
In reply to John SnowNotably, this method does not currently address the formatting issues present with the "errors" section in QAPIDoc and just vomits the text verbatim into the rST doc, with somewhat inconsistent results.
To be addressed in a future revision.
--- docs/sphinx/qapidoc.py | 6 ++++++ 1 file changed, 6 insertions(+)
docs/sphinx/qapidoc.py +6 -0
@@ -136,6 +136,12 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None: self.add_lines(section.text, section.info) self.ensure_blank_line() + def visit_errors(self, section: QAPIDoc.Section) -> None: + # FIXME: the formatting for errors may be inconsistent and may + # or may not require different newline placement to ensure + # proper rendering as a nested list. + self.add_lines(f":error:\n{section.text}", section.info) + def preamble(self, ent: QAPISchemaEntity) -> None: """ Generate option lines for qapi entity directives.
Signature · 1 line
2.47.0
PATCH 14/23 docs/qapidoc: add format_type() method
In reply to John SnowThis method is responsible for generating a type name for a given member with the correct annotations for the QAPI domain. Features and enums do not *have* types, so they return None. Everything else returns the type name with a "?" suffix if that type is optional, and ensconced in [brackets] if it's an array type.
--- docs/sphinx/qapidoc.py | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
docs/sphinx/qapidoc.py +28 -2
@@ -30,7 +30,7 @@ import re import sys import textwrap -from typing import List +from typing import List, Optional from docutils import nodes from docutils.parsers.rst import Directive, directives
@@ -38,7 +38,14 @@ from qapi.error import QAPIError, QAPISemError from qapi.gen import QAPISchemaVisitor from qapi.parser import QAPIDoc -from qapi.schema import QAPISchema, QAPISchemaEntity +from qapi.schema import ( + QAPISchema, + QAPISchemaArrayType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectTypeMember, +) from qapi.source import QAPISourceInfo from sphinx import addnodes
@@ -125,6 +132,25 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + def format_type(self, ent) -> Optional[str]: + if isinstance(ent, (QAPISchemaEnumMember, QAPISchemaFeature)): + return None + + qapi_type = ent + optional = False + if isinstance(ent, QAPISchemaObjectTypeMember): + qapi_type = ent.type + optional = ent.optional + + if isinstance(qapi_type, QAPISchemaArrayType): + ret = f"[{qapi_type.element_type.doc_type()}]" + else: + ret = qapi_type.doc_type() + if optional: + ret += "?" + + return ret + # Transmogrification helpers def visit_paragraph(self, section: QAPIDoc.Section) -> None:
Signature · 1 line
2.47.0
PATCH 15/23 docs/qapidoc: add add_field() and generate_field() helper methods
In reply to John SnowThese are simple rST generation methods that assist in getting the types and formatting correct for a field list entry. add_field() is a more raw, direct call while generate_field() is intended to be used for generating the correct field from a member object.
--- docs/sphinx/qapidoc.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
docs/sphinx/qapidoc.py +25 -0
@@ -44,6 +44,7 @@ QAPISchemaEntity, QAPISchemaEnumMember, QAPISchemaFeature, + QAPISchemaMember, QAPISchemaObjectTypeMember, ) from qapi.source import QAPISourceInfo
@@ -132,6 +133,20 @@ def ensure_blank_line(self) -> None: # +2: correct for zero/one index, then increment by one. self.add_line_raw("", fname, line + 2) + def add_field( + self, + kind: str, + name: str, + body: str, + info: QAPISourceInfo, + typ: Optional[str] = None, + ) -> None: + if typ: + text = f":{kind} {typ} {name}: {body}" + else: + text = f":{kind} {name}: {body}" + self.add_lines(text, info) + def format_type(self, ent) -> Optional[str]: if isinstance(ent, (QAPISchemaEnumMember, QAPISchemaFeature)): return None
@@ -151,6 +166,16 @@ def format_type(self, ent) -> Optional[str]: return ret + def generate_field( + self, + kind: str, + member: QAPISchemaMember, + body: str, + info: QAPISourceInfo, + ) -> None: + typ = self.format_type(member) + self.add_field(kind, member.name, body, info, typ) + # Transmogrification helpers def visit_paragraph(self, section: QAPIDoc.Section) -> None:
Signature · 1 line
2.47.0
PATCH 16/23 docs/qapidoc: add visit_feature() method
In reply to John SnowThe format_type() method is adjusted here to return None for features, because Features don't have documented types. This makes generate_field() safe to use for features.
--- docs/sphinx/qapidoc.py | 9 +++++++++ 1 file changed, 9 insertions(+)
docs/sphinx/qapidoc.py +9 -0
@@ -187,6 +187,15 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None: self.add_lines(section.text, section.info) self.ensure_blank_line() + def visit_feature(self, section: QAPIDoc.ArgSection) -> None: + # FIXME - ifcond for features is not handled at all yet! + # Proposal: decorate the right-hand column with some graphical + # element to indicate conditional availability? + assert section.text # Guaranteed by parser.py + assert section.member + + self.generate_field("feat", section.member, section.text, section.info) + def visit_errors(self, section: QAPIDoc.Section) -> None: # FIXME: the formatting for errors may be inconsistent and may # or may not require different newline placement to ensure
Signature · 1 line
2.47.0
Re: [PATCH 16/23] docs/qapidoc: add visit_feature() method
Quoted 3 lines · John Snow
The
format_type()method is adjusted here to return None for features, because Features don't have documented types. This makesgenerate_field()safe to use for features.
I'm confused: the patch doesn't touch format_type().
Quoted 26 lines
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 9 +++++++++ 1 file changed, 9 insertions(+)docs/sphinx/qapidoc.py +9
-0@@ -187,6 +187,15 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None: self.add_lines(section.text, section.info) self.ensure_blank_line() + def visit_feature(self, section: QAPIDoc.ArgSection) -> None: + # FIXME - ifcond for features is not handled at all yet! + # Proposal: decorate the right-hand column with some graphical + # element to indicate conditional availability? + assert section.text # Guaranteed by parser.py + assert section.member + + self.generate_field("feat", section.member, section.text, section.info) + def visit_errors(self, section: QAPIDoc.Section) -> None: # FIXME: the formatting for errors may be inconsistent and may # or may not require different newline placement to ensure
Re: [PATCH 16/23] docs/qapidoc: add visit_feature() method
Quoted 8 lines · Markus Armbruster
Quoted 3 lines · John Snow
The
format_type()method is adjusted here to return None for features, because Features don't have documented types. This makesgenerate_field()safe to use for features.I'm confused: the patch doesn't touch
format_type().
Patch refactoring booboo, sorry for the confusion.
Quoted 31 lines
docs/sphinx/qapidoc.py def visit_paragraph(self, section: QAPIDoc.Section)
Quoted 12 lines
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 9 +++++++++ 1 file changed, 9 insertions(+)docs/sphinx/qapidoc.py +0
-0@@ -187,6 +187,15 @@ def visit_paragraph(self, section: QAPIDoc.Section)-> None:
docs/sphinx/qapidoc.py def visit_paragraph(self, section: QAPIDoc.Section) -> None:
Quoted 11 lines
docs/sphinx/qapidoc.py +8
-0self.add_lines(section.text, section.info) self.ensure_blank_line() + def visit_feature(self, section: QAPIDoc.ArgSection) -> None: + # FIXME - ifcond for features is not handled at all yet! + # Proposal: decorate the right-hand column with some graphical + # element to indicate conditional availability? + assert section.text # Guaranteed by parser.py + assert section.member + + self.generate_field("feat", section.member, section.text,section.info)
Quoted 4 lines
docs/sphinx/qapidoc.py +1
-0+ def visit_errors(self, section: QAPIDoc.Section) -> None: # FIXME: the formatting for errors may be inconsistent and may # or may not require different newline placement to ensure
PATCH 17/23 docs/qapidoc: record current documented entity in transmogrifier
In reply to John SnowKeep a record of which entity we're working on documenting for the purposes of being able to change certain generative features conditionally and create stronger assertions.
If you find yourself asking: "Wait, but where does the current entity actually get recorded?!", you're right! That part comes with the visit_entity() implementation, which gets added later.
--- docs/sphinx/qapidoc.py | 6 ++++++ 1 file changed, 6 insertions(+)
docs/sphinx/qapidoc.py +6 -0
@@ -74,9 +74,15 @@ def dedent(text: str) -> str: class Transmogrifier: def __init__(self, schema): + self._curr_ent = None self._result = StringList() self.indent = 0 + @property + def entity(self) -> QAPISchemaEntity: + assert self._curr_ent is not None + return self._curr_ent + # General-purpose rST generation functions def get_indent(self) -> str:
Signature · 1 line
2.47.0
Re: [PATCH 17/23] docs/qapidoc: record current documented entity in transmogrifier
Quoted 7 lines · John Snow
Keep a record of which entity we're working on documenting for the purposes of being able to change certain generative features conditionally and create stronger assertions.
If you find yourself asking: "Wait, but where does the current entity actually get recorded?!", you're right! That part comes with the
visit_entity()implementation, which gets added later.
That's fine, except your title promises "record current documented entity". Perhaps "Prepare to record entity being transmogrified".
Quoted 25 lines
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 6 ++++++ 1 file changed, 6 insertions(+)docs/sphinx/qapidoc.py +6
-0@@ -74,9 +74,15 @@ def dedent(text: str) -> str: class Transmogrifier: def __init__(self, schema): + self._curr_ent = None self._result = StringList() self.indent = 0 + @property + def entity(self) -> QAPISchemaEntity: + assert self._curr_ent is not None + return self._curr_ent + # General-purpose rST generation functions def get_indent(self) -> str:
Re: [PATCH 17/23] docs/qapidoc: record current documented entity in transmogrifier
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · John Snow
Keep a record of which entity we're working on documenting for the purposes of being able to change certain generative features conditionally and create stronger assertions.
Quoted 7 lines
Quoted 3 lines
If you find yourself asking: "Wait, but where does the current entity actually get recorded?!", you're right! That part comes with the
visit_entity()implementation, which gets added later.That's fine, except your title promises "record current documented entity". Perhaps "Prepare to record entity being transmogrified".
Sometimes, I think I exist in a time and space that doesn't actually exist; a hallucination of multiple timelines that might exist only in my own consciousness. I'd be worried if I wasn't an ardent solipsist.
--js
Quoted 28 lines
Quoted 26 lines
Signed-off-by John Snow <[email protected]>--- docs/sphinx/qapidoc.py | 6 ++++++ 1 file changed, 6 insertions(+)docs/sphinx/qapidoc.py +6
-0@@ -74,9 +74,15 @@ def dedent(text: str) -> str: class Transmogrifier: def __init__(self, schema): + self._curr_ent = None self._result = StringList() self.indent = 0 + @property + def entity(self) -> QAPISchemaEntity: + assert self._curr_ent is not None + return self._curr_ent + # General-purpose rST generation functions def get_indent(self) -> str:
Re: [PATCH 17/23] docs/qapidoc: record current documented entity in transmogrifier
Quoted 12 lines · John Snow
Quoted 10 lines · Markus Armbruster
Quoted 7 lines · John Snow
Keep a record of which entity we're working on documenting for the purposes of being able to change certain generative features conditionally and create stronger assertions.
If you find yourself asking: "Wait, but where does the current entity actually get recorded?!", you're right! That part comes with the
visit_entity()implementation, which gets added later.
Quoted 7 lines
Quoted 3 lines
That's fine, except your title promises "record current documented entity". Perhaps "Prepare to record entity being transmogrified".
Sometimes, I think I exist in a time and space that doesn't actually exist; a hallucination of multiple timelines that might exist only in my own consciousness. I'd be worried if I wasn't an ardent solipsist.
I'm glad I haven't caused you worry! Perhaps I should try this solipsism thing myself some day.
[...]
PATCH 18/23 docs/qapidoc: add visit_returns() method
In reply to John SnowGenerates :returns: fields for explicit returns statements. Note that this does not presently handle undocumented returns, which is handled in a later commit.
--- docs/sphinx/qapidoc.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
docs/sphinx/qapidoc.py +14 -0
@@ -41,6 +41,7 @@ from qapi.schema import ( QAPISchema, QAPISchemaArrayType, + QAPISchemaCommand, QAPISchemaEntity, QAPISchemaEnumMember, QAPISchemaFeature,
@@ -202,6 +203,19 @@ def visit_feature(self, section: QAPIDoc.ArgSection) -> None: self.generate_field("feat", section.member, section.text, section.info) + def visit_returns(self, section: QAPIDoc.Section) -> None: + assert isinstance(self.entity, QAPISchemaCommand) + rtype = self.entity.ret_type + # q_empty can produce None, but we won't be documenting anything + # without an explicit return statement in the doc block, and we + # should not have any such explicit statements when there is no + # return value. + assert rtype + + typ = self.format_type(rtype) + assert section.text # We don't expect empty returns sections. + self.add_field("returns", typ, section.text, section.info) + def visit_errors(self, section: QAPIDoc.Section) -> None: # FIXME: the formatting for errors may be inconsistent and may # or may not require different newline placement to ensure
Signature · 1 line
2.47.0
PATCH 19/23 docs/qapidoc: add visit_member() method
In reply to John SnowThis method is used for generating the "members" of a wide variety of things, including structs, unions, enums, alternates, etc. The field name it uses to do so is dependent on the type of entity the "member" belongs to.
--- docs/sphinx/qapidoc.py | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
docs/sphinx/qapidoc.py +27 -0
@@ -74,6 +74,16 @@ def dedent(text: str) -> str: class Transmogrifier: + # Field names used for different entity types: + field_types = { + "enum": "value", + "struct": "memb", + "union": "memb", + "event": "memb", + "command": "arg", + "alternate": "choice", + } + def __init__(self, schema): self._curr_ent = None self._result = StringList()
@@ -84,6 +94,10 @@ def entity(self) -> QAPISchemaEntity: assert self._curr_ent is not None return self._curr_ent + @property + def member_field_type(self) -> str: + return self.field_types[self.entity.meta] + # General-purpose rST generation functions def get_indent(self) -> str:
@@ -194,6 +208,19 @@ def visit_paragraph(self, section: QAPIDoc.Section) -> None: self.add_lines(section.text, section.info) self.ensure_blank_line() + def visit_member(self, section: QAPIDoc.ArgSection) -> None: + # TODO: ifcond for members + # TODO?: features for members (documented at entity-level, + # but sometimes defined per-member. Should we add such + # information to member descriptions when we can?) + assert section.text + self.generate_field( + self.member_field_type, + section.member, + section.text, + section.info, + ) + def visit_feature(self, section: QAPIDoc.ArgSection) -> None: # FIXME - ifcond for features is not handled at all yet! # Proposal: decorate the right-hand column with some graphical
Signature · 1 line
2.47.0
PATCH 20/23 docs/qapidoc: add visit_sections() method
In reply to John SnowImplement the actual main dispatch method that processes and handles the list of doc sections for a given QAPI entity.
--- docs/sphinx/qapidoc.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
docs/sphinx/qapidoc.py +23 -0
@@ -278,6 +278,29 @@ def preamble(self, ent: QAPISchemaEntity) -> None: self.ensure_blank_line() + def visit_sections(self, ent: QAPISchemaEntity) -> None: + sections = ent.doc.all_sections if ent.doc else [] + + # Add sections *in the order they are documented*: + for section in sections: + if section.tag == QAPIDoc.Tag.UNTAGGED: + self.visit_paragraph(section) + elif section.tag == QAPIDoc.Tag.MEMBER: + self.visit_member(section) + elif section.tag == QAPIDoc.Tag.FEATURE: + self.visit_feature(section) + elif section.tag in (QAPIDoc.Tag.SINCE, QAPIDoc.Tag.TODO): + # Since is handled in preamble, TODO is skipped intentionally. + pass + elif section.tag == QAPIDoc.Tag.RETURNS: + self.visit_returns(section) + elif section.tag == QAPIDoc.Tag.ERRORS: + self.visit_errors(section) + else: + assert False + + self.ensure_blank_line() + # Transmogrification core methods def visit_module(self, path: str) -> None:
Signature · 1 line
2.47.0
PATCH 21/23 docs/qapidoc: add visit_entity()
In reply to John SnowFinally, the core entry method for a qapi entity.
--- docs/sphinx/qapidoc.py | 13 +++++++++++++ 1 file changed, 13 insertions(+)
docs/sphinx/qapidoc.py +13 -0
@@ -356,6 +356,19 @@ def visit_freeform(self, doc) -> None: self.add_lines(text, info) self.ensure_blank_line() + def visit_entity(self, ent): + assert ent is not None + + try: + self._curr_ent = ent + # This line gets credited to the start of the /definition/. + self.add_line(f".. qapi:{ent.meta}:: {ent.name}", ent.info) + with self.indented(): + self.preamble(ent) + self.visit_sections(ent) + finally: + self._curr_ent = None + # Disable black auto-formatter until re-enabled: # fmt: off
Signature · 1 line
2.47.0
PATCH 22/23 docs/qapidoc: implement transmogrify() method
In reply to John SnowThis is the true top-level processor for the new transmogrifier; responsible both for generating the intermediate rST and then running the nested parse on that generated document to produce the final docutils tree that is then - very finally - postprocessed by sphinx for final rendering to HTML &c.
--- docs/sphinx/qapidoc.py | 47 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)
docs/sphinx/qapidoc.py +46 -1
@@ -2,6 +2,7 @@ # # QEMU qapidoc QAPI file parsing extension # +# Copyright (c) 2024 Red Hat # Copyright (c) 2020 Linaro # # This work is licensed under the terms of the GNU GPLv2 or later.
@@ -53,12 +54,15 @@ from sphinx import addnodes from sphinx.directives.code import CodeBlock from sphinx.errors import ExtensionError +from sphinx.util import logging from sphinx.util.docutils import switch_source_input from sphinx.util.nodes import nested_parse_with_titles __version__ = "1.0" +logger = logging.getLogger(__name__) + def dedent(text: str) -> str: # Adjust indentation to make description text parse as paragraph.
@@ -89,6 +93,10 @@ def __init__(self, schema): self._result = StringList() self.indent = 0 + @property + def result(self) -> StringList: + return self._result + @property def entity(self) -> QAPISchemaEntity: assert self._curr_ent is not None
@@ -824,7 +832,43 @@ def new_serialno(self): return "qapidoc-%d" % env.new_serialno("qapidoc") def transmogrify(self, schema) -> nodes.Element: - raise NotImplementedError + logger.info("Transmogrifying QAPI to rST ...") + vis = Transmogrifier(schema) + modules = set() + + for doc in schema.docs: + module_source = doc.info.fname + if module_source not in modules: + vis.visit_module(module_source) + modules.add(module_source) + + if doc.symbol: + ent = schema.lookup_entity(doc.symbol) + assert ent + vis.visit_entity(ent) + else: + vis.visit_freeform(doc) + + logger.info("Transmogrification complete.") + + contentnode = nodes.section() + content = vis.result + titles_allowed = True + + logger.info("Transmogrifier running nested parse ...") + with switch_source_input(self.state, content): + if titles_allowed: + node: nodes.Element = nodes.section() + node.document = self.state.document + nested_parse_with_titles(self.state, content, contentnode) + else: + node = nodes.paragraph() + node.document = self.state.document + self.state.nested_parse(content, 0, contentnode) + logger.info("Transmogrifier's nested parse completed.") + sys.stdout.flush() + + return contentnode def legacy(self, schema) -> nodes.Element: vis = QAPISchemaGenRSTVisitor(self)
@@ -958,6 +1002,7 @@ def run(self) -> List[nodes.Node]: def setup(app): """Register qapi-doc directive with Sphinx""" + app.setup_extension("qapi-domain") app.add_config_value("qapidoc_srctree", None, "env") app.add_directive("qapi-doc", QAPIDocDirective) app.add_directive("qmp-example", QMPExample)
Signature · 1 line
2.47.0
PATCH 23/23 docs/qapidoc: add transmogrifier test document
In reply to John Snow--- docs/index.rst | 1 + docs/qapi/index.rst | 53 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 docs/qapi/index.rst
docs/index.rst +1 -0
@@ -18,3 +18,4 @@ Welcome to QEMU's documentation! interop/index specs/index devel/index + qapi/index
docs/qapi/index.rst new file +53 -0
53 lines not on this page · raw message
Signature · 1 line
2.47.0
Quoted 15 lines · John Snow
based-on: https://patchew.org/QEMU/[email protected]/
Hi!
This series is a very, very barebones implementation for the new QAPI doc generator. It does not have many features that I presented on at KVM Forum; the point of this patch set is instead to present a stripped down basis for ongoing work so we can discuss on-list with full context of the code available to do so.
The documentation this series generates is *not suitable* for replacing the current document generator, it has a few glaring omissions - on purpose - those features have been factored out intentionally so they can be reviewed with fuller context and more careful review.
Quoted 5 lines
What this series does:
- Adds the new "Transmogrifier" rST generator to
qapidoc.py, which generates an in-memory rST document using qapi-domain directives.
- Adds a test document that showcases this new transmogrifier.
Note to other reviewers: transmogrifier output is
docs/manual/qapi/index.html.
Quoted 8 lines
What this series very notably does not do (yet):
- "ifcond" data for anything other than top-level entities is not considered or rendered. This means "if" statements for features and members are entirely absent.
- The inliner is not present at all. This series renders only documentation exactly as it is exists in the source files.
This item is not even a regression.
Quoted 5 lines
- *branches* are themselves not considered at all; they're skipped entirely for now. They will be included alongside the inliner in either a subsequent series or a followup to this series.
- Undocumented members and return statements are not autogenerated.
The current doc generator auto-generates missing member documentation ("Not documented"). It doesn't auto-generate missing returns documentation. I explored auto-generating them, but shelved my work to not interfere with yours.
Quoted 2 lines
- Pseudofeatures (Things like allow-oob) are not generated as documented features.
What exactly are "pseudofeatures"?
Quoted 2 lines
- Documentation culling: all entities are documented whether or not they're relevant to the wire format.
Also not a regression.
Quoted 7 lines
My goal in doing it this way is to save the "fancy" features for later so we can focus on reviewing and tightening up the core functionality of the transmogrifier. Once we're on steadier ground, I will re-add the fanciful features while adjusting the qapi-domain mechanisms. Once everything looks "roughly right, give or take some minor nits", I will switch back to the qapi-domain series itself for review before we merge everything together.
Makes sense to me.
[...]
Quoted 7 lines
docs/index.rst | 1 + docs/qapi/index.rst | 53 ++++++ docs/sphinx/qapidoc.py | 419 ++++++++++++++++++++++++++++++++++++++--- scripts/qapi/parser.py | 97 +++++++--- scripts/qapi/source.py | 4 +- 5 files changed, 524 insertions(+), 50 deletions(-) create mode 100644 docs/qapi/index.rst
The changes to the QAPI generator core (scripts/qapi/) are small, and spread over just four patches:
qapi/source: allow multi-line QAPISourceInfo advancing qapi/schema: add __repr__ to QAPIDoc.Section
qapi: expand tags to all doc sections qapi/parser: adjust info location for doc body section
Quoted 29 lines · Markus Armbruster
Quoted 1 line · John Snow
based-on:
https://patchew.org/QEMU/[email protected]/
Quoted 14 lines
Hi!
This series is a very, very barebones implementation for the new QAPI doc generator. It does not have many features that I presented on at KVM Forum; the point of this patch set is instead to present a stripped down basis for ongoing work so we can discuss on-list with full context of the code available to do so.
The documentation this series generates is *not suitable* for replacing the current document generator, it has a few glaring omissions - on purpose - those features have been factored out intentionally so they can be reviewed with fuller context and more careful review.
Quoted 5 lines
What this series does:
- Adds the new "Transmogrifier" rST generator to
qapidoc.py, which generates an in-memory rST document using qapi-domain directives.
- Adds a test document that showcases this new transmogrifier.Note to other reviewers: transmogrifier output is
docs/manual/qapi/index.html.Quoted 2 lines
What this series very notably does not do (yet):
Quoted 9 lines
Quoted 6 lines
- "ifcond" data for anything other than top-level entities is not considered or rendered. This means "if" statements for features and members are entirely absent.
- The inliner is not present at all. This series renders only documentation exactly as it is exists in the source files.
This item is not even a regression.
No; but the version of this series as sent also does not add "The members of ..." stubs, which would be a regression. I didn't necessarily intend for this to be merged as-is; more of a "part one, with additional tricky elements that require more careful thought isolated into separate patches for later".
where "later" means "in v2" or "as a follow-up series as we stage things in a development branch before final submission for inclusion to origin/master" or whatever the actual mechanism is. I don't have a strong vision there, really; I just wanted to nail down the basics out in the open even if that was just between you (Markus) and I and we have a gentleman's agreement that it looks tentatively OK.
Quoted 12 lines
Quoted 6 lines
- *branches* are themselves not considered at all; they're skipped entirely for now. They will be included alongside the inliner in either a subsequent series or a followup to this series.
- Undocumented members and return statements are not autogenerated.
The current doc generator auto-generates missing member documentation ("Not documented"). It doesn't auto-generate missing returns documentation. I explored auto-generating them, but shelved my work to not interfere with yours.
Quoted 5 lines
Quoted 2 lines
- Pseudofeatures (Things like allow-oob) are not generated as documented features.
What exactly are "pseudofeatures"?
What I've named things like allow-oob that aren't features, but ought to be documented. We may well decide to promote them to real-deal special features, or maybe not. My work-in-progress branch currently just adds "dummy" features to document them. We can discuss this later alongside the patch that implements this.
Quoted 35 lines
Quoted 3 lines
- Documentation culling: all entities are documented whether or not they're relevant to the wire format.
Also not a regression.
Quoted 7 lines
My goal in doing it this way is to save the "fancy" features for later so we can focus on reviewing and tightening up the core functionality of the transmogrifier. Once we're on steadier ground, I will re-add the fanciful features while adjusting the qapi-domain mechanisms. Once everything looks "roughly right, give or take some minor nits", I will switch back to the qapi-domain series itself for review before we merge everything together.
Makes sense to me.
[...]
Quoted 7 lines
docs/index.rst | 1 + docs/qapi/index.rst | 53 ++++++ docs/sphinx/qapidoc.py | 419 ++++++++++++++++++++++++++++++++++++++--- scripts/qapi/parser.py | 97 +++++++--- scripts/qapi/source.py | 4 +- 5 files changed, 524 insertions(+), 50 deletions(-) create mode 100644 docs/qapi/index.rstThe changes to the QAPI generator core (
scripts/qapi/) are small, and spread over just four patches:qapi/source: allow multi-line QAPISourceInfo advancing qapi/schema: add
__repr__to QAPIDoc.Section
qapi: expand tags to all doc sections qapi/parser: adjust info location for doc body section
Quoted 35 lines · John Snow
Quoted 33 lines · Markus Armbruster
Quoted 1 line · John Snow
based-on:
https://patchew.org/QEMU/[email protected]/
Quoted 14 lines
Hi!
This series is a very, very barebones implementation for the new QAPI doc generator. It does not have many features that I presented on at KVM Forum; the point of this patch set is instead to present a stripped down basis for ongoing work so we can discuss on-list with full context of the code available to do so.
The documentation this series generates is *not suitable* for replacing the current document generator, it has a few glaring omissions - on purpose - those features have been factored out intentionally so they can be reviewed with fuller context and more careful review.
Quoted 5 lines
What this series does:
- Adds the new "Transmogrifier" rST generator to
qapidoc.py, which generates an in-memory rST document using qapi-domain directives.
- Adds a test document that showcases this new transmogrifier.Note to other reviewers: transmogrifier output is
docs/manual/qapi/index.html.Quoted 6 lines
What this series very notably does not do (yet):
- "ifcond" data for anything other than top-level entities is not considered or rendered. This means "if" statements for features and members are entirely absent.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- The inliner is not present at all. This series renders only documentation exactly as it is exists in the source files.
This item is not even a regression.
No; but the version of this series as sent also does not add "The members of ..." stubs, which would be a regression.
Right.
Quoted 11 lines
I didn't necessarily intend for
this to be merged as-is; more of a "part one, with additional tricky elements that require more careful thought isolated into separate patches for later".where "later" means "in v2" or "as a follow-up series as we stage things in a development branch before final submission for inclusion to origin/master" or whatever the actual mechanism is. I don't have a strong vision there, really; I just wanted to nail down the basics out in the open even if that was just between you (Markus) and I and we have a gentleman's agreement that it looks tentatively OK.
Got it.
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
- *branches* are themselves not considered at all; they're skipped entirely for now. They will be included alongside the inliner in either a subsequent series or a followup to this series.
- Undocumented members and return statements are not autogenerated.
The current doc generator auto-generates missing member documentation ("Not documented"). It doesn't auto-generate missing returns documentation. I explored auto-generating them, but shelved my work to not interfere with yours.
Quoted 2 lines
- Pseudofeatures (Things like allow-oob) are not generated as documented features.
Quoted 8 lines
Quoted 2 lines
What exactly are "pseudofeatures"?
What I've named things like allow-oob that aren't features, but ought to be documented. We may well decide to promote them to real-deal special features, or maybe not. My work-in-progress branch currently just adds "dummy" features to document them. We can discuss this later alongside the patch that implements this.
I agree this is a digression, so feel free to ignore the remainder of my reply for now.
We have two kinds of flags in the QAPI schema language: features and ad hoc flags. The ad hoc flags are 'boxed' (commands and events), 'gen', 'success-response', 'allow-oob', 'allow-preconfig', 'coroutine' (commands only).
The flags sort into three buckets:
1. Code generation directives that do not affect the external interface, and thus should not be visible in introspection: 'boxed', 'gen', 'coroutine'.
2. Flags that are visible at the external interface, but don't affect code generation beyond making them visible in introspection: the non-special features.
3. Code generation directives that affect the external interface, and thus are (or should be) visible in introspection.
3a. The special features: are visible.
3b. 'allow-oob': is visible, but differently, because it predates special features.
3c. 'allow-preconfig': not visible.
3d. 'success-response': not visible, because we use it for QGA only, which doesn't provide introspection.
Bucket 3 could use cleanup.
[...]
Quoted 122 lines · Markus Armbruster
Quoted 1 line · John Snow
On Thu, Dec 19, 2024 at 7:31 AM Markus Armbruster <[email protected]>
Quoted 10 lines · wrote:
Quoted 10 lines
Quoted 1 line · John Snow
based-on:
https://patchew.org/QEMU/[email protected]/
Quoted 5 lines
Hi!
This series is a very, very barebones implementation for the new QAPI doc generator. It does not have many features that I presented on at
KVM
Quoted 1 line
Quoted 1 line
Quoted 1 line
Forum; the point of this patch set is instead to present a stripped
down
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
basis for ongoing work so we can discuss on-list with full context of the code available to do so.
The documentation this series generates is *not suitable* for
replacing
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
the current document generator, it has a few glaring omissions - on purpose - those features have been factored out intentionally so they can be reviewed with fuller context and more careful review.
What this series does:
- Adds the new "Transmogrifier" rST generator to
qapidoc.py, which generates an in-memory rST document using qapi-domain directives.
- Adds a test document that showcases this new transmogrifier.Note to other reviewers: transmogrifier output is
docs/manual/qapi/index.html.Quoted 6 lines
What this series very notably does not do (yet):
- "ifcond" data for anything other than top-level entities is not considered or rendered. This means "if" statements for features and members are entirely absent.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- The inliner is not present at all. This series renders only documentation exactly as it is exists in the source files.
This item is not even a regression.
No; but the version of this series as sent also does not add "The members of ..." stubs, which would be a regression.
Right.
Quoted 1 line
I didn't necessarily intend
for
Quoted 5 lines
this to be merged as-is; more of a "part one, with additional tricky elements that require more careful thought isolated into separate patches for later".
where "later" means "in v2" or "as a follow-up series as we stage things
in
Quoted 3 lines
a development branch before final submission for inclusion to origin/master" or whatever the actual mechanism is. I don't have a strong vision there, really; I just wanted to nail down the basics out in the
open
Quoted 1 line
even if that was just between you (Markus) and I and we have a
gentleman's
Quoted 1 line
agreement that it looks tentatively OK.
Got it.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- *branches* are themselves not considered at all; they're skipped entirely for now. They will be included alongside the inliner in either a subsequent series or a followup to this series.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
- Undocumented members and return statements are not autogenerated.
The current doc generator auto-generates missing member documentation ("Not documented"). It doesn't auto-generate missing returns documentation. I explored auto-generating them, but shelved my work to not interfere with yours.
Quoted 1 line
- Pseudofeatures (Things like allow-oob) are not generated as
documented
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
features.
What exactly are "pseudofeatures"?
What I've named things like allow-oob that aren't features, but ought to
be
Quoted 3 lines
documented. We may well decide to promote them to real-deal special features, or maybe not. My work-in-progress branch currently just adds "dummy" features to document them. We can discuss this later alongside
the
Quoted 1 line
patch that implements this.
I agree this is a digression, so feel free to ignore the remainder of my reply for now.
We have two kinds of flags in the QAPI schema language: features and ad hoc flags. The ad hoc flags are 'boxed' (commands and events), 'gen', 'success-response', 'allow-oob', 'allow-preconfig', 'coroutine' (commands only).
The flags sort into three buckets:
1. Code generation directives that do not affect the external interface, and thus should not be visible in introspection: 'boxed', 'gen', 'coroutine'.
2. Flags that are visible at the external interface, but don't affect code generation beyond making them visible in introspection: the non-special features.
3. Code generation directives that affect the external interface, and thus are (or should be) visible in introspection.
3a. The special features: are visible.
3b. 'allow-oob': is visible, but differently, because it predates special features.
Quoted 7 lines
3c. 'allow-preconfig': not visible.
3d. 'success-response': not visible, because we use it for QGA only, which doesn't provide introspection.
Bucket 3 could use cleanup.
Yep. We can get into it on the pseudofeatures patch. Your analysis here matches my understanding.
Quoted 3 lines
[...]