Add QMP lexer for annotated JSON to Sphinx
cover Add QMP lexer for annotated JSON to Sphinx
QMP isn't json. It turns out Pygments cares about this. Teach Pygments to go with the annotated flow.
John Snow (2):
sphinx: add qmp_lexer
docs/bitmaps: use QMP lexer instead of json
docs/conf.py | 4 +-- docs/interop/bitmaps.rst | 54 ++++++++++++++++++++-------------------- docs/sphinx/qmp_lexer.py | 34 +++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 29 deletions(-) create mode 100644 docs/sphinx/qmp_lexer.py
Signature · 1 line
2.20.1
PATCH 1/2 sphinx: add qmp_lexer
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality notations.
--- docs/conf.py | 4 ++-- docs/sphinx/qmp_lexer.py | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) create mode 100644 docs/sphinx/qmp_lexer.py
docs/conf.py +2 -2
@@ -41,7 +41,7 @@ except NameError: # add these directories to sys.path here. If the directory is relative to the # documentation root, use an absolute path starting from qemu_docdir. # -# sys.path.insert(0, os.path.join(qemu_docdir, "my_subdir")) +sys.path.insert(0, os.path.join(qemu_docdir, "sphinx")) # -- General configuration ------------------------------------------------
@@ -54,7 +54,7 @@ needs_sphinx = '1.3' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = [] +extensions = ['qmp_lexer'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates']
docs/sphinx/qmp_lexer.py new file +34 -0
@@ -0,0 +1,34 @@ +# QEMU Monitor Protocol Lexer Extension +# +# Copyright (C) 2019, Red Hat Inc. +# +# Authors: +# Eduardo Habkost <[email protected]> +# John Snow <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory. +"""qmp_lexer is a Sphinx extension that provides a QMP lexer for code blocks.""" + +from pygments.lexer import RegexLexer, DelegatingLexer +from pygments.lexers.data import JsonLexer +import pygments.token + +class QMPExampleMarkersLexer(RegexLexer): + """QMPExampleMarkersLexer highlights directionality flow indicators.""" + tokens = { + 'root': [ + (r'-> ', pygments.token.Generic.Prompt), + (r'<- ', pygments.token.Generic.Prompt), + ] + } + +class QMPExampleLexer(DelegatingLexer): + """QMPExampleLexer highlights annotated QMP examples.""" + def __init__(self, **options): + super(QMPExampleLexer, self).__init__(JsonLexer, QMPExampleMarkersLexer, + pygments.token.Error, **options) + +def setup(sphinx): + """For use by Sphinx() extensions API.""" + sphinx.add_lexer("QMP", QMPExampleLexer())
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 1/2] sphinx: add qmp_lexer
Quoted 7 lines · John Snow
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
Quoted 6 lines
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality notations.
Signed-off-by Eduardo Habkost <[email protected]>Signed-off-by John Snow <[email protected]>
Thanks for figuring out how to integrate it into Sphinx!
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 1/2] sphinx: add qmp_lexer
In reply to John SnowQuoted 18 lines · John Snow
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality notations.
docs/sphinx/qmp_lexer.py new file +0
-0
docs/sphinx/qmp_lexer.py line 10
Quoted 10 lines
docs/sphinx/qmp_lexer.py new file +10
-0@@ -0,0 +1,34 @@ +# QEMU Monitor Protocol Lexer Extension +# +# Copyright (C) 2019, Red Hat Inc. +# +# Authors: +# Eduardo Habkost <[email protected]> +# John Snow <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory.
Did you definitely mean 2-only and not 2-or-later ?
thanks
-- PMM
Re: [Qemu-devel] [PATCH 1/2] sphinx: add qmp_lexer
Quoted 23 lines · Peter Maydell
Quoted 22 lines · John Snow
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality notations.
docs/sphinx/qmp_lexer.py new file +4
-0@@ -0,0 +1,34 @@ +# QEMU Monitor Protocol Lexer Extension +# +# Copyright (C) 2019, Red Hat Inc. +#
Quoted 8 lines
docs/sphinx/qmp_lexer.py line 10
Quoted 6 lines
docs/sphinx/qmp_lexer.py new file +6
-0+# Authors: +# Eduardo Habkost <[email protected]> +# John Snow <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory.Did you definitely mean 2-only and not 2-or-later ?
John asked if I was OK with GPLv2 before submitting the patch, but I'm also OK with 2-or-later.
Signature · 1 line
Eduardo
Re: [Qemu-devel] [PATCH 1/2] sphinx: add qmp_lexer
In reply to Peter MaydellQuoted 32 lines · Peter Maydell
Quoted 18 lines · John Snow
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality notations.
docs/sphinx/qmp_lexer.py new file +0
-0docs/sphinx/qmp_lexer.py line 10
Quoted 10 lines
docs/sphinx/qmp_lexer.py new file +10
-0@@ -0,0 +1,34 @@ +# QEMU Monitor Protocol Lexer Extension +# +# Copyright (C) 2019, Red Hat Inc. +# +# Authors: +# Eduardo Habkost <[email protected]> +# John Snow <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory.Did you definitely mean 2-only and not 2-or-later ?
Quoted 3 lines
thanks
-- PMM
Copy-paste pulled from another Python script. 2 or later is fine by me; I can resend if desired (or, I'd be fine with anyone touching it up in post.)
Re: [Qemu-devel] [PATCH 1/2] sphinx: add qmp_lexer
Quoted 36 lines · John Snow
Quoted 23 lines · Peter Maydell
Quoted 22 lines · John Snow
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning, it doesn't bring back highlighting on non-strict json blocks.
We can alleviate this by creating a custom lexer for QMP examples that allows us to properly highlight these examples in a robust way, keeping our directionality notations.
docs/sphinx/qmp_lexer.py new file +4
-0@@ -0,0 +1,34 @@ +# QEMU Monitor Protocol Lexer Extension +# +# Copyright (C) 2019, Red Hat Inc. +#Quoted 8 lines
docs/sphinx/qmp_lexer.py line 10
Quoted 6 lines
docs/sphinx/qmp_lexer.py new file +6
-0+# Authors: +# Eduardo Habkost <[email protected]> +# John Snow <[email protected]> +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory.Did you definitely mean 2-only and not 2-or-later ?
Quoted 3 lines
Copy-paste pulled from another Python script. 2 or later is fine by me; I can resend if desired (or, I'd be fine with anyone touching it up in post.)
Our default-preference as a project is 2-or-later, so if you're both happy with that I think we should use that.
thanks
-- PMM
PATCH 2/2 docs/bitmaps: use QMP lexer instead of json
In reply to John SnowThe annotated style json we use in QMP documentation is not strict json and depending on the version of Sphinx (2.0+) or Pygments installed, might cause the build to fail.
Use the new QMP lexer.
Further, some versions of Sphinx can not apply custom lexers to "code" directives and require the use of "code-block" directives instead, so make that change at this time as well.
Tested under:
- Sphinx 1.3.6 and Pygments 2.4
- Sphinx 1.7.6 and Pygments 2.2
- Sphinx 2.0.1 and Pygments 2.4
--- docs/interop/bitmaps.rst | 54 ++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 27 deletions(-)
docs/interop/bitmaps.rst +27 -27
@@ -199,7 +199,7 @@ persistence, and recording state can be adjusted at creation time. to create a new, actively recording persistent bitmap: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-add", "arguments": {
@@ -220,7 +220,7 @@ persistence, and recording state can be adjusted at creation time. To create a new, disabled (``-recording``), transient bitmap that tracks changes in 32KiB segments: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-add", "arguments": {
@@ -254,7 +254,7 @@ Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. Remove a bitmap named ``bitmap0`` from node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-remove", "arguments": {
@@ -280,7 +280,7 @@ Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-clear", "arguments": {
@@ -309,7 +309,7 @@ begin being recorded. ``+busy`` bitmaps cannot be enabled. To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-enable", "arguments": {
@@ -347,7 +347,7 @@ writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-disable", "arguments": {
@@ -393,7 +393,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty. ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves a copy. - .. code:: json + .. code-block:: QMP -> { "execute": "block-dirty-bitmap-merge", "arguments": {
@@ -424,7 +424,7 @@ attached to nodes serving as the root for guest devices. API. This result highlights a bitmap ``bitmap0`` attached to the root node of device ``drive0``. - .. code:: json + .. code-block:: QMP -> { "execute": "query-block",
@@ -562,7 +562,7 @@ new, empty bitmap that records writes from this point in time forward. destination. These writes will be recorded in the bitmap accordingly. -.. code:: json +.. code-block:: QMP -> { "execute": "transaction",
@@ -650,7 +650,7 @@ Example: Resetting an Incremental Backup Anchor Point If we want to start a new backup chain with an existing bitmap, we can also use a transaction to reset the bitmap while making a new full backup: -.. code:: json +.. code-block:: QMP -> { "execute": "transaction",
@@ -730,7 +730,7 @@ Example: First Incremental Backup #. Issue an incremental backup command: - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -788,7 +788,7 @@ Example: Second Incremental Backup #. Issue a new incremental backup command. The only difference here is that we have changed the target image below. - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -869,7 +869,7 @@ image: #. Issue a new incremental backup command. Apart from the new destination image, there is no difference from the last two examples. - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -932,7 +932,7 @@ point in time. #. Create a full (anchor) backup for each drive, with accompanying bitmaps: - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1018,7 +1018,7 @@ point in time. #. Issue a multi-drive incremental push backup transaction: - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1121,7 +1121,7 @@ described above. This example demonstrates the single-job failure case: #. Attempt to create an incremental backup via QMP: - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -1139,7 +1139,7 @@ described above. This example demonstrates the single-job failure case: #. Receive a pair of events indicating failure: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1175,7 +1175,7 @@ described above. This example demonstrates the single-job failure case: #. Retry the command after fixing the underlying problem, such as freeing up space on the backup volume: - .. code:: json + .. code-block:: QMP -> { "execute": "drive-backup",
@@ -1193,7 +1193,7 @@ described above. This example demonstrates the single-job failure case: #. Receive confirmation that the job completed successfully: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1233,7 +1233,7 @@ and one succeeds: #. Issue the transaction to start a backup of both drives. - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1267,13 +1267,13 @@ and one succeeds: #. Receive notice that the Transaction was accepted, and jobs were launched: - .. code:: json + .. code-block:: QMP <- { "return": {} } #. Receive notice that the first job has completed: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1289,7 +1289,7 @@ and one succeeds: #. Receive notice that the second job has failed: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1365,7 +1365,7 @@ applied: #. Issue the multi-drive incremental backup transaction: - .. code:: json + .. code-block:: QMP -> { "execute": "transaction",
@@ -1401,13 +1401,13 @@ applied: #. Receive notice that the Transaction was accepted, and jobs were launched: - .. code:: json + .. code-block:: QMP <- { "return": {} } #. Receive notification that the backup job for ``drive1`` has failed: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...},
@@ -1434,7 +1434,7 @@ applied: #. Receive notification that the job for ``drive0`` has been cancelled: - .. code:: json + .. code-block:: QMP <- { "timestamp": {...}
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 2/2] docs/bitmaps: use QMP lexer instead of json
Quoted 10 lines · John Snow
The annotated style json we use in QMP documentation is not strict json and depending on the version of Sphinx (2.0+) or Pygments installed, might cause the build to fail.
Use the new QMP lexer.
Further, some versions of Sphinx can not apply custom lexers to "code" directives and require the use of "code-block" directives instead, so make that change at this time as well.
Quoted 7 lines
Tested under:
- Sphinx 1.3.6 and Pygments 2.4
- Sphinx 1.7.6 and Pygments 2.2
- Sphinx 2.0.1 and Pygments 2.4Reported-by Aarushi Mehta <[email protected]>Signed-off-by John Snow <[email protected]>
Signature · 1 line
Eduardo