Add the ABI documentation to the admin guide
Subsystems (2)
PATCH 10/12 docs: kernel_abi.py: use --enable-lineno for get_abi.pl
In reply to Mauro Carvalho ChehabJust like kernel-doc extension, we need to be able to identify what part of an imported document has issues, as reporting them as:
get_abi.pl rest --dir $srctree/Documentation/ABI/obsolete --rst-source:1689: ERROR: Unexpected indentation.
Makes a lot harder for someone to fix.
It should be noticed that it the line which will be reported is the line where the "What:" definition is, and not the line with actually has an error.
--- Documentation/sphinx/kernel_abi.py | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
Documentation/sphinx/kernel_abi.py +14 -4
@@ -35,6 +35,7 @@ import codecs import os import subprocess import sys +import re from os import path
@@ -92,7 +93,7 @@ class KernelCmd(Directive): env = doc.settings.env cwd = path.dirname(doc.current_source) - cmd = "get_abi.pl rest --dir " + cmd = "get_abi.pl rest --enable-lineno --dir " cmd += self.arguments[0] srctree = path.abspath(os.environ["srctree"])
@@ -136,7 +137,7 @@ class KernelCmd(Directive): % (self.name, ErrorString(exc))) return out - def nestedParse(self, lines, fname): + def nestedParse(self, lines, f): content = ViewList() node = nodes.section()
@@ -146,8 +147,17 @@ class KernelCmd(Directive): code_block += "\n " + l lines = code_block + "\n\n" - for c, l in enumerate(lines.split("\n")): - content.append(l, fname, c) + line_regex = re.compile("^#define LINENO (\S+)\#([0-9]+)$") + ln = 0 + + for line in lines.split("\n"): + match = line_regex.search(line) + if match: + f = match.group(1) + # sphinx counts lines from 0 + ln = int(match.group(2)) - 1 + else: + content.append(line, f, ln) buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
Signature · 1 line
2.21.0
PATCH 6/12 docs: kernel_abi.py: fix UTF-8 support
In reply to Mauro Carvalho ChehabThe parser breaks with UTF-8 characters with Sphinx 1.4.
--- Documentation/sphinx/kernel_abi.py | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-)
Documentation/sphinx/kernel_abi.py +7 -13
@@ -1,4 +1,5 @@ -# -*- coding: utf-8; mode: python -*- +# coding=utf-8 +# u""" kernel-abi ~~~~~~~~~~
@@ -28,6 +29,7 @@ u""" """ +import codecs import sys import os from os import path
@@ -43,14 +45,6 @@ from docutils.utils.error_reporting import ErrorString __version__ = '1.0' -# We can't assume that six is installed -PY3 = sys.version_info[0] == 3 -PY2 = sys.version_info[0] == 2 -if PY3: - # pylint: disable=C0103, W0622 - unicode = str - basestring = str - def setup(app): app.add_directive("kernel-abi", KernelCmd)
@@ -115,12 +109,12 @@ class KernelCmd(Directive): cmd , stdout = subprocess.PIPE , stderr = subprocess.PIPE - , universal_newlines = True , **kwargs ) out, err = proc.communicate() - if err: - self.warn(err) + + out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8') + if proc.returncode != 0: raise self.severe( u"command '%s' failed with return code %d"
@@ -129,7 +123,7 @@ class KernelCmd(Directive): except OSError as exc: raise self.severe(u"problems with '%s' directive: %s." % (self.name, ErrorString(exc))) - return unicode(out) + return out def nestedParse(self, lines, fname): content = ViewList()
Signature · 1 line
2.21.0
cover Add the ABI documentation to the admin guide
Hi Greg/Jon,
That's the rebased patches with add the ABI documentation to the admin-guide. It is rebased on the top of driver-core/driver-core-next, and it can work both on the "old" mode, where the scripts will avoid codes that would cause troubles for ReST, or at the new mode, with parses the description to the ReST output untouched.
Patches 1 to 3 improves the get_abi.pl for it to parse ReST format, and to report where the errors/warnings occur at the ABI file.
Patch 4 is implements support for a Sphinx log facility (info).
Patches 5 to 10 adds the new kernel ABI extension, making it ready to parse files in ether "normal" or "rst-source" mode.
Patch 11 addresses a Sphinx issue, with is there since old vesions, but it gets worse on Sphins 2.0: basically, the ReST parser is lazy:
if it receives too much stuff, it silently ignores the end of the content. So, the extension needs to split the content on small chunks.
IMO, it is safe to apply patches 1 to 11 via Greg's tree. The only potential conflict would be at patch 4, but looking at docs-next, I don't see any changes at Documentation/scripts/kernelog.py.
Patch 12 actually enables everything.
If Patch 12 gets merged via Greg's tree, it may rise a single-line trivial conflict with docs-next, as it will be adding kernel_abi to the list of extensions at Documentation/conf.py, and Jon is likely adding another extension. too.
So, it could be wise postpone patch 12 to be merged after both trees got merged upstream.
Patch 12 adds ABI with ReST-scaping code. I'll submit soon a second version for the RFC patches with turns it to "clean" mode.
The entire ABI series (plus Documentation/features) is at:
https://git.linuxtv.org/mchehab/experimental.git/log/?h=abi_patches_v4.2
And the ReST output (with Sphinx 2.0 - with IMO provides a nicer output) is at:
https://www.infradead.org/~mchehab/rst_features/admin-guide/abi.html
Mauro Carvalho Chehab (12):
scripts/get_abi.pl: change script to allow parsing in ReST mode scripts/get_abi.pl: fix parsing on ReST mode scripts/get_abi.pl: Allow optionally record from where a line came
from
docs: kernellog.py: add support for info()
docs: kernel_abi.py: add a script to parse ABI documentation
docs: kernel_abi.py: fix UTF-8 support
docs: kernel_abi.py: make it compatible with Sphinx 1.7+
docs: kernel_abi.py: Update copyrights
docs: kernel_abi.py: add a SPDX header file
docs: kernel_abi.py: use --enable-lineno for get_abi.pl
docs: kernel_abi.py: Handle with a lazy Sphinx parser
docs: add ABI documentation to the admin-guide book
Documentation/admin-guide/abi-obsolete.rst | 10 ++ Documentation/admin-guide/abi-removed.rst | 4 + Documentation/admin-guide/abi-stable.rst | 13 ++ Documentation/admin-guide/abi-testing.rst | 19 +++ Documentation/admin-guide/abi.rst | 11 ++ Documentation/admin-guide/index.rst | 1 + Documentation/conf.py | 2 +- Documentation/sphinx/kernel_abi.py | 189 +++++++++++++++++++++ Documentation/sphinx/kernellog.py | 6 +- scripts/get_abi.pl | 104 ++++++++---- 10 files changed, 325 insertions(+), 34 deletions(-) create mode 100644 Documentation/admin-guide/abi-obsolete.rst create mode 100644 Documentation/admin-guide/abi-removed.rst create mode 100644 Documentation/admin-guide/abi-stable.rst create mode 100644 Documentation/admin-guide/abi-testing.rst create mode 100644 Documentation/admin-guide/abi.rst create mode 100644 Documentation/sphinx/kernel_abi.py
Signature · 1 line
2.21.0
PATCH 3/12 scripts/get_abi.pl: Allow optionally record from where a line came from
The get_abi.pl reads a lot of files and can join them on a single output file. Store where each "What:" output came from, in order to be able to optionally display it.
This is useful for the Sphinx extension, with can now be able to blame what ABI file has issues, and on what line the What: description with problems begin.
--- scripts/get_abi.pl | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
scripts/get_abi.pl +21 -1
@@ -10,6 +10,7 @@ use Fcntl ':mode'; my $help; my $man; my $debug; +my $enable_lineno; my $prefix="Documentation/ABI"; #
@@ -19,6 +20,7 @@ my $description_is_rst = 0; GetOptions( "debug|d+" => \$debug, + "enable-lineno" => \$enable_lineno, "rst-source!" => \$description_is_rst, "dir=s" => \$prefix, 'help|?' => \$help,
@@ -67,6 +69,7 @@ sub parse_abi { $data{$nametag}->{file} = $name; $data{$nametag}->{filepath} = $file; $data{$nametag}->{is_file} = 1; + $data{$nametag}->{line_no} = 1; my $type = $file; $type =~ s,.*/(.*)/.*,$1,;
@@ -126,6 +129,8 @@ sub parse_abi { if ($tag ne "" && $new_tag) { $tag = $new_tag; + $data{$what}->{line_no} = $ln; + if ($new_what) { @{$data{$what}->{label}} = @labels if ($data{$nametag}->{what}); @labels = ();
@@ -221,6 +226,12 @@ sub output_rest { my $file = $data{$what}->{file}; my $filepath = $data{$what}->{filepath}; + if ($enable_lineno) { + printf "#define LINENO %s%s#%s\n\n", + $prefix, $data{$what}->{file}, + $data{$what}->{line_no}; + } + my $w = $what; $w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
@@ -369,6 +380,10 @@ sub search_symbols { } } +# Ensure that the prefix will always end with a slash +# While this is not needed for find, it makes the patch nicer +# with --enable-lineno +$prefix =~ s,/?$,/,; # # Parses all ABI files located at $prefix dir
@@ -395,7 +410,8 @@ abi_book.pl - parse the Linux ABI files and produce a ReST book. =head1 SYNOPSIS -B<abi_book.pl> [--debug] [--man] [--help] --[(no-)rst-source] [--dir=<dir>] <COMAND> [<ARGUMENT>] +B<abi_book.pl> [--debug] [--enable-lineno] [--man] [--help] + [--(no-)rst-source] [--dir=<dir>] <COMAND> [<ARGUMENT>] Where <COMMAND> can be:
@@ -425,6 +441,10 @@ selecting between a rst-compliant source ABI (--rst-source), or a plain text that may be violating ReST spec, so it requres some escaping logic (--no-rst-source). +=item B<--enable-lineno> + +Enable output of #define LINENO lines. + =item B<--debug> Put the script in verbose mode, useful for debugging. Can be called multiple
Signature · 1 line
2.21.0
PATCH 7/12 docs: kernel_abi.py: make it compatible with Sphinx 1.7+
In reply to Mauro Carvalho ChehabThe same way kerneldoc.py needed changes to work with newer Sphinx, this script needs the same changes.
While here, reorganize the include order to match kerneldoc.py.
--- Documentation/sphinx/kernel_abi.py | 39 +++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 12 deletions(-)
Documentation/sphinx/kernel_abi.py +27 -12
@@ -30,18 +30,27 @@ u""" """ import codecs -import sys import os -from os import path import subprocess +import sys -from sphinx.ext.autodoc import AutodocReporter +from os import path -from docutils import nodes -from docutils.parsers.rst import Directive, directives +from docutils import nodes, statemachine from docutils.statemachine import ViewList +from docutils.parsers.rst import directives, Directive from docutils.utils.error_reporting import ErrorString +# +# AutodocReporter is only good up to Sphinx 1.7 +# +import sphinx + +Use_SSI = sphinx.__version__[:3] >= '1.7' +if Use_SSI: + from sphinx.util.docutils import switch_source_input +else: + from sphinx.ext.autodoc import AutodocReporter __version__ = '1.0'
@@ -139,11 +148,17 @@ class KernelCmd(Directive): content.append(l, fname, c) buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter - self.state.memo.title_styles = [] - self.state.memo.section_level = 0 - self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter) - try: - self.state.nested_parse(content, 0, node, match_titles=1) - finally: - self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf + + if Use_SSI: + with switch_source_input(self.state, content): + self.state.nested_parse(content, 0, node, match_titles=1) + else: + self.state.memo.title_styles = [] + self.state.memo.section_level = 0 + self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter) + try: + self.state.nested_parse(content, 0, node, match_titles=1) + finally: + self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf + return node.children
Signature · 1 line
2.21.0
PATCH 8/12 docs: kernel_abi.py: Update copyrights
In reply to Mauro Carvalho ChehabAs Markus doesn't want to maintain ths file, update it to put me as its maintainer.
--- Documentation/sphinx/kernel_abi.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Documentation/sphinx/kernel_abi.py +2 -1
@@ -7,7 +7,8 @@ u""" Implementation of the ``kernel-abi`` reST-directive. :copyright: Copyright (C) 2016 Markus Heiser - :copyright: Copyright (C) 2016 Mauro Carvalho Chehab + :copyright: Copyright (C) 2016-2019 Mauro Carvalho Chehab + :maintained-by: Mauro Carvalho Chehab <[email protected]> :license: GPL Version 2, June 1991 see Linux/COPYING for details. The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
Signature · 1 line
2.21.0
PATCH 9/12 docs: kernel_abi.py: add a SPDX header file
In reply to Mauro Carvalho ChehabThis file is released under GPL v2.
--- Documentation/sphinx/kernel_abi.py | 1 + 1 file changed, 1 insertion(+)
Documentation/sphinx/kernel_abi.py +1 -0
@@ -1,4 +1,5 @@ # coding=utf-8 +# SPDX-License-Identifier: GPL-2.0 # u""" kernel-abi
Signature · 1 line
2.21.0
PATCH 11/12 docs: kernel_abi.py: Handle with a lazy Sphinx parser
In reply to Mauro Carvalho ChehabThe Sphinx docutils parser is lazy: if the content is bigger than a certain number of lines, it silenlty stops parsing it, producing an incomplete content. This seems to be worse on newer Sphinx versions, like 2.0.
So, change the logic to parse the contents per input file.
--- Documentation/sphinx/kernel_abi.py | 39 ++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 13 deletions(-)
Documentation/sphinx/kernel_abi.py +26 -13
@@ -36,6 +36,7 @@ import os import subprocess import sys import re +import kernellog from os import path
@@ -79,12 +80,6 @@ class KernelCmd(Directive): "debug" : directives.flag } - def warn(self, message, **replace): - replace["fname"] = self.state.document.current_source - replace["line_no"] = replace.get("line_no", self.lineno) - message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace - self.state.document.settings.env.app.warn(message, prefix="") - def run(self): doc = self.state.document
@@ -110,7 +105,7 @@ class KernelCmd(Directive): shell_env["srctree"] = srctree lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env) - nodeList = self.nestedParse(lines, fname) + nodeList = self.nestedParse(lines, self.arguments[0]) return nodeList def runCmd(self, cmd, **kwargs):
@@ -137,9 +132,9 @@ class KernelCmd(Directive): % (self.name, ErrorString(exc))) return out - def nestedParse(self, lines, f): + def nestedParse(self, lines, fname): content = ViewList() - node = nodes.section() + node = nodes.section() if "debug" in self.options: code_block = "\n\n.. code-block:: rst\n :linenos:\n"
@@ -149,22 +144,42 @@ class KernelCmd(Directive): line_regex = re.compile("^#define LINENO (\S+)\#([0-9]+)$") ln = 0 + n = 0 + f = fname for line in lines.split("\n"): + n = n + 1 match = line_regex.search(line) if match: - f = match.group(1) + new_f = match.group(1) + + # Sphinx parser is lazy: it stops parsing contents in the + # middle, if it is too big. So, handle it per input file + if new_f != f and content: + self.do_parse(content, node) + content = ViewList() + + f = new_f + # sphinx counts lines from 0 ln = int(match.group(2)) - 1 else: content.append(line, f, ln) - buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter + kernellog.info(self.state.document.settings.env.app, "%s: parsed %i lines" % (fname, n)) + if content: + self.do_parse(content, node) + + return node.children + + def do_parse(self, content, node): if Use_SSI: with switch_source_input(self.state, content): self.state.nested_parse(content, 0, node, match_titles=1) else: + buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter + self.state.memo.title_styles = [] self.state.memo.section_level = 0 self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter)
@@ -172,5 +187,3 @@ class KernelCmd(Directive): self.state.nested_parse(content, 0, node, match_titles=1) finally: self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf - - return node.children
Signature · 1 line
2.21.0
PATCH 12/12 docs: add ABI documentation to the admin-guide book
In reply to Mauro Carvalho ChehabAs we don't want a generic Sphinx extension to execute commands, change the one proposed to Markus to call the abi_book.pl script.
Use a script to parse the Documentation/ABI directory and output it at the admin-guide.
--- Documentation/admin-guide/abi-obsolete.rst | 10 ++++++++++ Documentation/admin-guide/abi-removed.rst | 4 ++++ Documentation/admin-guide/abi-stable.rst | 13 +++++++++++++ Documentation/admin-guide/abi-testing.rst | 19 +++++++++++++++++++ Documentation/admin-guide/abi.rst | 11 +++++++++++ Documentation/admin-guide/index.rst | 1 + Documentation/conf.py | 2 +- 7 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/abi-obsolete.rst create mode 100644 Documentation/admin-guide/abi-removed.rst create mode 100644 Documentation/admin-guide/abi-stable.rst create mode 100644 Documentation/admin-guide/abi-testing.rst create mode 100644 Documentation/admin-guide/abi.rst
Documentation/admin-guide/abi-obsolete.rst new file +10 -0
@@ -0,0 +1,10 @@ +ABI obsolete symbols +==================== + +Documents interfaces that are still remaining in the kernel, but are +marked to be removed at some later point in time. + +The description of the interface will document the reason why it is +obsolete and when it can be expected to be removed. + +.. kernel-abi:: $srctree/Documentation/ABI/obsolete
Documentation/admin-guide/abi-removed.rst new file +4 -0
@@ -0,0 +1,4 @@ +ABI removed symbols +=================== + +.. kernel-abi:: $srctree/Documentation/ABI/removed
Documentation/admin-guide/abi-stable.rst new file +13 -0
@@ -0,0 +1,13 @@ +ABI stable symbols +================== + +Documents the interfaces that the developer has defined to be stable. + +Userspace programs are free to use these interfaces with no +restrictions, and backward compatibility for them will be guaranteed +for at least 2 years. + +Most interfaces (like syscalls) are expected to never change and always +be available. + +.. kernel-abi:: $srctree/Documentation/ABI/stable
Documentation/admin-guide/abi-testing.rst new file +19 -0
@@ -0,0 +1,19 @@ +ABI testing symbols +=================== + +Documents interfaces that are felt to be stable, +as the main development of this interface has been completed. + +The interface can be changed to add new features, but the +current interface will not break by doing this, unless grave +errors or security problems are found in them. + +Userspace programs can start to rely on these interfaces, but they must +be aware of changes that can occur before these interfaces move to +be marked stable. + +Programs that use these interfaces are strongly encouraged to add their +name to the description of these interfaces, so that the kernel +developers can easily notify them if any changes occur. + +.. kernel-abi:: $srctree/Documentation/ABI/testing
Documentation/admin-guide/abi.rst new file +11 -0
@@ -0,0 +1,11 @@ +===================== +Linux ABI description +===================== + +.. toctree:: + :maxdepth: 1 + + abi-stable + abi-testing + abi-obsolete + abi-removed
Documentation/admin-guide/index.rst +1 -0
@@ -16,6 +16,7 @@ etc. README kernel-parameters devices + abi This section describes CPU vulnerabilities and their mitigations.
Documentation/conf.py +1 -1
@@ -34,7 +34,7 @@ needs_sphinx = '1.3' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig'] +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig', 'kernel_abi'] # The name of the math extension changed on Sphinx 1.4 if (major == 1 and minor > 3) or (major > 1):
Signature · 1 line
2.21.0
PATCH 4/12 docs: kernellog.py: add support for info()
In reply to Mauro Carvalho ChehabAn extension may want to just inform about something. So, add support for it.
--- Documentation/sphinx/kernellog.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
Documentation/sphinx/kernellog.py +5 -1
@@ -25,4 +25,8 @@ def verbose(app, message): else: app.verbose(message) - +def info(app, message): + if UseLogging: + logger.info(message) + else: + app.info(message)
Signature · 1 line
2.21.0
PATCH 2/12 scripts/get_abi.pl: fix parsing on ReST mode
In reply to Mauro Carvalho ChehabWhen the source ABI file is using ReST notation, the script should handle whitespaces and lines with care, as otherwise the file won't be properly recognized.
Address the bugs that are on such part of the script.
--- scripts/get_abi.pl | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
scripts/get_abi.pl +14 -16
@@ -151,7 +151,8 @@ sub parse_abi { $content = $2; } while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} - $data{$what}->{$tag} .= $content; + + $data{$what}->{$tag} .= "$content\n" if ($content); } else { $data{$what}->{$tag} = $content; }
@@ -166,31 +167,28 @@ sub parse_abi { } if ($tag eq "description") { + my $content = $_; + while ($content =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} if (!$data{$what}->{description}) { - s/^($space)//; - if (m/^(\s*)(.*)/) { - my $sp = $1; - while ($sp =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} - my $content = "$sp$2"; - - $content =~ s/^($space)//; - - $data{$what}->{$tag} .= "$content"; + # Preserve initial spaces for the first line + if ($content =~ m/^(\s*)(.*)$/) { + $space = $1; + $content = $2; } + + $data{$what}->{$tag} .= "$content\n" if ($content); } else { - my $content = $_; if (m/^\s*\n/) { $data{$what}->{$tag} .= $content; next; } - while ($content =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} $space = "" if (!($content =~ s/^($space)//)); - # Compress spaces with tabs - $content =~ s<^ {8}> <\t>; - $content =~ s<^ {1,7}\t> <\t>; - $content =~ s< {1,7}\t> <\t>; +# # Compress spaces with tabs +# $content =~ s<^ {8}> <\t>; +# $content =~ s<^ {1,7}\t> <\t>; +# $content =~ s< {1,7}\t> <\t>; $data{$what}->{$tag} .= $content; } next;
Signature · 1 line
2.21.0
PATCH 5/12 docs: kernel_abi.py: add a script to parse ABI documentation
In reply to Mauro Carvalho ChehabThe ABI documentation is special: it is not plain text files, but, instead, files with an strict format, as specified by Documentation/ABI/README.
Add a parser for it.
--- Documentation/sphinx/kernel_abi.py | 155 +++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 Documentation/sphinx/kernel_abi.py
Documentation/sphinx/kernel_abi.py new file +155 -0
@@ -0,0 +1,155 @@ +# -*- coding: utf-8; mode: python -*- +u""" + kernel-abi + ~~~~~~~~~~ + + Implementation of the ``kernel-abi`` reST-directive. + + :copyright: Copyright (C) 2016 Markus Heiser + :copyright: Copyright (C) 2016 Mauro Carvalho Chehab + :license: GPL Version 2, June 1991 see Linux/COPYING for details. + + The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the + scripts/get_abi.pl script to parse the Kernel ABI files. + + Overview of directive's argument and options. + + .. code-block:: rst + + .. kernel-abi:: <ABI directory location> + :debug: + + The argument ``<ABI directory location>`` is required. It contains the + location of the ABI files to be parsed. + + ``debug`` + Inserts a code-block with the *raw* reST. Sometimes it is helpful to see + what reST is generated. + +""" + +import sys +import os +from os import path +import subprocess + +from sphinx.ext.autodoc import AutodocReporter + +from docutils import nodes +from docutils.parsers.rst import Directive, directives +from docutils.statemachine import ViewList +from docutils.utils.error_reporting import ErrorString + + +__version__ = '1.0' + +# We can't assume that six is installed +PY3 = sys.version_info[0] == 3 +PY2 = sys.version_info[0] == 2 +if PY3: + # pylint: disable=C0103, W0622 + unicode = str + basestring = str + +def setup(app): + + app.add_directive("kernel-abi", KernelCmd) + return dict( + version = __version__ + , parallel_read_safe = True + , parallel_write_safe = True + ) + +class KernelCmd(Directive): + + u"""KernelABI (``kernel-abi``) directive""" + + required_arguments = 1 + optional_arguments = 0 + has_content = False + final_argument_whitespace = True + + option_spec = { + "debug" : directives.flag + } + + def warn(self, message, **replace): + replace["fname"] = self.state.document.current_source + replace["line_no"] = replace.get("line_no", self.lineno) + message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace + self.state.document.settings.env.app.warn(message, prefix="") + + def run(self): + + doc = self.state.document + if not doc.settings.file_insertion_enabled: + raise self.warning("docutils: file insertion disabled") + + env = doc.settings.env + cwd = path.dirname(doc.current_source) + cmd = "get_abi.pl rest --dir " + cmd += self.arguments[0] + + srctree = path.abspath(os.environ["srctree"]) + + fname = cmd + + # extend PATH with $(srctree)/scripts + path_env = os.pathsep.join([ + srctree + os.sep + "scripts", + os.environ["PATH"] + ]) + shell_env = os.environ.copy() + shell_env["PATH"] = path_env + shell_env["srctree"] = srctree + + lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env) + nodeList = self.nestedParse(lines, fname) + return nodeList + + def runCmd(self, cmd, **kwargs): + u"""Run command ``cmd`` and return it's stdout as unicode.""" + + try: + proc = subprocess.Popen( + cmd + , stdout = subprocess.PIPE + , stderr = subprocess.PIPE + , universal_newlines = True + , **kwargs + ) + out, err = proc.communicate() + if err: + self.warn(err) + if proc.returncode != 0: + raise self.severe( + u"command '%s' failed with return code %d" + % (cmd, proc.returncode) + ) + except OSError as exc: + raise self.severe(u"problems with '%s' directive: %s." + % (self.name, ErrorString(exc))) + return unicode(out) + + def nestedParse(self, lines, fname): + content = ViewList() + node = nodes.section() + + if "debug" in self.options: + code_block = "\n\n.. code-block:: rst\n :linenos:\n" + for l in lines.split("\n"): + code_block += "\n " + l + lines = code_block + "\n\n" + + for c, l in enumerate(lines.split("\n")): + content.append(l, fname, c) + + buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter + self.state.memo.title_styles = [] + self.state.memo.section_level = 0 + self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter) + try: + self.state.nested_parse(content, 0, node, match_titles=1) + finally: + self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf + return node.children
Signature · 1 line
2.21.0
PATCH 1/12 scripts/get_abi.pl: change script to allow parsing in ReST mode
In reply to Mauro Carvalho ChehabRight now, several ABI files won't parse as ReST, as they contain severe violations to the spec, with makes the script to crash.
So, the code has a sanity logic with escapes bad code and cleans tags that can cause Sphinx to crash.
Add support for disabling this mode.
Right now, as enabling rst-mode causes crash, it is disabled by default.
--- scripts/get_abi.pl | 74 ++++++++++++++++++++++++++++++---------------- 1 file changed, 48 insertions(+), 26 deletions(-)
scripts/get_abi.pl +48 -26
@@ -12,8 +12,14 @@ my $man; my $debug; my $prefix="Documentation/ABI"; +# +# If true, assumes that the description is formatted with ReST +# +my $description_is_rst = 0; + GetOptions( "debug|d+" => \$debug, + "rst-source!" => \$description_is_rst, "dir=s" => \$prefix, 'help|?' => \$help, man => \$man
@@ -137,14 +143,15 @@ sub parse_abi { next; } if ($tag eq "description") { - next if ($content =~ m/^\s*$/); - if ($content =~ m/^(\s*)(.*)/) { - my $new_content = $2; - $space = $new_tag . $sep . $1; - while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} - $space =~ s/./ /g; - $data{$what}->{$tag} .= "$new_content\n"; + # Preserve initial spaces for the first line + $content = ' ' x length($new_tag) . $sep . $content; + $content =~ s,^(\s*):,$1 ,; + if ($content =~ m/^(\s*)(.*)$/) { + $space = $1; + $content = $2; } + while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} + $data{$what}->{$tag} .= $content; } else { $data{$what}->{$tag} = $content; }
@@ -160,11 +167,15 @@ sub parse_abi { if ($tag eq "description") { if (!$data{$what}->{description}) { - next if (m/^\s*\n/); + s/^($space)//; if (m/^(\s*)(.*)/) { - $space = $1; - while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} - $data{$what}->{$tag} .= "$2\n"; + my $sp = $1; + while ($sp =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {} + my $content = "$sp$2"; + + $content =~ s/^($space)//; + + $data{$what}->{$tag} .= "$content"; } } else { my $content = $_;
@@ -274,23 +285,27 @@ sub output_rest { print "Defined on file :ref:`$file <$fileref>`\n\n" if ($type ne "File"); my $desc = $data{$what}->{description}; - $desc =~ s/^\s+//; - - # Remove title markups from the description, as they won't work - $desc =~ s/\n[\-\*\=\^\~]+\n/\n/g; if (!($desc =~ /^\s*$/)) { - if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/ || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) { - # put everything inside a code block - $desc =~ s/\n/\n /g; - - print "::\n\n"; - print " $desc\n\n"; - } else { - # Escape any special chars from description - $desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g; - + if ($description_is_rst) { print "$desc\n\n"; + } else { + $desc =~ s/^\s+//; + + # Remove title markups from the description, as they won't work + $desc =~ s/\n[\-\*\=\^\~]+\n/\n\n/g; + + if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/ || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) { + # put everything inside a code block + $desc =~ s/\n/\n /g; + + print "::\n\n"; + print " $desc\n\n"; + } else { + # Escape any special chars from description + $desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g; + print "$desc\n\n"; + } } } else { print "DESCRIPTION MISSING for $what\n\n" if (!$data{$what}->{is_file});
@@ -382,7 +397,7 @@ abi_book.pl - parse the Linux ABI files and produce a ReST book. =head1 SYNOPSIS -B<abi_book.pl> [--debug] [--man] [--help] [--dir=<dir>] <COMAND> [<ARGUMENT>] +B<abi_book.pl> [--debug] [--man] [--help] --[(no-)rst-source] [--dir=<dir>] <COMAND> [<ARGUMENT>] Where <COMMAND> can be:
@@ -405,6 +420,13 @@ B<validate> - validate the ABI contents Changes the location of the ABI search. By default, it uses the Documentation/ABI directory. +=item B<--rst-source> and B<--no-rst-source> + +The input file may be using ReST syntax or not. Those two options allow +selecting between a rst-compliant source ABI (--rst-source), or a +plain text that may be violating ReST spec, so it requres some escaping +logic (--no-rst-source). + =item B<--debug> Put the script in verbose mode, useful for debugging. Can be called multiple
Signature · 1 line
2.21.0