Skip to contentSkip to conversation

Documentation/Sphinx

PATCH Superseded Jani Nikula 5 participants 45 messages May 20, 2016 – Jun 29, 2016

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Documentation/Sphinx

Hi Jon, all -

I've had a few moments of spare time to look into Sphinx. This is a sort of status report on the progress.

I suppose we all thought it would be easiest to use docproc for the Sphinx toolchain, for starters. I tried it, hard, and even sent a bunch of docproc prep patches. However I ran into some problems. Out-of-tree builds were a nightmare, as Sphinx has trouble to pick up some of the files from $(srctree) and some from $(objtree). Having dependencies on the source files work without rebuilding everything was getting tricky too.

I decided to look into writing a Sphinx extension for kernel-doc, and it turns out to be a really nice solution. We can keep all the .rst files in $(srctree), we don't have to explicitly specify the .rst files to process, there are no intermediate files, and Sphinx runs the kernel-doc script using the extension based on the directives in the .rst files. The extension tells Sphinx about the dependencies on the source files, and Sphinx handles rebuilding as needed. Out-of-tree builds just work. Sites like https://readthedocs.org/ can build the documentation, including kernel-doc, without extra tweaks. As a whole, the build becomes much simpler.

There are a few tradeoffs, of course. First, this requires that the EXPORT_SYMBOL markers are placed immediately after the function being exported, as kernel-doc will only look at one file at a time. This is the recommendation anyway. See the corresponding patch for further details ("kernel-doc: support printing exported and non-exported symbols"). Second, we lose support for the !C docproc directive to check that all kernel-doc comments in a file are used. This is probably something we'd like to have back in the future, but at this time I think it's an acceptable tradeoff wrt the gains.

With this, we can put any .rst files (including ones that have kernel-doc directives) anywhere under Documentation, add a link to them in Documentation/index.rst table of contents, and it will just work. It can't get much simpler than that.

At this time I've put most effort into the configuration and build side of things, solving the problems described above, and handling missing tools and packages gracefully. There are still issues to be ironed out in a) the kernel-doc script rst output and b) the xml template to rst conversion. These are somewhat orthogonal from each other and the build, and I expect some hand-editing will be required in the end.

The patches are available in the "sphinx" branch of [1], and I've set up a project at Read the Docs to build that into documentation [2] (mostly to test this approach also works and so I don't have to host this anywhere).

Any comments are welcome, but please do remember that I've focused on polishing the toolchain and build, not the output quality, with release early, release often in mind.

BR,
Jani.

[1] git://people.freedesktop.org/~jani/drm
[2] https://kernel.readthedocs.io/

Jani Nikula (9):
kernel-doc: fix use of uninitialized value
kernel-doc: support printing exported and non-exported symbols Documentation/sphinx: add basic working Sphinx configuration and build
Documentation: add .gitignore
Documentation/sphinx: add Sphinx kernel-doc directive extension Documentation/sphinx: configure the kernel-doc extension
Documentation: add kernel hacking rst
Documentation: add kernel api rst
Documentation: moar files

Jonathan Corbet (1):
sphinx: cheesy script to convert .tmpl files

 Documentation/.gitignore                  |    1 +
 Documentation/DocBook/Makefile            |    7 +-
 Documentation/Makefile.sphinx             |   63 +
 Documentation/conf.py                     |  384 +++++
 Documentation/crypto-API.rst              | 1870 +++++++++++++++++++++
 Documentation/filesystems.rst             |  314 ++++
 Documentation/gpu.rst                     | 2556 +++++++++++++++++++++++++++++
 Documentation/index.rst                   |   30 +
 Documentation/kernel-api.rst              |  419 +++++
 Documentation/kernel-hacking.rst          |  795 +++++++++
 Documentation/sphinx/convert_template.sed |   14 +
 Documentation/sphinx/kernel-doc.py        |   99 ++
 Documentation/sphinx/post_convert.sed     |   19 +
 Documentation/sphinx/tmplcvt              |   19 +
 Makefile                                  |    5 +-
 scripts/kernel-doc                        |   32 +-
 16 files changed, 6619 insertions(+), 8 deletions(-)
 create mode 100644 Documentation/.gitignore
 create mode 100644 Documentation/Makefile.sphinx
 create mode 100644 Documentation/conf.py
 create mode 100644 Documentation/crypto-API.rst
 create mode 100644 Documentation/filesystems.rst
 create mode 100644 Documentation/gpu.rst
 create mode 100644 Documentation/index.rst
 create mode 100644 Documentation/kernel-api.rst
 create mode 100644 Documentation/kernel-hacking.rst
 create mode 100644 Documentation/sphinx/convert_template.sed
 create mode 100644 Documentation/sphinx/kernel-doc.py
 create mode 100644 Documentation/sphinx/post_convert.sed
 create mode 100755 Documentation/sphinx/tmplcvt
Signature · 1 line
2.1.4
#1 lore raw

PATCH 1/10 kernel-doc: fix use of uninitialized value

I'm not quite sure why the errors below are happening, but this fixes them.

Use of uninitialized value in string ne at ./scripts/kernel-doc line 1819, <IN> line 6494. Use of uninitialized value $_[0] in join or string at ./scripts/kernel-doc line 1759, <IN> line 6494.

---
 scripts/kernel-doc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/kernel-doc +2 -1
index 2fc8fad5195e..babb374c043d 100755
@@ -1803,7 +1803,8 @@ sub output_function_rst(%) {
 	} else {
 	    print "      ``$parameter``\n";
 	}
-	if ($args{'parameterdescs'}{$parameter_name} ne $undescribed) {
+	if (defined($args{'parameterdescs'}{$parameter_name}) &&
+	    $args{'parameterdescs'}{$parameter_name} ne $undescribed) {
 	    my $oldprefix = $lineprefix;
 	    $lineprefix = "        ";
 	    output_highlight_rst($args{'parameterdescs'}{$parameter_name});
Signature · 1 line
2.1.4
#2 lore raw

PATCH 2/10 kernel-doc: support printing exported and non-exported symbols

In reply to Jani Nikula

Currently we use docproc to figure out which symbols are exported, and then docproc calls kernel-doc on specific functions, to get documentation on exported functions. According to git blame and docproc comments, this is due to historical reasons, as functions and their corresponding EXPORT_SYMBOL* may have been in different files. However for more than ten years the recommendation in CodingStyle has been to place the EXPORT_SYMBOL* immediately after the closing function brace line.

Additionally, the kernel-doc comments for functions are generally placed above the function definition in the .c files (i.e. where the EXPORT_SYMBOL* is) rather than above the declaration in the .h files. There are some exceptions to this, but AFAICT none of these are included in DocBook documentation using the "!E" docproc directive.

Therefore, assuming the EXPORT_SYMBOL* and kernel-doc are with the function definition, kernel-doc can extract the exported vs. not information by making two passes on the input file. Add support for that via the new -export and -internal parameters.

---
 scripts/kernel-doc | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)
scripts/kernel-doc +27 -2
index babb374c043d..0e4109bcaa41 100755
@@ -59,6 +59,12 @@ Output format selection (mutually exclusive):
   -text			Output plain text format.
 
 Output selection (mutually exclusive):
+  -export		Only output documentation for symbols that have been
+			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
+			in the same FILE.
+  -internal		Only output documentation for symbols that have NOT been
+			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
+			in the same FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -380,6 +386,7 @@ my $doc_block = $doc_com . 'DOC:\s*(.*)?';
 my $doc_split_start = '^\s*/\*\*\s*$';
 my $doc_split_sect = '\s*\*\s*(@[\w\s]+):(.*)';
 my $doc_split_end = '^\s*\*/\s*$';
+my $export_symbol = '^\s*EXPORT_SYMBOL(_GPL)?\s*\(\s*(\w+)\s*\)\s*;';
 
 my %constants;
 my %parameterdescs;
@@ -444,6 +451,12 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_only = 2;
 	$function = shift @ARGV;
 	$function_table{$function} = 1;
+    } elsif ($cmd eq "-export") { # only exported symbols
+	$function_only = 3;
+	%function_table = ()
+    } elsif ($cmd eq "-internal") { # only non-exported symbols
+	$function_only = 4;
+	%function_table = ()
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -1971,8 +1984,10 @@ sub output_declaration {
     my $functype = shift;
     my $func = "output_${functype}_$output_mode";
     if (($function_only==0) ||
-	( $function_only == 1 && defined($function_table{$name})) ||
-	( $function_only == 2 && !($functype eq "function" && defined($function_table{$name}))))
+	( ($function_only == 1 || $function_only == 3) &&
+	  defined($function_table{$name})) ||
+	( ($function_only == 2 || $function_only == 4) &&
+	  !($functype eq "function" && defined($function_table{$name}))))
     {
 	&$func(@_);
 	$section_counter++;
@@ -2677,6 +2692,16 @@ sub process_file($) {
 
     $. = 1;
 
+    # two passes for -export and -internal
+    if ($function_only == 3 || $function_only == 4) {
+	while (<IN>) {
+	    if (/$export_symbol/o) {
+		$function_table{$2} = 1;
+	    }
+	}
+	seek(IN, 0, 0);
+    }
+
     $section_counter = 0;
     while (<IN>) {
 	while (s/\\\s*$//) {
Signature · 1 line
2.1.4
#3 lore raw

PATCH 3/10 Documentation/sphinx: add basic working Sphinx configuration and build

In reply to Jani Nikula

Add basic configuration and makefile to build documentation from any .rst files under Documentation using Sphinx. For starters, there's just the placeholder index.rst.

At the top level Makefile, hook Sphinx documentation targets alongside (but independent of) the DocBook toolchain, having both be run on the various 'make *docs' targets.

All Sphinx processing is placed into Documentation/Makefile.sphinx. Both that and the Documentation/DocBook/Makefile are now expected to handle all the documentation targets, explicitly ignoring them if they're not relevant for that particular toolchain. The changes to the existing DocBook Makefile are kept minimal.

There is graceful handling of missing Sphinx and rst2pdf (which is needed for pdf output) by checking for the tool and python module, respectively, with informative messages to the user.

If the Read the Docs theme (sphinx_rtd_theme) is available, use it, but otherwise gracefully fall back to the Sphinx default theme, with an informative message to the user, and slightly less pretty HTML output.

Sphinx can now handle htmldocs, pdfdocs (if rst2pdf is available), epubdocs and xmldocs targets. The output documents are written into per output type subdirectories under Documentation/output.

Finally, you can pass options to sphinx-build using the SPHINXBUILD make variable. For example, 'make SPHINXOPTS=-v htmldocs' for more verbose output from Sphinx.

This is based on the original work by Jonathan Corbet, but he probably wouldn't recognize this as his own anymore.

---
 Documentation/DocBook/Makefile |   7 +-
 Documentation/Makefile.sphinx  |  61 +++++++
 Documentation/conf.py          | 378 +++++++++++++++++++++++++++++++++++++++++
 Documentation/index.rst        |  23 +++
 Makefile                       |   5 +-
 5 files changed, 469 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/Makefile.sphinx
 create mode 100644 Documentation/conf.py
 create mode 100644 Documentation/index.rst
Documentation/DocBook/Makefile +3 -4
index d70f9b68174e..e0c7e1e0590b 100644
@@ -33,10 +33,6 @@ PDF_METHOD	= $(prefer-db2x)
 PS_METHOD	= $(prefer-db2x)
 
 
-###
-# The targets that may be used.
-PHONY += xmldocs sgmldocs psdocs pdfdocs htmldocs mandocs installmandocs cleandocs
-
 targets += $(DOCBOOKS)
 BOOKS := $(addprefix $(obj)/,$(DOCBOOKS))
 xmldocs: $(BOOKS)
@@ -63,6 +59,9 @@ installmandocs: mandocs
 		sort -k 2 -k 1 | uniq -f 1 | sed -e 's: :/:' | \
 		xargs install -m 644 -t /usr/local/man/man9/
 
+# no-op for the DocBook toolchain
+epubdocs:
+
 ###
 #External programs used
 KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref
Documentation/Makefile.sphinx new file +61 -0

61 lines not on this page · raw message

Documentation/conf.py new file +378 -0

378 lines not on this page · raw message

Documentation/index.rst new file +23 -0
new file mode 100644
index 000000000000..71a276f34c7f
@@ -0,0 +1,23 @@
+.. The Linux Kernel documentation master file, created by
+   sphinx-quickstart on Fri Feb 12 13:51:46 2016.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to The Linux Kernel's documentation!
+============================================
+
+Nothing for you to see here *yet*. Please move along.
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
Makefile +4 -1
index 0f9cb36d45c2..85b3d1049054 100644
@@ -1385,8 +1385,11 @@ $(help-board-dirs): help-%:
 
 # Documentation targets
 # ---------------------------------------------------------------------------
-%docs: scripts_basic FORCE
+DOC_TARGETS := xmldocs sgmldocs psdocs pdfdocs htmldocs mandocs installmandocs epubdocs cleandocs
+PHONY += $(DOC_TARGETS)
+$(DOC_TARGETS): scripts_basic FORCE
 	$(Q)$(MAKE) $(build)=scripts build_docproc build_check-lc_ctype
+	$(Q)$(MAKE) $(build)=Documentation -f $(srctree)/Documentation/Makefile.sphinx $@
 	$(Q)$(MAKE) $(build)=Documentation/DocBook $@
 
 else # KBUILD_EXTMOD
Signature · 1 line
2.1.4
#4 lore raw

PATCH 4/10 Documentation: add .gitignore

In reply to Jani Nikula

The Sphinx output directory is generated.

---
 Documentation/.gitignore | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 Documentation/.gitignore
Documentation/.gitignore new file +1 -0
new file mode 100644
index 000000000000..53752db253e3
@@ -0,0 +1 @@
+output
Signature · 1 line
2.1.4
#5 lore raw

PATCH 5/10 Documentation/sphinx: add Sphinx kernel-doc directive extension

In reply to Jani Nikula

Add an extension to handle kernel-doc directives, to call kernel-doc according to the arguments and parameters given to the reStructuredText directive.

The syntax for the kernel-doc directive is:

.. kernel-doc:: FILENAME
:export:
:internal:
:functions: FUNCTION [FUNCTION ...]
:doc: SECTION TITLE

Of the directive options export, internal, functions, and doc, currently only one option may be given at a time.

The FILENAME is relative from the kernel source tree root.

The extension notifies Sphinx about the document dependency on FILENAME, causing the document to be rebuilt when the file has been changed.

---
 Documentation/sphinx/kernel-doc.py | 99 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 Documentation/sphinx/kernel-doc.py
Documentation/sphinx/kernel-doc.py new file +99 -0
new file mode 100644
index 000000000000..21f61373daf0
@@ -0,0 +1,99 @@
+# coding=utf-8
+#
+# Copyright © 2016 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+# Authors:
+#    Jani Nikula <[email protected]>
+
+import os
+import subprocess
+import sys
+
+from docutils import nodes, statemachine
+from docutils.parsers.rst import directives
+from sphinx.util.compat import Directive
+
+class KernelDocDirective(Directive):
+    """Extract kernel-doc comments from the specified file"""
+    required_argument = 1
+    optional_arguments = 4
+    option_spec = {
+        'doc': directives.unchanged_required,
+        'functions': directives.unchanged_required,
+        'export': directives.flag,
+        'internal': directives.flag,
+    }
+    has_content = False
+
+    def run(self):
+        env = self.state.document.settings.env
+        cmd = [env.config.kerneldoc_bin, '-rst']
+
+        filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
+
+        # Tell sphinx of the dependency
+        env.note_dependency(os.path.abspath(filename))
+
+        tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)
+        source = self.state_machine.input_lines.source(self.lineno - self.state_machine.input_offset - 1)
+
+        # FIXME: make this nicer and more robust against errors
+        if 'export' in self.options:
+            cmd += ['-export']
+        elif 'internal' in self.options:
+            cmd += ['-internal']
+        elif 'doc' in self.options:
+            cmd += ['-function', str(self.options.get('doc'))]
+        elif 'functions' in self.options:
+            for f in str(self.options.get('functions')).split(' '):
+                cmd += ['-function', f]
+
+        cmd += [filename]
+
+        try:
+            env.app.verbose('calling kernel-doc \'%s\'' % (" ".join(cmd)))
+
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            out, err = p.communicate()
+
+            if p.returncode != 0:
+                sys.stderr.write(err)
+
+                env.app.warn('kernel-doc \'%s\' failed with return code %d' % (" ".join(cmd), p.returncode))
+                return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
+            elif env.config.kerneldoc_verbosity > 0:
+                sys.stderr.write(err)
+
+            lines = statemachine.string2lines(out, tab_width, convert_whitespace=True)
+            self.state_machine.insert_input(lines, source)
+            return []
+        except Exception as e:
+            env.app.warn('kernel-doc \'%s\' processing failed with: %s' %
+                         (" ".join(cmd), str(e)))
+            return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
+
+def setup(app):
+    app.add_config_value('kerneldoc_bin', None, 'env')
+    app.add_config_value('kerneldoc_srctree', None, 'env')
+    app.add_config_value('kerneldoc_verbosity', 1, 'env')
+
+    app.add_directive('kernel-doc', KernelDocDirective)
Signature · 1 line
2.1.4
maintainer
#6 lore raw

Re: [PATCH 05/10] Documentation/sphinx: add Sphinx kernel-doc directive extension

Quoted 11 lines · On Fri, 20 May 2016 16:39:36 +0300 Jani Nikula <[email protected]> wrote:

Add an extension to handle kernel-doc directives, to call kernel-doc according to the arguments and parameters given to the reStructuredText directive.

The syntax for the kernel-doc directive is:

.. kernel-doc:: FILENAME
:export:
:internal:
:functions: FUNCTION [FUNCTION ...]
:doc: SECTION TITLE

So this seems just like what the doctor ordered - nice!

One little thought that has crossed my mind: it might be nice to be able to put a regex in :functions: so we could say something like:

:functions: atomic_.*

That requires a kernel-doc tweak too, of course. Someday.

Might it be worth running this by the Sphinx list just to see what they have to say?

jon

#7 lore raw

Re: [PATCH 05/10] Documentation/sphinx: add Sphinx kernel-doc directive extension

Am 03.06.2016 um 22:35 schrieb Jonathan Corbet <[email protected]>:

Quoted 15 lines
Quoted 11 lines · On Fri, 20 May 2016 16:39:36 +0300 Jani Nikula <[email protected]> wrote:

Add an extension to handle kernel-doc directives, to call kernel-doc according to the arguments and parameters given to the reStructuredText directive.

The syntax for the kernel-doc directive is:

.. kernel-doc:: FILENAME
:export:
:internal:
:functions: FUNCTION [FUNCTION ...]
:doc: SECTION TITLE

Quoted 8 lines

So this seems just like what the doctor ordered - nice!

One little thought that has crossed my mind: it might be nice to be able to put a regex in :functions: so we could say something like:

:functions: atomic_.*

That requires a kernel-doc tweak too, of course. Someday.

IMO it is better to be explicit ... wildcards (ranges) has the disadvantage that you can't check if any/all definition is missed ...

--Markus--

Quoted 5 lines

Might it be worth running this by the Sphinx list just to see what they have to say?

jon

#8 lore raw

PATCH 6/10 Documentation/sphinx: configure the kernel-doc extension

In reply to Jani Nikula

Tell Sphinx where to find the extension, and pass on the kernel src tree and kernel-doc paths to the extension.

With this, any .rst files under Documentation may contain the kernel-doc rst directive to include kernel-doc documentation from any source file.

While building, it may be handy to pass kernel-doc extension configuration on the command line. For example, 'make SPHINXOPTS="-D kerneldoc_verbosity=0" htmldocs' silences all stderr output from kernel-doc when the kernel-doc exit code is 0. (The stderr will be logged unconditionally when the exit code is non-zero.)

---
 Documentation/Makefile.sphinx |  4 +++-
 Documentation/conf.py         | 10 ++++++++--
 2 files changed, 11 insertions(+), 3 deletions(-)
Documentation/Makefile.sphinx +3 -1
index 25a7970bfe5c..8a662be9bd87 100644
@@ -25,7 +25,9 @@ HAVE_RST2PDF := $(shell if python -c "import rst2pdf" >/dev/null 2>&1; then echo
 # Internal variables.
 PAPEROPT_a4     = -D latex_paper_size=a4
 PAPEROPT_letter = -D latex_paper_size=letter
-ALLSPHINXOPTS   = -d $(BUILDDIR)/.doctrees $(PAPEROPT_$(PAPER)) -c $(srctree)/$(src) $(SPHINXOPTS) $(srctree)/$(src)
+KERNELDOC       = $(srctree)/scripts/kernel-doc
+KERNELDOC_CONF  = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
+ALLSPHINXOPTS   = -d $(BUILDDIR)/.doctrees $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) -c $(srctree)/$(src) $(SPHINXOPTS) $(srctree)/$(src)
 # the i18n builder cannot share the environment and doctrees with the others
 I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 
Documentation/conf.py +8 -2
index ff86f5e88049..0d33cb157716 100644
@@ -18,7 +18,7 @@ import os
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
-#sys.path.insert(0, os.path.abspath('.'))
+sys.path.insert(0, os.path.abspath('sphinx'))
 
 # -- General configuration ------------------------------------------------
 
@@ -28,7 +28,7 @@ import os
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ['kernel-doc']
 
 # Gracefully handle missing rst2pdf.
 try:
@@ -376,3 +376,9 @@ epub_exclude_files = ['search.html']
 pdf_documents = [
     ('index', u'Kernel', u'Kernel', u'J. Random Bozo'),
 ]
+
+# kernel-doc extension configuration for running Sphinx directly (e.g. by Read
+# the Docs). In a normal build, these are supplied from the Makefile via command
+# line arguments.
+kerneldoc_bin = '../scripts/kernel-doc'
+kerneldoc_srctree = '..'
Signature · 1 line
2.1.4
#9 lore raw

PATCH 7/10 sphinx: cheesy script to convert .tmpl files

In reply to Jani Nikula

From: Jonathan Corbet <[email protected]>

This script uses pandoc to convert existing DocBook template files to RST templates. A couple of sed scripts are need to massage things both before and after the conversion, but the result is then usable with no hand editing.

[Jani: Change usage to tmplcvt <in> <out>. Fix escaping for docproc directives. Add support the new kernel-doc extension.]

---
 Documentation/sphinx/convert_template.sed | 14 ++++++++++++++
 Documentation/sphinx/post_convert.sed     | 19 +++++++++++++++++++
 Documentation/sphinx/tmplcvt              | 19 +++++++++++++++++++
 3 files changed, 52 insertions(+)
 create mode 100644 Documentation/sphinx/convert_template.sed
 create mode 100644 Documentation/sphinx/post_convert.sed
 create mode 100755 Documentation/sphinx/tmplcvt
Documentation/sphinx/convert_template.sed new file +14 -0
new file mode 100644
index 000000000000..d53bb8220a26
@@ -0,0 +1,14 @@
+#
+# Pandoc doesn't grok <function> or <structname>, so convert them
+# ahead of time.
+#
+# Use "$bq" instead of "`" so that pandoc won't mess with it.
+#
+s%<function>\([^<(]\+\)()</function>%:c:func:$bq\1$bq%g
+s%<function>\([^<(]\+\)</function>%:c:func:$bq\1$bq%g
+s%<structname>struct *\([^<]\+\)</structname>%:ref:$bqstruct \1$bq%g
+s%<structname>\([^<]\+\)</structname>%:ref:$bqstruct \1$bq%g
+#
+# Wrap docproc directives in para and code blocks.
+#
+s%^\(!.*\)$%<para><code>DOCPROC: \1</code></para>%
Documentation/sphinx/post_convert.sed new file +19 -0
new file mode 100644
index 000000000000..ce17049f6a8e
@@ -0,0 +1,19 @@
+#
+# pandoc thinks that both "_" needs to be escaped.  Remove the extra
+# backslashes.  Also put in proper backquotes now that pandoc won't quote
+# them.
+#
+s/$bq/`/g
+s/\\_/_/g
+#
+# Unwrap docproc directives.
+#
+s/^``DOCPROC: !E\(.*\)``$/.. kernel-doc:: \1\n   :export:/
+s/^``DOCPROC: !I\(.*\)``$/.. kernel-doc:: \1\n   :internal:/
+s/^``DOCPROC: !F\([^ ]*\) \(.*\)``$/.. kernel-doc:: \1\n   :functions: \2/
+s/^``DOCPROC: !P\([^ ]*\) \(.*\)``$/.. kernel-doc:: \1\n   :doc: \2/
+s/^``DOCPROC: \(!.*\)``$/.. WARNING: DOCPROC directive not supported: \1/
+#
+# Trim trailing whitespace.
+#
+s/[[:space:]]*$//
Documentation/sphinx/tmplcvt new file +19 -0
new file mode 100755
index 000000000000..909a73065e0a
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# Convert a template file into something like RST
+#
+# fix <function>
+# feed to pandoc
+# fix \_
+# title line?
+#
+
+in=$1
+rst=$2
+tmp=$rst.tmp
+
+cp $in $tmp
+sed --in-place -f convert_template.sed $tmp
+pandoc -s -S -f docbook -t rst -o $rst $tmp
+sed --in-place -f post_convert.sed $rst
+rm $tmp
Signature · 1 line
2.1.4
#10 lore raw

PATCH 8/10 Documentation: add kernel hacking rst

In reply to Jani Nikula

Crude conversion. This one doesn't use kernel-doc.

---
 Documentation/index.rst          |   5 +-
 Documentation/kernel-hacking.rst | 795 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 799 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/kernel-hacking.rst
Documentation/index.rst +4 -1
index 71a276f34c7f..1ed1c4e91c27 100644
@@ -6,13 +6,16 @@
 Welcome to The Linux Kernel's documentation!
 ============================================
 
-Nothing for you to see here *yet*. Please move along.
+This is a hacked together set of pages generated from the kernel .tmpl files
+using sphinx.  There's lots of glitches and such still - lots of details to work
+out, but at least it's a start.
 
 Contents:
 
 .. toctree::
    :maxdepth: 2
 
+   kernel-hacking
 
 Indices and tables
 ==================
Documentation/kernel-hacking.rst new file +795 -0

795 lines not on this page · raw message

Signature · 1 line
2.1.4
#11 lore raw

PATCH 9/10 Documentation: add kernel api rst

In reply to Jani Nikula

Crude conversion. This one uses kernel-doc.

---
 Documentation/index.rst      |   1 +
 Documentation/kernel-api.rst | 419 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 420 insertions(+)
 create mode 100644 Documentation/kernel-api.rst
Documentation/index.rst +1 -0
index 1ed1c4e91c27..6ca68ef5c89a 100644
@@ -16,6 +16,7 @@ Contents:
    :maxdepth: 2
 
    kernel-hacking
+   kernel-api
 
 Indices and tables
 ==================
Documentation/kernel-api.rst new file +419 -0

419 lines not on this page · raw message

Signature · 1 line
2.1.4
#12 lore raw

PATCH 10/10 Documentation: moar files

In reply to Jani Nikula

Crude conversion. This has enough errors to break the pdf build.

---
 Documentation/crypto-API.rst  | 1870 ++++++++++++++++++++++++++++++
 Documentation/filesystems.rst |  314 +++++
 Documentation/gpu.rst         | 2556 +++++++++++++++++++++++++++++++++++++++++
 Documentation/index.rst       |    3 +
 4 files changed, 4743 insertions(+)
 create mode 100644 Documentation/crypto-API.rst
 create mode 100644 Documentation/filesystems.rst
 create mode 100644 Documentation/gpu.rst
Documentation/crypto-API.rst new file +1870 -0
new file mode 100644
index 000000000000..11853b0e6b41
@@ -0,0 +1,1870 @@
+=======================
+Linux Kernel Crypto API
+=======================
+
+:Author: Stephan Mueller
+:Author: Marek Vasut
+
+Kernel Crypto API Interface Specification
+=========================================
+
+Introduction
+------------
+
+The kernel crypto API offers a rich set of cryptographic ciphers as well
+as other data transformation mechanisms and methods to invoke these.
+This document contains a description of the API and provides example
+code.
+
+To understand and properly use the kernel crypto API a brief explanation
+of its structure is given. Based on the architecture, the API can be
+separated into different components. Following the architecture
+specification, hints to developers of ciphers are provided. Pointers to
+the API function call documentation are given at the end.
+
+The kernel crypto API refers to all algorithms as "transformations".
+Therefore, a cipher handle variable usually has the name "tfm". Besides
+cryptographic operations, the kernel crypto API also knows compression
+transformations and handles them the same way as ciphers.
+
+The kernel crypto API serves the following entity types:
+
+-  consumers requesting cryptographic services
+
+-  data transformation implementations (typically ciphers) that can be
+   called by consumers using the kernel crypto API
+
+This specification is intended for consumers of the kernel crypto API as
+well as for developers implementing ciphers. This API specification,
+however, does not discuss all API calls available to data transformation
+implementations (i.e. implementations of ciphers and other
+transformations (such as CRC or even compression algorithms) that can
+register with the kernel crypto API).
+
+Note: The terms "transformation" and cipher algorithm are used
+interchangeably.
+
+Terminology
+-----------
+
+The transformation implementation is an actual code or interface to
+hardware which implements a certain transformation with precisely
+defined behavior.
+
+The transformation object (TFM) is an instance of a transformation
+implementation. There can be multiple transformation objects associated
+with a single transformation implementation. Each of those
+transformation objects is held by a crypto API consumer or another
+transformation. Transformation object is allocated when a crypto API
+consumer requests a transformation implementation. The consumer is then
+provided with a structure, which contains a transformation object (TFM).
+
+The structure that contains transformation objects may also be referred
+to as a "cipher handle". Such a cipher handle is always subject to the
+following phases that are reflected in the API calls applicable to such
+a cipher handle:
+
+1. Initialization of a cipher handle.
+
+2. Execution of all intended cipher operations applicable for the handle
+   where the cipher handle must be furnished to every API call.
+
+3. Destruction of a cipher handle.
+
+When using the initialization API calls, a cipher handle is created and
+returned to the consumer. Therefore, please refer to all initialization
+API calls that refer to the data structure type a consumer is expected
+to receive and subsequently to use. The initialization API calls have
+all the same naming conventions of crypto_alloc_\*.
+
+The transformation context is private data associated with the
+transformation object.
+
+Kernel Crypto API Architecture
+==============================
+
+Cipher algorithm types
+----------------------
+
+The kernel crypto API provides different API calls for the following
+cipher types:
+
+-  Symmetric ciphers
+
+-  AEAD ciphers
+
+-  Message digest, including keyed message digest
+
+-  Random number generation
+
+-  User space interface
+
+Ciphers And Templates
+---------------------
+
+The kernel crypto API provides implementations of single block ciphers
+and message digests. In addition, the kernel crypto API provides
+numerous "templates" that can be used in conjunction with the single
+block ciphers and message digests. Templates include all types of block
+chaining mode, the HMAC mechanism, etc.
+
+Single block ciphers and message digests can either be directly used by
+a caller or invoked together with a template to form multi-block ciphers
+or keyed message digests.
+
+A single block cipher may even be called with multiple templates.
+However, templates cannot be used without a single cipher.
+
+See /proc/crypto and search for "name". For example:
+
+-  aes
+
+-  ecb(aes)
+
+-  cmac(aes)
+
+-  ccm(aes)
+
+-  rfc4106(gcm(aes))
+
+-  sha1
+
+-  hmac(sha1)
+
+-  authenc(hmac(sha1),cbc(aes))
+
+In these examples, "aes" and "sha1" are the ciphers and all others are
+the templates.
+
+Synchronous And Asynchronous Operation
+--------------------------------------
+
+The kernel crypto API provides synchronous and asynchronous API
+operations.
+
+When using the synchronous API operation, the caller invokes a cipher
+operation which is performed synchronously by the kernel crypto API.
+That means, the caller waits until the cipher operation completes.
+Therefore, the kernel crypto API calls work like regular function calls.
+For synchronous operation, the set of API calls is small and
+conceptually similar to any other crypto library.
+
+Asynchronous operation is provided by the kernel crypto API which
+implies that the invocation of a cipher operation will complete almost
+instantly. That invocation triggers the cipher operation but it does not
+signal its completion. Before invoking a cipher operation, the caller
+must provide a callback function the kernel crypto API can invoke to
+signal the completion of the cipher operation. Furthermore, the caller
+must ensure it can handle such asynchronous events by applying
+appropriate locking around its data. The kernel crypto API does not
+perform any special serialization operation to protect the caller's data
+integrity.
+
+Crypto API Cipher References And Priority
+-----------------------------------------
+
+A cipher is referenced by the caller with a string. That string has the
+following semantics:
+
+::
+
+        template(single block cipher)
+
+
+where "template" and "single block cipher" is the aforementioned
+template and single block cipher, respectively. If applicable,
+additional templates may enclose other templates, such as
+
+::
+
+        template1(template2(single block cipher)))
+
+
+The kernel crypto API may provide multiple implementations of a template
+or a single block cipher. For example, AES on newer Intel hardware has
+the following implementations: AES-NI, assembler implementation, or
+straight C. Now, when using the string "aes" with the kernel crypto API,
+which cipher implementation is used? The answer to that question is the
+priority number assigned to each cipher implementation by the kernel
+crypto API. When a caller uses the string to refer to a cipher during
+initialization of a cipher handle, the kernel crypto API looks up all
+implementations providing an implementation with that name and selects
+the implementation with the highest priority.
+
+Now, a caller may have the need to refer to a specific cipher
+implementation and thus does not want to rely on the priority-based
+selection. To accommodate this scenario, the kernel crypto API allows
+the cipher implementation to register a unique name in addition to
+common names. When using that unique name, a caller is therefore always
+sure to refer to the intended cipher implementation.
+
+The list of available ciphers is given in /proc/crypto. However, that
+list does not specify all possible permutations of templates and
+ciphers. Each block listed in /proc/crypto may contain the following
+information -- if one of the components listed as follows are not
+applicable to a cipher, it is not displayed:
+
+-  name: the generic name of the cipher that is subject to the
+   priority-based selection -- this name can be used by the cipher
+   allocation API calls (all names listed above are examples for such
+   generic names)
+
+-  driver: the unique name of the cipher -- this name can be used by the
+   cipher allocation API calls
+
+-  module: the kernel module providing the cipher implementation (or
+   "kernel" for statically linked ciphers)
+
+-  priority: the priority value of the cipher implementation
+
+-  refcnt: the reference count of the respective cipher (i.e. the number
+   of current consumers of this cipher)
+
+-  selftest: specification whether the self test for the cipher passed
+
+-  type:
+
+   -  skcipher for symmetric key ciphers
+
+   -  cipher for single block ciphers that may be used with an
+      additional template
+
+   -  shash for synchronous message digest
+
+   -  ahash for asynchronous message digest
+
+   -  aead for AEAD cipher type
+
+   -  compression for compression type transformations
+
+   -  rng for random number generator
+
+   -  givcipher for cipher with associated IV generator (see the geniv
+      entry below for the specification of the IV generator type used by
+      the cipher implementation)
+
+-  blocksize: blocksize of cipher in bytes
+
+-  keysize: key size in bytes
+
+-  ivsize: IV size in bytes
+
+-  seedsize: required size of seed data for random number generator
+
+-  digestsize: output size of the message digest
+
+-  geniv: IV generation type:
+
+   -  eseqiv for encrypted sequence number based IV generation
+
+   -  seqiv for sequence number based IV generation
+
+   -  chainiv for chain iv generation
+
+   -  <builtin> is a marker that the cipher implements IV generation and
+      handling as it is specific to the given cipher
+
+Key Sizes
+---------
+
+When allocating a cipher handle, the caller only specifies the cipher
+type. Symmetric ciphers, however, typically support multiple key sizes
+(e.g. AES-128 vs. AES-192 vs. AES-256). These key sizes are determined
+with the length of the provided key. Thus, the kernel crypto API does
+not provide a separate way to select the particular symmetric cipher key
+size.
+
+Cipher Allocation Type And Masks
+--------------------------------
+
+The different cipher handle allocation functions allow the specification
+of a type and mask flag. Both parameters have the following meaning (and
+are therefore not covered in the subsequent sections).
+
+The type flag specifies the type of the cipher algorithm. The caller
+usually provides a 0 when the caller wants the default handling.
+Otherwise, the caller may provide the following selections which match
+the the aforementioned cipher types:
+
+-  CRYPTO_ALG_TYPE_CIPHER Single block cipher
+
+-  CRYPTO_ALG_TYPE_COMPRESS Compression
+
+-  CRYPTO_ALG_TYPE_AEAD Authenticated Encryption with Associated Data
+   (MAC)
+
+-  CRYPTO_ALG_TYPE_BLKCIPHER Synchronous multi-block cipher
+
+-  CRYPTO_ALG_TYPE_ABLKCIPHER Asynchronous multi-block cipher
+
+-  CRYPTO_ALG_TYPE_GIVCIPHER Asynchronous multi-block cipher packed
+   together with an IV generator (see geniv field in the /proc/crypto
+   listing for the known IV generators)
+
+-  CRYPTO_ALG_TYPE_DIGEST Raw message digest
+
+-  CRYPTO_ALG_TYPE_HASH Alias for CRYPTO_ALG_TYPE_DIGEST
+
+-  CRYPTO_ALG_TYPE_SHASH Synchronous multi-block hash
+
+-  CRYPTO_ALG_TYPE_AHASH Asynchronous multi-block hash
+
+-  CRYPTO_ALG_TYPE_RNG Random Number Generation
+
+-  CRYPTO_ALG_TYPE_AKCIPHER Asymmetric cipher
+
+-  CRYPTO_ALG_TYPE_PCOMPRESS Enhanced version of
+   CRYPTO_ALG_TYPE_COMPRESS allowing for segmented compression /
+   decompression instead of performing the operation on one segment
+   only. CRYPTO_ALG_TYPE_PCOMPRESS is intended to replace
+   CRYPTO_ALG_TYPE_COMPRESS once existing consumers are converted.
+
+The mask flag restricts the type of cipher. The only allowed flag is
+CRYPTO_ALG_ASYNC to restrict the cipher lookup function to
+asynchronous ciphers. Usually, a caller provides a 0 for the mask flag.
+
+When the caller provides a mask and type specification, the caller
+limits the search the kernel crypto API can perform for a suitable
+cipher implementation for the given cipher name. That means, even when a
+caller uses a cipher name that exists during its initialization call,
+the kernel crypto API may not select it due to the used type and mask
+field.
+
+Internal Structure of Kernel Crypto API
+---------------------------------------
+
+The kernel crypto API has an internal structure where a cipher
+implementation may use many layers and indirections. This section shall
+help to clarify how the kernel crypto API uses various components to
+implement the complete cipher.
+
+The following subsections explain the internal structure based on
+existing cipher implementations. The first section addresses the most
+complex scenario where all other scenarios form a logical subset.
+
+Generic AEAD Cipher Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following ASCII art decomposes the kernel crypto API layers when
+using the AEAD cipher with the automated IV generation. The shown
+example is used by the IPSEC layer.
+
+For other use cases of AEAD ciphers, the ASCII art applies as well, but
+the caller may not use the AEAD cipher with a separate IV generator. In
+this case, the caller must generate the IV.
+
+The depicted example decomposes the AEAD cipher of GCM(AES) based on the
+generic C implementations (gcm.c, aes-generic.c, ctr.c, ghash-generic.c,
+seqiv.c). The generic implementation serves as an example showing the
+complete logic of the kernel crypto API.
+
+It is possible that some streamlined cipher implementations (like
+AES-NI) provide implementations merging aspects which in the view of the
+kernel crypto API cannot be decomposed into layers any more. In case of
+the AES-NI implementation, the CTR mode, the GHASH implementation and
+the AES cipher are all merged into one cipher implementation registered
+with the kernel crypto API. In this case, the concept described by the
+following ASCII art applies too. However, the decomposition of GCM into
+the individual sub-components by the kernel crypto API is not done any
+more.
+
+Each block in the following ASCII art is an independent cipher instance
+obtained from the kernel crypto API. Each block is accessed by the
+caller or by other blocks using the API functions defined by the kernel
+crypto API for the cipher implementation type.
+
+The blocks below indicate the cipher type as well as the specific logic
+implemented in the cipher.
+
+The ASCII art picture also indicates the call structure, i.e. who calls
+which component. The arrows point to the invoked block where the caller
+uses the API applicable to the cipher type specified for the block.
+
+::
+
+
+    kernel crypto API                                |   IPSEC Layer
+                                                     |
+    +-----------+                                    |
+    |           |            (1)
+    |   aead    | <-----------------------------------  esp_output
+    |  (seqiv)  | ---+
+    +-----------+    |
+                     | (2)
+    +-----------+    |
+    |           | <--+                (2)
+    |   aead    | <-----------------------------------  esp_input
+    |   (gcm)   | ------------+
+    +-----------+             |
+          | (3)               | (5)
+          v                   v
+    +-----------+       +-----------+
+    |           |       |           |
+    |  skcipher |       |   ahash   |
+    |   (ctr)   | ---+  |  (ghash)  |
+    +-----------+    |  +-----------+
+                     |
+    +-----------+    | (4)
+    |           | <--+
+    |   cipher  |
+    |   (aes)   |
+    +-----------+
+
+
+
+The following call sequence is applicable when the IPSEC layer triggers
+an encryption operation with the esp_output function. During
+configuration, the administrator set up the use of rfc4106(gcm(aes)) as
+the cipher for ESP. The following call sequence is now depicted in the
+ASCII art above:
+
+1. esp_output() invokes crypto_aead_encrypt() to trigger an
+   encryption operation of the AEAD cipher with IV generator.
+
+   In case of GCM, the SEQIV implementation is registered as GIVCIPHER
+   in crypto_rfc4106_alloc().
+
+   The SEQIV performs its operation to generate an IV where the core
+   function is seqiv_geniv().
+
+2. Now, SEQIV uses the AEAD API function calls to invoke the associated
+   AEAD cipher. In our case, during the instantiation of SEQIV, the
+   cipher handle for GCM is provided to SEQIV. This means that SEQIV
+   invokes AEAD cipher operations with the GCM cipher handle.
+
+   During instantiation of the GCM handle, the CTR(AES) and GHASH
+   ciphers are instantiated. The cipher handles for CTR(AES) and GHASH
+   are retained for later use.
+
+   The GCM implementation is responsible to invoke the CTR mode AES and
+   the GHASH cipher in the right manner to implement the GCM
+   specification.
+
+3. The GCM AEAD cipher type implementation now invokes the SKCIPHER API
+   with the instantiated CTR(AES) cipher handle.
+
+   During instantiation of the CTR(AES) cipher, the CIPHER type
+   implementation of AES is instantiated. The cipher handle for AES is
+   retained.
+
+   That means that the SKCIPHER implementation of CTR(AES) only
+   implements the CTR block chaining mode. After performing the block
+   chaining operation, the CIPHER implementation of AES is invoked.
+
+4. The SKCIPHER of CTR(AES) now invokes the CIPHER API with the AES
+   cipher handle to encrypt one block.
+
+5. The GCM AEAD implementation also invokes the GHASH cipher
+   implementation via the AHASH API.
+
+When the IPSEC layer triggers the esp_input() function, the same call
+sequence is followed with the only difference that the operation starts
+with step (2).
+
+Generic Block Cipher Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Generic block ciphers follow the same concept as depicted with the ASCII
+art picture above.
+
+For example, CBC(AES) is implemented with cbc.c, and aes-generic.c. The
+ASCII art picture above applies as well with the difference that only
+step (4) is used and the SKCIPHER block chaining mode is CBC.
+
+Generic Keyed Message Digest Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Keyed message digest implementations again follow the same concept as
+depicted in the ASCII art picture above.
+
+For example, HMAC(SHA256) is implemented with hmac.c and
+sha256_generic.c. The following ASCII art illustrates the
+implementation:
+
+::
+
+
+    kernel crypto API            |       Caller
+                                 |
+    +-----------+         (1)    |
+    |           | <------------------  some_function
+    |   ahash   |
+    |   (hmac)  | ---+
+    +-----------+    |
+                     | (2)
+    +-----------+    |
+    |           | <--+
+    |   shash   |
+    |  (sha256) |
+    +-----------+
+
+
+
+The following call sequence is applicable when a caller triggers an HMAC
+operation:
+
+1. The AHASH API functions are invoked by the caller. The HMAC
+   implementation performs its operation as needed.
+
+   During initialization of the HMAC cipher, the SHASH cipher type of
+   SHA256 is instantiated. The cipher handle for the SHA256 instance is
+   retained.
+
+   At one time, the HMAC implementation requires a SHA256 operation
+   where the SHA256 cipher handle is used.
+
+2. The HMAC instance now invokes the SHASH API with the SHA256 cipher
+   handle to calculate the message digest.
+
+Developing Cipher Algorithms
+============================
+
+Registering And Unregistering Transformation
+--------------------------------------------
+
+There are three distinct types of registration functions in the Crypto
+API. One is used to register a generic cryptographic transformation,
+while the other two are specific to HASH transformations and
+COMPRESSion. We will discuss the latter two in a separate chapter, here
+we will only look at the generic ones.
+
+Before discussing the register functions, the data structure to be
+filled with each, struct crypto_alg, must be considered -- see below
+for a description of this data structure.
+
+The generic registration functions can be found in
+include/linux/crypto.h and their definition can be seen below. The
+former function registers a single transformation, while the latter
+works on an array of transformation descriptions. The latter is useful
+when registering transformations in bulk.
+
+::
+
+       int crypto_register_alg(struct crypto_alg *alg);
+       int crypto_register_algs(struct crypto_alg *algs, int count);
+
+
+The counterparts to those functions are listed below.
+
+::
+
+       int crypto_unregister_alg(struct crypto_alg *alg);
+       int crypto_unregister_algs(struct crypto_alg *algs, int count);
+
+
+Notice that both registration and unregistration functions do return a
+value, so make sure to handle errors. A return code of zero implies
+success. Any return code < 0 implies an error.
+
+The bulk registration / unregistration functions require that struct
+crypto_alg is an array of count size. These functions simply loop over
+that array and register / unregister each individual algorithm. If an
+error occurs, the loop is terminated at the offending algorithm
+definition. That means, the algorithms prior to the offending algorithm
+are successfully registered. Note, the caller has no way of knowing
+which cipher implementations have successfully registered. If this is
+important to know, the caller should loop through the different
+implementations using the single instance \*_alg functions for each
+individual implementation.
+
+Single-Block Symmetric Ciphers [CIPHER]
+---------------------------------------
+
+Example of transformations: aes, arc4, ...
+
+This section describes the simplest of all transformation
+implementations, that being the CIPHER type used for symmetric ciphers.
+The CIPHER type is used for transformations which operate on exactly one
+block at a time and there are no dependencies between blocks at all.
+
+Registration specifics
+~~~~~~~~~~~~~~~~~~~~~~
+
+The registration of [CIPHER] algorithm is specific in that struct
+crypto_alg field .cra_type is empty. The .cra_u.cipher has to be
+filled in with proper callbacks to implement this transformation.
+
+See struct cipher_alg below.
+
+Cipher Definition With struct cipher_alg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Struct cipher_alg defines a single block cipher.
+
+Here are schematics of how these functions are called when operated from
+other part of the kernel. Note that the .cia_setkey() call might happen
+before or after any of these schematics happen, but must not happen
+during any of these are in-flight.
+
+::
+
+             KEY ---.    PLAINTEXT ---.
+                    v                 v
+              .cia_setkey() -> .cia_encrypt()
+                                      |
+                                      '-----> CIPHERTEXT
+
+
+Please note that a pattern where .cia_setkey() is called multiple times
+is also valid:
+
+::
+
+
+      KEY1 --.    PLAINTEXT1 --.         KEY2 --.    PLAINTEXT2 --.
+             v                 v                v                 v
+       .cia_setkey() -> .cia_encrypt() -> .cia_setkey() -> .cia_encrypt()
+                               |                                  |
+                               '---> CIPHERTEXT1                  '---> CIPHERTEXT2
+
+
+Multi-Block Ciphers
+-------------------
+
+Example of transformations: cbc(aes), ecb(arc4), ...
+
+This section describes the multi-block cipher transformation
+implementations. The multi-block ciphers are used for transformations
+which operate on scatterlists of data supplied to the transformation
+functions. They output the result into a scatterlist of data as well.
+
+Registration Specifics
+~~~~~~~~~~~~~~~~~~~~~~
+
+The registration of multi-block cipher algorithms is one of the most
+standard procedures throughout the crypto API.
+
+Note, if a cipher implementation requires a proper alignment of data,
+the caller should use the functions of crypto_skcipher_alignmask() to
+identify a memory alignment mask. The kernel crypto API is able to
+process requests that are unaligned. This implies, however, additional
+overhead as the kernel crypto API needs to perform the realignment of
+the data which may imply moving of data.
+
+Cipher Definition With struct blkcipher_alg and ablkcipher_alg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Struct blkcipher_alg defines a synchronous block cipher whereas struct
+ablkcipher_alg defines an asynchronous block cipher.
+
+Please refer to the single block cipher description for schematics of
+the block cipher usage.
+
+Specifics Of Asynchronous Multi-Block Cipher
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are a couple of specifics to the asynchronous interface.
+
+First of all, some of the drivers will want to use the Generic
+ScatterWalk in case the hardware needs to be fed separate chunks of the
+scatterlist which contains the plaintext and will contain the
+ciphertext. Please refer to the ScatterWalk interface offered by the
+Linux kernel scatter / gather list implementation.
+
+Hashing [HASH]
+--------------
+
+Example of transformations: crc32, md5, sha1, sha256,...
+
+Registering And Unregistering The Transformation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are multiple ways to register a HASH transformation, depending on
+whether the transformation is synchronous [SHASH] or asynchronous
+[AHASH] and the amount of HASH transformations we are registering. You
+can find the prototypes defined in include/crypto/internal/hash.h:
+
+::
+
+       int crypto_register_ahash(struct ahash_alg *alg);
+
+       int crypto_register_shash(struct shash_alg *alg);
+       int crypto_register_shashes(struct shash_alg *algs, int count);
+
+
+The respective counterparts for unregistering the HASH transformation
+are as follows:
+
+::
+
+       int crypto_unregister_ahash(struct ahash_alg *alg);
+
+       int crypto_unregister_shash(struct shash_alg *alg);
+       int crypto_unregister_shashes(struct shash_alg *algs, int count);
+
+
+Cipher Definition With struct shash_alg and ahash_alg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here are schematics of how these functions are called when operated from
+other part of the kernel. Note that the .setkey() call might happen
+before or after any of these schematics happen, but must not happen
+during any of these are in-flight. Please note that calling .init()
+followed immediately by .finish() is also a perfectly valid
+transformation.
+
+::
+
+       I)   DATA -----------.
+                            v
+             .init() -> .update() -> .final()      ! .update() might not be called
+                         ^    |         |            at all in this scenario.
+                         '----'         '---> HASH
+
+       II)  DATA -----------.-----------.
+                            v           v
+             .init() -> .update() -> .finup()      ! .update() may not be called
+                         ^    |         |            at all in this scenario.
+                         '----'         '---> HASH
+
+       III) DATA -----------.
+                            v
+                        .digest()                  ! The entire process is handled
+                            |                        by the .digest() call.
+                            '---------------> HASH
+
+
+Here is a schematic of how the .export()/.import() functions are called
+when used from another part of the kernel.
+
+::
+
+       KEY--.                 DATA--.
+            v                       v                  ! .update() may not be called
+        .setkey() -> .init() -> .update() -> .export()   at all in this scenario.
+                                 ^     |         |
+                                 '-----'         '--> PARTIAL_HASH
+
+       ----------- other transformations happen here -----------
+
+       PARTIAL_HASH--.   DATA1--.
+                     v          v
+                 .import -> .update() -> .final()     ! .update() may not be called
+                             ^    |         |           at all in this scenario.
+                             '----'         '--> HASH1
+
+       PARTIAL_HASH--.   DATA2-.
+                     v         v
+                 .import -> .finup()
+                               |
+                               '---------------> HASH2
+
+
+Specifics Of Asynchronous HASH Transformation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some of the drivers will want to use the Generic ScatterWalk in case the
+implementation needs to be fed separate chunks of the scatterlist which
+contains the input data. The buffer containing the resulting hash will
+always be properly aligned to .cra_alignmask so there is no need to
+worry about this.
+
+User Space Interface
+====================
+
+Introduction
+------------
+
+The concepts of the kernel crypto API visible to kernel space is fully
+applicable to the user space interface as well. Therefore, the kernel
+crypto API high level discussion for the in-kernel use cases applies
+here as well.
+
+The major difference, however, is that user space can only act as a
+consumer and never as a provider of a transformation or cipher
+algorithm.
+
+The following covers the user space interface exported by the kernel
+crypto API. A working example of this description is libkcapi that can
+be obtained from [1]. That library can be used by user space
+applications that require cryptographic services from the kernel.
+
+Some details of the in-kernel kernel crypto API aspects do not apply to
+user space, however. This includes the difference between synchronous
+and asynchronous invocations. The user space API call is fully
+synchronous.
+
+[1] http://www.chronox.de/libkcapi.html
+
+User Space API General Remarks
+------------------------------
+
+The kernel crypto API is accessible from user space. Currently, the
+following ciphers are accessible:
+
+-  Message digest including keyed message digest (HMAC, CMAC)
+
+-  Symmetric ciphers
+
+-  AEAD ciphers
+
+-  Random Number Generators
+
+The interface is provided via socket type using the type AF_ALG. In
+addition, the setsockopt option type is SOL_ALG. In case the user space
+header files do not export these flags yet, use the following macros:
+
+::
+
+    #ifndef AF_ALG
+    #define AF_ALG 38
+    #endif
+    #ifndef SOL_ALG
+    #define SOL_ALG 279
+    #endif
+
+
+A cipher is accessed with the same name as done for the in-kernel API
+calls. This includes the generic vs. unique naming schema for ciphers as
+well as the enforcement of priorities for generic names.
+
+To interact with the kernel crypto API, a socket must be created by the
+user space application. User space invokes the cipher operation with the
+send()/write() system call family. The result of the cipher operation is
+obtained with the read()/recv() system call family.
+
+The following API calls assume that the socket descriptor is already
+opened by the user space application and discusses only the kernel
+crypto API specific invocations.
+
+To initialize the socket interface, the following sequence has to be
+performed by the consumer:
+
+1. Create a socket of type AF_ALG with the struct sockaddr_alg
+   parameter specified below for the different cipher types.
+
+2. Invoke bind with the socket descriptor
+
+3. Invoke accept with the socket descriptor. The accept system call
+   returns a new file descriptor that is to be used to interact with the
+   particular cipher instance. When invoking send/write or recv/read
+   system calls to send data to the kernel or obtain data from the
+   kernel, the file descriptor returned by accept must be used.
+
+In-place Cipher operation
+-------------------------
+
+Just like the in-kernel operation of the kernel crypto API, the user
+space interface allows the cipher operation in-place. That means that
+the input buffer used for the send/write system call and the output
+buffer used by the read/recv system call may be one and the same. This
+is of particular interest for symmetric cipher operations where a
+copying of the output data to its final destination can be avoided.
+
+If a consumer on the other hand wants to maintain the plaintext and the
+ciphertext in different memory locations, all a consumer needs to do is
+to provide different memory pointers for the encryption and decryption
+operation.
+
+Message Digest API
+------------------
+
+The message digest type to be used for the cipher operation is selected
+when invoking the bind syscall. bind requires the caller to provide a
+filled struct sockaddr data structure. This data structure must be
+filled as follows:
+
+::
+
+    struct sockaddr_alg sa = {
+        .salg_family = AF_ALG,
+        .salg_type = "hash", /* this selects the hash logic in the kernel */
+        .salg_name = "sha1" /* this is the cipher name */
+    };
+
+
+The salg_type value "hash" applies to message digests and keyed message
+digests. Though, a keyed message digest is referenced by the appropriate
+salg_name. Please see below for the setsockopt interface that explains
+how the key can be set for a keyed message digest.
+
+Using the send() system call, the application provides the data that
+should be processed with the message digest. The send system call allows
+the following flags to be specified:
+
+-  MSG_MORE: If this flag is set, the send system call acts like a
+   message digest update function where the final hash is not yet
+   calculated. If the flag is not set, the send system call calculates
+   the final message digest immediately.
+
+With the recv() system call, the application can read the message digest
+from the kernel crypto API. If the buffer is too small for the message
+digest, the flag MSG_TRUNC is set by the kernel.
+
+In order to set a message digest key, the calling application must use
+the setsockopt() option of ALG_SET_KEY. If the key is not set the HMAC
+operation is performed without the initial HMAC state change caused by
+the key.
+
+Symmetric Cipher API
+--------------------
+
+The operation is very similar to the message digest discussion. During
+initialization, the struct sockaddr data structure must be filled as
+follows:
+
+::
+
+    struct sockaddr_alg sa = {
+        .salg_family = AF_ALG,
+        .salg_type = "skcipher", /* this selects the symmetric cipher */
+        .salg_name = "cbc(aes)" /* this is the cipher name */
+    };
+
+
+Before data can be sent to the kernel using the write/send system call
+family, the consumer must set the key. The key setting is described with
+the setsockopt invocation below.
+
+Using the sendmsg() system call, the application provides the data that
+should be processed for encryption or decryption. In addition, the IV is
+specified with the data structure provided by the sendmsg() system call.
+
+The sendmsg system call parameter of struct msghdr is embedded into the
+struct cmsghdr data structure. See recv(2) and cmsg(3) for more
+information on how the cmsghdr data structure is used together with the
+send/recv system call family. That cmsghdr data structure holds the
+following information specified with a separate header instances:
+
+-  specification of the cipher operation type with one of these flags:
+
+   -  ALG_OP_ENCRYPT - encryption of data
+
+   -  ALG_OP_DECRYPT - decryption of data
+
+-  specification of the IV information marked with the flag ALG_SET_IV
+
+The send system call family allows the following flag to be specified:
+
+-  MSG_MORE: If this flag is set, the send system call acts like a
+   cipher update function where more input data is expected with a
+   subsequent invocation of the send system call.
+
+Note: The kernel reports -EINVAL for any unexpected data. The caller
+must make sure that all data matches the constraints given in
+/proc/crypto for the selected cipher.
+
+With the recv() system call, the application can read the result of the
+cipher operation from the kernel crypto API. The output buffer must be
+at least as large as to hold all blocks of the encrypted or decrypted
+data. If the output data size is smaller, only as many blocks are
+returned that fit into that output buffer size.
+
+AEAD Cipher API
+---------------
+
+The operation is very similar to the symmetric cipher discussion. During
+initialization, the struct sockaddr data structure must be filled as
+follows:
+
+::
+
+    struct sockaddr_alg sa = {
+        .salg_family = AF_ALG,
+        .salg_type = "aead", /* this selects the symmetric cipher */
+        .salg_name = "gcm(aes)" /* this is the cipher name */
+    };
+
+
+Before data can be sent to the kernel using the write/send system call
+family, the consumer must set the key. The key setting is described with
+the setsockopt invocation below.
+
+In addition, before data can be sent to the kernel using the write/send
+system call family, the consumer must set the authentication tag size.
+To set the authentication tag size, the caller must use the setsockopt
+invocation described below.
+
+Using the sendmsg() system call, the application provides the data that
+should be processed for encryption or decryption. In addition, the IV is
+specified with the data structure provided by the sendmsg() system call.
+
+The sendmsg system call parameter of struct msghdr is embedded into the
+struct cmsghdr data structure. See recv(2) and cmsg(3) for more
+information on how the cmsghdr data structure is used together with the
+send/recv system call family. That cmsghdr data structure holds the
+following information specified with a separate header instances:
+
+-  specification of the cipher operation type with one of these flags:
+
+   -  ALG_OP_ENCRYPT - encryption of data
+
+   -  ALG_OP_DECRYPT - decryption of data
+
+-  specification of the IV information marked with the flag ALG_SET_IV
+
+-  specification of the associated authentication data (AAD) with the
+   flag ALG_SET_AEAD_ASSOCLEN. The AAD is sent to the kernel together
+   with the plaintext / ciphertext. See below for the memory structure.
+
+The send system call family allows the following flag to be specified:
+
+-  MSG_MORE: If this flag is set, the send system call acts like a
+   cipher update function where more input data is expected with a
+   subsequent invocation of the send system call.
+
+Note: The kernel reports -EINVAL for any unexpected data. The caller
+must make sure that all data matches the constraints given in
+/proc/crypto for the selected cipher.
+
+With the recv() system call, the application can read the result of the
+cipher operation from the kernel crypto API. The output buffer must be
+at least as large as defined with the memory structure below. If the
+output data size is smaller, the cipher operation is not performed.
+
+The authenticated decryption operation may indicate an integrity error.
+Such breach in integrity is marked with the -EBADMSG error code.
+
+AEAD Memory Structure
+~~~~~~~~~~~~~~~~~~~~~
+
+The AEAD cipher operates with the following information that is
+communicated between user and kernel space as one data stream:
+
+-  plaintext or ciphertext
+
+-  associated authentication data (AAD)
+
+-  authentication tag
+
+The sizes of the AAD and the authentication tag are provided with the
+sendmsg and setsockopt calls (see there). As the kernel knows the size
+of the entire data stream, the kernel is now able to calculate the right
+offsets of the data components in the data stream.
+
+The user space caller must arrange the aforementioned information in the
+following order:
+
+-  AEAD encryption input: AAD \|\| plaintext
+
+-  AEAD decryption input: AAD \|\| ciphertext \|\| authentication tag
+
+The output buffer the user space caller provides must be at least as
+large to hold the following data:
+
+-  AEAD encryption output: ciphertext \|\| authentication tag
+
+-  AEAD decryption output: plaintext
+
+Random Number Generator API
+---------------------------
+
+Again, the operation is very similar to the other APIs. During
+initialization, the struct sockaddr data structure must be filled as
+follows:
+
+::
+
+    struct sockaddr_alg sa = {
+        .salg_family = AF_ALG,
+        .salg_type = "rng", /* this selects the symmetric cipher */
+        .salg_name = "drbg_nopr_sha256" /* this is the cipher name */
+    };
+
+
+Depending on the RNG type, the RNG must be seeded. The seed is provided
+using the setsockopt interface to set the key. For example, the
+ansi_cprng requires a seed. The DRBGs do not require a seed, but may be
+seeded.
+
+Using the read()/recvmsg() system calls, random numbers can be obtained.
+The kernel generates at most 128 bytes in one call. If user space
+requires more data, multiple calls to read()/recvmsg() must be made.
+
+WARNING: The user space caller may invoke the initially mentioned accept
+system call multiple times. In this case, the returned file descriptors
+have the same state.
+
+Zero-Copy Interface
+-------------------
+
+In addition to the send/write/read/recv system call family, the AF_ALG
+interface can be accessed with the zero-copy interface of
+splice/vmsplice. As the name indicates, the kernel tries to avoid a copy
+operation into kernel space.
+
+The zero-copy operation requires data to be aligned at the page
+boundary. Non-aligned data can be used as well, but may require more
+operations of the kernel which would defeat the speed gains obtained
+from the zero-copy interface.
+
+The system-interent limit for the size of one zero-copy operation is 16
+pages. If more data is to be sent to AF_ALG, user space must slice the
+input into segments with a maximum size of 16 pages.
+
+Zero-copy can be used with the following code example (a complete
+working example is provided with libkcapi):
+
+::
+
+    int pipes[2];
+
+    pipe(pipes);
+    /* input data in iov */
+    vmsplice(pipes[1], iov, iovlen, SPLICE_F_GIFT);
+    /* opfd is the file descriptor returned from accept() system call */
+    splice(pipes[0], NULL, opfd, NULL, ret, 0);
+    read(opfd, out, outlen);
+
+
+Setsockopt Interface
+--------------------
+
+In addition to the read/recv and send/write system call handling to send
+and retrieve data subject to the cipher operation, a consumer also needs
+to set the additional information for the cipher operation. This
+additional information is set using the setsockopt system call that must
+be invoked with the file descriptor of the open cipher (i.e. the file
+descriptor returned by the accept system call).
+
+Each setsockopt invocation must use the level SOL_ALG.
+
+The setsockopt interface allows setting the following data using the
+mentioned optname:
+
+-  ALG_SET_KEY -- Setting the key. Key setting is applicable to:
+
+   -  the skcipher cipher type (symmetric ciphers)
+
+   -  the hash cipher type (keyed message digests)
+
+   -  the AEAD cipher type
+
+   -  the RNG cipher type to provide the seed
+
+-  ALG_SET_AEAD_AUTHSIZE -- Setting the authentication tag size for
+   AEAD ciphers. For a encryption operation, the authentication tag of
+   the given size will be generated. For a decryption operation, the
+   provided ciphertext is assumed to contain an authentication tag of
+   the given size (see section about AEAD memory layout below).
+
+User space API example
+----------------------
+
+Please see [1] for libkcapi which provides an easy-to-use wrapper around
+the aforementioned Netlink kernel interface. [1] also contains a test
+application that invokes all libkcapi API calls.
+
+[1] http://www.chronox.de/libkcapi.html
+
+Programming Interface
+=====================
+
+Please note that the kernel crypto API contains the AEAD givcrypt API
+(crypto_aead_giv\* and aead_givcrypt_\* function calls in
+include/crypto/aead.h). This API is obsolete and will be removed in the
+future. To obtain the functionality of an AEAD cipher with internal IV
+generation, use the IV generator as a regular cipher. For example,
+rfc4106(gcm(aes)) is the AEAD cipher with external IV generation and
+seqniv(rfc4106(gcm(aes))) implies that the kernel crypto API generates
+the IV. Different IV generators are available.
+
+Block Cipher Context Data Structures
+------------------------------------
+
+.. kernel-doc:: include/linux/crypto.h
+   :doc: Block Cipher Context Data Structures
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request
+
+Block Cipher Algorithm Definitions
+----------------------------------
+
+.. kernel-doc:: include/linux/crypto.h
+   :doc: Block Cipher Algorithm Definitions
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_alg
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: ablkcipher_alg
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_alg
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: blkcipher_alg
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: cipher_alg
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: rng_alg
+
+Symmetric Key Cipher API
+------------------------
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :doc: Symmetric Key Cipher API
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_alloc_skcipher
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_free_skcipher
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_has_skcipher
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_ivsize
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_blocksize
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_setkey
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_reqtfm
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_encrypt
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_decrypt
+
+Symmetric Key Cipher Request Handle
+-----------------------------------
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :doc: Symmetric Key Cipher Request Handle
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: crypto_skcipher_reqsize
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: skcipher_request_set_tfm
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: skcipher_request_alloc
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: skcipher_request_free
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: skcipher_request_set_callback
+
+.. kernel-doc:: include/crypto/skcipher.h
+   :functions: skcipher_request_set_crypt
+
+Asynchronous Block Cipher API - Deprecated
+------------------------------------------
+
+.. kernel-doc:: include/linux/crypto.h
+   :doc: Asynchronous Block Cipher API
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_alloc_ablkcipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_free_ablkcipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_has_ablkcipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_ivsize
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_blocksize
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_setkey
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_reqtfm
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_encrypt
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_decrypt
+
+Asynchronous Cipher Request Handle - Deprecated
+-----------------------------------------------
+
+.. kernel-doc:: include/linux/crypto.h
+   :doc: Asynchronous Cipher Request Handle
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_ablkcipher_reqsize
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: ablkcipher_request_set_tfm
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: ablkcipher_request_alloc
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: ablkcipher_request_free
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: ablkcipher_request_set_callback
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: ablkcipher_request_set_crypt
+
+Authenticated Encryption With Associated Data (AEAD) Cipher API
+---------------------------------------------------------------
+
+.. kernel-doc:: include/crypto/aead.h
+   :doc: Authenticated Encryption With Associated Data (AEAD) Cipher API
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_alloc_aead
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_free_aead
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_ivsize
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_authsize
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_blocksize
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_setkey
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_setauthsize
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_encrypt
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_decrypt
+
+Asynchronous AEAD Request Handle
+--------------------------------
+
+.. kernel-doc:: include/crypto/aead.h
+   :doc: Asynchronous AEAD Request Handle
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: crypto_aead_reqsize
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request_set_tfm
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request_alloc
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request_free
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request_set_callback
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request_set_crypt
+
+.. kernel-doc:: include/crypto/aead.h
+   :functions: aead_request_set_ad
+
+Synchronous Block Cipher API - Deprecated
+-----------------------------------------
+
+.. kernel-doc:: include/linux/crypto.h
+   :doc: Synchronous Block Cipher API
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_alloc_blkcipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_free_blkcipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_has_blkcipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_name
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_ivsize
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_blocksize
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_setkey
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_encrypt
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_encrypt_iv
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_decrypt
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_decrypt_iv
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_set_iv
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_blkcipher_get_iv
+
+Single Block Cipher API
+-----------------------
+
+.. kernel-doc:: include/linux/crypto.h
+   :doc: Single Block Cipher API
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_alloc_cipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_free_cipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_has_cipher
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_cipher_blocksize
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_cipher_setkey
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_cipher_encrypt_one
+
+.. kernel-doc:: include/linux/crypto.h
+   :functions: crypto_cipher_decrypt_one
+
+Message Digest Algorithm Definitions
+------------------------------------
+
+.. kernel-doc:: include/crypto/hash.h
+   :doc: Message Digest Algorithm Definitions
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: hash_alg_common
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: ahash_alg
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: shash_alg
+
+Asynchronous Message Digest API
+-------------------------------
+
+.. kernel-doc:: include/crypto/hash.h
+   :doc: Asynchronous Message Digest API
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_alloc_ahash
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_free_ahash
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_init
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_digestsize
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_reqtfm
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_reqsize
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_setkey
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_finup
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_final
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_digest
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_export
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_ahash_import
+
+Asynchronous Hash Request Handle
+--------------------------------
+
+.. kernel-doc:: include/crypto/hash.h
+   :doc: Asynchronous Hash Request Handle
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: ahash_request_set_tfm
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: ahash_request_alloc
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: ahash_request_free
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: ahash_request_set_callback
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: ahash_request_set_crypt
+
+Synchronous Message Digest API
+------------------------------
+
+.. kernel-doc:: include/crypto/hash.h
+   :doc: Synchronous Message Digest API
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_alloc_shash
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_free_shash
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_blocksize
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_digestsize
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_descsize
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_setkey
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_digest
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_export
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_import
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_init
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_update
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_final
+
+.. kernel-doc:: include/crypto/hash.h
+   :functions: crypto_shash_finup
+
+Crypto API Random Number API
+----------------------------
+
+.. kernel-doc:: include/crypto/rng.h
+   :doc: Random number generator API
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_alloc_rng
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_rng_alg
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_free_rng
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_rng_generate
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_rng_get_bytes
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_rng_reset
+
+.. kernel-doc:: include/crypto/rng.h
+   :functions: crypto_rng_seedsize
+
+.. WARNING: DOCPROC directive not supported: !Cinclude/crypto/rng.h
+
+Asymmetric Cipher API
+---------------------
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :doc: Generic Public Key API
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: akcipher_alg
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: akcipher_request
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_alloc_akcipher
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_free_akcipher
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_set_pub_key
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_set_priv_key
+
+Asymmetric Cipher Request Handle
+--------------------------------
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: akcipher_request_alloc
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: akcipher_request_free
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: akcipher_request_set_callback
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: akcipher_request_set_crypt
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_maxsize
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_encrypt
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_decrypt
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_sign
+
+.. kernel-doc:: include/crypto/akcipher.h
+   :functions: crypto_akcipher_verify
+
+Code Examples
+=============
+
+Code Example For Symmetric Key Cipher Operation
+-----------------------------------------------
+
+::
+
+
+    struct tcrypt_result {
+        struct completion completion;
+        int err;
+    };
+
+    /* tie all data structures together */
+    struct skcipher_def {
+        struct scatterlist sg;
+        struct crypto_skcipher *tfm;
+        struct skcipher_request *req;
+        struct tcrypt_result result;
+    };
+
+    /* Callback function */
+    static void test_skcipher_cb(struct crypto_async_request *req, int error)
+    {
+        struct tcrypt_result *result = req->data;
+
+        if (error == -EINPROGRESS)
+            return;
+        result->err = error;
+        complete(&result->completion);
+        pr_info("Encryption finished successfully\n");
+    }
+
+    /* Perform cipher operation */
+    static unsigned int test_skcipher_encdec(struct skcipher_def *sk,
+                         int enc)
+    {
+        int rc = 0;
+
+        if (enc)
+            rc = crypto_skcipher_encrypt(sk->req);
+        else
+            rc = crypto_skcipher_decrypt(sk->req);
+
+        switch (rc) {
+        case 0:
+            break;
+        case -EINPROGRESS:
+        case -EBUSY:
+            rc = wait_for_completion_interruptible(
+                &sk->result.completion);
+            if (!rc && !sk->result.err) {
+                reinit_completion(&sk->result.completion);
+                break;
+            }
+        default:
+            pr_info("skcipher encrypt returned with %d result %d\n",
+                rc, sk->result.err);
+            break;
+        }
+        init_completion(&sk->result.completion);
+
+        return rc;
+    }
+
+    /* Initialize and trigger cipher operation */
+    static int test_skcipher(void)
+    {
+        struct skcipher_def sk;
+        struct crypto_skcipher *skcipher = NULL;
+        struct skcipher_request *req = NULL;
+        char *scratchpad = NULL;
+        char *ivdata = NULL;
+        unsigned char key[32];
+        int ret = -EFAULT;
+
+        skcipher = crypto_alloc_skcipher("cbc-aes-aesni", 0, 0);
+        if (IS_ERR(skcipher)) {
+            pr_info("could not allocate skcipher handle\n");
+            return PTR_ERR(skcipher);
+        }
+
+        req = skcipher_request_alloc(skcipher, GFP_KERNEL);
+        if (!req) {
+            pr_info("could not allocate skcipher request\n");
+            ret = -ENOMEM;
+            goto out;
+        }
+
+        skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+                          test_skcipher_cb,
+                          &sk.result);
+
+        /* AES 256 with random key */
+        get_random_bytes(&key, 32);
+        if (crypto_skcipher_setkey(skcipher, key, 32)) {
+            pr_info("key could not be set\n");
+            ret = -EAGAIN;
+            goto out;
+        }
+
+        /* IV will be random */
+        ivdata = kmalloc(16, GFP_KERNEL);
+        if (!ivdata) {
+            pr_info("could not allocate ivdata\n");
+            goto out;
+        }
+        get_random_bytes(ivdata, 16);
+
+        /* Input data will be random */
+        scratchpad = kmalloc(16, GFP_KERNEL);
+        if (!scratchpad) {
+            pr_info("could not allocate scratchpad\n");
+            goto out;
+        }
+        get_random_bytes(scratchpad, 16);
+
+        sk.tfm = skcipher;
+        sk.req = req;
+
+        /* We encrypt one block */
+        sg_init_one(&sk.sg, scratchpad, 16);
+        skcipher_request_set_crypt(req, &sk.sg, &sk.sg, 16, ivdata);
+        init_completion(&sk.result.completion);
+
+        /* encrypt data */
+        ret = test_skcipher_encdec(&sk, 1);
+        if (ret)
+            goto out;
+
+        pr_info("Encryption triggered successfully\n");
+
+    out:
+        if (skcipher)
+            crypto_free_skcipher(skcipher);
+        if (req)
+            skcipher_request_free(req);
+        if (ivdata)
+            kfree(ivdata);
+        if (scratchpad)
+            kfree(scratchpad);
+        return ret;
+    }
+
+
+Code Example For Use of Operational State Memory With SHASH
+-----------------------------------------------------------
+
+::
+
+
+    struct sdesc {
+        struct shash_desc shash;
+        char ctx[];
+    };
+
+    static struct sdescinit_sdesc(struct crypto_shash *alg)
+    {
+        struct sdescsdesc;
+        int size;
+
+        size = sizeof(struct shash_desc) + crypto_shash_descsize(alg);
+        sdesc = kmalloc(size, GFP_KERNEL);
+        if (!sdesc)
+            return ERR_PTR(-ENOMEM);
+        sdesc->shash.tfm = alg;
+        sdesc->shash.flags = 0x0;
+        return sdesc;
+    }
+
+    static int calc_hash(struct crypto_shashalg,
+                 const unsigned chardata, unsigned int datalen,
+                 unsigned chardigest) {
+        struct sdescsdesc;
+        int ret;
+
+        sdesc = init_sdesc(alg);
+        if (IS_ERR(sdesc)) {
+            pr_info("trusted_key: can't alloc %s\n", hash_alg);
+            return PTR_ERR(sdesc);
+        }
+
+        ret = crypto_shash_digest(&sdesc->shash, data, datalen, digest);
+        kfree(sdesc);
+        return ret;
+    }
+
+
+Code Example For Random Number Generator Usage
+----------------------------------------------
+
+::
+
+
+    static int get_random_numbers(u8 *buf, unsigned int len)
+    {
+        struct crypto_rngrng = NULL;
+        chardrbg = "drbg_nopr_sha256"; /* Hash DRBG with SHA-256, no PR */
+        int ret;
+
+        if (!buf || !len) {
+            pr_debug("No output buffer provided\n");
+            return -EINVAL;
+        }
+
+        rng = crypto_alloc_rng(drbg, 0, 0);
+        if (IS_ERR(rng)) {
+            pr_debug("could not allocate RNG handle for %s\n", drbg);
+            return -PTR_ERR(rng);
+        }
+
+        ret = crypto_rng_get_bytes(rng, buf, len);
+        if (ret < 0)
+            pr_debug("generation of random numbers failed\n");
+        else if (ret == 0)
+            pr_debug("RNG returned no data");
+        else
+            pr_debug("RNG returned %d bytes of data\n", ret);
+
+    out:
+        crypto_free_rng(rng);
+        return ret;
+    }
Documentation/filesystems.rst new file +314 -0

314 lines not on this page · raw message

Documentation/gpu.rst new file +2556 -0

816 lines not on this page · raw message

Documentation/index.rst +3 -0
index 6ca68ef5c89a..7b520cda5a5c 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.1.4

Part of the diff is not shown here. View the raw message.

#13 lore raw
In reply to Jani Nikula
Quoted 6 lines · Jani Nikula

At this time I've put most effort into the configuration and build side of things, solving the problems described above, and handling missing tools and packages gracefully. There are still issues to be ironed out in a) the kernel-doc script rst output and b) the xml template to rst conversion. These are somewhat orthogonal from each other and the build, and I expect some hand-editing will be required in the end.

Jon -

I've added a significant amount of polish. It works and it looks pretty. The code is in the "sphinx-wip" branch of [1], gitweb at [2], and Read the Docs built copy at [3].

There's roughly 30 commits there that I think are getting ready for merging. The rest is scripted conversion of some DocBook templates for demonstration purposes; I think the final conversion should be done by the authors of the documents when they are ready to ditch the DocBook templates in the same commit.

I also think I've now spearheaded this pretty close to a point where more benefits can be gained by merging the work to docs-next, and collaborating there.

Let me know what you think.

BR,
Jani.

PS. I never thought I'd do this much perl coding in my life. It didn't kill me, I must be stronger now.

[1] git://people.freedesktop.org/~jani/drm
[2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-wip
[3] https://kernel.readthedocs.io/

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#14 lore raw
Quoted 20 lines · Jani Nikula
Quoted 6 lines · Jani Nikula

At this time I've put most effort into the configuration and build side of things, solving the problems described above, and handling missing tools and packages gracefully. There are still issues to be ironed out in a) the kernel-doc script rst output and b) the xml template to rst conversion. These are somewhat orthogonal from each other and the build, and I expect some hand-editing will be required in the end.

Jon -

I've added a significant amount of polish. It works and it looks pretty. The code is in the "sphinx-wip" branch of [1], gitweb at [2], and Read the Docs built copy at [3].

There's roughly 30 commits there that I think are getting ready for merging. The rest is scripted conversion of some DocBook templates for demonstration purposes; I think the final conversion should be done by the authors of the documents when they are ready to ditch the DocBook templates in the same commit.

Quoted 5 lines

I also think I've now spearheaded this pretty close to a point where more benefits can be gained by merging the work to docs-next, and collaborating there.

Let me know what you think.

I've looked both through the patches in Jani's branch and the resulting docs. Spotted 2 nitpicks, which are already fixed, and it looks imo _really_ good.

I think next steps is to get this merged into docs-next, with a stable tag, so that I can pull it into drm-misc. There's still piles of little polish to do to make it really pretty, but that's almost all in the kernel-doc comments or .rst files itself. I think for starting we could convert a few docs already, or just add the index.rst and leave conversions to maintainers. At least for gpu.tmpl I expect some need for cordination with ongoing work.
-Daniel

Quoted 16 lines

BR,
Jani.

PS. I never thought I'd do this much perl coding in my life. It didn't kill me, I must be stronger now.

[1] git://people.freedesktop.org/~jani/drm
[2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-wip
[3] https://kernel.readthedocs.io/

--
Jani Nikula, Intel Open Source Technology Center

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#15 lore raw

Hi,

sorry for my temporary absence, I have been on holiday the last weeks :-)

Am 30.05.2016 um 11:10 schrieb Daniel Vetter <[email protected]>:

Quoted 31 lines · Jani Nikula
Quoted 20 lines · Jani Nikula
Quoted 6 lines

At this time I've put most effort into the configuration and build side of things, solving the problems described above, and handling missing tools and packages gracefully. There are still issues to be ironed out in a) the kernel-doc script rst output and b) the xml template to rst conversion. These are somewhat orthogonal from each other and the build, and I expect some hand-editing will be required in the end.

Jon -

I've added a significant amount of polish. It works and it looks pretty. The code is in the "sphinx-wip" branch of [1], gitweb at [2], and Read the Docs built copy at [3].

There's roughly 30 commits there that I think are getting ready for merging. The rest is scripted conversion of some DocBook templates for demonstration purposes; I think the final conversion should be done by the authors of the documents when they are ready to ditch the DocBook templates in the same commit.

Quoted 5 lines

I also think I've now spearheaded this pretty close to a point where more benefits can be gained by merging the work to docs-next, and collaborating there.

Let me know what you think.

I've looked both through the patches in Jani's branch and the resulting docs. Spotted 2 nitpicks, which are already fixed, and it looks imo _really_ good.

Quoted 8 lines

I think next steps is to get this merged into docs-next, with a stable tag, so that I can pull it into drm-misc. There's still piles of little polish to do to make it really pretty, but that's almost all in the kernel-doc comments or .rst files itself. I think for starting we could convert a few docs already, or just add the index.rst and leave conversions to maintainers. At least for gpu.tmpl I expect some need for cordination with ongoing work.
-Daniel

Here my 5cents about Jani's patch series:

1. Migration implementations should not be a part of the kernel tree

2. sed/pandoc migration fits not for all the XML documentation and has disadvantages. Jonathan asked it before: http://article.gmane.org/gmane.linux.documentation/37533

I repeat myself: Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where dbxml comes into play

- reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23

- writing reST has many bugs and leaks (you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

* dbxml brings the ability to chunk one large XML book into small
reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

this might helpfull e.g. if you have to convert single-column informal-tables to lists or other things ... in short; dbxml and it's hooks are the key to hack everything you need in a full automated DocBook-->reST migration workflow.

3. we also discussed before, that ASCII art tables are ugly, because the produce confusing diffs, for this I wrote the flat-table directive (dbxml migrates tables to flat-tables / pandoc can't).
https://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table

Quoted 4 lines

With this, we can put any .rst files (including ones that have kernel-doc directives) anywhere under Documentation, add a link to them in Documentation/index.rst table of contents, and it will just work. It can't get much simpler than that.

4. We discussed it / I already mentioned that each document shipped in it's own sphinx project. Bundling all documents into one sphinx-project will work for 4 or 5 small documents, but not for the whole documentation. BTW all XML documents are currently separated DocBook projects .. so why should we merge them into one big project? Making one index-file for the different and small ".txt" files seems OK, but not for the XML docs.

5. In general, the markup of the linux kernel's source code comments remains unchanged and the reST markup within the comments is passed through the output. A closer lookup to the *kernel-doc* and *reST* markup revals, that there are some conflicts between reST (inline) markup and kernel-doc markup. Determined by the historical development of the kernel-doc comments, the *classic* kernel-doc comments contain characters like ``*`` or strings with e.g. leading/trailing underscore (``_``), which are inline markups in reST. Here a schort example from a *classic* comment::

<SNIP> -----
* In contrast to the other drm_get_*_name functions this one here returns a
* const pointer and hence is threadsafe.
<SNAP> -----

In reST markup, the wildcard in the string ``drm_get_*_name`` has to be
masked: ``drm_get_\\*_name``. Some more examples from reST markup:

* Emphasis "*": like ``*emphasis*`` or ``**emphasis strong**``
* Leading "_" : is a *anchor* in reST markup (``_foo``).
* Trailing "_: is a reference in reST markup (``foo_``).
* interpreted text: "`"
* inline literals: "``"
* substitution references: "|"

These special strings has to be masked in the output and can't be used as *plain-text markup*. To get in use of the fully reST markup (stop masking special characters) we need some options in the sources documentation, comments like

/* parse-markup: reST */

which influence the behavior.

Quoted 3 lines

Sites like https://readthedocs.org/ can build the documentation, including kernel-doc, without extra tweaks. As a whole, the build becomes much simpler.

6. This fail assessments I also had before. RTD has limits in resources and in flexibility, that's why I moved to github pages

https://github.com/return42/sphkerneldoc/issues/1

I will stop here ... I think it is good that everyone make its own experience, BUT ...

IMO it is a misjudgment to think that changing the markup and it's toolchain is only a series of patches.

Sorry if my words are unpleasant, this is not my intend, but IMO *the view of the whole* and concepts are missed.

Many of the facts mentioned above have been covered in my POC at
https://github.com/return42/sphkerneldoc ... On others, like 5. I'am working on ....

Quoted 1 line

I've had a few moments of spare time to look into Sphinx.

... a comprehensive solution needs time and will not be done in a hurry. Please give me a week or may be two, then I could present a much more comprehensive solution.

Again: my english is not the best, if some of my words has been unpleasant I'am sorry for this, it's not my intend.

--Markus--

Quoted 24 lines
Quoted 17 lines

BR,
Jani.

PS. I never thought I'd do this much perl coding in my life. It didn't kill me, I must be stronger now.

[1] git://people.freedesktop.org/~jani/drm
[2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-wip
[3] https://kernel.readthedocs.io/

--
Jani Nikula, Intel Open Source Technology Center

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#16 lore raw
Quoted 3 lines · Markus Heiser

Here my 5cents about Jani's patch series:

1. Migration implementations should not be a part of the kernel tree

If you're referring to the conversion scripts, I don't care either way. It's probably helpful to have them until everything is converted, and we can dispose of them afterwards.

Quoted 14 lines

2. sed/pandoc migration fits not for all the XML documentation and has disadvantages. Jonathan asked it before: http://article.gmane.org/gmane.linux.documentation/37533

I repeat myself: Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where dbxml comes into play

- reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23

- writing reST has many bugs and leaks (you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

Quoted 8 lines

* dbxml brings the ability to chunk one large XML book into small
reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

this might helpfull e.g. if you have to convert single-column informal-tables to lists or other things ... in short; dbxml and it's hooks are the key to hack everything you need in a full automated DocBook-->reST migration workflow.

I am not proposing to merge the documents that I've converted mostly as samples in the branch. I needed something to demonstrate the build is sane.

The authors of the DocBook documents should make the conversions as they see fit, when they see fit, with the tools they see fit, probably with some manual work on top.

Quoted 4 lines

3. we also discussed before, that ASCII art tables are ugly, because the produce confusing diffs, for this I wrote the flat-table directive (dbxml migrates tables to flat-tables / pandoc can't).
https://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table

See the above. Any authors that think the Sphinx support I've added is good enough can go ahead and switch.

I think it's safe for me to say the GPU documents won't wait for further extension directives or conversion tools. Some others will definitely want to have the flat table extension before switching.

Quoted 11 lines
Quoted 4 lines

With this, we can put any .rst files (including ones that have kernel-doc directives) anywhere under Documentation, add a link to them in Documentation/index.rst table of contents, and it will just work. It can't get much simpler than that.

4. We discussed it / I already mentioned that each document shipped in it's own sphinx project. Bundling all documents into one sphinx-project will work for 4 or 5 small documents, but not for the whole documentation. BTW all XML documents are currently separated DocBook projects .. so why should we merge them into one big project? Making one index-file for the different and small ".txt" files seems OK, but not for the XML docs.

FWIW I locally converted all the DocBook documents (except media) and it works just fine, and to me it looks like exactly what we should have. One of the goals was to have nice cross-referencing between the documents (e.g. from GPU to kernel or device driver API). And it works.

This does not exclude having *additional* indexes or Sphinx config files for subsystems or subprojects to build a subset of the documentation for specific needs. It's up to the authors of the documents to decide.

For PDF documents, adding the documents separately in pdf_documents seems to be the right thing to do.

Quoted 24 lines

5. In general, the markup of the linux kernel's source code comments remains unchanged and the reST markup within the comments is passed through the output. A closer lookup to the *kernel-doc* and *reST* markup revals, that there are some conflicts between reST (inline) markup and kernel-doc markup. Determined by the historical development of the kernel-doc comments, the *classic* kernel-doc comments contain characters like ``*`` or strings with e.g. leading/trailing underscore (``_``), which are inline markups in reST. Here a schort example from a *classic* comment::

<SNIP> -----
* In contrast to the other drm_get_*_name functions this one here returns a
* const pointer and hence is threadsafe.
<SNAP> -----

In reST markup, the wildcard in the string ``drm_get_*_name`` has to be
masked: ``drm_get_\\*_name``. Some more examples from reST markup:

* Emphasis "*": like ``*emphasis*`` or ``**emphasis strong**``
* Leading "_" : is a *anchor* in reST markup (``_foo``).
* Trailing "_: is a reference in reST markup (``foo_``).
* interpreted text: "`"
* inline literals: "``"
* substitution references: "|"

Quoted 8 lines

These special strings has to be masked in the output and can't be used as *plain-text markup*. To get in use of the fully reST markup (stop masking special characters) we need some options in the sources documentation, comments like

/* parse-markup: reST */

which influence the behavior.

I find it totally unacceptable to require explicitly marking kernel-doc comments or source files as being reStructuredText.

Note that it's all opt-in already. If you add a .rst file that includes kernel-doc via the kernel-doc extension, you better make sure the comments parse as reStructuredText and render nicely. I'm willing to do much of the job for all the things that I care about.

Besides, if you look at the results, you'll find it looks mostly good without any fixes. In the sample documents, I've erred on the side of having a few markup hickups here and there while most of it works perfectly well as reStructuredText. I think this is exactly what we should do, declare it all reStructuredText and fix issues as we go.

Quoted 9 lines
Quoted 3 lines

Sites like https://readthedocs.org/ can build the documentation, including kernel-doc, without extra tweaks. As a whole, the build becomes much simpler.

6. This fail assessments I also had before. RTD has limits in resources and in flexibility, that's why I moved to github pages

https://github.com/return42/sphkerneldoc/issues/1

I'm just personally using Read the Docs to ensure they can build the documentation as "pure" Sphinx (they won't use the Makefiles), and so I don't have to host the docs anywhere myself. Sure, I can't add *all* the files there because it exceeds the build limits. They might cater for us if they want to carry the kernel documentation going forward, but most likely the output should be at kernel.org anyway. (Plus we'll have freedesktop.org and 01.org for the GPU documentation too.)

Quoted 8 lines

I will stop here ... I think it is good that everyone make its own experience, BUT ...

IMO it is a misjudgment to think that changing the markup and it's toolchain is only a series of patches.

Sorry if my words are unpleasant, this is not my intend, but IMO *the view of the whole* and concepts are missed.

I'm not sure what to say.

For one thing, a concrete series of patches to add Sphinx support to the kernel, integrating it with the build system and kernel-doc and everything is *exactly* what is needed. This is what I've done. It's here now. It works. Anyone can take my git tree, run 'make htmldocs' and see the results for themselves.

Quoted 1 line

From your message it remains unclear to me what "view of the whole" and

"concepts" and "comprehensive solution" I have missed.

Quoted 9 lines

Many of the facts mentioned above have been covered in my POC at
https://github.com/return42/sphkerneldoc ... On others, like 5. I'am working on ....

Quoted 1 line

I've had a few moments of spare time to look into Sphinx.

... a comprehensive solution needs time and will not be done in a hurry. Please give me a week or may be two, then I could present a much more comprehensive solution.

Please do not underestimate the productivity of my moments of spare time. ;)

My view of the whole is that we've been talking about adding lightweight markup support for the better part of a year now, and I'm getting pretty tired of talking...

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#17 lore raw

I concur with Jani on all points, just want to follow-up here.

Quoted 10 lines · Jani Nikula
Quoted 9 lines

Many of the facts mentioned above have been covered in my POC at
https://github.com/return42/sphkerneldoc ... On others, like 5. I'am working on ....

Quoted 1 line

I've had a few moments of spare time to look into Sphinx.

... a comprehensive solution needs time and will not be done in a hurry. Please give me a week or may be two, then I could present a much more comprehensive solution.

Quoted 6 lines

Please do not underestimate the productivity of my moments of spare time. ;)

My view of the whole is that we've been talking about adding lightweight markup support for the better part of a year now, and I'm getting pretty tired of talking...

Tbh I'd like to get this in soonish, so that we have about 1 month of time before 4.8 for actually polishing gpu documentation. Maybe there's some room for more, better tables, other extensions, polish the kernel-doc script a bit more. But what we have now works, I think it does address everything we've discussed over the past year that we absolutely need. I think it makes it possible to extend for all the more optional use-cases we need to be able to convert, and we can add those extensions later on. And with Jani's big pile of kernel-doc patches we now also have someone who understands that perl script, which is awesome.

In short I want to go nuts improving the docs themselves and stop discussing the tooling to build them. Can we please make this happen?

Note that Jani's already started to throw out our old ascidoc hacks in the topic/kerneldoc branch in the drm-intel.git repo, and we'll switch over the autobuilder for the 01.org docs as soon as that's done. We're committed, I want this ;-)

Thanks, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#18 lore raw
In reply to Jani Nikula

Am 30.05.2016 um 16:46 schrieb Jani Nikula <[email protected]>:

Quoted 32 lines · Markus Heiser
Quoted 3 lines

Here my 5cents about Jani's patch series:

1. Migration implementations should not be a part of the kernel tree

If you're referring to the conversion scripts, I don't care either way. It's probably helpful to have them until everything is converted, and we can dispose of them afterwards.

Quoted 14 lines

2. sed/pandoc migration fits not for all the XML documentation and has disadvantages. Jonathan asked it before: http://article.gmane.org/gmane.linux.documentation/37533

I repeat myself: Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where dbxml comes into play

- reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23

- writing reST has many bugs and leaks (you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

Quoted 8 lines

* dbxml brings the ability to chunk one large XML book into small
reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

this might helpfull e.g. if you have to convert single-column informal-tables to lists or other things ... in short; dbxml and it's hooks are the key to hack everything you need in a full automated DocBook-->reST migration workflow.

Quoted 7 lines

I am not proposing to merge the documents that I've converted mostly as samples in the branch. I needed something to demonstrate the build is sane.

The authors of the DocBook documents should make the conversions as they see fit, when they see fit, with the tools they see fit, probably with some manual work on top.

OK

Quoted 6 lines
Quoted 5 lines

3. we also discussed before, that ASCII art tables are ugly, because the produce confusing diffs, for this I wrote the flat-table directive (dbxml migrates tables to flat-tables / pandoc can't).
https://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table

Quoted 6 lines

See the above. Any authors that think the Sphinx support I've added is good enough can go ahead and switch.

I think it's safe for me to say the GPU documents won't wait for further extension directives or conversion tools. Some others will definitely want to have the flat table extension before switching.

OK

Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

With this, we can put any .rst files (including ones that have kernel-doc directives) anywhere under Documentation, add a link to them in Documentation/index.rst table of contents, and it will just work. It can't get much simpler than that.

Quoted 6 lines

4. We discussed it / I already mentioned that each document shipped in it's own sphinx project. Bundling all documents into one sphinx-project will work for 4 or 5 small documents, but not for the whole documentation. BTW all XML documents are currently separated DocBook projects .. so why should we merge them into one big project? Making one index-file for the different and small ".txt" files seems OK, but not for the XML docs.

Quoted 3 lines

FWIW I locally converted all the DocBook documents (except media) and it works just fine, and to me it looks like exactly what we should have.

With DocBook, it was hard to separate a file into small chunks (see media for how it is done). With Sphinx, it is common to split a document in small chunks (along parts, chapters, sections ...) Thats why I recommend chunking documents (from the beginning).

Quoted 2 lines

One of the goals was to have nice cross-referencing between the documents (e.g. from GPU to kernel or device driver API). And it works.

For this, Sphinx-doc brings intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

Quoted 3 lines

This does not exclude having *additional* indexes or Sphinx config files for subsystems or subprojects to build a subset of the documentation for specific needs. It's up to the authors of the documents to decide.

OK, I think we have a bit different point of view .. IMO it is better to give the authors a *scalable structure* to place their's documentation.

Quoted 2 lines

For PDF documents, adding the documents separately in pdf_documents seems to be the right thing to do.

I can't recommend to use rst2pdf (it is less maintained), use default sphinx LaTeX toolchain.

Quoted 32 lines
Quoted 32 lines

5. In general, the markup of the linux kernel's source code comments remains unchanged and the reST markup within the comments is passed through the output. A closer lookup to the *kernel-doc* and *reST* markup revals, that there are some conflicts between reST (inline) markup and kernel-doc markup. Determined by the historical development of the kernel-doc comments, the *classic* kernel-doc comments contain characters like ``*`` or strings with e.g. leading/trailing underscore (``_``), which are inline markups in reST. Here a schort example from a *classic* comment::

<SNIP> -----
* In contrast to the other drm_get_*_name functions this one here returns a
* const pointer and hence is threadsafe.
<SNAP> -----

In reST markup, the wildcard in the string ``drm_get_*_name`` has to be
masked: ``drm_get_\\*_name``. Some more examples from reST markup:

* Emphasis "*": like ``*emphasis*`` or ``**emphasis strong**``
* Leading "_" : is a *anchor* in reST markup (``_foo``).
* Trailing "_: is a reference in reST markup (``foo_``).
* interpreted text: "`"
* inline literals: "``"
* substitution references: "|"

These special strings has to be masked in the output and can't be used as *plain-text markup*. To get in use of the fully reST markup (stop masking special characters) we need some options in the sources documentation, comments like

      /* parse-markup: reST */
Quoted 8 lines
Quoted 1 line

which influence the behavior.

I find it totally unacceptable to require explicitly marking kernel-doc comments or source files as being reStructuredText.
Note that it's all opt-in already. If you add a .rst file that includes kernel-doc via the kernel-doc extension, you better make sure the comments parse as reStructuredText and render nicely. I'm willing to do much of the job for all the things that I care about.

We have a different POV ... I try to build up a documentation project, which could use all given kernel-doc markups without any change, where reST is an "addition". Your approach is to fix kernel-doc comments if they are referred by a kernl-doc directive in a .rst document. There is nothing wrong about your approach, but I try to build a whole source code documentation like the one I started here:
http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

Benefit: Beside a kernel-doc directive, authors (from everywhere, not only within the kernel-src-tree) can refer kernel-docs with intersphinx Here is a example, see "cross references" at:
http://return42.github.io/sphkerneldoc/articles/linux_src_doc.html?cross-references

Quoted 16 lines

Besides, if you look at the results, you'll find it looks mostly good without any fixes. In the sample documents, I've erred on the side of having a few markup hickups here and there while most of it works perfectly well as reStructuredText. I think this is exactly what we should do, declare it all reStructuredText and fix issues as we go.

Quoted 9 lines
Quoted 3 lines

Sites like https://readthedocs.org/ can build the documentation, including kernel-doc, without extra tweaks. As a whole, the build becomes much simpler.

6. This fail assessments I also had before. RTD has limits in resources and in flexibility, that's why I moved to github pages

https://github.com/return42/sphkerneldoc/issues/1

Quoted 7 lines

I'm just personally using Read the Docs to ensure they can build the documentation as "pure" Sphinx (they won't use the Makefiles), and so I don't have to host the docs anywhere myself. Sure, I can't add *all* the files there because it exceeds the build limits. They might cater for us if they want to carry the kernel documentation going forward, but most likely the output should be at kernel.org anyway. (Plus we'll have freedesktop.org and 01.org for the GPU documentation too.)

There a some similar discussions on RTD ... yes, they make exceptions for *special* users, but its all harry, use your default publishing sides and forget RTD ;-)

Quoted 10 lines
Quoted 8 lines

I will stop here ... I think it is good that everyone make its own experience, BUT ...

IMO it is a misjudgment to think that changing the markup and it's toolchain is only a series of patches.

Sorry if my words are unpleasant, this is not my intend, but IMO *the view of the whole* and concepts are missed.

I'm not sure what to say.

As I said, we have a different POV, which is not in contradiction, as your requirements are a subset of mine.

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

Quoted 2 lines

Please do not underestimate the productivity of my moments of spare time. ;)

1:0 for you ;-)

Am 30.05.2016 um 17:29 schrieb Daniel Vetter <[email protected]>:

Quoted 10 lines

I concur with Jani on all points, just want to follow-up here.

Tbh I'd like to get this in soonish, so that we have about 1 month of time before 4.8 for actually polishing gpu documentation. Maybe there's some room for more, better tables, other extensions, polish the kernel-doc script a bit more. But what we have now works, I think it does address everything we've discussed over the past year that we absolutely need. I think it makes it possible to extend for all the more optional use-cases we need to be able to convert, and we can add those extensions later on.

OK, I understand your needs, I don't wan't to stop you writing documentation with reST. May use Jani's approach for a first toolchain

Quoted 3 lines

And with Jani's big pile of kernel-doc
patches we now also have someone who understands that perl script, which is awesome.

?

https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

Quoted 7 lines

In short I want to go nuts improving the docs themselves and stop discussing the tooling to build them. Can we please make this happen?

Note that Jani's already started to throw out our old ascidoc hacks in the topic/kerneldoc branch in the drm-intel.git repo, and we'll switch over the autobuilder for the 01.org docs as soon as that's done. We're committed, I want this ;-)

Am 30.05.2016 um 16:46 schrieb Jani Nikula <[email protected]>:

Quoted 1 line

I'm getting pretty tired of talking...

Yes, we are all tired in this discussions ... I see, you are in need to switch over ... for this, may use your toolchain ... but I have one please, do not change more than you really need and be open for changing the toolchain later.

I will inform you about any substantial progress within my solution ... may we can merge our experience soon or later, but don't stop writing documentation.

Thank's for your patience ... :-)

--Markus--

#19 lore raw
Quoted 10 lines · Markus Heiser

Am 30.05.2016 um 16:46 schrieb Jani Nikula <[email protected]>:

Quoted 7 lines

I am not proposing to merge the documents that I've converted mostly as samples in the branch. I needed something to demonstrate the build is sane.

The authors of the DocBook documents should make the conversions as they see fit, when they see fit, with the tools they see fit, probably with some manual work on top.

OK

To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I propose to merge at this time. There's the Sphinx configuration, kernel build integration, Sphinx kernel-doc extension, tons of kernel-doc updates, etc. There is no DocBook tmpl conversion; all of that is left to the authors (owners, maintainers) of the documents, but this enables them to focus on that part.

I was planning on sending out the patches after some feedback here.

[1] git://people.freedesktop.org/~jani/drm
[2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-for-docs-next

Quoted 4 lines

With DocBook, it was hard to separate a file into small chunks (see media for how it is done). With Sphinx, it is common to split a document in small chunks (along parts, chapters, sections ...) Thats why I recommend chunking documents (from the beginning).

Agreed, but up to the authors.

Quoted 4 lines
Quoted 2 lines

One of the goals was to have nice cross-referencing between the documents (e.g. from GPU to kernel or device driver API). And it works.

For this, Sphinx-doc brings intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

If the kernel is split to several intersphinx "prefixes", we won't know the prefix of the link targets when we're generating the references in kernel-doc. Also, can be deferred to follow-up work if someone figures out the how.

Quoted 2 lines

I can't recommend to use rst2pdf (it is less maintained), use default sphinx LaTeX toolchain.

I think we'll use whatever works, rst2pdf seemed to work for now, but we can change if needed.

Quoted 7 lines
Quoted 6 lines

I find it totally unacceptable to require explicitly marking kernel-doc comments or source files as being reStructuredText.
Note that it's all opt-in already. If you add a .rst file that includes kernel-doc via the kernel-doc extension, you better make sure the comments parse as reStructuredText and render nicely. I'm willing to do much of the job for all the things that I care about.

Quoted 7 lines

We have a different POV ... I try to build up a documentation project, which could use all given kernel-doc markups without any change, where reST is an "addition". Your approach is to fix kernel-doc comments if they are referred by a kernl-doc directive in a .rst document. There is nothing wrong about your approach, but I try to build a whole source code documentation like the one I started here:
http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

That looks nice, but I'll argue it would not be much worse even if you assumed it's all rst.

The bigger point is, if you expect people to tag each source file or kernel-doc comment with "rst", you'll end up with a mess where some places have that tag, some not, but it's not conclusive about whether they actually *are* rst or not. (And you've had tons of patch churn to add those tags to get there.)

The kernel-doc comments are written by humans who will screw it up anyway. (Apologies for the distrust, fellow developers, but I've been reading too many of your fine kernel-doc comments lately.) People will happily cargo cult rst and current kernel-doc and javadoc and doxygen and whatnot in a fruit salad. The only thing that will help in the end is keeping the rules simple and consistent and having the feedback from the tools.

Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series. Do remember that this is a one time conversion. It needs to be good enough that there's not too much manual editing involved, but it doesn't need to be perfect. Some degree of editing will be required no matter what, not least because the DocBook has also been written by humans, and the battle against the GIGO principle is a lost one.

Quoted 5 lines
Quoted 3 lines

And with Jani's big pile of kernel-doc
patches we now also have someone who understands that perl script, which is awesome.

?

I must also question any sentence that implies I understand perl. ;)

Quoted 7 lines
Quoted 7 lines

In short I want to go nuts improving the docs themselves and stop discussing the tooling to build them. Can we please make this happen?

Note that Jani's already started to throw out our old ascidoc hacks in the topic/kerneldoc branch in the drm-intel.git repo, and we'll switch over the autobuilder for the 01.org docs as soon as that's done. We're committed, I want this ;-)

It's done, we're feeding this to our integration tree and dogfooding.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#20 lore raw

Em Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <[email protected]> escreveu:

Quoted 9 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series.

With regards to media, my plan is to merge create a topic branch based on Kernel 4.7-rc1 at:
https://git.linuxtv.org/media_tree.git/

As none of the Jani's patches seem to affect the media API docs, it seems I don't need to merge back from Jon's -next branch.

There, I intend to add Markus patches with the conversion from the DocBook to rst, plus the flat-table extension logic.

Then, I'll work to manually fix what's needed and I'll add the automation scripting logic that we have at the DocBook Makefile to work with the new media rst files.

Lastly, once the job's done, I'll drop Documentation/DocBook/media.

Markus,

With that regards, could you please send the patches to me?

Thanks,
Mauro

#21 lore raw

Am 30.05.2016 um 23:23 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 27 lines

Em Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <[email protected]> escreveu:

Quoted 9 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series.

With regards to media, my plan is to merge create a topic branch based on Kernel 4.7-rc1 at:
https://git.linuxtv.org/media_tree.git/

As none of the Jani's patches seem to affect the media API docs, it seems I don't need to merge back from Jon's -next branch.

There, I intend to add Markus patches with the conversion from the DocBook to rst, plus the flat-table extension logic.

Then, I'll work to manually fix what's needed and I'll add the automation scripting logic that we have at the DocBook Makefile to work with the new media rst files.

Quoted 5 lines

Lastly, once the job's done, I'll drop Documentation/DocBook/media.

Markus,

With that regards, could you please send the patches to me?

Yes. What is your timeline ... is it OK if I send you a patch in the next two weeks? ... first I wan't to finish my other work / I'am just back from holiday .. a lot of work to do :-o

--Markus--

Quoted 3 lines

Thanks,
Mauro

#22 lore raw

Em Tue, 31 May 2016 12:16:25 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 31 lines

Am 30.05.2016 um 23:23 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 29 lines

Em Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <[email protected]> escreveu:

Quoted 9 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series.

With regards to media, my plan is to merge create a topic branch based on Kernel 4.7-rc1 at:
https://git.linuxtv.org/media_tree.git/

As none of the Jani's patches seem to affect the media API docs, it seems I don't need to merge back from Jon's -next branch.

There, I intend to add Markus patches with the conversion from the DocBook to rst, plus the flat-table extension logic.

Then, I'll work to manually fix what's needed and I'll add the automation scripting logic that we have at the DocBook Makefile to work with the new media rst files.

Lastly, once the job's done, I'll drop Documentation/DocBook/media.

Quoted 7 lines
Quoted 3 lines

Markus,

With that regards, could you please send the patches to me?

Yes. What is your timeline ... is it OK if I send you a patch in the next two weeks? ... first I wan't to finish my other work / I'am just back from holiday .. a lot of work to do :-o

Hi Markus,

I'm wanting to start working on it next week, if possible.

I created an experimental branch on my tree for such work, where I'm merging from both Jon's doc-next tree and from media tree at:
https://git.linuxtv.org//mchehab/experimental.git/log/?h=docs-next

Could you please rebase your work with the media DocBook and with the flat-table support to be on the top of it?

Thanks!
Mauro

#23 lore raw

Am 24.06.2016 um 12:40 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 50 lines

Em Tue, 31 May 2016 12:16:25 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 31 lines

Am 30.05.2016 um 23:23 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 29 lines

Em Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <[email protected]> escreveu:

Quoted 9 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series.

With regards to media, my plan is to merge create a topic branch based on Kernel 4.7-rc1 at:
https://git.linuxtv.org/media_tree.git/

As none of the Jani's patches seem to affect the media API docs, it seems I don't need to merge back from Jon's -next branch.

There, I intend to add Markus patches with the conversion from the DocBook to rst, plus the flat-table extension logic.

Then, I'll work to manually fix what's needed and I'll add the automation scripting logic that we have at the DocBook Makefile to work with the new media rst files.

Lastly, once the job's done, I'll drop Documentation/DocBook/media.

Quoted 7 lines
Quoted 3 lines

Markus,

With that regards, could you please send the patches to me?

Yes. What is your timeline ... is it OK if I send you a patch in the next two weeks? ... first I wan't to finish my other work / I'am just back from holiday .. a lot of work to do :-o

Hi Markus,

I'm wanting to start working on it next week, if possible.

I created an experimental branch on my tree for such work, where I'm merging from both Jon's doc-next tree and from media tree at:
https://git.linuxtv.org//mchehab/experimental.git/log/?h=docs-next

Quoted 5 lines

Could you please rebase your work with the media DocBook and with the flat-table support to be on the top of it?

Thanks!
Mauro

Hi Mauro,

sorry for my late reply, I finished the man-page builder last weekend.

Quoted 1 line

I'm merging from both Jon's doc-next tree and from media tree

Currently I'am working on a rebase on top of the Jon's docs-next branch (with "s" in "docs-" / think it was only a typo).

Hopefully I get this done in the next 2 days.

I will send a pull request when I'am finished / thanks for your patience until then.

Regards

--Markus--

#24 lore raw

Em Mon, 27 Jun 2016 08:15:28 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 64 lines

Am 24.06.2016 um 12:40 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 50 lines

Em Tue, 31 May 2016 12:16:25 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 31 lines

Am 30.05.2016 um 23:23 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 29 lines

Em Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <[email protected]> escreveu:

Quoted 9 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series.

With regards to media, my plan is to merge create a topic branch based on Kernel 4.7-rc1 at:
https://git.linuxtv.org/media_tree.git/

As none of the Jani's patches seem to affect the media API docs, it seems I don't need to merge back from Jon's -next branch.

There, I intend to add Markus patches with the conversion from the DocBook to rst, plus the flat-table extension logic.

Then, I'll work to manually fix what's needed and I'll add the automation scripting logic that we have at the DocBook Makefile to work with the new media rst files.

Lastly, once the job's done, I'll drop Documentation/DocBook/media.

Quoted 7 lines
Quoted 3 lines

Markus,

With that regards, could you please send the patches to me?

Yes. What is your timeline ... is it OK if I send you a patch in the next two weeks? ... first I wan't to finish my other work / I'am just back from holiday .. a lot of work to do :-o

Hi Markus,

I'm wanting to start working on it next week, if possible.

I created an experimental branch on my tree for such work, where I'm merging from both Jon's doc-next tree and from media tree at:
https://git.linuxtv.org//mchehab/experimental.git/log/?h=docs-next

Quoted 5 lines

Could you please rebase your work with the media DocBook and with the flat-table support to be on the top of it?

Thanks!
Mauro

Hi Mauro,

sorry for my late reply, I finished the man-page builder last weekend.

Quoted 1 line

I'm merging from both Jon's doc-next tree and from media tree

Quoted 7 lines

Currently I'am working on a rebase on top of the Jon's docs-next branch (with "s" in "docs-" / think it was only a typo).

Hopefully I get this done in the next 2 days.

I will send a pull request when I'am finished / thanks for your patience until then.

Ok!

Thanks for your work!

Regards,
Mauro

#25 lore raw

Am 27.06.2016 um 19:08 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 75 lines

Em Mon, 27 Jun 2016 08:15:28 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 64 lines

Am 24.06.2016 um 12:40 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 50 lines

Em Tue, 31 May 2016 12:16:25 +0200
Markus Heiser <[email protected]> escreveu:

Quoted 31 lines

Am 30.05.2016 um 23:23 schrieb Mauro Carvalho Chehab <[email protected]>:

Quoted 29 lines

Em Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <[email protected]> escreveu:

Quoted 9 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series.

With regards to media, my plan is to merge create a topic branch based on Kernel 4.7-rc1 at:
https://git.linuxtv.org/media_tree.git/

As none of the Jani's patches seem to affect the media API docs, it seems I don't need to merge back from Jon's -next branch.

There, I intend to add Markus patches with the conversion from the DocBook to rst, plus the flat-table extension logic.

Then, I'll work to manually fix what's needed and I'll add the automation scripting logic that we have at the DocBook Makefile to work with the new media rst files.

Lastly, once the job's done, I'll drop Documentation/DocBook/media.

Quoted 7 lines
Quoted 3 lines

Markus,

With that regards, could you please send the patches to me?

Yes. What is your timeline ... is it OK if I send you a patch in the next two weeks? ... first I wan't to finish my other work / I'am just back from holiday .. a lot of work to do :-o

Hi Markus,

I'm wanting to start working on it next week, if possible.

I created an experimental branch on my tree for such work, where I'm merging from both Jon's doc-next tree and from media tree at:
https://git.linuxtv.org//mchehab/experimental.git/log/?h=docs-next

Quoted 5 lines

Could you please rebase your work with the media DocBook and with the flat-table support to be on the top of it?

Thanks!
Mauro

Hi Mauro,

sorry for my late reply, I finished the man-page builder last weekend.

Quoted 1 line

I'm merging from both Jon's doc-next tree and from media tree

Quoted 7 lines

Currently I'am working on a rebase on top of the Jon's docs-next branch (with "s" in "docs-" / think it was only a typo).

Hopefully I get this done in the next 2 days.

I will send a pull request when I'am finished / thanks for your patience until then.

Quoted 6 lines

Ok!

Thanks for your work!

Regards,
Mauro

Hi Mauro,

you could pull it from (see [1]).

https://github.com/return42/linux.git docs-next/linux-doc-reST

I tested it local with your experimental/docs-next and it works. If you don't want to build the whole documentation every time (target htmldocs), use

  $ make books/linux_tv.html
  ... 
  $ make books/linux_tv.clean

May you read the Documentation/books_migrated/README.txt. Since Documentation/books_migrated is only for the period of transition move the linux_tv folder to Documentation/ when you start your review.

If you want the "media" folder back, rename it:

mv Documentation/books_migrated/linux_tv Documentation/media

But take into account, that the target and dist-name is the name of the folder

  $ make books/media.html	
  ...
  SPHINX  books/medi.html --> file:///share/linux-test/Documentation/dist/books/media

BTW, if you do so, we have to modify the ref in Documentation/index.rst

If you build the book, you will see messages like this one:

/share/linux-test/Documentation/books_migrated/linux_tv/media/dvb/FE_GET_EVENT.rst:19: WARNING: Inline emphasis start-string without end-string.

These are marginal issues, ignore them for the first, we fix these in one step later. First, check if you find any *systematic error* and report me .. hopefully there are none, but who knows ..

If you have any problems or questions, regardless with sphinx-doc installation, layout issues, with the reST markup or whatever ... don't hesitate to contact me directly.

One question: is it right, that no man-pages are created from the "media" content? I tried the xmlto on the DocBook-XML media but no man-pages has been created ...

BTW: the man-page build (target books/name.man) is 10^2-10^3 times faster than the xmlto, no loud fan anymore ... but if you don't have man-pages you wan't notice a change ;-)

[1] http://article.gmane.org/gmane.linux.kernel/2255354

-- Markus --

#26 lore raw
In reply to Jani Nikula

Am 30.05.2016 um 22:05 schrieb Jani Nikula <[email protected]>:

Quoted 34 lines · Markus Heiser
Quoted 10 lines

Am 30.05.2016 um 16:46 schrieb Jani Nikula <[email protected]>:

Quoted 7 lines

I am not proposing to merge the documents that I've converted mostly as samples in the branch. I needed something to demonstrate the build is sane.

The authors of the DocBook documents should make the conversions as they see fit, when they see fit, with the tools they see fit, probably with some manual work on top.

OK

To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I propose to merge at this time. There's the Sphinx configuration, kernel build integration, Sphinx kernel-doc extension, tons of kernel-doc updates, etc. There is no DocBook tmpl conversion; all of that is left to the authors (owners, maintainers) of the documents, but this enables them to focus on that part.

I was planning on sending out the patches after some feedback here.

[1] git://people.freedesktop.org/~jani/drm
[2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-for-docs-next

Quoted 4 lines

With DocBook, it was hard to separate a file into small chunks (see media for how it is done). With Sphinx, it is common to split a document in small chunks (along parts, chapters, sections ...) Thats why I recommend chunking documents (from the beginning).

Agreed, but up to the authors.

Quoted 3 lines
Quoted 2 lines

One of the goals was to have nice cross-referencing between the documents (e.g. from GPU to kernel or device driver API). And it works.

Quoted 6 lines
Quoted 1 line

For this, Sphinx-doc brings intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

If the kernel is split to several intersphinx "prefixes", we won't know the prefix of the link targets when we're generating the references in kernel-doc. Also, can be deferred to follow-up work if someone figures out the how.

It stands to reason that each book should be placed in a separate folder. This simple role simplifies much, e.g. chunking, "prefixes" for the intersphinx, a place for images of this book .. etc. It is also the base to have one sphinx-doc project (individual config set) for each book.

Please, place each (DocBook) reST-book at least into a separate folder.

Quoted 5 lines
Quoted 2 lines

I can't recommend to use rst2pdf (it is less maintained), use default sphinx LaTeX toolchain.

I think we'll use whatever works, rst2pdf seemed to work for now, but we can change if needed.

The discussion in the past was dominated by the fear, that something on the sphinx-doc could not be maintained in the future ... I don't share theses fears, if needed, I also have no problem to repair or throw my damaged toolchain away ;-)

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines

I find it totally unacceptable to require explicitly marking kernel-doc comments or source files as being reStructuredText.
Note that it's all opt-in already. If you add a .rst file that includes kernel-doc via the kernel-doc extension, you better make sure the comments parse as reStructuredText and render nicely. I'm willing to do much of the job for all the things that I care about.

Quoted 7 lines

We have a different POV ... I try to build up a documentation project, which could use all given kernel-doc markups without any change, where reST is an "addition". Your approach is to fix kernel-doc comments if they are referred by a kernl-doc directive in a .rst document. There is nothing wrong about your approach, but I try to build a whole source code documentation like the one I started here:
http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

Quoted 2 lines

That looks nice, but I'll argue it would not be much worse even if you assumed it's all rst.

A superficial look on the HTML output may give the impression. But in the log you will find tons of errors and warnings. My experience is, that authors will not consult logs if there are tons of errors from the beginning, which carries a decrease in quality. IMO not a good starting point.

Quoted 5 lines

The bigger point is, if you expect people to tag each source file or kernel-doc comment with "rst", you'll end up with a mess where some places have that tag, some not, but it's not conclusive about whether they actually *are* rst or not. (And you've had tons of patch churn to add those tags to get there.)

At the end, only sources which have been modified to reST need one line (in the first lines) :

/* parse-markup: reST */

which announce the reST markup in this file, normaly this needs no additional patches, except the author forget to announce his movement to reST ...

Quoted 7 lines

The kernel-doc comments are written by humans who will screw it up anyway. (Apologies for the distrust, fellow developers, but I've been reading too many of your fine kernel-doc comments lately.) People will happily cargo cult rst and current kernel-doc and javadoc and doxygen and whatnot in a fruit salad. The only thing that will help in the end is keeping the rules simple and consistent and having the feedback from the tools.

You are right, I have seen tons of individual markups in the kernel-doc comments. In the past some authors ignored the description in the kernel-doc-nano-howto. The "/* parse-markup: reST */" will be only one addition more to the kernel-doc-nano-howto they could ignore ;-)

Quoted 5 lines
Quoted 4 lines

I worry a little bit in that reST will be only one more toolchain beside DocBook .. in the long term, kernel's documentation should get rid of all the DocBook artifacts and for this a more comprehensive solution is needed.

Quoted 8 lines

We agree on the end goal, eradicate DocBook. I must say that in my experiments, apart from the media docs, almost everything converts surprisingly nicely or IMO "good enough" with just the tmplcvt script in this series. Do remember that this is a one time conversion. It needs to be good enough that there's not too much manual editing involved, but it doesn't need to be perfect. Some degree of editing will be required no matter what, not least because the DocBook has also been written by humans, and the battle against the GIGO principle is a lost one.

and I feel like Don Quichotte :-)

-- Markus--

#27 lore raw
Quoted 15 lines · Markus Heiser
Quoted 15 lines
Quoted 14 lines
Quoted 6 lines

I find it totally unacceptable to require explicitly marking kernel-doc comments or source files as being reStructuredText.
Note that it's all opt-in already. If you add a .rst file that includes kernel-doc via the kernel-doc extension, you better make sure the comments parse as reStructuredText and render nicely. I'm willing to do much of the job for all the things that I care about.

We have a different POV ... I try to build up a documentation project, which could use all given kernel-doc markups without any change, where reST is an "addition". Your approach is to fix kernel-doc comments if they are referred by a kernl-doc directive in a .rst document. There is nothing wrong about your approach, but I try to build a whole source code documentation like the one I started here:
http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

Quoted 8 lines
Quoted 2 lines

That looks nice, but I'll argue it would not be much worse even if you assumed it's all rst.

A superficial look on the HTML output may give the impression. But in the log you will find tons of errors and warnings. My experience is, that authors will not consult logs if there are tons of errors from the beginning, which carries a decrease in quality. IMO not a good starting point.

0-day builds all docs, and checks for new warnings. Even in today's gpu.tmpl build there's a massive pile of warnings, so yes developers don't look. But 0-day does, and then developers look at the nice mails from 0-day. It mostly works to keep out new fail I think. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#28 lore raw

Am 31.05.2016 um 10:07 schrieb Daniel Vetter <[email protected]>:

Quoted 26 lines
Quoted 15 lines · Markus Heiser
Quoted 15 lines
Quoted 14 lines
Quoted 6 lines

I find it totally unacceptable to require explicitly marking kernel-doc comments or source files as being reStructuredText.
Note that it's all opt-in already. If you add a .rst file that includes kernel-doc via the kernel-doc extension, you better make sure the comments parse as reStructuredText and render nicely. I'm willing to do much of the job for all the things that I care about.

We have a different POV ... I try to build up a documentation project, which could use all given kernel-doc markups without any change, where reST is an "addition". Your approach is to fix kernel-doc comments if they are referred by a kernl-doc directive in a .rst document. There is nothing wrong about your approach, but I try to build a whole source code documentation like the one I started here:
http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

Quoted 8 lines
Quoted 2 lines

That looks nice, but I'll argue it would not be much worse even if you assumed it's all rst.

A superficial look on the HTML output may give the impression. But in the log you will find tons of errors and warnings. My experience is, that authors will not consult logs if there are tons of errors from the beginning, which carries a decrease in quality. IMO not a good starting point.

Quoted 4 lines

0-day builds all docs, and checks for new warnings. Even in today's gpu.tmpl build there's a massive pile of warnings, so yes developers don't look. But 0-day does, and then developers look at the nice mails from 0-day. It mostly works to keep out new fail I think.

In general, I'am not very happy with workarounds like this. IMO these are workarounds are often, rewards bunglers and punish those with more work, who want make thinks right. There might be situations where 0-day build is the only/best solution. But *here* we are talking about one additional comment line the author adds, when he modify his source comments from kernel-doc to reST markup .. IMO not very hard.

This one line helps the doc-builder to distinguish between *vintage* kernel-doc comments and those with reST additions in.

With the announcement of the markup, we can use all existing kernel-doc markups "as is" for building a complete src-code documentation, with thousands of errors less in the log (experience from my POC). IMO a great benefit ... given by just one additional comment line to distinguish between vintage and reST ...

BTW it's is not uncommon to announce the markup in projects with mixed markups in the source code comments.

A few words about my point of view / my thought:

I strict separate markup from the doc-building tools. The decision in favor for reST is not done because sphinx-doc is a great doc-bulding tool, it is done because reST is a easy to write / read markup with a clear and expressive syntax definition. The *best* builder for this markup is sphinx-doc, so it is only natural, that the decision for the builder falls to sphinx-doc.

But kernel documentation is not a project from scratch, there was the *vintage* kernel-doc markup first. Therefore, reST is only an additional markup!

With this POV you can add reST (or any other) markup to the doc-building process. The Linux kernel draws on diverse projects, some of them may use different markups and these maintainers are not interested in changing there whole markup when they contribute to the kernel ... but may be there is someone how want's to add an other additional markup support to the doc-building process.

I'am not interested in supporting additional markups beside reST. But this is a scalable future-proof solution, which only needs an announcement of the the markup. The one additional comment line we are talking about.

--Markus--

Quoted 5 lines

-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#29 lore raw
Quoted 6 lines · Markus Heiser

Am 31.05.2016 um 10:07 schrieb Daniel Vetter <[email protected]>:

Quoted 4 lines

0-day builds all docs, and checks for new warnings. Even in today's gpu.tmpl build there's a massive pile of warnings, so yes developers don't look. But 0-day does, and then developers look at the nice mails from 0-day. It mostly works to keep out new fail I think.

Quoted 6 lines

In general, I'am not very happy with workarounds like this. IMO these are workarounds are often, rewards bunglers and punish those with more work, who want make thinks right. There might be situations where 0-day build is the only/best solution. But *here* we are talking about one additional comment line the author adds, when he modify his source comments from kernel-doc to reST markup .. IMO not very hard.

That "one line" translates to nearly 50000 kernel-doc comments in more than 6000 files. If you expect people to add a tag in each file/comment, it will never happen. If we assume it's all rst, we can at least start converting.

I quickly wrote a small "kernel-doc-rst-lint" script (70 lines of python) based on rst-lint [1] that runs kernel-doc on a file and reports all the kernel-doc and rst-lint errors in the output. This can be run as a "checker" in the kernel build with

$ make CHECKER=scripts/kernel-doc-rst-lint C=1

and it can provide better and more direct warnings on kernel-doc/rst errors than a full Sphinx build does.

BR,
Jani.

[1] https://pypi.python.org/pypi/restructuredtext_lint

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#30 lore raw

Am 31.05.2016 um 12:30 schrieb Jani Nikula <[email protected]>:

Quoted 14 lines · Markus Heiser
Quoted 6 lines

Am 31.05.2016 um 10:07 schrieb Daniel Vetter <[email protected]>:

Quoted 4 lines

0-day builds all docs, and checks for new warnings. Even in today's gpu.tmpl build there's a massive pile of warnings, so yes developers don't look. But 0-day does, and then developers look at the nice mails from 0-day. It mostly works to keep out new fail I think.

Quoted 6 lines

In general, I'am not very happy with workarounds like this. IMO these are workarounds are often, rewards bunglers and punish those with more work, who want make thinks right. There might be situations where 0-day build is the only/best solution. But *here* we are talking about one additional comment line the author adds, when he modify his source comments from kernel-doc to reST markup .. IMO not very hard.

Quoted 4 lines

That "one line" translates to nearly 50000 kernel-doc comments in more than 6000 files. If you expect people to add a tag in each file/comment, it will never happen. If we assume it's all rst, we can at least start converting.

I have the impression that we misunderstand us ...

You will add this line only to these files where you have changed the markup from *vintage* kerenel-doc to reST. In my solution, you can change the markup on every comment, but you don't have to .. it is enough to add one line at the top of the file.

It's hard to describe something without an example, let my finish my work and after this I can show it by example. Then you will see, that the impact is less then you fear.

Quoted 9 lines

I quickly wrote a small "kernel-doc-rst-lint" script (70 lines of python) based on rst-lint [1] that runs kernel-doc on a file and reports all the kernel-doc and rst-lint errors in the output. This can be run as a "checker" in the kernel build with

$ make CHECKER=scripts/kernel-doc-rst-lint C=1

and it can provide better and more direct warnings on kernel-doc/rst errors than a full Sphinx build does.

I haven't tested [1], but I assume that it covers only docutils-reST not the Sphinx-doc superset (thats might be the reason why you see less errors) ... anyway it could be convenient tool.

--Markus--

PS: I looked closer to [1], it uses the docutils puplischer ..

from docutils.core import Publisher

with a /dev/null like stream

document.reporter.stream = None

The errors you get from this are the same you get from a rst2xxx tool ...

|  >>> import restructuredtext_lint
|  >>> errors = restructuredtext_lint.lint("""
|  ... Hello World
|  ... =======
|  ... 
|  ... :ref:`label_name` 
|  ... """)
|  >>> errors[0].astext()
|  u'None:3: (WARNING/2) Title underline too short.\n\nHello World\n======='
|  >>> errors[1].astext()
|  u'None:5: (INFO/1) No role entry for "ref" in module "docutils.parsers.rst.languages.en".\nTrying "ref" as canonical role name.'
Quoted 11 lines

BR,
Jani.

[1] https://pypi.python.org/pypi/restructuredtext_lint

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
maintainer
#31 lore raw

Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

In reply to Jani Nikula
Quoted 5 lines · On Mon, 30 May 2016 23:05:34 +0300 Jani Nikula <[email protected]> wrote:
Quoted 2 lines

I can't recommend to use rst2pdf (it is less maintained), use default sphinx LaTeX toolchain.

I think we'll use whatever works, rst2pdf seemed to work for now, but we can change if needed.

I really like the idea of using rst2pdf and keeping the huge latex dependency out of the mix. I am a bit concerned, though; I've been able to crash it in my experiments here. We may want to have the ability to support either chain eventually; otherwise, we might just end up picking up maintenance of rst2pdf at some point so that it works properly for us.

jon

#32 lore raw

Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

Am 03.06.2016 um 22:47 schrieb Jonathan Corbet <[email protected]>:

Quoted 9 lines
Quoted 5 lines · On Mon, 30 May 2016 23:05:34 +0300 Jani Nikula <[email protected]> wrote:
Quoted 2 lines

I can't recommend to use rst2pdf (it is less maintained), use default sphinx LaTeX toolchain.

I think we'll use whatever works, rst2pdf seemed to work for now, but we can change if needed.

Quoted 7 lines

I really like the idea of using rst2pdf and keeping the huge latex dependency out of the mix. I am a bit concerned, though; I've been able to crash it in my experiments here. We may want to have the ability to support either chain eventually; otherwise, we might just end up picking up maintenance of rst2pdf at some point so that it works properly for us.

jon

I looked closer to rst2pdf, it supports only the docutils reST, but not the sphinx superset ...

<SNIP rst2pdf>-------------

$ rst2pdf  index.rst
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:27: (ERROR/3) Unknown directive type "toctree".

.. toctree::
:maxdepth: 1

kernel-doc-intro
kernel-doc-syntax
<SNAP>-------------

rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:" are a part of the (extended) reST syntax from sphinx, thats why standard docutils (like rst2*) will not work ...

Quoted 18 lines

Am 18.04.2016 um 10:10 schrieb Markus Heiser <[email protected]>:
Re: Kernel docs: muddying the waters a bit

BTW a few words about differences between DockBook and reST (Sphinx).

With DocBook you write *books*, the protocol (the DocBook application) has no facility to *chunk* and interconnect several documents. The external ENTITY is a workaround on the SGML layer, not on XML nor on the DB-application layer. Thats the reason, why so many XML-tools don't handle this entities and many DocBook to (e.g.) reST tools are fail.

With **standard** reST it is nearly the same, except there is a "include" directive on the application layer. But this directive is very simple, comparable to the C preprocessor "#include" directive.

With the **superset** reST-markup of Sphinx-doc you get a the "toctree" directive, which lets you control how a document-tree should be build.

Quoted 8 lines

http://www.sphinx-doc.org/en/stable/markup/toctree.html

@Mauro: you mentioned a docutils (rst2*) experience in your mail
http://marc.info/?l=linux-doc&m=145735316012094&w=2

Because the "toctree" directive -- and other directives we use -- are a part of a superset of the **standard** reST, the standard docutils (like rst2*) will not work.

-- Markus --

#33 lore raw

Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

Quoted 8 lines · Markus Heiser

I looked closer to rst2pdf, it supports only the docutils reST, but not the sphinx superset ...

<SNIP rst2pdf>-------------

$ rst2pdf  index.rst
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:27: (ERROR/3) Unknown directive type "toctree".

This doesn't actually happen if you run rst2pdf through Sphinx.

Quoted 11 lines

.. toctree::
:maxdepth: 1

kernel-doc-intro
kernel-doc-syntax
<SNAP>-------------

rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:" are a part of the (extended) reST syntax from sphinx, thats why standard docutils (like rst2*) will not work ...

You will get warnings like

[WARNING] styles.py:548 Using undefined style 'function', aliased to style 'normal'. [WARNING] styles.py:548 Using undefined style 'type', aliased to style 'normal'.

but it's a matter of defining a custom rst2pdf stylesheet. It's json with a sort of inheritance model, so it should be easy to just extend the existing stylesheet.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#34 lore raw

Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

Am 07.06.2016 um 08:44 schrieb Jani Nikula <[email protected]>:

Quoted 12 lines · Markus Heiser
Quoted 8 lines

I looked closer to rst2pdf, it supports only the docutils reST, but not the sphinx superset ...

<SNIP rst2pdf>-------------

$ rst2pdf  index.rst
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:27: (ERROR/3) Unknown directive type "toctree".

This doesn't actually happen if you run rst2pdf through Sphinx.

Aaah, thanks for hinting. With this in mind, I think rst2pdf is a good choice and the minor bugs in could be handled. In the long term a XeTeX builder as an alternative to produce layouts with higher quality would be nice.

I (also) added rst2pdf:

https://github.com/return42/linux/commit/d88c4981f86fed85e54ee4d4955b35eb9eaac063

-- M --

Quoted 26 lines
Quoted 11 lines

.. toctree::
:maxdepth: 1

kernel-doc-intro
kernel-doc-syntax
<SNAP>-------------

rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:" are a part of the (extended) reST syntax from sphinx, thats why standard docutils (like rst2*) will not work ...

You will get warnings like

[WARNING] styles.py:548 Using undefined style 'function', aliased to style 'normal'. [WARNING] styles.py:548 Using undefined style 'type', aliased to style 'normal'.

but it's a matter of defining a custom rst2pdf stylesheet. It's json with a sort of inheritance model, so it should be easy to just extend the existing stylesheet.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
maintainer
#35 lore raw
In reply to Jani Nikula
Quoted 4 lines · On Mon, 30 May 2016 23:05:34 +0300 Jani Nikula <[email protected]> wrote:

To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I propose to merge at this time. There's the Sphinx configuration, kernel build integration, Sphinx kernel-doc extension, tons of kernel-doc updates, etc.

OK, I do believe that I am ready to do that. Many, many thanks for doing this work! Please drop me a note when you think that the branch is ready to go.

Quoted 3 lines

There is no DocBook tmpl conversion; all of that is left
to the authors (owners, maintainers) of the documents, but this enables them to focus on that part.

I would like to have some actual documents in there in the near future, so that interested folks have something to look at and start with. Maybe I'll do that myself with some of the docbooks without active maintainers, or even, maybe, some of the .txt files :)

Thanks,

jon

#36 lore raw
Quoted 11 lines · Jonathan Corbet
Quoted 4 lines · On Mon, 30 May 2016 23:05:34 +0300 Jani Nikula <[email protected]> wrote:

To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I propose to merge at this time. There's the Sphinx configuration, kernel build integration, Sphinx kernel-doc extension, tons of kernel-doc updates, etc.

OK, I do believe that I am ready to do that. Many, many thanks for doing this work! Please drop me a note when you think that the branch is ready to go.

I've just fixed the last bug in the line number support. I guess next week Jani will double-check that and then send out the patch bomb for you. We have more ideas and work, but I think this is a very good baseline to get going.

Quoted 8 lines
Quoted 3 lines

There is no DocBook tmpl conversion; all of that is left
to the authors (owners, maintainers) of the documents, but this enables them to focus on that part.

I would like to have some actual documents in there in the near future, so that interested folks have something to look at and start with. Maybe I'll do that myself with some of the docbooks without active maintainers, or even, maybe, some of the .txt files :)

I definitely want to get gpu.tmpl converted for 4.8, but that's probably better done in drm-misc instead of doc-next for coordination with ongoing drm work. I'd also like to unify dma-buffer-sharing.txt with the kerneldoc we have into one .rst. Jani has done all the testing and development on a full conversion of all .tmpl files (except media), so I guess we could convert all the others directly in doc-next.

One bit to think of is how much we want to split things up. For gpu and related documentation I think we'll go with Documentation/gpu/ and then put an overview.rst, drm_kms.rst, drm_kms_helper.rst drm_uapi.rst and so on in there. gpu.tmpl is way too big imo, the only reason we had it all merged is to keep cross-referencing working. But with sphinx that works across source files, and we can split things up more. Many other docs seem fairly small in comparison, so maybe ok to keep as-is.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#37 lore raw
In reply to Jonathan Corbet
Quoted 11 lines · Jonathan Corbet
Quoted 4 lines · On Mon, 30 May 2016 23:05:34 +0300 Jani Nikula <[email protected]> wrote:

To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I propose to merge at this time. There's the Sphinx configuration, kernel build integration, Sphinx kernel-doc extension, tons of kernel-doc updates, etc.

OK, I do believe that I am ready to do that. Many, many thanks for doing this work! Please drop me a note when you think that the branch is ready to go.

In case you missed the patch bomb in your inbox, I think it's ready now. ;)

Quoted 8 lines
Quoted 3 lines

There is no DocBook tmpl conversion; all of that is left
to the authors (owners, maintainers) of the documents, but this enables them to focus on that part.

I would like to have some actual documents in there in the near future, so that interested folks have something to look at and start with. Maybe I'll do that myself with some of the docbooks without active maintainers, or even, maybe, some of the .txt files :)

So we'll be working on gpu.tmpl next and I'll look at Documentation/kernel-doc-nano-HOWTO.txt too.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
maintainer
#38 lore raw
In reply to Daniel Vetter
Quoted 2 lines · On Mon, 30 May 2016 11:10:26 +0200 Daniel Vetter <[email protected]> wrote:

I think next steps is to get this merged into docs-next, with a stable tag, so that I can pull it into drm-misc.

So, I want to take another look at this, which probably will need another day or two before it can happen. First impression, though, is that this is great, so I'm expecting that I'll be applying it.

Not sure about the stable tag, though? It doesn't really seem like stable material?

jon

#39 lore raw
Quoted 6 lines · Jonathan Corbet
Quoted 2 lines · On Mon, 30 May 2016 11:10:26 +0200 Daniel Vetter <[email protected]> wrote:

I think next steps is to get this merged into docs-next, with a stable tag, so that I can pull it into drm-misc.

Quoted 6 lines

So, I want to take another look at this, which probably will need another day or two before it can happen. First impression, though, is that this is great, so I'm expecting that I'll be applying it.

Not sure about the stable tag, though? It doesn't really seem like stable material?

Oh, I meant a git tag of a stable (non-rebasing) branch that I can pull into drm-misc so that we can apply the gpu.tmpl->gpu.rst conversion on top. Not a cc: stable tag. Too many kinds of stable tags I guess ;-)
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
maintainer
#40 lore raw
In reply to Jani Nikula

[So I'm finally trying to get into this for real, hopefully I won't be interrupted too many times...expect a few mails as I catch up.]

Quoted 4 lines · On Fri, 20 May 2016 16:39:31 +0300 Jani Nikula <[email protected]> wrote:

There are a few tradeoffs, of course. First, this requires that the EXPORT_SYMBOL markers are placed immediately after the function being exported, as kernel-doc will only look at one file at a time. This is the recommendation anyway.

As I understand it, the technical reasons that kept some markers in separate files should no longer be relevant, so this is probably OK. It would be nice to have a sense for how many sites need to be fixed.

Quoted 4 lines

Second, we lose support for the !C docproc directive to check that all kernel-doc comments in a file are used. This is probably something we'd like to have back in the future, but at this time I think it's an acceptable tradeoff wrt the gains.

This is maybe a job for a separate tool. A related issue is the (fairly frequent) "oh look, none of the comments in $FILE are being used" realization that seems to happen fairly often. It would be nice to check for that, but that's going to be hard to shoehorn into Sphinx.

jon

#41 lore raw
Quoted 10 lines · Jonathan Corbet
Quoted 5 lines

Second, we lose support for the !C docproc directive to check that all kernel-doc comments in a file are used. This is probably something we'd like to have back in the future, but at this time I think it's an acceptable tradeoff wrt the gains.

This is maybe a job for a separate tool. A related issue is the (fairly frequent) "oh look, none of the comments in $FILE are being used" realization that seems to happen fairly often. It would be nice to check for that, but that's going to be hard to shoehorn into Sphinx.

I think much more valuable would be a tool that checks whether comments are pulled in anywhere (for a given source file), over the entire tree. Very often entire subsystems carg-cult kernel-doc, but never use it in a .tmpl, which means 0-day won't notice, and neither anyone else.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
maintainer
#42 lore raw
Quoted 10 lines · On Fri, 3 Jun 2016 22:24:03 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 4 lines

This is maybe a job for a separate tool. A related issue is the (fairly frequent) "oh look, none of the comments in $FILE are being used" realization that seems to happen fairly often. It would be nice to check for that, but that's going to be hard to shoehorn into Sphinx.

I think much more valuable would be a tool that checks whether comments are pulled in anywhere (for a given source file), over the entire tree. Very often entire subsystems carg-cult kernel-doc, but never use it in a .tmpl, which means 0-day won't notice, and neither anyone else.

That is kind of what I was trying to get at, yes :)

jon

#43 lore raw
Quoted 8 lines · Jonathan Corbet
Quoted 5 lines · On Fri, 3 Jun 2016 22:24:03 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 4 lines

This is maybe a job for a separate tool. A related issue is the (fairly frequent) "oh look, none of the comments in $FILE are being used" realization that seems to happen fairly often. It would be nice to check for that, but that's going to be hard to shoehorn into Sphinx.

Quoted 7 lines
Quoted 5 lines

I think much more valuable would be a tool that checks whether comments are pulled in anywhere (for a given source file), over the entire tree. Very often entire subsystems carg-cult kernel-doc, but never use it in a .tmpl, which means 0-day won't notice, and neither anyone else.

That is kind of what I was trying to get at, yes :)

For the 0-day, I've already got a preliminary kernel-doc-rst-lint script, which can be used as a checker in the kernel build. This would catch all kernel-doc comments, whether they're included to documentation or not, and complain about kernel-doc and rst issues. It won't be as comprehensive as a full Sphinx build in terms of validation, but it'll cover more kernel-doc comments. It should catch the silly errors early.

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#44 lore raw
In reply to Jonathan Corbet
Quoted 6 lines · Jonathan Corbet

[So I'm finally trying to get into this for real, hopefully I won't be interrupted too many times...expect a few mails as I catch up.]

On Fri, 20 May 2016 16:39:31 +0300
Jani Nikula <[email protected]> wrote:

Quoted 8 lines
Quoted 4 lines

There are a few tradeoffs, of course. First, this requires that the EXPORT_SYMBOL markers are placed immediately after the function being exported, as kernel-doc will only look at one file at a time. This is the recommendation anyway.

As I understand it, the technical reasons that kept some markers in separate files should no longer be relevant, so this is probably OK. It would be nice to have a sense for how many sites need to be fixed.

Actually, mostly this is not a problem due to EXPORT_SYMBOL placement, but rather due to kernel-doc comments being placed in header files above function declarations while EXPORT_SYMBOL is where it should be next to the function definition.

I don't think we can force people to move the kernel-doc comments for exported functions from header files next to the function definitions.

The straightforward fix to this is to add an optional filename parameter to the kernel-doc extension :export: argument, to pass additional files to kernel-doc where to look for the EXPORT_SYMBOLs. For example:

.. kernel-doc:: include/drm/foo.h
:export: drivers/gpu/drm/foo/foo.c drivers/gpu/drm/foo/bar.c

This would instruct kernel-doc to extract documentation from include/drm/foo.h for all functions that have been exported using EXPORT_SYMBOL (or _GPL) in include/drm/foo.h, drivers/gpu/drm/foo/foo.c, or drivers/gpu/drm/foo/bar.c.

We have something along these lines in docproc already with the !D directive, so nothing altogether surprising.

If my quick grep-fu serves me right, there are about a thousand exported symbols with kernel-doc comments in the headers. It's a ballpark figure. They come in batches; a small fraction of that many filenames in a fraction of the :export: statements would cover most of them.

Before this fix, the workaround is to name the functions with :functions: argument instead of using :export:.

I'm hoping this is not a blocker for merging the series. If the proposed fix is acceptable, I'll get it done before v4.8.

BR,
Jani.

The ugly greps:

$ git grep "^EXPORT_SYMBOL" | sed 's/^[^(]*(\([a-zA-Z0-9_]*\)).*/\1/' | sort > exports
$ git grep -h -A1 "^/\*\*$" -- *.h | grep -v "^\(/\*\*\|--\)" | sed 's/^ \*[         ]*\([a-zA-Z0-9_][a-zA-Z0-9_]*\).*/\1/;' | sort > comments
$ comm -1 -2 comments exports | wc -l
952
Signature · 1 line
Jani Nikula, Intel Open Source Technology Center