Skip to contentSkip to conversation

Documentation/sphinx follow-up, again

PATCH In review Jani Nikula 2 participants 13 messages Jun 23, 2016

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover Documentation/sphinx follow-up, again

The first 9 patches are just a resend of [1], there's two patches on top here.

Pull details below.

BR,
Jani.

[1] http://mid.gmane.org/[email protected]

The following changes since commit 8569de68e79e94cce6709831edd94accb6942ade:

docs: kernel-doc: Add "example" and "note" to the magic section types (2016-06-09 13:35:05 -0600)

are available in the git repository at:

git://people.freedesktop.org/~jani/drm sphinx-for-docs-next

for you to fetch changes up to ebc88ef05c825024a5d95285459b8c842c095c0f:

Documentation: add top level 'make help' output for Sphinx (2016-06-23 15:11:51 +0300)

----------------------------------------------------------------
Jani Nikula (11):
kernel-doc: remove old debug cruft from dump_section()
kernel-doc: do not warn about duplicate default section names
kernel-doc: add missing semi-colons in option parsing
kernel-doc: abstract filename mapping
kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs
kernel-doc: unify all EXPORT_SYMBOL scanning to one place Documentation/sphinx: remove unnecessary temporary variable Documentation/sphinx: use a more sensible string split in kernel-doc extension Documentation/sphinx: add support for specifying extra export files Documentation/sphinx: drop modindex, we don't have python modules
Documentation: add top level 'make help' output for Sphinx

 Documentation/DocBook/Makefile     |  4 +-
 Documentation/Makefile.sphinx      |  8 ++++
 Documentation/index.rst            |  2 -
 Documentation/sphinx/kernel-doc.py | 18 +++++---
 Makefile                           |  2 +
 scripts/kernel-doc                 | 88 +++++++++++++++++++++++++++-----------
 6 files changed, 87 insertions(+), 35 deletions(-)
Signature · 1 line
2.1.4
#1 lore raw

PATCH v2 1/11 kernel-doc: remove old debug cruft from dump_section()

No functional changes.

---
 scripts/kernel-doc | 3 ---
 1 file changed, 3 deletions(-)
scripts/kernel-doc +0 -3
index 27757c21551a..ac18eb5ed776 100755
@@ -529,21 +529,18 @@ sub dump_section {
     my $contents = join "\n", @_;
 
     if ($name =~ m/$type_param/) {
-#	print STDERR "parameter def '$1' = '$contents'\n";
 	$name = $1;
 	$parameterdescs{$name} = $contents;
 	$sectcheck = $sectcheck . $name . " ";
         $parameterdesc_start_lines{$name} = $new_start_line;
         $new_start_line = 0;
     } elsif ($name eq "@\.\.\.") {
-#	print STDERR "parameter def '...' = '$contents'\n";
 	$name = "...";
 	$parameterdescs{$name} = $contents;
 	$sectcheck = $sectcheck . $name . " ";
         $parameterdesc_start_lines{$name} = $new_start_line;
         $new_start_line = 0;
     } else {
-#	print STDERR "other section '$name' = '$contents'\n";
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
 	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
 	    ++$warnings;
Signature · 1 line
2.1.4
#2 lore raw

PATCH v2 2/11 kernel-doc: do not warn about duplicate default section names

In reply to Jani Nikula

Since

commit 32217761ee9db0215350dfe1ca4e66f312fb8c54

Author: Jani Nikula <[email protected]>
Date:   Sun May 29 09:40:44 2016 +0300

kernel-doc: concatenate contents of colliding sections

we started getting (more) errors on duplicate section names, especially on the default section name "Description":

include/net/mac80211.h:3174: warning: duplicate section name 'Description'

This is usually caused by a slightly unorthodox placement of parameter descriptions, like in the above case, and kernel-doc resetting back to the default section more than once within a kernel-doc comment.

Ignore warnings on the duplicate section name automatically assigned by kernel-doc, and only consider explicitly user assigned duplicate section names an issue.

---
 scripts/kernel-doc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
scripts/kernel-doc +5 -2
index ac18eb5ed776..710615f3a4ff 100755
@@ -542,8 +542,11 @@ sub dump_section {
         $new_start_line = 0;
     } else {
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
-	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
-	    ++$warnings;
+	    # Only warn on user specified duplicate section names.
+	    if ($name ne $section_default) {
+		print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
+		++$warnings;
+	    }
 	    $sections{$name} .= $contents;
 	} else {
 	    $sections{$name} = $contents;
Signature · 1 line
2.1.4
#3 lore raw

PATCH v2 3/11 kernel-doc: add missing semi-colons in option parsing

In reply to Jani Nikula
---
 scripts/kernel-doc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/kernel-doc +2 -2
index 710615f3a4ff..a6f82c812c15 100755
@@ -484,10 +484,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_table{$function} = 1;
     } elsif ($cmd eq "-export") { # only exported symbols
 	$output_selection = OUTPUT_EXPORTED;
-	%function_table = ()
+	%function_table = ();
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$output_selection = OUTPUT_INTERNAL;
-	%function_table = ()
+	%function_table = ();
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
Signature · 1 line
2.1.4
#4 lore raw

PATCH v2 4/11 kernel-doc: abstract filename mapping

In reply to Jani Nikula

Reduce duplication in follow-up work. No functional changes.

---
 scripts/kernel-doc | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)
scripts/kernel-doc +18 -9
index a6f82c812c15..516d95fcefb7 100755
@@ -2730,26 +2730,35 @@ sub local_unescape($) {
 	return $text;
 }
 
-sub process_file($) {
+sub map_filename($) {
     my $file;
-    my $identifier;
-    my $func;
-    my $descr;
-    my $in_purpose = 0;
-    my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
-    my $leading_space;
 
     if (defined($ENV{'SRCTREE'})) {
 	$file = "$ENV{'SRCTREE'}" . "/" . $orig_file;
-    }
-    else {
+    } else {
 	$file = $orig_file;
     }
+
     if (defined($source_map{$file})) {
 	$file = $source_map{$file};
     }
 
+    return $file;
+}
+
+sub process_file($) {
+    my $file;
+    my $identifier;
+    my $func;
+    my $descr;
+    my $in_purpose = 0;
+    my $initial_section_counter = $section_counter;
+    my ($orig_file) = @_;
+    my $leading_space;
+
+    $file = map_filename($orig_file);
+
     if (!open(IN,"<$file")) {
 	print STDERR "Error: Cannot open file $file\n";
 	++$errors;
Signature · 1 line
2.1.4
#5 lore raw

PATCH v2 5/11 kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs

In reply to Jani Nikula

If the kernel-doc comments for functions are not in the same file as the EXPORT_SYMBOL statements, the -export and -internal output selections do not work as expected. This is typically the case when the kernel-doc comments are in header files next to the function declarations and the EXPORT_SYMBOL statements are next to the function definitions in the source files.

Let the user specify additional source files in which to look for the EXPORT_SYMBOLs using the new -export-file FILE option, which may be given multiple times.

The pathological example for this is include/net/mac80211.h, which has all the kernel-doc documentation for the exported functions defined in a plethora of source files net/mac80211/*.c.

---
 scripts/kernel-doc | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)
scripts/kernel-doc +37 -2
index 516d95fcefb7..9708a87c7069 100755
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive):
 Output selection (mutually exclusive):
   -export		Only output documentation for symbols that have been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-			in the same FILE.
+                        in the same FILE or any -export-file FILE.
   -internal		Only output documentation for symbols that have NOT been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-			in the same FILE.
+                        in the same FILE or any -export-file FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -76,6 +76,9 @@ Output selection modifiers:
   -no-doc-sections	Do not output DOC: sections.
   -enable-lineno        Enable output of #define LINENO lines. Only works with
                         reStructuredText format.
+  -export-file FILE     Specify an additional FILE in which to look for
+                        EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL(). To be used with
+                        -export or -internal. May be specified multiple times.
 
 Other parameters:
   -v			Verbose output, more warnings and other information.
@@ -336,6 +339,8 @@ use constant {
 my $output_selection = OUTPUT_ALL;
 my $show_not_found = 0;
 
+my @export_file_list;
+
 my @build_time;
 if (defined($ENV{'KBUILD_BUILD_TIMESTAMP'}) &&
     (my $seconds = `date -d"${ENV{'KBUILD_BUILD_TIMESTAMP'}}" +%s`) ne '') {
@@ -488,6 +493,9 @@ while ($ARGV[0] =~ m/^-(.*)/) {
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$output_selection = OUTPUT_INTERNAL;
 	%function_table = ();
+    } elsif ($cmd eq "-export-file") {
+	my $file = shift @ARGV;
+	push(@export_file_list, $file);
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -2747,6 +2755,25 @@ sub map_filename($) {
     return $file;
 }
 
+sub process_export_file($) {
+    my ($orig_file) = @_;
+    my $file = map_filename($orig_file);
+
+    if (!open(IN,"<$file")) {
+	print STDERR "Error: Cannot open file $file\n";
+	++$errors;
+	return;
+    }
+
+    while (<IN>) {
+	if (/$export_symbol/) {
+	    $function_table{$2} = 1;
+	}
+    }
+
+    close(IN);
+}
+
 sub process_file($) {
     my $file;
     my $identifier;
@@ -3081,6 +3108,14 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
 	close(SOURCE_MAP);
 }
 
+if ($output_selection == OUTPUT_EXPORTED ||
+    $output_selection == OUTPUT_INTERNAL) {
+    foreach (@export_file_list) {
+	chomp;
+	process_export_file($_);
+    }
+}
+
 foreach (@ARGV) {
     chomp;
     process_file($_);
Signature · 1 line
2.1.4
#6 lore raw

PATCH v2 6/11 kernel-doc: unify all EXPORT_SYMBOL scanning to one place

In reply to Jani Nikula

Scan all input files for EXPORT_SYMBOLs along with the explicitly specified export files before actually parsing anything.

---
 scripts/kernel-doc | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)
scripts/kernel-doc +5 -13
index 9708a87c7069..932b3f34ff06 100755
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive):
 Output selection (mutually exclusive):
   -export		Only output documentation for symbols that have been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-                        in the same FILE or any -export-file FILE.
+                        in any input FILE or -export-file FILE.
   -internal		Only output documentation for symbols that have NOT been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-                        in the same FILE or any -export-file FILE.
+                        in any input FILE or -export-file FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -2792,17 +2792,6 @@ sub process_file($) {
 	return;
     }
 
-    # two passes for -export and -internal
-    if ($output_selection == OUTPUT_EXPORTED ||
-	$output_selection == OUTPUT_INTERNAL) {
-	while (<IN>) {
-	    if (/$export_symbol/o) {
-		$function_table{$2} = 1;
-	    }
-	}
-	seek(IN, 0, 0);
-    }
-
     $. = 1;
 
     $section_counter = 0;
@@ -3110,6 +3099,9 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
 
 if ($output_selection == OUTPUT_EXPORTED ||
     $output_selection == OUTPUT_INTERNAL) {
+
+    push(@export_file_list, @ARGV);
+
     foreach (@export_file_list) {
 	chomp;
 	process_export_file($_);
Signature · 1 line
2.1.4
#7 lore raw

PATCH v2 7/11 Documentation/sphinx: remove unnecessary temporary variable

In reply to Jani Nikula

Leftover cruft. No functional changes.

---
 Documentation/sphinx/kernel-doc.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
Documentation/sphinx/kernel-doc.py +1 -2
index 4adfb0e91ecc..2856376cc62c 100644
@@ -59,7 +59,6 @@ class KernelDocDirective(Directive):
         env.note_dependency(os.path.abspath(filename))
 
         tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)
-        source = filename
 
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
@@ -105,7 +104,7 @@ class KernelDocDirective(Directive):
                     lineoffset = int(match.group(1)) - 1
                     # we must eat our comments since the upset the markup
                 else:
-                    result.append(line, source, lineoffset)
+                    result.append(line, filename, lineoffset)
                     lineoffset += 1
 
             node = nodes.section()
Signature · 1 line
2.1.4
#8 lore raw

PATCH v2 8/11 Documentation/sphinx: use a more sensible string split in kernel-doc extension

In reply to Jani Nikula

Using the default str.split doesn't return empty strings like the current version does.

---
 Documentation/sphinx/kernel-doc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/sphinx/kernel-doc.py +1 -1
index 2856376cc62c..d6a76f9a0ba5 100644
@@ -68,7 +68,7 @@ class KernelDocDirective(Directive):
         elif 'doc' in self.options:
             cmd += ['-function', str(self.options.get('doc'))]
         elif 'functions' in self.options:
-            for f in str(self.options.get('functions')).split(' '):
+            for f in str(self.options.get('functions')).split():
                 cmd += ['-function', f]
 
         cmd += [filename]
Signature · 1 line
2.1.4
#9 lore raw

PATCH v2 9/11 Documentation/sphinx: add support for specifying extra export files

In reply to Jani Nikula

Let the user specify file patterns where to look for the EXPORT_SYMBOLs in addition to the file with kernel-doc comments. This is directly based on the -export-file FILE option added to kernel-doc in "kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs", but we extend that with globbing patterns in the Sphinx extension.

The file patterns are added as options to the :export: and :internal:
arguments of the kernel-doc directive. For example, to extract the documentation of exported functions from include/net/mac80211.h:

.. kernel-doc:: include/net/mac80211.h
:export: net/mac80211/*.c

Without the file pattern, no exported functions would be found, as the EXPORT_SYMBOLs are placed in the various source files under net/mac80211.

The matched files are also added as dependencies on the document in Sphinx, as they may affect the output. This is one of the reasons to do the globbing in the Sphinx extension instead of in scripts/kernel-doc.

The file pattern remains optional, and is not needed if the kernel-doc comments and EXPORT_SYMBOLs are placed in the source file passed in as the main argument to the kernel-doc directive. This is the most common case across the kernel source tree.

---
 Documentation/sphinx/kernel-doc.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
Documentation/sphinx/kernel-doc.py +11 -2
index d6a76f9a0ba5..dedb24e3bb7a 100644
@@ -31,6 +31,7 @@ import os
 import subprocess
 import sys
 import re
+import glob
 
 from docutils import nodes, statemachine
 from docutils.statemachine import ViewList
@@ -44,8 +45,8 @@ class KernelDocDirective(Directive):
     option_spec = {
         'doc': directives.unchanged_required,
         'functions': directives.unchanged_required,
-        'export': directives.flag,
-        'internal': directives.flag,
+        'export': directives.unchanged,
+        'internal': directives.unchanged,
     }
     has_content = False
 
@@ -54,6 +55,7 @@ class KernelDocDirective(Directive):
         cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
 
         filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
+        export_file_patterns = []
 
         # Tell sphinx of the dependency
         env.note_dependency(os.path.abspath(filename))
@@ -63,14 +65,21 @@ class KernelDocDirective(Directive):
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
             cmd += ['-export']
+            export_file_patterns = str(self.options.get('export')).split()
         elif 'internal' in self.options:
             cmd += ['-internal']
+            export_file_patterns = str(self.options.get('internal')).split()
         elif 'doc' in self.options:
             cmd += ['-function', str(self.options.get('doc'))]
         elif 'functions' in self.options:
             for f in str(self.options.get('functions')).split():
                 cmd += ['-function', f]
 
+        for pattern in export_file_patterns:
+            for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
+                env.note_dependency(os.path.abspath(f))
+                cmd += ['-export-file', f]
+
         cmd += [filename]
 
         try:
Signature · 1 line
2.1.4
#10 lore raw

PATCH v2 10/11 Documentation/sphinx: drop modindex, we don't have python modules

In reply to Jani Nikula

The modindex is for python modules.

---
 Documentation/index.rst | 2 --
 1 file changed, 2 deletions(-)
Documentation/index.rst +0 -2
index 71a276f34c7f..1139a7ea39ec 100644
@@ -18,6 +18,4 @@ Indices and tables
 ==================
 
 * :ref:`genindex`
-* :ref:`modindex`
 * :ref:`search`
-
Signature · 1 line
2.1.4
#11 lore raw

PATCH v2 11/11 Documentation: add top level 'make help' output for Sphinx

In reply to Jani Nikula

While there's slight overlap with the DocBook help now, this can stay intact when the DocBook help goes away.

---
 Documentation/DocBook/Makefile | 4 ++--
 Documentation/Makefile.sphinx  | 8 ++++++++
 Makefile                       | 2 ++
 3 files changed, 12 insertions(+), 2 deletions(-)
Documentation/DocBook/Makefile +2 -2
index e0c7e1e0590b..496d4295ec38 100644
@@ -218,7 +218,7 @@ silent_gen_xml = :
 ###
 # Help targets as used by the top-level makefile
 dochelp:
-	@echo  ' Linux kernel internal documentation in different formats:'
+	@echo  ' Linux kernel internal documentation in different formats (DocBook):'
 	@echo  '  htmldocs        - HTML'
 	@echo  '  pdfdocs         - PDF'
 	@echo  '  psdocs          - Postscript'
@@ -227,7 +227,7 @@ dochelp:
 	@echo  '  installmandocs  - install man pages generated by mandocs'
 	@echo  '  cleandocs       - clean all generated DocBook files'
 	@echo
-	@echo  'make DOCBOOKS="s1.xml s2.xml" [target] Generate only docs s1.xml s2.xml'
+	@echo  '  make DOCBOOKS="s1.xml s2.xml" [target] Generate only docs s1.xml s2.xml'
 	@echo  '  valid values for DOCBOOKS are: $(DOCBOOKS)'
 
 
Documentation/Makefile.sphinx +8 -0
index addf32309bc3..810c11f0d37f 100644
@@ -60,4 +60,12 @@ installmandocs:
 cleandocs:
 	$(Q)rm -rf $(BUILDDIR)
 
+dochelp:
+	@echo  ' Linux kernel internal documentation in different formats (Sphinx):'
+	@echo  '  htmldocs        - HTML'
+	@echo  '  pdfdocs         - PDF'
+	@echo  '  epubdocs        - EPUB'
+	@echo  '  xmldocs         - XML'
+	@echo  '  cleandocs       - clean all generated files'
+
 endif # HAVE_SPHINX
Makefile +2 -0
index 8908a51b895a..f7f393c40886 100644
@@ -1364,6 +1364,8 @@ help:
 	@$(MAKE) $(build)=$(package-dir) help
 	@echo  ''
 	@echo  'Documentation targets:'
+	@$(MAKE) -f $(srctree)/Documentation/Makefile.sphinx dochelp
+	@echo  ''
 	@$(MAKE) -f $(srctree)/Documentation/DocBook/Makefile dochelp
 	@echo  ''
 	@echo  'Architecture specific targets ($(SRCARCH)):'
Signature · 1 line
2.1.4
maintainer
#12 lore raw
In reply to Jani Nikula
Quoted 2 lines · On Thu, 23 Jun 2016 15:28:00 +0300 Jani Nikula <[email protected]> wrote:

The first 9 patches are just a resend of [1], there's two patches on top here.

OK, these all look good to me. Applied to the docs tree, sorry for the delay.

jon