Documentation/sphinx follow-up
Subsystems (2)
cover Documentation/sphinx follow-up
Hi Jon -
Thanks for merging the main Sphinx series! I greet you with another set of patches on top. ;)
The main things here are reducing the noise on duplicate sections and adding better support for extracting exported symbols when EXPORT_SYMBOLs and kernel-doc are in separate files. And then there are some drive-by cleanups and fixes on top.
I'm still working on rewriting Documentation/kernel-doc-nano-HOWTO.txt to reflect all the changes, but didn't want to let these wait for that.
BR,
Jani.
PS. One wrinkle I spotted is that in the kernel-doc directive, :functions: and :internal: also match structs and other types. This is just kernel-doc the script shining its awesomeness through to the upper layer; I just totally missed this earlier. I added :doc: to make the distinction for documents, even though internally that uses kernel-doc -function parameter as well.
Should we rename :functions: to, mmh, maybe :symbols: (for want of a better word) while we still can, or leave that in as a historical curiosity? I'm not sure there's much value in adding a separate :types:
(or something) for non-functions either? And should :internal: keep returning non-functions as well? Or make that return just functions and have another argument to get the types? Thoughts?
Jani Nikula (9):
kernel-doc: remove old debug cruft from dump_section()
kernel-doc: do not warn about duplicate default section names
kernel-doc: add missing semi-colons in option parsing
kernel-doc: abstract filename mapping
kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs
kernel-doc: unify all EXPORT_SYMBOL scanning to one place Documentation/sphinx: remove unnecessary temporary variable Documentation/sphinx: use a more sensible string split in kernel-doc
extension
Documentation/sphinx: add support for specifying extra export files
Documentation/sphinx/kernel-doc.py | 18 +++++--- scripts/kernel-doc | 88 +++++++++++++++++++++++++++----------- 2 files changed, 75 insertions(+), 31 deletions(-)
Signature · 1 line
2.1.4
PATCH 1/9 kernel-doc: remove old debug cruft from dump_section()
No functional changes.
--- scripts/kernel-doc | 3 --- 1 file changed, 3 deletions(-)
scripts/kernel-doc +0 -3
@@ -529,21 +529,18 @@ sub dump_section { my $contents = join "\n", @_; if ($name =~ m/$type_param/) { -# print STDERR "parameter def '$1' = '$contents'\n"; $name = $1; $parameterdescs{$name} = $contents; $sectcheck = $sectcheck . $name . " "; $parameterdesc_start_lines{$name} = $new_start_line; $new_start_line = 0; } elsif ($name eq "@\.\.\.") { -# print STDERR "parameter def '...' = '$contents'\n"; $name = "..."; $parameterdescs{$name} = $contents; $sectcheck = $sectcheck . $name . " "; $parameterdesc_start_lines{$name} = $new_start_line; $new_start_line = 0; } else { -# print STDERR "other section '$name' = '$contents'\n"; if (defined($sections{$name}) && ($sections{$name} ne "")) { print STDERR "${file}:$.: warning: duplicate section name '$name'\n"; ++$warnings;
Signature · 1 line
2.1.4
PATCH 2/9 kernel-doc: do not warn about duplicate default section names
In reply to Jani NikulaSince
commit 32217761ee9db0215350dfe1ca4e66f312fb8c54
Author: Jani Nikula <[email protected]> Date: Sun May 29 09:40:44 2016 +0300
kernel-doc: concatenate contents of colliding sections
we started getting (more) errors on duplicate section names, especially on the default section name "Description":
include/net/mac80211.h:3174: warning: duplicate section name 'Description'
This is usually caused by a slightly unorthodox placement of parameter descriptions, like in the above case, and kernel-doc resetting back to the default section more than once within a kernel-doc comment.
Ignore warnings on the duplicate section name automatically assigned by kernel-doc, and only consider explicitly user assigned duplicate section names an issue.
--- scripts/kernel-doc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
scripts/kernel-doc +5 -2
@@ -542,8 +542,11 @@ sub dump_section { $new_start_line = 0; } else { if (defined($sections{$name}) && ($sections{$name} ne "")) { - print STDERR "${file}:$.: warning: duplicate section name '$name'\n"; - ++$warnings; + # Only warn on user specified duplicate section names. + if ($name ne $section_default) { + print STDERR "${file}:$.: warning: duplicate section name '$name'\n"; + ++$warnings; + } $sections{$name} .= $contents; } else { $sections{$name} = $contents;
Signature · 1 line
2.1.4
PATCH 3/9 kernel-doc: add missing semi-colons in option parsing
In reply to Jani Nikula--- scripts/kernel-doc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/kernel-doc +2 -2
@@ -484,10 +484,10 @@ while ($ARGV[0] =~ m/^-(.*)/) { $function_table{$function} = 1; } elsif ($cmd eq "-export") { # only exported symbols $output_selection = OUTPUT_EXPORTED; - %function_table = () + %function_table = (); } elsif ($cmd eq "-internal") { # only non-exported symbols $output_selection = OUTPUT_INTERNAL; - %function_table = () + %function_table = (); } elsif ($cmd eq "-v") { $verbose = 1; } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
Signature · 1 line
2.1.4
PATCH 4/9 kernel-doc: abstract filename mapping
In reply to Jani NikulaReduce duplication in follow-up work. No functional changes.
--- scripts/kernel-doc | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-)
scripts/kernel-doc +18 -9
@@ -2730,26 +2730,35 @@ sub local_unescape($) { return $text; } -sub process_file($) { +sub map_filename($) { my $file; - my $identifier; - my $func; - my $descr; - my $in_purpose = 0; - my $initial_section_counter = $section_counter; my ($orig_file) = @_; - my $leading_space; if (defined($ENV{'SRCTREE'})) { $file = "$ENV{'SRCTREE'}" . "/" . $orig_file; - } - else { + } else { $file = $orig_file; } + if (defined($source_map{$file})) { $file = $source_map{$file}; } + return $file; +} + +sub process_file($) { + my $file; + my $identifier; + my $func; + my $descr; + my $in_purpose = 0; + my $initial_section_counter = $section_counter; + my ($orig_file) = @_; + my $leading_space; + + $file = map_filename($orig_file); + if (!open(IN,"<$file")) { print STDERR "Error: Cannot open file $file\n"; ++$errors;
Signature · 1 line
2.1.4
PATCH 5/9 kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs
In reply to Jani NikulaIf the kernel-doc comments for functions are not in the same file as the EXPORT_SYMBOL statements, the -export and -internal output selections do not work as expected. This is typically the case when the kernel-doc comments are in header files next to the function declarations and the EXPORT_SYMBOL statements are next to the function definitions in the source files.
Let the user specify additional source files in which to look for the EXPORT_SYMBOLs using the new -export-file FILE option, which may be given multiple times.
The pathological example for this is include/net/mac80211.h, which has all the kernel-doc documentation for the exported functions defined in a plethora of source files net/mac80211/*.c.
--- scripts/kernel-doc | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-)
scripts/kernel-doc +37 -2
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive): Output selection (mutually exclusive): -export Only output documentation for symbols that have been exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() - in the same FILE. + in the same FILE or any -export-file FILE. -internal Only output documentation for symbols that have NOT been exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() - in the same FILE. + in the same FILE or any -export-file FILE. -function NAME Only output documentation for the given function(s) or DOC: section title(s). All other functions and DOC: sections are ignored. May be specified multiple times.
@@ -76,6 +76,9 @@ Output selection modifiers: -no-doc-sections Do not output DOC: sections. -enable-lineno Enable output of #define LINENO lines. Only works with reStructuredText format. + -export-file FILE Specify an additional FILE in which to look for + EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL(). To be used with + -export or -internal. May be specified multiple times. Other parameters: -v Verbose output, more warnings and other information.
@@ -336,6 +339,8 @@ use constant { my $output_selection = OUTPUT_ALL; my $show_not_found = 0; +my @export_file_list; + my @build_time; if (defined($ENV{'KBUILD_BUILD_TIMESTAMP'}) && (my $seconds = `date -d"${ENV{'KBUILD_BUILD_TIMESTAMP'}}" +%s`) ne '') {
@@ -488,6 +493,9 @@ while ($ARGV[0] =~ m/^-(.*)/) { } elsif ($cmd eq "-internal") { # only non-exported symbols $output_selection = OUTPUT_INTERNAL; %function_table = (); + } elsif ($cmd eq "-export-file") { + my $file = shift @ARGV; + push(@export_file_list, $file); } elsif ($cmd eq "-v") { $verbose = 1; } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -2747,6 +2755,25 @@ sub map_filename($) { return $file; } +sub process_export_file($) { + my ($orig_file) = @_; + my $file = map_filename($orig_file); + + if (!open(IN,"<$file")) { + print STDERR "Error: Cannot open file $file\n"; + ++$errors; + return; + } + + while (<IN>) { + if (/$export_symbol/) { + $function_table{$2} = 1; + } + } + + close(IN); +} + sub process_file($) { my $file; my $identifier;
@@ -3081,6 +3108,14 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) { close(SOURCE_MAP); } +if ($output_selection == OUTPUT_EXPORTED || + $output_selection == OUTPUT_INTERNAL) { + foreach (@export_file_list) { + chomp; + process_export_file($_); + } +} + foreach (@ARGV) { chomp; process_file($_);
Signature · 1 line
2.1.4
PATCH 6/9 kernel-doc: unify all EXPORT_SYMBOL scanning to one place
In reply to Jani NikulaScan all input files for EXPORT_SYMBOLs along with the explicitly specified export files before actually parsing anything.
--- scripts/kernel-doc | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-)
scripts/kernel-doc +5 -13
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive): Output selection (mutually exclusive): -export Only output documentation for symbols that have been exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() - in the same FILE or any -export-file FILE. + in any input FILE or -export-file FILE. -internal Only output documentation for symbols that have NOT been exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() - in the same FILE or any -export-file FILE. + in any input FILE or -export-file FILE. -function NAME Only output documentation for the given function(s) or DOC: section title(s). All other functions and DOC: sections are ignored. May be specified multiple times.
@@ -2792,17 +2792,6 @@ sub process_file($) { return; } - # two passes for -export and -internal - if ($output_selection == OUTPUT_EXPORTED || - $output_selection == OUTPUT_INTERNAL) { - while (<IN>) { - if (/$export_symbol/o) { - $function_table{$2} = 1; - } - } - seek(IN, 0, 0); - } - $. = 1; $section_counter = 0;
@@ -3110,6 +3099,9 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) { if ($output_selection == OUTPUT_EXPORTED || $output_selection == OUTPUT_INTERNAL) { + + push(@export_file_list, @ARGV); + foreach (@export_file_list) { chomp; process_export_file($_);
Signature · 1 line
2.1.4
PATCH 7/9 Documentation/sphinx: remove unnecessary temporary variable
In reply to Jani NikulaLeftover cruft. No functional changes.
--- Documentation/sphinx/kernel-doc.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
Documentation/sphinx/kernel-doc.py +1 -2
@@ -59,7 +59,6 @@ class KernelDocDirective(Directive): env.note_dependency(os.path.abspath(filename)) tab_width = self.options.get('tab-width', self.state.document.settings.tab_width) - source = filename # FIXME: make this nicer and more robust against errors if 'export' in self.options:
@@ -105,7 +104,7 @@ class KernelDocDirective(Directive): lineoffset = int(match.group(1)) - 1 # we must eat our comments since the upset the markup else: - result.append(line, source, lineoffset) + result.append(line, filename, lineoffset) lineoffset += 1 node = nodes.section()
Signature · 1 line
2.1.4
PATCH 8/9 Documentation/sphinx: use a more sensible string split in kernel-doc extension
In reply to Jani NikulaUsing the default str.split doesn't return empty strings like the current version does.
--- Documentation/sphinx/kernel-doc.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/sphinx/kernel-doc.py +1 -1
@@ -68,7 +68,7 @@ class KernelDocDirective(Directive): elif 'doc' in self.options: cmd += ['-function', str(self.options.get('doc'))] elif 'functions' in self.options: - for f in str(self.options.get('functions')).split(' '): + for f in str(self.options.get('functions')).split(): cmd += ['-function', f] cmd += [filename]
Signature · 1 line
2.1.4
PATCH 9/9 Documentation/sphinx: add support for specifying extra export files
In reply to Jani NikulaLet the user specify file patterns where to look for the EXPORT_SYMBOLs in addition to the file with kernel-doc comments. This is directly based on the -export-file FILE option added to kernel-doc in "kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs", but we extend that with globbing patterns in the Sphinx extension.
The file patterns are added as options to the :export: and :internal:
arguments of the kernel-doc directive. For example, to extract the documentation of exported functions from include/net/mac80211.h:
.. kernel-doc:: include/net/mac80211.h
:export: net/mac80211/*.c
Without the file pattern, no exported functions would be found, as the EXPORT_SYMBOLs are placed in the various source files under net/mac80211.
The matched files are also added as dependencies on the document in Sphinx, as they may affect the output. This is one of the reasons to do the globbing in the Sphinx extension instead of in scripts/kernel-doc.
The file pattern remains optional, and is not needed if the kernel-doc comments and EXPORT_SYMBOLs are placed in the source file passed in as the main argument to the kernel-doc directive. This is the most common case across the kernel source tree.
--- Documentation/sphinx/kernel-doc.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
Documentation/sphinx/kernel-doc.py +11 -2
@@ -31,6 +31,7 @@ import os import subprocess import sys import re +import glob from docutils import nodes, statemachine from docutils.statemachine import ViewList
@@ -44,8 +45,8 @@ class KernelDocDirective(Directive): option_spec = { 'doc': directives.unchanged_required, 'functions': directives.unchanged_required, - 'export': directives.flag, - 'internal': directives.flag, + 'export': directives.unchanged, + 'internal': directives.unchanged, } has_content = False
@@ -54,6 +55,7 @@ class KernelDocDirective(Directive): cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno'] filename = env.config.kerneldoc_srctree + '/' + self.arguments[0] + export_file_patterns = [] # Tell sphinx of the dependency env.note_dependency(os.path.abspath(filename))
@@ -63,14 +65,21 @@ class KernelDocDirective(Directive): # FIXME: make this nicer and more robust against errors if 'export' in self.options: cmd += ['-export'] + export_file_patterns = str(self.options.get('export')).split() elif 'internal' in self.options: cmd += ['-internal'] + export_file_patterns = str(self.options.get('internal')).split() elif 'doc' in self.options: cmd += ['-function', str(self.options.get('doc'))] elif 'functions' in self.options: for f in str(self.options.get('functions')).split(): cmd += ['-function', f] + for pattern in export_file_patterns: + for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern): + env.note_dependency(os.path.abspath(f)) + cmd += ['-export-file', f] + cmd += [filename] try:
Signature · 1 line
2.1.4