Add Documentation/features to admin-guide and x86
Subsystems (2)
PATCH 2/4 scripts/get_feat.pl: handle ".." special case
In reply to Mauro Carvalho ChehabThe status ".." Means that the feature can't be implemented on a given architecture.
The problem is that this doesn't show anything at the output, so replace it by "---", with is a markup for a long hyphen.
--- scripts/get_feat.pl | 2 ++ 1 file changed, 2 insertions(+)
scripts/get_feat.pl +2 -0
@@ -141,6 +141,8 @@ sub parse_feat { $max_size_arch = length($a); } + $status = "---" if ($status =~ m/^\.\.$/); + $archs{$a} = 1; $arch_table{$a} = $status; next;
Signature · 1 line
2.21.0
PATCH 1/4 scripts/get_feat.pl: add a script to handle Documentation/features
In reply to Mauro Carvalho ChehabThe Documentation/features contains a set of parseable files. It is not worth converting them to ReST format, as they're useful the way it is. It is, however, interesting to parse them and produce output on different formats:
1) Output the contents of a feature in ReST format;
2) Output what features a given architecture supports;
3) Output a matrix with features x architectures.
--- scripts/get_feat.pl | 472 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 472 insertions(+) create mode 100755 scripts/get_feat.pl
scripts/get_feat.pl new file +472 -0
@@ -0,0 +1,472 @@ +#!/usr/bin/perl +# SPDX-License-Identifier: GPL-2.0 + +use strict; +use Pod::Usage; +use Getopt::Long; +use File::Find; +use Fcntl ':mode'; + +my $help; +my $man; +my $debug; +my $arch; +my $feat; +my $prefix="Documentation/features"; + +GetOptions( + "debug|d+" => \$debug, + "dir=s" => \$prefix, + 'help|?' => \$help, + 'arch=s' => \$arch, + 'feat=s' => \$feat, + man => \$man +) or pod2usage(2); + +pod2usage(1) if $help; +pod2usage(-exitstatus => 0, -verbose => 2) if $man; + +pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2); + +my ($cmd, $arg) = @ARGV; + +pod2usage(2) if ($cmd ne "current" && $cmd ne "rest" && $cmd ne "validate"); + +require Data::Dumper if ($debug); + +my %data; +my %archs; + +# +# Displays an error message, printing file name and line +# +sub parse_error($$$$) { + my ($file, $ln, $msg, $data) = @_; + + $data =~ s/\s+$/\n/; + + print STDERR "Warning: file $file#$ln:\n\t$msg"; + + if ($data ne "") { + print STDERR ". Line\n\t\t$data"; + } else { + print STDERR "\n"; + } +} + +# +# Parse a features file, storing its contents at %data +# + +my $h_name = "Feature"; +my $h_kconfig = "Kconfig"; +my $h_description = "Description"; +my $h_subsys = "Subsystem"; +my $h_status = "Status"; +my $h_arch = "Architecture"; + +my $max_size_name = length($h_name); +my $max_size_kconfig = length($h_kconfig); +my $max_size_description = length($h_description); +my $max_size_subsys = length($h_subsys); +my $max_size_status = length($h_status); +my $max_size_arch = length($h_arch); + +sub parse_feat { + my $file = $File::Find::name; + + my $mode = (stat($file))[2]; + return if ($mode & S_IFDIR); + return if ($file =~ m,($prefix)/arch-support.txt,); + return if (!($file =~ m,arch-support.txt$,)); + + my $subsys = ""; + $subsys = $2 if ( m,.*($prefix)/([^/]+).*,); + + if (length($subsys) > $max_size_subsys) { + $max_size_subsys = length($subsys); + } + + my $name; + my $kconfig; + my $description; + my $comments = ""; + my $last_status; + my $ln; + my %arch_table; + + print STDERR "Opening $file\n" if ($debug > 1); + open IN, $file; + + while(<IN>) { + $ln++; + + if (m/^\#\s+Feature\s+name:\s*(.*\S)/) { + $name = $1; + if (length($name) > $max_size_name) { + $max_size_name = length($name); + } + next; + } + if (m/^\#\s+Kconfig:\s*(.*\S)/) { + $kconfig = $1; + if (length($kconfig) > $max_size_kconfig) { + $max_size_kconfig = length($kconfig); + } + next; + } + if (m/^\#\s+description:\s*(.*\S)/) { + $description = $1; + if (length($description) > $max_size_description) { + $max_size_description = length($description); + } + next; + } + next if (m/^\\s*$/); + next if (m/^\s*\-+\s*$/); + next if (m/^\s*\|\s*arch\s*\|\s*status\s*\|\s*$/); + + if (m/^\#\s*(.*)/) { + $comments .= "$1\n"; + next; + } + if (m/^\s*\|\s*(\S+):\s*\|\s*(\S+)\s*\|\s*$/) { + my $a = $1; + my $status = $2; + + if (length($status) > $max_size_status) { + $max_size_status = length($status); + } + if (length($a) > $max_size_arch) { + $max_size_arch = length($a); + } + + $archs{$a} = 1; + $arch_table{$a} = $status; + next; + } + + #Everything else is an error + parse_error($file, $ln, "line is invalid", $_); + } + close IN; + + if (!$name) { + parse_error($file, $ln, "Feature name not found", ""); + return; + } + + parse_error($file, $ln, "Subsystem not found", "") if (!$subsys); + parse_error($file, $ln, "Kconfig not found", "") if (!$kconfig); + parse_error($file, $ln, "Description not found", "") if (!$description); + + if (!%arch_table) { + parse_error($file, $ln, "Architecture table not found", ""); + return; + } + + $data{$name}->{where} = $file; + $data{$name}->{subsys} = $subsys; + $data{$name}->{kconfig} = $kconfig; + $data{$name}->{description} = $description; + $data{$name}->{comments} = $comments; + $data{$name}->{table} = \%arch_table; +} + +# +# Output feature(s) for a given architecture +# +sub output_arch_table { + my $title = "Feature status on $arch architecture"; + + print "=" x length($title) . "\n"; + print "$title\n"; + print "=" x length($title) . "\n\n"; + + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_status; + print " "; + print "=" x $max_size_description; + print "\n"; + printf "%-${max_size_subsys}s ", $h_subsys; + printf "%-${max_size_name}s ", $h_name; + printf "%-${max_size_kconfig}s ", $h_kconfig; + printf "%-${max_size_status}s ", $h_status; + printf "%-${max_size_description}s\n", $h_description; + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_status; + print " "; + print "=" x $max_size_description; + print "\n"; + + foreach my $name (sort { + ($data{$a}->{subsys} cmp $data{$b}->{subsys}) || + ($data{$a}->{name} cmp $data{$b}->{name}) + } keys %data) { + next if ($feat && $name ne $feat); + + my %arch_table = %{$data{$name}->{table}}; + printf "%-${max_size_subsys}s ", $data{$name}->{subsys}; + printf "%-${max_size_name}s ", $name; + printf "%-${max_size_kconfig}s ", $data{$name}->{kconfig}; + printf "%-${max_size_status}s ", $arch_table{$arch}; + printf "%-${max_size_description}s\n", $data{$name}->{description}; + } + + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_status; + print " "; + print "=" x $max_size_description; + print "\n"; +} + +# +# Output a feature on all architectures +# +sub output_feature { + my $title = "Feature $feat"; + + print "=" x length($title) . "\n"; + print "$title\n"; + print "=" x length($title) . "\n\n"; + + print ":Subsystem: $data{$feat}->{subsys} \n" if ($data{$feat}->{subsys}); + print ":Kconfig: $data{$feat}->{kconfig} \n" if ($data{$feat}->{kconfig}); + + my $desc = $data{$feat}->{description}; + $desc =~ s/^([a-z])/\U$1/; + $desc =~ s/\.?\s*//; + print "\n$desc.\n\n"; + + my $com = $data{$feat}->{comments}; + $com =~ s/^\s+//; + $com =~ s/\s+$//; + if ($com) { + print "Comments\n"; + print "--------\n\n"; + print "$com\n\n"; + } + + print "=" x $max_size_arch; + print " "; + print "=" x $max_size_status; + print "\n"; + + printf "%-${max_size_arch}s ", $h_arch; + printf "%-${max_size_status}s", $h_status . "\n"; + + print "=" x $max_size_arch; + print " "; + print "=" x $max_size_status; + print "\n"; + + my %arch_table = %{$data{$feat}->{table}}; + foreach my $arch (sort keys %arch_table) { + printf "%-${max_size_arch}s ", $arch; + printf "%-${max_size_status}s\n", $arch_table{$arch}; + } + + print "=" x $max_size_arch; + print " "; + print "=" x $max_size_status; + print "\n"; +} + +# +# Output all features for all architectures +# + +sub matrix_lines { + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + + foreach my $arch (sort keys %archs) { + my $len = $max_size_status; + + $len = length($arch) if ($len < length($arch)); + + print "=" x $len; + print " "; + } + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_description; + print "\n"; +} + +sub output_matrix { + + my $title = "Feature List (feature x architecture)"; + + print "=" x length($title) . "\n"; + print "$title\n"; + print "=" x length($title) . "\n\n"; + + matrix_lines; + + printf "%-${max_size_subsys}s ", $h_subsys; + printf "%-${max_size_name}s ", $h_name; + + foreach my $arch (sort keys %archs) { + printf "%-${max_size_status}s ", $arch; + } + printf "%-${max_size_kconfig}s ", $h_kconfig; + printf "%-${max_size_description}s\n", $h_description; + + matrix_lines; + + foreach my $name (sort { + ($data{$a}->{subsys} cmp $data{$b}->{subsys}) || + ($data{$a}->{name} cmp $data{$b}->{name}) + } keys %data) { + printf "%-${max_size_subsys}s ", $data{$name}->{subsys}; + printf "%-${max_size_name}s ", $name; + + my %arch_table = %{$data{$name}->{table}}; + + foreach my $arch (sort keys %arch_table) { + my $len = $max_size_status; + + $len = length($arch) if ($len < length($arch)); + + printf "%-${len}s ", $arch_table{$arch}; + } + printf "%-${max_size_kconfig}s ", $data{$name}->{kconfig}; + printf "%-${max_size_description}s\n", $data{$name}->{description}; + } + + matrix_lines; +} + + +# +# Parses all feature files located at $prefix dir +# +find({wanted =>\&parse_feat, no_chdir => 1}, $prefix); + +print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug); + +# +# Handles the command +# +if ($cmd eq "current") { + $arch = qx(uname -m | sed 's/x86_64/x86/' | sed 's/i386/x86/'); + $arch =~s/\s+$//; +} + +if ($cmd ne "validate") { + if ($arch) { + output_arch_table; + } elsif ($feat) { + output_feature; + } else { + output_matrix; + } +} + +__END__ + +=head1 NAME + +get_feat.pl - parse the Linux Feature files and produce a ReST book. + +=head1 SYNOPSIS + +B<get_feat.pl> [--debug] [--man] [--help] [--dir=<dir>] + [--arch=<arch>] [--feat=<feature>] <COMAND> [<ARGUMENT>] + +Where <COMMAND> can be: + +=over 8 + +B<current> - output features for this machine's architecture + +B<rest> - output features in ReST markup language + +B<validate> - validate the feature contents + +=back + +=head1 OPTIONS + +=over 8 + +=item B<--arch> + +Output features for an specific architecture, optionally filtering for +a single specific feature. + +=item B<--feat> + +Output features for a single specific architecture. + +=item B<--dir> + +Changes the location of the Feature files. By default, it uses +the Documentation/features directory. + +=item B<--debug> + +Put the script in verbose mode, useful for debugging. Can be called multiple +times, to increase verbosity. + +=item B<--help> + +Prints a brief help message and exits. + +=item B<--man> + +Prints the manual page and exits. + +=back + +=head1 DESCRIPTION + +Parse the Linux feature files from Documentation/features (by default), +optionally producing results at ReST format. + +It supports output data per architecture, per feature or a +feature x arch matrix. + +When used with B<rest> command, it will use either one of the tree formats: + +If neither B<--arch> or B<--feature> arguments are used, it will output a +matrix with features per architecture. + +If B<--arch> argument is used, it will output the features availability for +a given architecture. + +If B<--feat> argument is used, it will output the content of the feature +file using ReStructured Text markup. + +=head1 BUGS + +Report bugs to Mauro Carvalho Chehab <[email protected]> + +=head1 COPYRIGHT + +Copyright (c) 2019 by Mauro Carvalho Chehab <[email protected]>. + +License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>. + +This is free software: you are free to change and redistribute it. +There is NO WARRANTY, to the extent permitted by law. + +=cut
Signature · 1 line
2.21.0
PATCH 4/4 docs: admin-guide, x86: add a features list
In reply to Mauro Carvalho ChehabAdd a feature list matrix at the admin-guide and a x86-specific feature list to the respective Kernel books.
--- Documentation/admin-guide/features.rst | 3 +++ Documentation/admin-guide/index.rst | 1 + Documentation/conf.py | 2 +- Documentation/x86/features.rst | 3 +++ Documentation/x86/index.rst | 1 + 5 files changed, 9 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/features.rst create mode 100644 Documentation/x86/features.rst
Documentation/admin-guide/features.rst new file +3 -0
@@ -0,0 +1,3 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. kernel-feat:: $srctree/Documentation/features
Documentation/admin-guide/index.rst +1 -0
@@ -17,6 +17,7 @@ etc. kernel-parameters devices abi + features This section describes CPU vulnerabilities and their mitigations.
Documentation/conf.py +1 -1
@@ -34,7 +34,7 @@ needs_sphinx = '1.3' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig', 'kernel_abi'] +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig', 'kernel_abi', 'kernel_feat'] # The name of the math extension changed on Sphinx 1.4 if (major == 1 and minor > 3) or (major > 1):
Documentation/x86/features.rst new file +3 -0
@@ -0,0 +1,3 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. kernel-feat:: $srctree/Documentation/features x86
Documentation/x86/index.rst +1 -0
@@ -29,3 +29,4 @@ x86-specific Documentation usb-legacy-support i386/index x86_64/index + features
Signature · 1 line
2.21.0
PATCH 3/4 sphinx/kernel_feat.py: add a script to parse feature files
In reply to Mauro Carvalho ChehabThe feature files have a special well-defined format. Add a script that parses them, allowing to search for a feature and/or by an architecture and to produce ReST-compatible outputs.
--- Documentation/sphinx/kernel_feat.py | 169 ++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 Documentation/sphinx/kernel_feat.py
Documentation/sphinx/kernel_feat.py new file +169 -0
@@ -0,0 +1,169 @@ +# coding=utf-8 +# SPDX-License-Identifier: GPL-2.0 +# +u""" + kernel-feat + ~~~~~~~~~~~ + + Implementation of the ``kernel-feat`` reST-directive. + + :copyright: Copyright (C) 2016 Markus Heiser + :copyright: Copyright (C) 2016-2019 Mauro Carvalho Chehab + :maintained-by: Mauro Carvalho Chehab <[email protected]> + :license: GPL Version 2, June 1991 see Linux/COPYING for details. + + The ``kernel-feat`` (:py:class:`KernelFeat`) directive calls the + scripts/get_feat.pl script to parse the Kernel ABI files. + + Overview of directive's argument and options. + + .. code-block:: rst + + .. kernel-feat:: <ABI directory location> + :debug: + + The argument ``<ABI directory location>`` is required. It contains the + location of the ABI files to be parsed. + + ``debug`` + Inserts a code-block with the *raw* reST. Sometimes it is helpful to see + what reST is generated. + +""" + +import codecs +import os +import subprocess +import sys + +from os import path + +from docutils import nodes, statemachine +from docutils.statemachine import ViewList +from docutils.parsers.rst import directives, Directive +from docutils.utils.error_reporting import ErrorString + +# +# AutodocReporter is only good up to Sphinx 1.7 +# +import sphinx + +Use_SSI = sphinx.__version__[:3] >= '1.7' +if Use_SSI: + from sphinx.util.docutils import switch_source_input +else: + from sphinx.ext.autodoc import AutodocReporter + +__version__ = '1.0' + +def setup(app): + + app.add_directive("kernel-feat", KernelFeat) + return dict( + version = __version__ + , parallel_read_safe = True + , parallel_write_safe = True + ) + +class KernelFeat(Directive): + + u"""KernelFeat (``kernel-feat``) directive""" + + required_arguments = 1 + optional_arguments = 2 + has_content = False + final_argument_whitespace = True + + option_spec = { + "debug" : directives.flag + } + + def warn(self, message, **replace): + replace["fname"] = self.state.document.current_source + replace["line_no"] = replace.get("line_no", self.lineno) + message = ("%(fname)s:%(line_no)s: [kernel-feat WARN] : " + message) % replace + self.state.document.settings.env.app.warn(message, prefix="") + + def run(self): + + doc = self.state.document + if not doc.settings.file_insertion_enabled: + raise self.warning("docutils: file insertion disabled") + + env = doc.settings.env + cwd = path.dirname(doc.current_source) + cmd = "get_feat.pl rest --dir " + cmd += self.arguments[0] + + if len(self.arguments) > 1: + cmd += " --arch " + self.arguments[1] + + srctree = path.abspath(os.environ["srctree"]) + + fname = cmd + + # extend PATH with $(srctree)/scripts + path_env = os.pathsep.join([ + srctree + os.sep + "scripts", + os.environ["PATH"] + ]) + shell_env = os.environ.copy() + shell_env["PATH"] = path_env + shell_env["srctree"] = srctree + + lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env) + nodeList = self.nestedParse(lines, fname) + return nodeList + + def runCmd(self, cmd, **kwargs): + u"""Run command ``cmd`` and return it's stdout as unicode.""" + + try: + proc = subprocess.Popen( + cmd + , stdout = subprocess.PIPE + , stderr = subprocess.PIPE + , **kwargs + ) + out, err = proc.communicate() + + out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8') + + if proc.returncode != 0: + raise self.severe( + u"command '%s' failed with return code %d" + % (cmd, proc.returncode) + ) + except OSError as exc: + raise self.severe(u"problems with '%s' directive: %s." + % (self.name, ErrorString(exc))) + return out + + def nestedParse(self, lines, fname): + content = ViewList() + node = nodes.section() + + if "debug" in self.options: + code_block = "\n\n.. code-block:: rst\n :linenos:\n" + for l in lines.split("\n"): + code_block += "\n " + l + lines = code_block + "\n\n" + + for c, l in enumerate(lines.split("\n")): + content.append(l, fname, c) + + buf = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter + + if Use_SSI: + with switch_source_input(self.state, content): + self.state.nested_parse(content, 0, node, match_titles=1) + else: + self.state.memo.title_styles = [] + self.state.memo.section_level = 0 + self.state.memo.reporter = AutodocReporter(content, self.state.memo.reporter) + try: + self.state.nested_parse(content, 0, node, match_titles=1) + finally: + self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf + + return node.children
Signature · 1 line
2.21.0
cover Add Documentation/features to admin-guide and x86
Jon, Greg & all:
Not sure what tree this would fit better.
The first 2 patches on this series reimplement the logic at:
Documentation/features/list-arch.sh
I opted to not remove the old script, as someone could depend on its specific format. The new script does the same with:
./scripts/get_feat.pl current
The difference is that it outputs with using ascii table artwork.
The way it works is that it parse all feature files and produce 3 different types of contents, depending on its arguments:
- a feature x arch matrix:
./scripts/get_feat.pl rest
- a per-architecture feature table:
./scripts/get_feat.pl current or ./scripts/get_feat.pl rest --arch=arm64
- a per-feature table:
./scripts/get_feat.pl rest --feat=perf-regs
All outputs are compatible with the ReST format.
Patch 3 adds a new Sphinx plugin with handles its output.
Patch 4 adds the feature x arch matrix at the admin-guide.
Patch 4 also adds the features supported on x86 at the x86 arch guide.
IMHO, it makes sense to have a similar table on all other architectures, but the best is to wait for the next Kernel version, in order to see what arch conversion files got included.
So far, I didn't add the per-feature table anywhere.
Mauro Carvalho Chehab (4):
scripts/get_feat.pl: add a script to handle Documentation/features scripts/get_feat.pl: handle ".." special case sphinx/kernel_feat.py: add a script to parse feature files
docs: admin-guide, x86: add a features list
Documentation/admin-guide/features.rst | 3 + Documentation/admin-guide/index.rst | 1 + Documentation/conf.py | 2 +- Documentation/sphinx/kernel_feat.py | 169 +++++++++ Documentation/x86/features.rst | 3 + Documentation/x86/index.rst | 1 + scripts/get_feat.pl | 474 +++++++++++++++++++++++++ 7 files changed, 652 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/features.rst create mode 100644 Documentation/sphinx/kernel_feat.py create mode 100644 Documentation/x86/features.rst create mode 100755 scripts/get_feat.pl
Signature · 1 line
2.21.0