Improvements to the documentation build system
Subsystems (2)
PATCH 8/10 scripts/documentation-file-ref-check: teach about .txt -> .yaml renames
In reply to Mauro Carvalho ChehabAt DT, files are being renamed to jason. Teach the script how to handle such renames when used in fix mode.
--- scripts/documentation-file-ref-check | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
scripts/documentation-file-ref-check +14 -5
@@ -165,13 +165,22 @@ foreach my $ref (keys %broken_ref) { # usual reason for breakage: DT file moved around if ($ref =~ /devicetree/) { - my $search = $new; - $search =~ s,^.*/,,; - $f = qx(find Documentation/devicetree/ -iname "*$search*") if ($search); + # usual reason for breakage: DT file renamed to .yaml if (!$f) { - # Manufacturer name may have changed - $search =~ s/^.*,//; + my $new_ref = $ref; + $new_ref =~ s/\.txt$/.yaml/; + $f=$new_ref if (-f $new_ref); + } + + if (!$f) { + my $search = $new; + $search =~ s,^.*/,,; $f = qx(find Documentation/devicetree/ -iname "*$search*") if ($search); + if (!$f) { + # Manufacturer name may have changed + $search =~ s/^.*,//; + $f = qx(find Documentation/devicetree/ -iname "*$search*") if ($search); + } } }
Signature · 1 line
2.21.0
PATCH 7/10 scripts/documentation-file-ref-check: improve tools ref handling
In reply to Mauro Carvalho ChehabThere's a false positive on perf/util:
tools/perf/util/s390-cpumsf.c: Documentation/perf.data-file-format.txt
The file is there at tools/perf/Documentation/, but the logic with detects relative documentation references inside tools is not capable of detecting it.
So, improve it.
--- scripts/documentation-file-ref-check | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/documentation-file-ref-check +1 -1
@@ -127,7 +127,7 @@ while (<IN>) { if ($f =~ m/tools/) { my $path = $f; $path =~ s,(.*)/.*,$1,; - next if (grep -e, glob("$path/$ref $path/$fulref")); + next if (grep -e, glob("$path/$ref $path/../$ref $path/$fulref")); } # Discard known false-positives
Signature · 1 line
2.21.0
PATCH 6/10 scripts/documentation-file-ref-check: exclude false-positives
In reply to Mauro Carvalho ChehabThere are at least two cases where a documentation file was gone for good, but the text still mentions it:
1) drivers/vhost/vhost.c:
the reference for Documentation/virtual/lguest/lguest.c is just to give credits to the original work that vhost replaced;
2) Documentation/scsi/scsi_mid_low_api.txt:
It gives credit and mentions the old Documentation/Configure.help file that used to be part of Kernel 2.4.x
As we don't want to keep the script to keep pinpoint to those every time, let's add a logic at the script to allow it to ignore valid false-positives like the above.
--- scripts/documentation-file-ref-check | 13 +++++++++++++ 1 file changed, 13 insertions(+)
scripts/documentation-file-ref-check +13 -0
@@ -8,6 +8,14 @@ use warnings; use strict; use Getopt::Long qw(:config no_auto_abbrev); +# NOTE: only add things here when the file was gone, but the text wants +# to mention a past documentation file, for example, to give credits for +# the original work. +my %false_positives = ( + "Documentation/scsi/scsi_mid_low_api.txt" => "Documentation/Configure.help", + "drivers/vhost/vhost.c" => "Documentation/virtual/lguest/lguest.c", +); + my $scriptname = $0; $scriptname =~ s,.*/([^/]+/),$1,;
@@ -122,6 +130,11 @@ while (<IN>) { next if (grep -e, glob("$path/$ref $path/$fulref")); } + # Discard known false-positives + if (defined($false_positives{$f})) { + next if ($false_positives{$f} eq $fulref); + } + if ($fix) { if (!($ref =~ m/(scripts|Kconfig|Kbuild)/)) { $broken_ref{$ref}++;
Signature · 1 line
2.21.0
PATCH 5/10 scripts/documentation-file-ref-check: better handle translations
In reply to Mauro Carvalho ChehabOnly seek for translation renames inside the translation directory.
--- scripts/documentation-file-ref-check | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
scripts/documentation-file-ref-check +7 -3
@@ -141,6 +141,10 @@ print "Auto-fixing broken references. Please double-check the results\n"; foreach my $ref (keys %broken_ref) { my $new =$ref; + my $basedir = "."; + # On translations, only seek inside the translations directory + $basedir = $1 if ($ref =~ m,(Documentation/translations/[^/]+),); + # get just the basename $new =~ s,.*/,,;
@@ -161,18 +165,18 @@ foreach my $ref (keys %broken_ref) { # usual reason for breakage: file renamed to .rst if (!$f) { $new =~ s/\.txt$/.rst/; - $f=qx(find . -iname $new) if ($new); + $f=qx(find $basedir -iname $new) if ($new); } # usual reason for breakage: use dash or underline if (!$f) { $new =~ s/[-_]/[-_]/g; - $f=qx(find . -iname $new) if ($new); + $f=qx(find $basedir -iname $new) if ($new); } # Wild guess: seek for the same name on another place if (!$f) { - $f = qx(find . -iname $new) if ($new); + $f = qx(find $basedir -iname $new) if ($new); } my @find = split /\s+/, $f;
Signature · 1 line
2.21.0
PATCH 9/10 docs: by default, build docs a lot faster with Sphinx >= 1.7
In reply to Mauro Carvalho ChehabSince Sphinx version 1.7, it is possible to use "-jauto" in order to speedup documentation builds. On older versions, while -j was already supported, one would need to set the number of threads manually.
So, if SPHINXOPTS is not provided, add -jauto, in order to speed up the build. That makes it *a lot* times faster than without -j.
If one really wants to slow things down, it can just use:
make SPHINXOPTS=-j1 htmldocs
--- Documentation/Makefile | 2 ++ 1 file changed, 2 insertions(+)
Documentation/Makefile +2 -0
@@ -28,6 +28,8 @@ ifeq ($(HAVE_SPHINX),0) else # HAVE_SPHINX +SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version |"; while (<IN>) { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN') + # User-friendly check for pdflatex and latexmk HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi) HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
Signature · 1 line
2.21.0
cover Improvements to the documentation build system
Hi Jon,
This series contain some improvements for the building system.
I sent already several of the patches here. They're rebased on the top of your docs-next tree:
patch 1: gets rid of a warning since version 1.8 (I guess it starts appearing with 1.8.6);
patches 2 to 4: improve the pre-install script;
patches 5 to 8: improve the script with checks broken doc references;
patch 9: by default, use "-jauto" with Sphinx 1.7 or upper, in order to speed up the build.
patch 10 changes the recommended Sphinx version to 1.7.9. It keeps the minimal supported version to 1.3.
Patch 4 contains a good description of the improvements made at the build system.
If you prefer, you can pull those patches (and the next series I'm submitting you) from my development git tree:
https://git.linuxtv.org/mchehab/experimental.git/log/?h=fix_doc_links_v2
Regards,
Mauro
-
Mauro Carvalho Chehab (10):
docs: cdomain.py: get rid of a warning since version 1.8 scripts/sphinx-pre-install: make activate hint smarter scripts/sphinx-pre-install: get rid of RHEL7 explicity check scripts/sphinx-pre-install: always check if version is compatible with
build
scripts/documentation-file-ref-check: better handle translations scripts/documentation-file-ref-check: exclude false-positives scripts/documentation-file-ref-check: improve tools ref handling scripts/documentation-file-ref-check: teach about .txt -> .yaml
renames
docs: by default, build docs a lot faster with Sphinx >= 1.7
docs: requirements.txt: recommend Sphinx 1.7.9
Documentation/Makefile | 7 +++ Documentation/doc-guide/sphinx.rst | 17 +++--- Documentation/sphinx/cdomain.py | 5 +- Documentation/sphinx/requirements.txt | 4 +- scripts/documentation-file-ref-check | 44 ++++++++++++---- scripts/sphinx-pre-install | 75 +++++++++++++++------------ 6 files changed, 97 insertions(+), 55 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/10 docs: cdomain.py: get rid of a warning since version 1.8
There's a new warning about a deprecation function. Add a logic at cdomain.py to avoid that.
--- Documentation/sphinx/cdomain.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Documentation/sphinx/cdomain.py +4 -1
@@ -48,7 +48,10 @@ major, minor, patch = sphinx.version_info[:3] def setup(app): - app.override_domain(CDomain) + if (major == 1 and minor < 8): + app.override_domain(CDomain) + else: + app.add_domain(CDomain, override=True) return dict( version = __version__,
Signature · 1 line
2.21.0
PATCH 4/10 scripts/sphinx-pre-install: always check if version is compatible with build
In reply to Mauro Carvalho ChehabCall the script every time a make docs target is selected, on a simplified check mode.
With this change, the script will set two vars:
$min_version - obtained from needs_sphinx var inside
conf.py (currently, '1.3')
$rec_version - obtained from sphinx/requirements.txt.
With those changes, a target like "make htmldocs" will do:
1) If no sphinx-build/sphinx-build3 is found, it will run the script on normal mode as before, checking for all system dependencies and providing install hints for the needed programs and will abort the build;
2) If no sphinx-build/sphinx-build3 is found, but there is a sphinx_${VER}/bin/activate file, and if ${VER} >= $min_version (string comparation), it will run in full mode, and will recommend to activate the virtualenv. If there are multiple virtualenvs, it will string sort the versions, recommending the highest version and will abort the build;
3) If Sphinx is detected but has a version lower than $min_version, it will run in full mode - with will recommend creating a virtual env using sphinx/requirements.txt, and will abort the build.
4) If Sphinx is detected and version is lower than $rec_version, it will run in full mode and will recommend creating a virtual env using sphinx/requirements.txt.
In this case, it **won't** abort the build.
5) If Sphinx is detected and version is equal or righer than $rec_version it will return just after detecting the version ("quick mode"), not checking if are there any missing dependencies.
Just like before, if one wants to install Sphinx from the distro, it has to call the script manually and use --no-virtualenv argument to get the hints for his OS:
You should run:
sudo dnf install -y python3-sphinx python3-sphinx_rtd_theme
While here, add a small help for the three optional arguments for the script.
--- Documentation/Makefile | 5 +++++ scripts/sphinx-pre-install | 46 +++++++++++++++++++++++++------------- 2 files changed, 35 insertions(+), 16 deletions(-)
Documentation/Makefile +5 -0
@@ -70,12 +70,14 @@ quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) $(abspath $(BUILDDIR)/$3/$4) htmldocs: + @./scripts/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var))) linkcheckdocs: @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var))) latexdocs: + @./scripts/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var))) ifeq ($(HAVE_PDFLATEX),0)
@@ -87,14 +89,17 @@ pdfdocs: else # HAVE_PDFLATEX pdfdocs: latexdocs + @./scripts/sphinx-pre-install --version-check $(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit;) endif # HAVE_PDFLATEX epubdocs: + @./scripts/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var))) xmldocs: + @./scripts/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var))) endif # HAVE_SPHINX
scripts/sphinx-pre-install +30 -16
91 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH 2/10 scripts/sphinx-pre-install: make activate hint smarter
In reply to Mauro Carvalho ChehabIt is possible that multiple Sphinx virtualenvs are installed on a given kernel tree. Change the logic to get the latest version of those, as this is probably what the user wants.
--- scripts/sphinx-pre-install | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-)
scripts/sphinx-pre-install +14 -8
@@ -1,7 +1,7 @@ #!/usr/bin/perl use strict; -# Copyright (c) 2017 Mauro Carvalho Chehab <[email protected]> +# Copyright (c) 2017-2019 Mauro Carvalho Chehab <[email protected]> # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License
@@ -15,6 +15,7 @@ use strict; my $conf = "Documentation/conf.py"; my $requirement_file = "Documentation/sphinx/requirements.txt"; +my $virtenv_prefix = "sphinx_"; # # Static vars
@@ -28,7 +29,8 @@ my $need_symlink = 0; my $need_sphinx = 0; my $rec_sphinx_upgrade = 0; my $install = ""; -my $virtenv_dir = "sphinx_"; +my $virtenv_dir = ""; +my $min_version; # # Command line arguments
@@ -229,7 +231,6 @@ sub get_sphinx_fname() sub check_sphinx() { - my $min_version; my $rec_version; my $cur_version;
@@ -255,7 +256,7 @@ sub check_sphinx() die "Can't get recommended sphinx version from $requirement_file" if (!$min_version); - $virtenv_dir .= $rec_version; + $virtenv_dir = $virtenv_prefix . $rec_version; my $sphinx = get_sphinx_fname(); return if ($sphinx eq "");
@@ -612,18 +613,23 @@ sub check_needs() which("sphinx-build-3"); } if ($need_sphinx || $rec_sphinx_upgrade) { - my $activate = "$virtenv_dir/bin/activate"; - if (-e "$ENV{'PWD'}/$activate") { + my $min_activate = "$ENV{'PWD'}/${virtenv_prefix}${min_version}/bin/activate"; + my @activates = glob "$ENV{'PWD'}/${virtenv_prefix}*/bin/activate"; + + @activates = sort {$b cmp $a} @activates; + + if (scalar @activates > 0 && $activates[0] ge $min_activate) { printf "\nNeed to activate virtualenv with:\n"; - printf "\t. $activate\n"; + printf "\t. $activates[0]\n"; } else { + my $rec_activate = "$virtenv_dir/bin/activate"; my $virtualenv = findprog("virtualenv-3"); $virtualenv = findprog("virtualenv-3.5") if (!$virtualenv); $virtualenv = findprog("virtualenv") if (!$virtualenv); $virtualenv = "virtualenv" if (!$virtualenv); printf "\t$virtualenv $virtenv_dir\n"; - printf "\t. $activate\n"; + printf "\t. $rec_activate\n"; printf "\tpip install -r $requirement_file\n"; $need++ if (!$rec_sphinx_upgrade);
Signature · 1 line
2.21.0
PATCH 3/10 scripts/sphinx-pre-install: get rid of RHEL7 explicity check
In reply to Mauro Carvalho ChehabRHEL8 was already launched. This test won't get it, and will do the wrong thing. Ok, we could fix it, but now we check Sphinx version to ensure that it matches the minimal (1.3), so there's no need for an explicit check there.
--- scripts/sphinx-pre-install | 13 ------------- 1 file changed, 13 deletions(-)
scripts/sphinx-pre-install +0 -13
@@ -581,19 +581,6 @@ sub check_needs() print "Unknown OS\n"; } - # RHEL 7.x and clones have Sphinx version 1.1.x and incomplete texlive - if (($system_release =~ /Red Hat Enterprise Linux/) || - ($system_release =~ /CentOS/) || - ($system_release =~ /Scientific Linux/) || - ($system_release =~ /Oracle Linux Server/)) { - $virtualenv = 1; - $pdf = 0; - - printf("NOTE: On this distro, Sphinx and TexLive shipped versions are incompatible\n"); - printf("with doc build. So, use Sphinx via a Python virtual environment.\n\n"); - printf("This script can't install a TexLive version that would provide PDF.\n"); - } - # Check for needed programs/tools check_sphinx(); check_perl_module("Pod::Usage", 0);
Signature · 1 line
2.21.0
PATCH 10/10 docs: requirements.txt: recommend Sphinx 1.7.9
In reply to Mauro Carvalho ChehabAs discussed at the linux-doc ML, while we'll still support version 1.3, it is time to recommend a more modern version.
So, let's switch the minimal requirements to Sphinx 1.7.9, as it has the "-jauto" flag, with makes a lot faster when building documentation.
--- Documentation/doc-guide/sphinx.rst | 17 ++++++++--------- Documentation/sphinx/requirements.txt | 4 ++-- 2 files changed, 10 insertions(+), 11 deletions(-)
Documentation/doc-guide/sphinx.rst +8 -9
@@ -27,8 +27,7 @@ Sphinx Install ============== The ReST markups currently used by the Documentation/ files are meant to be -built with ``Sphinx`` version 1.3 or higher. If you desire to build -PDF output, it is recommended to use version 1.4.6 or higher. +built with ``Sphinx`` version 1.3 or higher. There's a script that checks for the Sphinx requirements. Please see :ref:`sphinx-pre-install` for further details.
@@ -56,13 +55,13 @@ or ``virtualenv``, depending on how your distribution packaged Python 3. those expressions are written using LaTeX notation. It needs texlive installed with amdfonts and amsmath in order to evaluate them. -In summary, if you want to install Sphinx version 1.4.9, you should do:: +In summary, if you want to install Sphinx version 1.7.9, you should do:: - $ virtualenv sphinx_1.4 - $ . sphinx_1.4/bin/activate - (sphinx_1.4) $ pip install -r Documentation/sphinx/requirements.txt + $ virtualenv sphinx_1.7.9 + $ . sphinx_1.7.9/bin/activate + (sphinx_1.7.9) $ pip install -r Documentation/sphinx/requirements.txt -After running ``. sphinx_1.4/bin/activate``, the prompt will change, +After running ``. sphinx_1.7.9/bin/activate``, the prompt will change, in order to indicate that you're using the new environment. If you open a new shell, you need to rerun this command to enter again at the virtual environment before building the documentation.
@@ -105,8 +104,8 @@ command line options for your distro:: You should run: sudo dnf install -y texlive-luatex85 - /usr/bin/virtualenv sphinx_1.4 - . sphinx_1.4/bin/activate + /usr/bin/virtualenv sphinx_1.7.9 + . sphinx_1.7.9/bin/activate pip install -r Documentation/sphinx/requirements.txt Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
Documentation/sphinx/requirements.txt +2 -2
@@ -1,3 +1,3 @@ -docutils==0.12 -Sphinx==1.4.9 +docutils +Sphinx==1.7.9 sphinx_rtd_theme
Signature · 1 line
2.21.0
Jonathan Corbet
maintainerQuoted 4 lines · On Wed, 29 May 2019 20:09:22 -0300 Mauro Carvalho Chehab…
This series contain some improvements for the building system.
I sent already several of the patches here. They're rebased on the top of your docs-next tree:
The set is now applied...
Quoted 2 lines
patch 1: gets rid of a warning since version 1.8 (I guess it starts appearing with 1.8.6);
This one I'd already picked up before.
Quoted 6 lines
patches 2 to 4: improve the pre-install script;
patches 5 to 8: improve the script with checks broken doc references;
patch 9: by default, use "-jauto" with Sphinx 1.7 or upper, in order to speed up the build.
I put in the tweak we discussed here.
Quoted 5 lines
patch 10 changes the recommended Sphinx version to 1.7.9. It keeps the minimal supported version to 1.3.
Patch 4 contains a good description of the improvements made at the build system.
Thanks,
jon
Re: [PATCH 09/10] docs: by default, build docs a lot faster with Sphinx >= 1.7
In reply to Mauro Carvalho ChehabQuoted 23 lines · Mauro Carvalho Chehab
Since Sphinx version 1.7, it is possible to use "-jauto" in order to speedup documentation builds. On older versions, while -j was already supported, one would need to set the number of threads manually.
So, if SPHINXOPTS is not provided, add -jauto, in order to speed up the build. That makes it *a lot* times faster than without -j.
If one really wants to slow things down, it can just use:
make SPHINXOPTS=-j1 htmldocs
Signed-off-by Mauro Carvalho Chehab <[email protected]>--- Documentation/Makefile | 2 ++ 1 file changed, 2 insertions(+)Documentation/Makefile +0
-0
Documentation/Makefile ifeq ($(HAVE_SPHINX),0) · line 32
Quoted 5 lines
Documentation/Makefile +2
-0@@ -28,6 +28,8 @@ ifeq ($(HAVE_SPHINX),0) else # HAVE_SPHINX +SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version |"; while (<IN>) { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN') +
Setting SPHINXOPTS like this means you can't pass additional Sphinx options without also dropping -jauto. Which means whenever you want to use SPHINXOPTS for what it's meant for, you also need to provide -jauto to get the same result.
BR,
Jani.
Quoted 3 lines
Documentation/Makefile +0
-0# User-friendly check for pdflatex and latexmk HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi) HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
Signature · 1 line
Jani Nikula, Intel Open Source Graphics Center