Prepare Sphinx to build media PDF books

This patch series fix Sphinx to allow it to build the media documentation as a PDF file.

The first patch is actually a bug fix: one of the previous patch broke compilation for PDF as a hole, as it added an extra parenthesis to a function call.

The second patch just removes a left over code for rst2pdf.

The other patches change from "pdflatex" to "xelatex" and address several issues that prevent building the media books.

Jon,

I think this patch series belong to docs-next. Feel free to merge them there, if you agree. There's one extra patch that touches Documentation/conf.py, re-adding the media book to the PDF build, but IMHO this one would be better to be merged via the media tree, after the fixes inside the media documentation to fix the build.

I'm sending the media-specific patches on a separate patch series, meant to be merged via the media tree.

As on the previous experimental patch series, I'm pushing the entire stuff on my development tree, at:
https://git.linuxtv.org//mchehab/experimental.git/log/?h=docs-next

The generated PDF file is at:
https://mchehab.fedorapeople.org/media.pdf

Please notice that lots of tables are broken. Fixing them would require manual work, as we'll need to add tags to specify the column size via tabularcolumns, long tables should use the cssclass:: longtable, and very wide tables will need to be rotated and size-adjusted.

Anyway, at least *some* PDF support for media books are now possible.

Mauro Carvalho Chehab (9):
docs-rst: fix a breakage when building PDF documents
docs-rst: remove a rst2pdf left over code
docs-rst: allow generating some LaTeX pages in landscape
docs-rst: improve output for .. notes:: on LaTeX
docs-rst: Don't mangle with UTF-8 chars on LaTeX/PDF output
docs-rst: better adjust margins and font size
docs-rst: parse-heraders.pl: escape LaTeX characters
docs-rst: Don't go to interactive mode on errors
docs-rst: enable the Sphinx math extension

 Documentation/Makefile.sphinx         |  8 ++---
 Documentation/conf.py                 | 66 +++++++++++++++++++++++++++++------
 Documentation/sphinx/parse-headers.pl |  2 +-
 3 files changed, 60 insertions(+), 16 deletions(-)

2.7.4

changeset 606b9ac81a63 ("doc-rst: generic way to build only sphinx sub-folders") accidentally broke PDF generation by adding an extra ")". Remove it.

---
 Documentation/Makefile.sphinx | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index ea0664cece12..fdef3a4bc8c7 100644
@@ -71,7 +71,7 @@ ifeq ($(HAVE_PDFLATEX),0)
 	$(warning The 'pdflatex' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
 	@echo "  SKIP    Sphinx $@ target."
 else # HAVE_PDFLATEX
-	@$(call loop_cmd,sphinx,latex,.,latex,.))
+	@$(call loop_cmd,sphinx,latex,.,latex,.)
 	$(Q)$(MAKE) -C $(BUILDDIR)/latex
 endif # HAVE_PDFLATEX
 

2.7.4

The usage of rst2pdf was replaced by pdflatex on a previous patch. Remove the left-over code at conf.py.

---
 Documentation/conf.py | 7 -------
 1 file changed, 7 deletions(-)

index 5c06b018ad1d..2c60df7e5b79 100644
@@ -31,13 +31,6 @@ from load_config import loadConfig
 # ones.
 extensions = ['kernel-doc', 'rstFlatTable', 'kernel_include']
 
-# Gracefully handle missing rst2pdf.
-try:
-    import rst2pdf
-    extensions += ['rst2pdf.pdfbuilder']
-except ImportError:
-    pass
-
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 

2.7.4

Portrait is too small for some tables used at the media docs. So, allow documents to tell Sphinx to generate some pages in landscape by using:

.. raw:: latex

    \begin{landscape}

<some stuff>

.. raw:: latex

    \end{landscape}

---
 Documentation/conf.py | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

index 2c60df7e5b79..42175e87e425 100644
@@ -246,16 +246,19 @@ htmlhelp_basename = 'TheLinuxKerneldoc'
 
 latex_elements = {
 # The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
+'papersize': 'a4paper',
 
 # The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
-
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
+'pointsize': '10pt',
 
 # Latex figure (float) alignment
 #'figure_align': 'htbp',
+
+# Additional stuff for the LaTeX preamble.
+    'preamble': '''
+        % Allow generate some pages in landscape
+        \\usepackage{lscape}
+     '''
 }
 
 # Grouping the document tree into LaTeX files. List of tuples

2.7.4

The output for those notes are bad in pdf, as they're not in a box with a different color. Also, it causes the output to not build if the note is inside a table.

Change its implementation to avoid the above troubles.

The logic there came from:
https://stackoverflow.com/questions/606746/how-to-customize-an-existing-latex-environment-without-interfering-with-other-en

---
 Documentation/conf.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

index 42175e87e425..f91acc78e20b 100644
@@ -258,6 +258,29 @@ latex_elements = {
     'preamble': '''
         % Allow generate some pages in landscape
         \\usepackage{lscape}
+
+        % Put notes in gray color and let them be inside a table
+
+        \\definecolor{MyGray}{rgb}{0.80,0.80,0.80}
+
+        \\makeatletter\\newenvironment{graybox}{%
+           \\begin{lrbox}{\\@tempboxa}\\begin{minipage}{\\columnwidth}}{\\end{minipage}\\end{lrbox}%
+           \\colorbox{MyGray}{\\usebox{\\@tempboxa}}
+        }\\makeatother
+
+        \\makeatletter
+        \\renewenvironment{notice}[2]{
+          \\begin{graybox}
+          \\bf\\it
+          \\def\\py@noticetype{#1}
+          \\par\\strong{#2}
+          \\csname py@noticestart@#1\\endcsname
+        }
+	{
+          \\csname py@noticeend@\\py@noticetype\\endcsname
+          \\end{graybox}
+        }
+	\\makeatother
      '''
 }
 

2.7.4

pdflatex doesn't accept using some UTF-8 chars, like "equal or less than" or "equal or greater than" chars. However, the media documents use them. So, we need to use XeLaTeX for conversion, and a font that accepts such characters.

---
 Documentation/Makefile.sphinx |  6 +++---
 Documentation/conf.py         | 11 +++++++++++
 2 files changed, 14 insertions(+), 3 deletions(-)

index fdef3a4bc8c7..16a3502c9e40 100644
@@ -29,7 +29,7 @@ else ifneq ($(DOCBOOKS),)
 else # HAVE_SPHINX
 
 # User-friendly check for pdflatex
-HAVE_PDFLATEX := $(shell if which pdflatex >/dev/null 2>&1; then echo 1; else echo 0; fi)
+HAVE_PDFLATEX := $(shell if which xelatex >/dev/null 2>&1; then echo 1; else echo 0; fi)
 
 # Internal variables.
 PAPEROPT_a4     = -D latex_paper_size=a4

@@ -68,11 +68,11 @@ htmldocs:
 
 pdfdocs:
 ifeq ($(HAVE_PDFLATEX),0)
-	$(warning The 'pdflatex' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
+	$(warning The 'xelatex' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
 	@echo "  SKIP    Sphinx $@ target."
 else # HAVE_PDFLATEX
 	@$(call loop_cmd,sphinx,latex,.,latex,.)
-	$(Q)$(MAKE) -C $(BUILDDIR)/latex
+	$(Q)$(MAKE) PDFLATEX=xelatex -C $(BUILDDIR)/latex
 endif # HAVE_PDFLATEX
 
 epubdocs:

index f91acc78e20b..e254198b1dbf 100644
@@ -254,6 +254,10 @@ latex_elements = {
 # Latex figure (float) alignment
 #'figure_align': 'htbp',
 
+# Don't mangle with UTF-8 chars
+'inputenc': '',
+'utf8extra': '',
+
 # Additional stuff for the LaTeX preamble.
     'preamble': '''
         % Allow generate some pages in landscape

@@ -281,6 +285,13 @@ latex_elements = {
           \\end{graybox}
         }
 	\\makeatother
+
+	% Use some font with UTF-8 support with XeLaTeX
+        \\usepackage{fontspec}
+        \\setsansfont{DejaVu Serif}
+        \\setromanfont{DejaVu Sans}
+        \\setmonofont{DejaVu Sans Mono}
+
      '''
 }
 

2.7.4

As we have big tables, reduce the left/right margins and decrease the point size to 8pt. Visually, it is still good enough, and now less tables are too big to be displayed.

---
 Documentation/conf.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

index e254198b1dbf..ac5230fcda4d 100644
@@ -249,7 +249,7 @@ latex_elements = {
 'papersize': 'a4paper',
 
 # The font size ('10pt', '11pt' or '12pt').
-'pointsize': '10pt',
+'pointsize': '8pt',
 
 # Latex figure (float) alignment
 #'figure_align': 'htbp',

@@ -260,6 +260,9 @@ latex_elements = {
 
 # Additional stuff for the LaTeX preamble.
     'preamble': '''
+	% Adjust margins
+	\\usepackage[margin=0.5in, top=1in, bottom=1in]{geometry}
+
         % Allow generate some pages in landscape
         \\usepackage{lscape}
 

2.7.4

Let's escape the LaTeX characters, to avoid troubles when outputing them.

---
 Documentation/sphinx/parse-headers.pl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 34bd9e2630b0..74089b0da798 100755
@@ -220,7 +220,7 @@ $data =~ s/\n\s+\n/\n\n/g;
 #
 # Add escape codes for special characters
 #
-$data =~ s,([\_\`\*\<\>\&\\\\:\/\|]),\\$1,g;
+$data =~ s,([\_\`\*\<\>\&\\\\:\/\|\%\$\#\{\}\~\^]),\\$1,g;
 
 $data =~ s,DEPRECATED,**DEPRECATED**,g;
 

2.7.4

When building for LaTeX, it stops and enters into interactive mode on errors. Don't do that, as there are some non-fatal errors on media books when using Sphinx 1.4.x that we don't know how fix yet.

---
 Documentation/Makefile.sphinx | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 16a3502c9e40..ba4efb1f68f3 100644
@@ -72,7 +72,7 @@ ifeq ($(HAVE_PDFLATEX),0)
 	@echo "  SKIP    Sphinx $@ target."
 else # HAVE_PDFLATEX
 	@$(call loop_cmd,sphinx,latex,.,latex,.)
-	$(Q)$(MAKE) PDFLATEX=xelatex -C $(BUILDDIR)/latex
+	$(Q)$(MAKE) PDFLATEX=xelatex LATEXOPTS="-interaction=nonstopmode" -C $(BUILDDIR)/latex
 endif # HAVE_PDFLATEX
 
 epubdocs:

2.7.4

This extension will be used by the media books.

The name of the math image extension changed on Sphinx 1.4.x, according with:
http://www.sphinx-doc.org/en/stable/ext/math.html#module-sphinx.ext.imgmath

Let's autodetect, to keep building with versions < 1.4.

---
 Documentation/conf.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

index ac5230fcda4d..39b9c4a26f6e 100644
@@ -14,6 +14,11 @@
 
 import sys
 import os
+import sphinx
+
+# Get Sphinx version
+major, minor, patch = map(int, sphinx.__version__.split("."))
+
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the

@@ -31,6 +36,12 @@ from load_config import loadConfig
 # ones.
 extensions = ['kernel-doc', 'rstFlatTable', 'kernel_include']
 
+# The name of the math extension changed on Sphinx 1.4
+if minor > 3:
+    extensions.append("sphinx.ext.imgmath")
+else:
+    extensions.append("sphinx.ext.pngmath")
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 

2.7.4

It's now in docs-next. I was able to build some nice-looking docs with it without too much (additional) pain...

The conf.py patch makes me a bit nervous, in that I feel like I spent a fair amount of time explaining docs merge conflicts to Linus during the merge window, and would rather not do that again. Can we keep it aside, with the idea that one of us will put it in toward the end of the 4.9 merge window?

Meanwhile, let's see how this xelatex thing works out. Thanks for making all this work!

jon

Em Thu, 18 Aug 2016 17:21:27 -0600
Jonathan Corbet <[email protected]> escreveu:

Good!

Yeah, sure. I'm actually planning to pull from your docs-next branch at the media master tree before starting picking other patches on it, and wait for your pull request before sending mine on the next merge tree.

Anytime!

Thanks,
Mauro

Markus,

Em Thu, 18 Aug 2016 17:21:27 -0600
Jonathan Corbet <[email protected]> escreveu:

I'm noticing a very weird behavior when I'm building documentation on my server. There, I'm using this command:

	$ make cleandocs; make V=1 DOCBOOKS="" SPHINXDIRS=media SPHINX_CONF="conf.py" htmldocs

This is what happens on my local machine:
http://pastebin.com/VGqvDa7T

And this is the result of the same command on my server, accessed via ssh:
http://pastebin.com/1MFi5LEG

As you can see, it seems that internally sphinx is calling a make -C Documentation/output/latex, with is very bad, because it takes a lot of extra time to run and produces an useless output. It also produces a wrong output, as it would be calling pdflatex, instead of xelatex.

Do you have any glue about what's going on?

Also, if I use the "-j33" sphinx option, it complains:

WARNING: the kernel_include extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit
WARNING: doing serial read

Btw, we need to add support to build just one PDF file, as we did with the htmldocs.

Thanks,
Mauro

Am 24.08.2016 um 12:42 schrieb Mauro Carvalho Chehab <[email protected]>:

Hi Mauro,

if you build a sub-folder, the conf.py is the default. You don't need to name conf.py it explicit and you can leave the DOCBOOKS env.

$ make V=1 SPHINXDIRS=media cleandocs htmldocs

or less verbose:

$ make SPHINXDIRS=media cleandocs htmldocs

But this does not answer your question ;)

Seems to build fine. But this is not "make V=1" log.

Same here, it is not a "make V=1" log. The errors like:

WARNING: inline latex u"L' = L ^{\\frac{1}{2.19921875}}": latex exited with error

are dubious first. Which branch did you compile. It seems you are using "inline latex" ... this seems not in Jon's docs-next. I checked your experimental docs-next, there is a related markup, so I think you compiling this branch.

.. math::

L' = L ^{\frac{1}{2.19921875}}

So I guess the error message is related to one of the sphinx-extensions:

# The name of the math extension changed on Sphinx 1.4 if minor > 3:
extensions.append("sphinx.ext.imgmath")
else:
extensions.append("sphinx.ext.pngmath")

Since there is a log "Running Sphinx v1.4.6" (both, desktop and server) I guess it is related to the sphinx.ext.imgmath extension.

I haven't tested math-extensions yet, I will give it a try and send you my experience later. In the meantime you can check your math-extensions on desktop and server ...

In general I guess:

0.) you compiling different branches

or

1.) on your desktop the math-extension miss some latex stuff and does not run, so you get no errors (or it runs perfect without any error).

or / and

2.) on your server the math-extension runs and has some problems with the "L' = L ^{\frac{1}{2.19921875}}" expression.

As said, I haven't tested math-extensions yet. But the math extension uses "latex" as default / not xelatex

* imgmath_latex: http://www.sphinx-doc.org/en/stable/ext/math.html#module-sphinx.ext.imgmath
* pngmath_latex: https://linuxtv.org/downloads/sphinx-1.2.3/ext/math.html#module-sphinx.ext.pngmath

No ;-) ... check the math-extension process as described above.

Yes I know, it is the same with the kernel_doc extension, I can send a patch for both.

You mean, when you build a subfolder (SPHINXDIRS=media), you wanted to build a PDF with only media stuff in .. right? .. thats what I suggested in one of my last mails .. I can sent a patch for this.

-- Markus --

Hi Markus,

Em Wed, 24 Aug 2016 13:46:48 +0200
Markus Heiser <[email protected]> escreveu:

Yeah, I know. I added the SPHINX_CONF there because my end goal is to use the nitpick config, after cleaning it up ;)

Thanks! that was the case...

I had already those two extensions that are needed by math:

	texlive-amsmath-svn30645.2.14-24.fc24.1.noarch
	texlive-amsfonts-svn29208.3.04-24.fc24.1.noarch

But it was missing this one on the server:
texlive-anyfontsize.noarch

That will be very much appreciated, thanks!

Yes.

Please do that.

Thanks!
Mauro