PDF misery

So, FYI, I have Vegard's PDF-generation fix applied here, but I've not pushed it yet (even though I think it's fine) because I wanted to be sure that all was well with PDF generation. Bear with me as I ramble for a bit...

I'll confess to not paying as much attention to PDF output as I should. Partly that's from a perception that not many people care about it (though a few really do), and partly because PDF generation has always been a bit of a house of cards that turns into a real pain whenever anybody looks at it hard.

Somebody looked at it hard.

My fairly standard Fedora 39 system can't do a pdfdocs build at all. There are a couple of untoward things that get in the way.

One is the dreaded Latex "too deeply nested" message that causes everything to explode while generating filesystems.pdf. The problem, specifically, is .../filesystems/xfs/xfs-online-fsck-design.rst, which blows the limits in two ways. This patch "fixes" it - this clearly isn't something to apply, it's just a demonstration of the problem:

-----

index 352516feef6f..1762a9415ec1 100644
@@ -739,15 +739,6 @@ For a given fstests filesystem configuration:
         7. Subtract a small quantity
         8. Randomize the contents
 
-        * ...test the reactions of:
-
-          1. The kernel verifiers to stop obviously bad metadata
-          2. Offline checking (``xfs_repair -n``)
-          3. Offline repair (``xfs_repair``)
-          4. Online checking (``xfs_scrub -n``)
-          5. Online repair (``xfs_scrub``)
-          6. Both repair tools (``xfs_scrub`` and then ``xfs_repair`` if online repair doesn't succeed)
-
 This is quite the combinatoric explosion!
 
 Fortunately, having this much test coverage makes it easy for XFS developers to

@@ -2093,7 +2084,6 @@ of the records, so naturally the xfarray must also support sorting the record
 set prior to bulk loading.
 
 Case Study: Sorting xfarrays
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The sorting algorithm used in the xfarray is actually a combination of adaptive
 quicksort and a heapsort subalgorithm in the spirit of

-----
The first change removes the deepest level of nesting in an impressive set of enumerated lists; the second removes the nth-level subsection heading markup for "Sorting xfarrays". Having either of those present will cause Latex to blow a fuse.

I'm surprised that nobody else is reporting this problem. I honestly don't see a fix other than changing the organization of the document to avoid both deeply nested itemized lists and section structure, which is less than entirely desirable. I think there are good reasons for avoiding structures that deep, but limitations in the tooling aren't one of them.

Incidentally, on F38, I also run into a similar Latex error in .../userspace-api/media/v4l/pixfmt-rgb.html, which has a set of tables from hell. On F39, that somehow works. Weird.

The *other* problem is that PDF generation of the Chinese, Korean, or Japanese translations fails with:

xdvipdfmx:fatal: Invalid TTC index number

This, I am thinking, is likely some sort of F39 bug. xdvipdfmx is finding the CJK fonts just fine, but then something clearly goes wrong. I'll try to find the time to narrow that down and perhaps put in a proper bug report.

Meanwhile, I wish I knew what to do to make the PDF build a bit more robust. I wish that rst2pdf or rinohtype would progress to the point where they could be used, and Latex could be taken out of the picture, but no such luck.

Oh well...sorry for rambling at you...I wish I had a good plan for making this whole thing better.

jon

Em Sat, 17 Feb 2024 14:29:24 -0700
Jonathan Corbet <[email protected]> escreveu:

Yeah, LaTeX Sphinx output is really tricky, requiring workarounds from time to time to address some issues.

...

The maximum indentation level seems easily fixable. See my PoC at the end of this e-mail.

From:
- https://tex.stackexchange.com/questions/571826/too-deeply-nested
- https://stackoverflow.com/questions/1935952/maximum-nesting-level-of-lists-in-latex

And some local tests I did on Fedora 39, the default enum indentation limit is 7.

This is configurable by adding \setlistdepth after using enum item, e. g.:

	\usepackage{enumitem}
	\setlistdepth{9}

I guess we can add something like that to latex_elements at conf.py to increase the maximum limit.

I remember in the past I had to increase some LaTeX memory limits, but it seems that this is not required anymore.

I haven't tested rst2pdf for a while on Kernel builds. It works fine when producing IGT documentation, but the docs there are too trivial to cause issues.

Didn't try rinohtype yet.

Ideally, if one of those would work, that would be a lot better, as LaTeX makes it a lot more complex than it should be.

Regards,
Mauro

---

The way the .tex file below is, it builds fine with:

	$ xelatex  -halt-on-error /tmp/test.tex
	This is XeTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023) (preloaded format=xelatex)
	...
	Output written on test2.pdf (1 page).
	Transcript written on test2.log.

If you remove this line:

	\setlistdepth{9}

You'll see the error:

	$ xelatex  -halt-on-error /tmp/test_without_setlistdepth.tex
	This is XeTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023) (preloaded format=xelatex)
	...

! LaTeX Error: Too deeply nested.

See the LaTeX manual or LaTeX Companion for explanation. Type H <return> for immediate help.
...

l.60 \item
item level 7
No pages of output.
Transcript written on test_without_setlistdepth.log.

And it indicates that it fails when level == 7.

-

\documentclass{article}

\usepackage{enumitem}
\RequirePackage{enumitem}

\setlistdepth{9}

\setlist{nolistsep,

         leftmargin = *}

%
%
\newlist{myEnumerate}{enumerate}{9}
\setlist[myEnumerate]{
font = {\bfseries} ,
topsep = 0pt }
\setlist[myEnumerate,1]{label=\arabic* ---,ref=\arabic*} \setlist[myEnumerate,2]{label=\arabic{myEnumeratei}.\arabic*),ref=\themyEnumeratei.\arabic*} \setlist[myEnumerate,3]{label=\roman*),ref=\themyEnumerateii.\roman*} \setlist[myEnumerate,4]{label=\Roman*.,ref=\themyEnumerateiii.\Roman*} \setlist[myEnumerate,5]{label=\themyEnumerateiiii.\roman*.,ref = \themyEnumerateiiii.\roman*} \setlist[myEnumerate,6]{label=(\arabic*)} \setlist[myEnumerate,7]{label=(\Roman*)} \setlist[myEnumerate,8]{label=(\Alph*)} \setlist[myEnumerate,9]{label=(\roman*)} \newcommand{\ben}{\begin{myEnumerate}}
\newcommand{\een}{\end{myEnumerate}}
%
\newlist{myItemize}{itemize}{9}

\setlist[myItemize]{
    topsep      = 0pt }

\setlist[myItemize,1]{label=\textbullet}
\setlist[myItemize,2]{label=---}
\setlist[myItemize,3]{label=---}
\setlist[myItemize,4]{label=---}
\setlist[myItemize,5]{label=---}
\setlist[myItemize,6]{label=---}
\setlist[myItemize,7]{label=---}
\setlist[myItemize,8]{label=---}
\setlist[myItemize,9]{label=---}
\newcommand{\bit}{\begin{myItemize}}
\newcommand{\eit}{\end{myItemize}}

\begin{document}

\section{Main section}

\section{Subsection}
\ben
\item item level 1
\ben
\item item level 2
\bit
\item item level 3
\bit
\item item level 4
\bit
\item item level 5
\bit
\item item level 6
\bit
\item item level 7
\bit
\item item level 8
\eit
\eit
\eit
\eit
\eit
\eit
\item item level 2
\een
\item item level 1
\een

\end{document}

Hi Jon,

A few comments on PDF of CJK docs.

I think this is because xdvipdfmx accesses NotoSansCJK-VF.ttc, which is a variable font. xdvipdfmx/xetex can't work with such fonts (yet). See note at the bottom for more info on variable fonts support.

It sounds like you have google-noto-sans-cjk-vf-fonts installed on your system besides google-noto-sans-cjk-fonts.

What does

fc-list | grep NotoSansCJK-VF.ttc

say?

I'm wondering why xdvipdfmx behaves that way despite the fontconfig setting with:

fc-match "Noto Sans CJK SC"

returning:

NotoSansCJK-Regular.ttc: "Noto Sans CJK SC" "Regular"

This might be a bug in xdvipdfmx worth reporting. Or there might be glitches in the fontconfig setting.

Uninstalling google-noto-sans-cjk-vf-fonts (and google-noto-serif-cjk-vf-fonts in case you have it) should resolve the issue for you, that is if you can safely uninstall it/them.

* Note on variable fonts support:

In the announcement of LaTeX fontspec package v2.9a release [1], there is an entry:

- Support variable fonts under LuaLaTeX.

It is not explicitly mentioned there, but in the opening paragraph of Part III "Selecting font features" Chapter 7 "Variable fonts" of fontspec's package documentation, there is a sentence which reads:

Currently OpenType variable fonts are only supported in LuaTEX, while Multiple Master fonts only work with XETEX.

As the xeCJK package requires xelatex, variable fonts are not in our options, I guess.

[1]: https://ctan.org/ctan-ann/id/[email protected]

HTH, Akira

Em Sun, 18 Feb 2024 18:58:47 +0900
Akira Yokosawa <[email protected]> escreveu:

Tried the above on my desktop with Fedora 39:

<snip>

$ fc-list | grep NotoSansCJK-VF.ttc
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=Bold
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=Bold
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=Bold
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=Bold
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=Bold
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=Black
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=Black
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=Black
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=Black
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=Black
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=Thin,Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=Thin,Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=Thin,Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=Thin,Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=Thin,Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=Regular
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=DemiLight
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=DemiLight
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=Light
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=Light
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=DemiLight
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK TC:style=Medium
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=DemiLight
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=DemiLight
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=Light
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK SC:style=Medium
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=Light
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=Light
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK HK:style=Medium
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK KR:style=Medium
/usr/share/fonts/google-noto-sans-cjk-vf-fonts/NotoSansCJK-VF.ttc: Noto Sans CJK JP:style=Medium
$ fc-match "Noto Sans CJK SC"
NotoSansCJK-VF.ttc: "Noto Sans CJK SC" "Regular"
$ sudo dnf remove google-noto-sans-cjk-vf-fonts google-noto-serif-cjk-vf-fonts -y
...
Removed:
  default-fonts-cjk-sans-4.0-9.fc39.noarch              default-fonts-cjk-serif-4.0-9.fc39.noarch  google-noto-sans-cjk-vf-fonts-1:2.004-5.fc39.noarch 
  google-noto-serif-cjk-vf-fonts-1:2.002-2.fc39.noarch 

Done!

$ fc-list | grep NotoSansCJK-VF.ttc
$ fc-match "Noto Sans CJK SC"
Vera.ttf: "Bitstream Vera Sans" "Regular"
</snip>

Thanks,
Mauro

No output at all, even though I had google-noto-sans-cjk-vf-fonts installed.

Getting rid of google-noto-sans-cjk-vf-fonts did indeed make the problem go away.

So it seems. I guess $WE should probably change scripts/sphinx-pre-install to give a warning; I wonder if some defensive changes to Documentation/sphinx/kerneldoc-preamble.sty would also make sense? My TeXpert days (such as they were) are decades behind me at this point, so I'll be slow to do the latter part if it comes to that.

Thanks for your help on this!

jon

This won't help with the immediate issue at hand, but I am trying to put together something of a "docs CI" system that can pick up mailing list patches and do builds against all the various OS/Sphinx/target combinations.

At least that was my takeaway from the recent couple of regressions I
introduced: I need (much) better tests than just running 'make htmldocs' once or twice in my native environment.

I don't know. Tooling limitations notwithstanding, the good reasons are still there. HTML has no <h7> and that was clearly a "human" decision, not a technological limit.

I think the "bug" here is really that Sphinx generated LaTeX code that doesn't build -- and that it didn't warn about the problem in other modes (maybe it does in newer versions, or maybe we overlooked the warnings?).

If we'd had those warnings then we could have stopped those constructs from entering the documentation sources to start with.

We could maybe use the ".. only::" directive to "#ifdef out" the parts that don't work with various output formats and then incrementally work towards removing them while ensuring that the whole build still succeeds.

Vegard

With the attached patch I get:

$ make htmldocs SPHINXDIRS=filesystems/xfs
make[3]: Nothing to be done for 'html'.
Using alabaster theme
source directory: filesystems/xfs
WARNING: xfs-online-fsck-design.rst:2096: heading "Case Study: Sorting xfarrays" has nesting level 7; consider restructuring

Maybe we can do the same for itemize lists.

I didn't check what impact this potentially has on running time. Also only tested with Sphinx 4.3.2.

Thoughts?

Vegard

index 5830b01c5642..1cfcea4e487d 100644
@@ -55,7 +55,8 @@ needs_sphinx = '2.4.4'
 extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include',
               'kfigure', 'sphinx.ext.ifconfig', 'automarkup',
               'maintainers_include', 'sphinx.ext.autosectionlabel',
-              'kernel_abi', 'kernel_feat', 'translations']
+              'kernel_abi', 'kernel_feat', 'translations',
+              'kvalidate']
 
 if major >= 3:
     if (major > 3) or (minor > 0 or patch >= 2):

Hi,

On 2024/02/18 6:29, Jonathan Corbet wrote:
...

So, there is a handy answer in the Sphinx documentation.

Just look at https://www.sphinx-doc.org/en/master/latex.html
and search 'maxlistdepth'.

You can set this variable in conf.py, say,

    'maxlistdepth': '9',

Now the deep lists in xfs-online-fsck-design.rst can be built into PDF.

Problem solved.

It's getting late here, so can anybody prepare a proper patch with my

?

Thanks, Akira

Em Mon, 19 Feb 2024 00:49:08 +0900
Akira Yokosawa <[email protected]> escreveu:

Hmm... there are some other interesting options there, like the verbatim ones. One of the problems that required lots of changes at the media uAPI part of the docs is because code blocks are reproduced as-is, without wrapping, making them to either be truncated or going past the margin. That's specially painful on code blocks inside tables.

The way it was solved was this this patch:

3b4c963243b1 ("docs: conf.py: adjust the LaTeX document output")

plus several changes on media docs with redesign changes (before and after the above patch), plus several and several macros at the .rst files to change font size and carefully adjusting columns sizes on tables.

It sounds that since then other verbatim options got added. So, perhaps the PDF output could be improved on newer Sphinx versions.

Good catch!

Thanks,
Mauro

So I had actually tried this, and it failed miserably for me. Now it works.

As far as I can tell, my failing was that I tossed in that change and the \usepackage{enumitem} change at the same time; doing both doesn't work. I know better than to change multiple things at a time, but given how long a pdfdocs build attempt takes, one wants to cover all the bases...

I'll put something in, yes.

Thanks,

jon