Sphinx version dependencies?

Darrick has sent in patches to convert the ext4 documentation to use rst and to be built as part of the full kernel documentation thanks. In addition to that, he's imported the on-disk documentation from the ext4 wiki into the kernel sources, so hopefully we can keep it more up to date.

When I was experimenting with this, I had to actually build the kernel docs using Sphinx for the first time. I'm using Debian testing, so at first I blindly followed the instructions by ./scripts/sphinx-pre-install:

Detected OS: Debian GNU/Linux unstable (sid).

	/usr/local/bin/virtualenv sphinx_1.4
	. sphinx_1.4/bin/activate
	pip install -r Documentation/sphinx/requirements.txt

But when I did that, Sphinx had heartburn over the ext4.rst file.

./include/linux/spi/spi.h:373: ERROR: Unexpected indentation. /usr/projects/linux/ext4/Documentation/filesystems/ext4/ext4.rst:139: ERROR: Malformed table. Column span alignment problem in table line 5.
...

After consulting with Darrick, it appears the problem is that Sphinx8 1.4.9 was the problem. This is the version that Documentation/sphinx/requirements.txt calls for. He did his rst conversion work using Ubuntu 18.04's Sphinx 1.6.7.

As it turns out Debian testing/unstable already has Sphinx 1.7.5 in its repository, so if I simply install Sphinx 1.7.5, it works fine. That's what I've done for now.

So that leaves me with some questions:

* Is there a reason why scripts/sphinx-pre-install suggested using a Python virtual environment and installing Sphinx 1.4.9 instead of using the distro's pre-packaged Sphinx for Debian unstable/testing?

* Why does Documentation/sphinx/requirements.txt asking for such an old version of Sphinx?

* Is it a requirement that *.rst files that are checked into the kernel repo have to work with Sphinx 1.4.9? Or is it sufficient that it works with Sphinx 1.6.7 and 1.7.5 (which are the prepackaged Debian and Ubuntu versions). And it looks like Fedora 28 has Sphinx
1.7.2 if I'm not mistaken. How many versions of Sphinx are various
automated build/test systems using?

Thanks,

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hmmm, apparently it's choking on the table heading borders not matching the text:

====== ===========
Foo Bar
====== =========== <-- need to extend to EOL
Blah Blah blah blah blah
====== ===========

Though weirdly while I /can/ get this error to reproduce with the virtualenv 1.4.9 release, I can't get it to reproduce with the 1.3.6 or 1.6.7 ubuntu packages. Maybe it's a python3 thing, maybe not? Seems pretty fragile to me.

Anyway, I'll fix ext4.rst and resubmit that part to get this moving.

--D

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Am Donnerstag, den 19.07.2018, 12:04 -0700 schrieb Darrick J. Wong:

Not a py2 or py3 thing. I *guess* it is caused by changes in the underlying docutils.

Yes, this is a very confusing and we have to look at reST, docutils, Sphinx-doc, virtualenv py2 & py3, all these distros and the philosophy of the Linux community to understand.

Yes, this is a very confusing and we have to look at reST, docutils, Sphinx-doc, virtualenv py2 & py3, all these distros and the philosophy of the Linux community to understand.

reST: Is the markup.

docutils (python): Is the reST reference implementation with some changes in
2017[1].

Sphinx-doc (python): reST application based on the docutils lib
(docutils>=0.11)[2]. Take in mind, this dependency opens a new array of confusion, cause its fulfillment is based on what is installed on your system (or in your virtualenv, see below). Side node: the pdf-builder (LaTeX, to be correct) is in a continues redesign since the last two years and was horrible in older Sphinx-doc versions. E.g. take a look at the current 1.8.0 development (section deprecated)[4].

python: py2 will retire in 2020 (py3 was released in 2008). All py applications
we using are py2 and py3 compatible. If we have no bugs, py2 or py3 should never be the cause.

distros: Nearly all distros using using py2 as there default python interpreter
(mostly since there package managers are written/tested with py2).

virtualenv: A virtualenv is build up in the $HOME folder where Sphinx-doc and
docutils are installed from pypi, the python package manager[3]. Take in mind, using another package in parallel to the system package manager manager opens another array of confusion.

Normally I would say, lets use (and test against) state of the art. This is easily done by installing py3 and using virtualenv. But this seems not match the philosophy of the whole Linux community.

As one told me, the philosophy is to build the Kernel with less installations from the web by using what the distro ships. IMO this might be right for the Kernel but not for applications building viewing formats. When it comes to build PDF you will realize how naive this approach is. IMO there is also no need to build viewing formats as we can read the documentation in plain text as we can do it since the beginning ... and this is one reason why we changed from XML to a plain text markup.

The script named 'sphinx-pre-install' is the crutch that supports this philosophy.

We have this discussion over and over again an I believe we will have it again and again, as long as we do not change our POV .. building viewing formats is an application not a part of the Kernel.

Anyway, I hope this helps to clarify, why we have those problems.

[1] http://docutils.sourceforge.net/HISTORY.html
[2] https://github.com/sphinx-doc/sphinx/blob/master/setup.py#L22
[3] https://pypi.org/
[4] https://github.com/sphinx-doc/sphinx/blob/master/CHANGES

-- Markus --

It's clear we have an over-constrained problem. On the one hand, we want to make sure that the Linux kernel (but maybe not the docs? unclear?) can build on a wide variety of Linux distros, from those that are shipping (in some cases) decade-old, antiquated tools in Enterprise Distrobutions, to bleeding edge Community Distro's. On the other, we want to be able to use the latest and greatest features. And on yet a third hand, we need to deal with the fact that historically, developers have hated it when they have to download wierd tools, often ones that are not packaged by the distro, such as cmake, imake, etc., etc. (Fortunately with Python virtualenv makes things relatively painless; unlike in the past. But I think we sometimes have biases on past experience.)

My observation though is that at the moment sphinx-pre-install doesn't really seem to support any one philosophy consistently. On the one hand, if you are running Debian unstable, and you don't have the distro-packaged Sphinx installed, it tells you to use virtualenv, and force-installs Sphinx 1.4.9. On the other hand, if you happen to have the debian-packaged Sphinx installed already it says, "go forth and build!" This means that from the perspective of a Kernel developer, we don't know whether someone will be running Sphinx 1.4.9, or some other random Distro version.

One approach might be to build the virtualenv setup and download directly into build sequence. But the problem with that is it would break hermetic build systems that some distros use, which (by design) are not connected to the network for security reasons.

Another solution is to specify that the kernel docs *must* build on 1.4.9, and change sphinx-pre-install to always require this. This might require imposing a requirement on distro's to package more than one version of Sphinx. (There is precedence, at one point Debian packaged something like four different versions of autoconf, because it's needed because autoconf is not necessary backwards *or* forwards compatible.)

Yet another solution is to impose the burden on kernel developers and tell them that the kernel docs must be built on a variety of Sphinx versions, and that over time, changes will break the docs even if nothing has changed. So in addition to testing on a large range of Sphinx versions, over time we will have to continuously update the kernel docs to deal with backwards incompatible changes in how Sphinx processes the .rst format.

A forth solution is to force everyone to use something correleted to what the community distro's are packaging, but roughly every 12-18 months, we leapfrog to a newer version of Sphinx. This decreases the continuous upgrade burden, but doesn't make it go away.

I'm not entirely sure what's the best approach. Right now I just want to understand --- do I have to make ext4.rst work against one, or many versions of Sphinx? And which version(s) of Sphinx do I need to concern myself with? If that turns out to be an onerous burden, I'm sure I won't be the only person complaining. :-)

- Ted

Am Freitag, den 20.07.2018, 09:12 -0400 schrieb Theodore Y. Ts'o:

In that case ...

... its clear; the table was malformed. A markup error which is not detected by older versions of docutils (very special case).

In general I recommend to install an up-to-date Sphinx-doc in a virtualenv .. this is what I do in my projects ... and where I haven't seen those version problems ;)

If we have errors downward we have to handle them. The main thing is the markup is correct.

-- Markus --

... except that newer verions are A-OK with it. Apparently 1.3.x was OK with it, and 1.6.x and 1.7.x were ok with it. ***ONLY*** Sphinx 1.4.9 blew up on the "malformed table".

So in this case, Darrick has come up with a patch that is makes it OK with 1.4.9 without breaking on 1.7.5 --- and obviously, doing something that makes it broadly portable is the right thing.

I'm asking a larger question, which is moving forward, which is more important? Make it work with Sphinx 1.4.9? Or making it Sphinx work with Sphinx 1.7.5?

And should we change Documentation/sphinx/requirements.txt to require something newer, such as Sphinx 1.7.5? And should we require that Ubuntu 18.04 which is using Sphinx 1.6.8 use a virtualenv and use download Sphinx 1.6.8?

My understanding that the Sphinx developers make no guarantees that if we follow some external, version-indepedent spec, that it will work on Sphinx version N, as well as Sphinx version N+1. (In the ideal world, if there was such an independent spec for .rst format files, and a compliant .rst file doesn't work for Sphinx version N, it's a bug, and we should expect somebody --- perhaps the Distro's --- to backport the fix from Sphinx version N+1 to Sphinx version N.) E.g., is there an equivalent for ANSI C 1999 standard for .rst files?

- Ted

Am Freitag, den 20.07.2018, 10:52 -0400 schrieb Theodore Y. Ts'o:

Are you sure that it was not due to the docutils version? I can't reproduce it but the table parser is a part of docutils.

Right, fix it by the markup .. is what I recommend.

The requirements.txt came from commit fb947f3f47 [1] (inital 24071ac1a6). Where Jon and Mauro decided to tag explicit versions ...

docutils==0.12
Sphinx==1.4.9
sphinx_rtd_theme

Maybe it is time to switch to something like .. ?

Sphinx>=1.4.9
sphinx_rtd_theme

I don't know. Mauro has tested on many distros, he has more experience with the wide range of distros then I.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fb947f3f47

The reST markup is specified here:

http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html

but the (last) example of the simple table does not match your "1.4.9" experience.

-- Markus --

No idea. With the virtualenv instructions I get:

$ pip list | egrep -i '(sphinx|docutils)'
docutils         0.12   
Sphinx           1.4.9  
sphinx-rtd-theme 0.4.0  

With Ubuntu 18.04 I get:

$ dpkg -l | egrep -i '(sphinx|docutils)' | awk '{print $2, $3}' | sort -k 2
docutils-common 0.14+dfsg-3
python3-docutils 0.14+dfsg-3
python3-sphinx-rtd-theme 0.2.4-1
sphinx-rtd-theme-common 0.2.4-1
python3-alabaster 0.7.8-1
libjs-sphinxdoc 1.6.7-1ubuntu1
python3-sphinx 1.6.7-1ubuntu1
sphinx-common 1.6.7-1ubuntu1

Ok, newer docutils, maybe that's what it is?

With Ubuntu 16.04 I get:

$ dpkg -l | egrep -i '(sphinx|docutils)' | awk '{print $2, $3}' | sort -k 2
docutils-common 0.12+dfsg-1
python-docutils 0.12+dfsg-1
python-sphinx-rtd-theme 0.1.9-1
sphinx-rtd-theme-common 0.1.9-1
python-alabaster 0.7.7-1
libjs-sphinxdoc 1.3.6-2ubuntu1.1
python-sphinx 1.3.6-2ubuntu1.1
sphinx-common 1.3.6-2ubuntu1.1

and now I'm just confused since 16.04 has the same version of docutils and an older sphinx and runs fine; but 18.04 has newer docutils and newer sphinx and runs fine.

Yes. This makes writing broadly portable markup difficult -- originally I did not take the '=' all the way to the right edge of the table because I saw that last example in the above document and assumed that it wasn't necessary to extend the '=' all the way to the right edge. Neither Ubuntu system choked on it, so is this a bug in upstream? Some strange patch added by the distro? Something that ended up in the python wheel? Or a bug in the spec?

--D

Am Freitag, den 20.07.2018, 09:44 -0700 schrieb Darrick J. Wong:

Same here .. I'am completely confused by distros etc. .. I will never be able to control that. Thats why I recommend the virtualenv workflow (as I wrote Christoph):

$ sudo apt install python3-virtualenv

To setup up a environment for building htmldocs:

$ python3 -m virtualenv py3env
$ ./py3env/bin/pip install -r ./Documentation/sphinx/requirements.txt

To build htmldocs with:

 $ SPHINXBUILD=./py3env/bin/sphinx-build make htmldocs

If the env is no longer needed:

 $ rm -r py3env

Am Freitag, den 20.07.2018, 09:44 -0700 schrieb Darrick J. Wong:

I guess in the version history of docutils ;) .. use workflow above to escape from distro chaos. Even missing '=' will be OK.

-- Markus --

Well yes, but it's the virtualenv workflow that produced build errors for Ted; that's what would seem to need fixing?

--D

What would delight me if there was a fixed docutils and Sphinx version which is the **only** thing which subsystem maintainers need to test against. If it fails for that version, then I reject the patch; and if it works on that version (say, 1.4.9), and it fails on some other version that a Distro wants to use for its hermetic build environment (say, 1.7.5), I can tell them, "not my problem, feel free to send me a patch that makes things work for 1.7.5, and doesn't break on 1.4.9 --- or package 1.4.9 for your distro build systems."

I don't really care what the mandated version is --- although given that Fedora and Debian seem to be using 1.7.5, maybe that's the right answer, and too bad for the enterprise distro build systems --- that's why they get paid the big bucks. I just want to know what I'm obliged to test against.

So if Documentation/sphinx/requirements.txt is the only thing which is guaranteed to work, that's fine. But it might be good to document that somewhere.

- Ted

Hm, I read it again and I guess this is more ore less what the documentation will say, see 'Sphinx Install' section. [1]

- distros are fragile
- Documentation/sphinx/requirements.txt is the reference environment

.. even if this needs a virtualenv on specific distros. But what is with the offline build systems Christoph mentioned? I can't see a solution on the sometimes contradictory requirements on a *reference env* and the package reposetorie of a distro.

[1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html?#sphinx-install

--Markus--

[I'm mostly AFK at the moment, so haven't been following this discussion in anything approaching real time].

The above is roughly what I've been aiming for; I'd really like for people to be able to just build the docs without having to go beyond their distribution's repos. It gets challenging at times because Kids These Days seem to have a different view on issues like long-term compatibility. But we've mostly done it; it's pretty rare that we've had docs build problems that have been tied to a specific version of Sphinx.

You all just got lucky :)

jon

Sorry, but this is complete and utter bullshit.

If that was the case we could skip all that RST stuff and just stick to text files. If we want fancy docs it better behave like the rest of the kernel and just works. python virtual environments are just not going to cut it as they lead to a long-term maintainance nightmare.

Am Freitag, den 20.07.2018, 08:04 -0700 schrieb Christoph Hellwig:

Maybe .. I'am not a Kernel developer .. I only try to help with the doc tool- chain. But if I don't want to install anything from the web, the plain text is not as bad.

Really a nightmare? This is what I do on my ubuntu/debian box ... I do prefer py3 so I installed *virtualenv-py3* once from distro's package manager:

$ sudo apt install python3-virtualenv

To setup up a environment for building htmldocs:

$ python3 -m virtualenv py3env
$ ./py3env/bin/pip install -r ./Documentation/sphinx/requirements.txt

now I have all installed in the in the py3env folder. To build htmldocs with:

 $ SPHINXBUILD=./py3env/bin/sphinx-build make htmldocs

If the env is no longer needed:

 $ rm -r py3env

-- Markus --

It is (broken, that is, not bulls*). Building the documentation as part of the kernel build *is* an application.

The point of documentation is that we record knowledge and then anyone ought to be able to read it, no matter what platform they choose, right? So that means that for any given user, if their Linux distro packages the doc build tools they ought to be able to build the fancy html stuff and read it. If they're more accustomed to virtualenv and pip, they ought to be able to build the fancy html and read it too.

Anyway, this whole thread got started because Ted followed the instructions, applied my Documentation/ patches, and the build broke.

egg/wheel/whatever python calls it now.

It will be once we start uploading filesystem format documentation that's full of juicy tables and diagrams...

This isn't going to fly in $corporate hermetically sealed kernel build environment. It's ok to add things to the list of packages pulled in by the provisioning system prior to a kernel build, but we cannot support downloading python wheels for kernel builds. Therefore the distro packages have to work.

Which is fine, because the problem here is that whatever pip installs is less forgiving than the distro packaged versions. :)

--D