What kernel documentation could be

The development community has put a lot of work into the kernel's documentation directory in recent years, with visible results. But the kernel's documentation still falls far short of the standard set by many other projects, and there is a great deal of "tribal knowledge" in our community that is not set down. I'd like to look at the successes and failures of the work so far, but intent to focus on a discussion of what our documentation should be and what we can do to get it there.

Questions to discuss include:

- Bringing an coherent overall structure to Documentation/

- Making it easier for developers to improve the documentation.

- What we would like from Sphinx and what we can do to make it happen

- Making the docs build system less ugly and more maintainable

- Integrating rustdoc documentation

jon

Hi Jon,

I've been wondering recently if it would help to have a place where we can record documentation wishes.

It has happened quite a few times that my searches for particular pieces of documentation in the kernel were unfruitful, or that the related documentation left me none the wiser (one example, which is just an example and not finger-pointing, is the runtime PM documentation in Documentation/power/runtime_pm.rst that feels like reading a calculus-related PHD thesis when all you want is to understand how to add two numbers). I rarely had time to write documentation patches as a result, and my pains were thus left unnoticed by maintainers and developers, who may be willing to improve the documentation if they knew that users were struggling.

Regards,

Laurent Pinchart

[yes, I'm running even slower than usual...]

One of the conclusions that came out of the LSFMM session on MM documentation was that having an overall structure in place makes it easier for people to fill in pieces. If we could impose more of this kind of structure on our docs as a whole, one obvious way to express wishes of this sort would be to add an empty section where the need exists and hope that it inspires somebody to fill it in.

Thanks,

jon

Hi John,

Em Fri, 17 Jun 2022 14:57:10 -0600
Jonathan Corbet <[email protected]> escreveu:

Fully agreed. We dedicated a lot of resources on the past years to convert documentation to ReST and on kernel-doc, but not so much on placing the topics at the right order.

There are a huge amount of such documentation that describe border cases, or have just kernel-doc tags without any (or very small) descriptions. Kernel-doc markups are important, as they help to keep the documentation updated, but explaining subsystem's principles is equally important, as it can save a lot of time from maintainers if the contributors are aware on how and why the subsystem's kAPIs were organized the way it is.

We should imagine documentation as if this is a series of books describing every aspect of the kernel. So, I would expect it to be organized with some structure that would place the topics on some order that would make easier for people to read, being either new contributors or experienced developers that need to touch other parts of the Kernel that are out of his past experiences.

For instance to the kAPI documentation, I would expect something like:

- How to contribute / Submission process;
- An introductory chapter that would describe the most used components that all developers need to know, like:

- how to write a driver;
- fs principles;
- scheduler principles;
- mm most used functions (kmalloc & friends);
- wait queues;
- kernel threads;
- softirqs;
- arch-dependent items to consider (like bit order), weak memory model, etc;
- ...

IMO, we could take a look at the index of some already-existing books like LDD 3, Linux Kernel Development and others in order to get a rough idea about the items to be covered there.
- Bus-related subsystems (USB, PCI, I2C, ...);
- Then, place per-subsystem's documentation, all having their texts explaining basic principles.

IMO, we should write an index file and a couple of new ReST files with an skeleton for the above, and then ask people to help filling the blanks.

A well-prepared skeleton would make life easier. From time to time, we have discussions and patches shifting documentation between different directories.

The big missing feature on Sphinx itself is with regards to per-C-type domain. So, if we have one struct and one function both called "foo", the cross-references will be broken. This issue is known since Sphinx 3.1, and there are already patches fixing it since then (I remember testing them) but, up to today, the Sphinx upstream patches meant to fix it weren't applied yet (as far as I can tell).

Another problem is with regards to the documentation's build time.

One feature we're not using yet is intersphinx. Right now, we can build parts of the documentation with:

make SPHINXDIRS="foo" htmldocs

But, if "foo" have cross-references to "bar", the build will produce warnings and the documentation's cross-references to "bar" won't work.

Properly setting interphinx would allow this to work, as such references would point, instead, to some remote place (like kernel.org).

This could be used to help improving the documentation's build time during doc development.

Also, I would love to have a good way to just produce html (and pdf) from the documents I'm actually working with. The way I do it right now is that I create a "Documentation/foo" directory, adding there just the docs I'm currently working with, placing them on a fake index.rst file.

This way, I can build them really fast, without needing to rebuild everything at the same book. Perhaps something like that could be automated - or Sphinx itself could support something like:

make htmldocs SPHINXFILES="foo.rst bar.rst"

I guess shifting the minimal Sphinx version would help to remove some bad things there.

One thing that probably be solved on a different way is to have a better solution for things like:

Functions for feature 1
=======================

.. kernel-doc:: include/some_header.h
:doc: Feature 1

.. kernel-doc:: include/some_header.h
:functions:
func1
func2

Functions for feature 2
=======================

.. kernel-doc:: include/some_header.h
:doc: Feature 2

.. kernel-doc:: include/some_header.h
:functions:
func3
func4

Perhaps we could change Kernel-doc in a way that doing just:

.. kernel-doc:: include/some_header.h

would be enough.

Won't comment much about that, as never touched any of those.

Life would be a lot easier on this side if rustdoc could support ReST.

Regards,
Mauro

Not sure what you mean by this. Do you mean writing ReST for Rust documentation? For the files inside Documentation/rust/, we already use ReST [*] like the rest. For the source code docs (the ones handled by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate Doc/s or similar?

[*] Though it would be nice to be able to use Markdown (but that is a different topic).

Cheers,
Miguel

Em Sat, 18 Jun 2022 13:03:38 +0200
Miguel Ojeda <[email protected]> escreveu:

I meant that rustdoc could generate ReST output, being included inside the Kernel documentation on a way similar to kerneldoc. This would integrate better with our building system, which can produce output docs on 3 different formats (html, epub and pdf).

Regards,
Mauro

That may lose some of the advantages of the rustdoc output (tailored to Rust code, navigable source code view, etc.).

It may be possible to encode a lot of the information in ReST and then replicate functionality with Sphinx/plugins, possibly having a Rust domain etc., but I am not sure it would be worth it (for the HTML output at least).

Cheers,
Miguel

https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing

Integrating that needs to be done carefully, though, to not make a mess of it.

Yeah, currently that leads to parsing the header four times by kernel-doc the script. The solution would be to finally convert the script to a proper python Sphinx extension that can do caching. (This is how it works in Hawkmoth, FWIW.)

The order in nicely flowing documentation is not necessarily the same as the order in nicely flowing source code. I expect it to be much more acceptable to tweak the rst to achieve this than to do source code rearrangement to generate nice documentation.

BR,
Jani.

Jani Nikula, Intel Open Source Graphics Center

Em Thu, 23 Jun 2022 12:18:50 +0300
Jani Nikula <[email protected]> escreveu:

Yes.

That's one solution, but see: there is already a python extension that currently calls kernel-doc everytime. It could, instead, cache the rst returned by its first run (or a parsed version of it) and use the cached results the other 3 times.

Porting kernel-doc to python could be doable, but not trivial, due to several reasons:

- it should keep running standalone, as otherwise debugging parsing issues on kernel-doc would be a lot harder. In particular, kernel-doc --none is really helpful to report kernel-doc tag errors;
- regressions will likely be introduced on a change like that;
- regular contributors to kernel-doc will need to ramp up with the newer version;
- a port like that could increase the script run time, as the optimizations and regular expressions there could behave different on python.

True, but independently if the script would be rewritten in python or not, one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention there the symbols that belong to each part of the document, e. g. something like:

/**
* DOC: foo
*
* Some comments...
*
* symbols:
* foo
* bar
*/

One advantage is that all documentation will be on a single place, so hopefully it would be easier to maintain.

Also, on documents using DOC: with such new symbols tag, kernel-doc could validate if all documented symbols are singly included at all DOC: sessions and if any symbols are missed/renamed/removed.

Regards,
Mauro

This would mean having kernel-doc the perl script output meta information about the constructs, complicating it even more.

This is trivial and sensible to do in the python extension too.

Any change can cause regressions... but previously a diff on the html outputs generated by both versions has been an effective way of verifying the changes.

You say it like it's a bad thing. :p

Sometimes I regret not throwing kernel-doc the script in the curb while doing the Sphinx conversion.

This is certainly possible. It's somewhat countered by the python extension being able to cache stuff.

I believe preprocessing of what's supposed to be rst leads to problems. It's difficult to define proper escaping to ensure stuff gets parsed at the correct stage in the correct way. It should just be rst.

Part of the reason the old docbook thing was such a big mess were the "impedance mismatches" at each separate stage of processing.

In general, stick to the mechanisms provided by rst and Sphinx. Extend on them using Sphinx extensions, not by manually parsing rst.

BR,
Jani.

Jani Nikula, Intel Open Source Graphics Center

I've been pondering on this for a bit, and would like to do it, but I don't know when I might find the time for it.

Yes, of course. As Jani noted, that's just how you would do it, not a problem.

The nice thing is ... we already have a really nice regression test in the form of the current docs build and diff.

We have those? That script is a nightmare and nobody goes near it if they can possibly avoid it. I would expect to have more contributors with a decent Python version that doesn't include 25 years of regex accretion.

It could also decrease it by improving caching opportunities, getting rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc.

I've actually, in a spare moment or two, been doing some profiling of the kernel docs build and trying to track down the sources of the slowness. I am thinking that nearly 700 *million* calls to the iterator for the C-domain symbol list might have something to do with it...

I'm not quite sure I get this...you want to put the TOC tree in the source comments? This looks like the kind of thing that nobody ever remembers to update, but maybe I'm missing something.

Thanks,

jon

Em Thu, 23 Jun 2022 07:40:45 -0600
Jonathan Corbet <[email protected]> escreveu:

There is already a version of kernel-doc written in Python, made by Markus Heiser:

https://github.com/return42/linuxdoc/blob/master/linuxdoc/kernel_doc.py

It could be a starting point.

True. We can test the results with both versions and even check how performance is affected.

Well, the kernel-doc version from Markus in python is ~23% bigger than our current kernel-doc. Granted, it could be due to comments and blank lines, but basically, the same regexes that are in perl would also be need to be replicated in python, as parsing a C code with regexes is not a trivial task.

So, in terms of complexity, I doubt much would change by porting it to python.

Now, one of the things that the kernel-doc does is that it has the parse code and the output logic, which actually has an an abstraction to let it to produce results on different formats (currently, none, man and rst - we dropped some other formats from it). This abstraction increases its complexity. This is something that doesn't need to be replicated on any ports.

Wow, that's a lot!

No, it won't generate a TOC tree. It would instead reorder how kernel-doc would output the symbols.

That's no different than what we have already at the .kernel-doc directives, e. g. it would be a replacement for:

.. kernel-doc:: include/some_header.h
:doc: foo

.. kernel-doc:: include/some_header.h
:functions:
foo
bar

The problem we currently have is that the above pattern means that one or more .rst files would contain a list of symbols that are actually documented at some_header.h. People updating such file will very likely forget to update .rst files, leading to missing documentation. Also, the same symbol could be included on different .rst files.

I remember I had to fix myself duplicated symbols inclusion while trying to reduce the docs build time on a few *.h files that were included hundreds of times. I bet if someone checks again, duplicated inclusions of the same symbol and missed symbols that are documented on their sources, but aren't included at any .rst file.

Currently, detecting it is very hard, because the symbol lists are on different files and the same header are sometimes included on different rst files. By placing the symbol list inside the source file, it makes very simple for kernel-doc to check if all documented symbols are inside the "DOC:" markups, producing errors otherwise.

Regards,
Mauro

Hi Jon, Mauro,

with great interest I have read this thread !

If I can help, I'm happy to participate again. TBH: The kernel-doc implementation from mine could only be a piece of the puzzle to improve the process (not without raising other questions).

But similar to Jani, I see the priority problems on a different level level.

We have too many demands that reach into the smallest niches. I believe that some parts must be generalized and some request must be abandoned.

In my opinion, many problems are homemade, to take just one example:

IMO It is unnecessary that the build-chain must run on all platforms and with all distributions.

Who observes the Sphinx-doc & docutils development since (>15) years is aware that with various (old) Sphinx-doc & docutils versions no stable results can be produced, not without complicating the build-chain. And this is exactly the situation we are facing today.

The build chain of documentation has nothing to do with kernel development (at least in my opinion) and should be decoupled from it:
maintaining one defined build environment is enough work ... this becomes especially clear if you (as Jani recommends) rely more on sphinx-modules and widely used tools.

Another point on which I now have a clear view are the regular
expressions: no one likes them (me too), but I can't think of a general solution (parser) given the number of requirements for parsing source code we have.

I would like to support the kernel community again, but at the moment I have difficulties to follow the many exceptions. I would be happy if you keep me in CC .. may be I find my place again back in the kernel-doc development :-)

Thanks and with best regards

-- Markus --

Am 24.06.22 um 09:33 schrieb Mauro Carvalho Chehab:

The counterargument to this is that we want as many developers as possible to be able to build the docs and contribute to them. We can't complain that developers have broken the docs build if we don't do what we can to help them do the build themselves.

One of our longstanding contributors is on Sphinx 1.8.5:

https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/

I would like to narrow our range of supported versions, but I really don't want to cut people out.

The real question, perhaps, is whether requiring people to set up a virtualenv to run a supported version would be too much. It's something I worry about.

I fear you're right. There are nice C parsers for Python, but they don't work at the level we need; we need the comments, unexpanded macros, etc.

That said, the regex expression in kernel-doc is out of hand, with no documentation, lots of duplication, etc.

We would be glad to have you back.

Remember that the reason I was hesitant to take your kernel-doc rewrite at the time was that we were already thrashing too many things and I wanted to hold at least one piece stable. Hopefully we're past that phase now...:)

Thanks,

jon

I know for on that I will not setup pything venvs as they are massive pain. Anyting that isn't available in a somewhat recent distro using the package manage is a complete no-go. And software that changes so frequently that this question even arises seem rather suspect to me in general for something like documentation that should have a wide range of contributors.

Em Mon, 27 Jun 2022 09:27:54 -0600
Jonathan Corbet <[email protected]> escreveu:

Agreed.

If he migrates to Leap 15.4, he'll either get 2.3.1 (default package) or 4.2.0 (alternative package):

	https://download.opensuse.org/distribution/leap/15.4/repo/oss/noarch/python3-Sphinx-2.3.1-150400.1.7.noarch.rpm
	https://download.opensuse.org/distribution/leap/15.4/repo/oss/noarch/python3-Sphinx_4_2_0-4.2.0-150400.11.6.noarch.rpm

Installing distro-specific packages likely covers a broader group of developers than requiring virtualenv/venv.

However, if we decide to move on, removing the version-dependent hacks from the build system, it would mean that the minimal version would be 3.1.0. On such case, there won't be Sphinx packages on some LTS distros. Developers using such distros will be forced to use virtualenv/venv.

In any case, the script which checks if sphinx-build runs can recommend either way. Yet, as the current default is venv, tests are needed to check if it recommending python3-sphinx packages still work on all supported distros.

Such script could also be changed to detect if the distro has a native package >= 3.1.0, recommending it by default, falling back to virtualenv/venv otherwise.

Regards,
Mauro

Hi Mauro,

Ubuntu 20.04 LTS also has 1.8.5.
Looks like I'll get 4.3.2 after upgrading to 22.04 LTS (which is only about two months old).

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Em Tue, 28 Jun 2022 09:57:29 +0200
Geert Uytterhoeven <[email protected]> escreveu:

Yeah, LTS distros are the main reason why the default recommendation is to install Sphinx via virtualenv/venv.

With RHEL9, SUSE 15 SP4 (plus CenOS and openSUSE Leap versions of them), plus Ubuntu 22.04, maybe we can change the default to recommend installing it via distro-provided packages. Yet, Debian bulseye has 2.5.0. Only bookworm, scheduled for mid 2023, will come with Sphinx > 2.

Regards,
Mauro

Hi Mauro,

Debian current Stable (Bullseye) has python3-sphinx version 3.4.3 (python-sphinx is version 1.8.4).

Cheers,
Stephen Rothwell

Just for the curious ...

The Sphinx C domain code makes this elaborate tree data structure out of all the identifier names it has picked up - including the names of function parameters when they are provided in prototypes for some reason. This is the structure that is consulted whenever we want to resolve a cross-reference, which is fairly often with automarkup enabled.

How does it work? The algorithm is, as far as I can tell:

- Serialize the whole tree in a sort of breadth-first traversal

- Make a linear pass through the *entire* list, comparing the identifier name with each entry and accumulating a list of all the matches found.

- Return the first match and throw away the rest.

The result is O(n^2) behavior and, in the kernel docs build, n gets to be fairly large.

I went in with a crowbar and sledgehammer and replaced some of the list searches with a dict lookup, resulting in about a 20% speedup in the full htmldocs build with Sphinx 5.0.2. A couple of automarkup optimizations result in about a 27% speedup overall. And I didn't break any cross-references.

I think it's possible to do better, but this is a start. I'll post the automarkup changes as soon as I can, but I need to verify them across the whole range of Sphinx versions. For the Sphinx stuff, I'll need to turn my hatchetwork into something presentable and figure out how to contribute it upstream, but it seems worth the effort.

Thanks,

jon

Em Fri, 24 Jun 2022 16:57:56 -0600
Jonathan Corbet <[email protected]> escreveu:

Great to have a 20% speedup here! Yet, I would expect an even better performance improvement by replacing 700 million calls from linear search to dict, as it would change from O(n^2) to O(1) at the average case [1], but the speedup gain actually depends on the actual number of symbols we have defined.

[1] According with:
https://wiki.python.org/moin/TimeComplexity#dict
Worse case is O(n) - when collisions are common which is unlikely.

Great!

Indeed.

Regards,
Mauro

I said "some" - this was a proof-of-concept hack. The complexity of their symbol lookup code is ... daunting ... and not something that gets fixed in one place. More research is required...

jon

Note that at least for me, the issue with building docs isn't hugely performance related.

Yes, yes, it would be good if it was faster. But..

The primary problems I see with building the docs (and thus checking them, the same way I do an allmodconfig build test) is

(a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.

Seriously. There's a very solid reason why I require the standard kernel build to be entirely warning-free. Warnings are *BAD*. Even a single false-positive warning invalidates all other warnings.

And the doc build isn't about some "single warning" thing.

(b) it requires some unusual build tools

Now, (b) may some historical oddity, but at least if you have Fedora installed and it still has sphinx version 2.2.11 or something like that.

And when you try to build docs, the makefile gives you some random pip install thing that is entirely bogus.

The proper fix is to actually remove that old sphinx environment entirely, and install "python3-sphinx" instead, but that's not at all what the "to upgrade Sphinx" docs in the kernel say when you try to make the docs.

Anyway, (b) is one of those "the docs about the docs are wrong" things and ironic - but easily fixable if you know about it.

But (a) then makes it all entirely pointless as far as I'm concerned. The doc build could take five seconds, and I *still* wouldn't bother, because building docs doesn't actually do anything useful for me. It's just noise.

Linus

Em Sat, 25 Jun 2022 11:11:40 -0700
Linus Torvalds <[email protected]> escreveu:

When the environment is set for building docs, almost all output there are actually due to build errors/warnings that got introduced by patches touching documentation that were never build-tested.

There are 4 components that report errors during "make htmldocs".

1. Documentation cross-reference check.

	$ scripts/documentation-file-ref-check --warn
	Warning: Documentation/admin-guide/kernel-parameters.txt references a file that doesn't exist: Documentation/virt/kvm/amd-memory-encryption.rst
	Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst
	...

Those warnings are result of build-untested patches moving/renaming/removing documentation without updating all documentation.

On almost all Kernel versions, I usually send a couple of patch series addressing them, but incomplete patches that forget about updating references keep being merged.

Without htmldocs build tests by patch authors, this is an endless job.

2. ABI documentation check:

	$ ./scripts/get_abi.pl validate
	Warning: /sys/bus/iio/devices/iio:deviceX/fault_ovuv is defined 2 times:  Documentation/ABI/testing/sysfs-bus-iio-temperature-max31865:0  Documentation/ABI/testing/sysfs-bus-iio-temperature-max31856:14
	Warning: /sys/bus/iio/devices/iio:deviceX/in_filter_notch_center_frequency is defined 2 times:  Documentation/ABI/testing/sysfs-bus-iio:1948  Documentation/ABI/testing/sysfs-bus-iio-temperature-max31865:12
	Warning: /sys/bus/iio/devices/triggerX/sampling_frequency is defined 2 times:  Documentation/ABI/testing/sysfs-bus-iio:96  Documentation/ABI/testing/sysfs-bus-iio-timer-stm32:92
	Warning: /sys/devices/system/cpu/cpuX/topology/core_id is defined 2 times:  Documentation/ABI/stable/sysfs-devices-system-cpu:38  Documentation/ABI/testing/sysfs-devices-system-cpu:69

Those warnings indicate problems with userspace API definitions, as the same symbol has different meanings.

Talking about ABI, the output is even worse if the script is used to check if the ABIs used on a running system are all documented or not. On my laptop (Kernel 5.18.5-200.fc36.x86_64), I'm getting, for instance, 2815 undocumented or badly-specified sysfs ABI symbols:

	$ sudo ./scripts/get_abi.pl undefined >errors
	Reading /sys directory contents...done.
	Converting ABI What fields into regexes...done.
	4:59: processing sysfs files... done
	$ wc -l errors
	2815

3. kernel-doc warnings.

Those indicate kernel-doc markups with troubles. The output is the same as running:

	$ ./scripts/kernel-doc --none ./include/linux/fscache.h
	./include/linux/fscache.h:269: warning: Function parameter or member 'cookie' not described in 'fscache_use_cookie'
	./include/linux/fscache.h:269: warning: Excess function parameter 'object' description in 'fscache_use_cookie'
	./include/linux/fscache.h:286: warning: Function parameter or member 'cookie' not described in 'fscache_unuse_cookie'
	./include/linux/fscache.h:286: warning: Excess function parameter 'object' description in 'fscache_unuse_cookie'

In this particular case, kernel-doc markup describes a non-existent "object" symbol, while the function prototype has "cookie". Probably result of some rename at the function prototype that was not validated via html build.

4. Sphinx warnings.

Most of them are one line warnings:

Documentation/arm/samsung-s3c24xx/cpufreq.rst:2: WARNING: Explicit markup ends without a blank line; unexpected unindent. Documentation/admin-guide/device-mapper/writecache.rst:23: WARNING: Unexpected indentation. Documentation/vm/highmem:166: ./include/linux/highmem.h:154: WARNING: Inline emphasis start-string without end-string. Documentation/vm/highmem:166: ./include/linux/highmem.h:157: WARNING: Inline emphasis start-string without end-string.

But on errors, Sphinx is very noisy, as it outputs the offending lines or tables.

For instance, on next-20220620, some patch broke a table. The error there is very noisy, as it outputs the entire broken table.

All the Sphinx warnings could be avoided if people start testing documentation changes. A policy of making kernel htmldocs warning-free would have solved all the above issues.

I can submit some patches addressing issues along this week against linux-next (or against your tree?), but without developers trying to build documentation, we'll keep having issues.

That's likely the case. You probably installed Sphinx via pip on, let's say, Fedora 35. After upgrading to Fedora 36, python version got incremented, and the past install won't work anymore, because it would be trying to use patch libraries from a different directory.

Since this changeset:

a8b380c379ef ("scripts: sphinx-pre-install: only ask to activate valid venvs")

the documentation build checks if this command works:

	sphinx-build --version

If it doesn't, it will give instructions about how to install Sphinx.

The script which checks for Sphinx availability is capable of either recommend distro-specific Sphinx package or the usage of python venv.

As Sphinx release cycle is fast, it was opted to use venv by default, so, it currently recommends the usage of pip.

The same script called at htmldocs build time can be used to recommend the usage of the distro-provided package. Running it in Fedora, it would give:

./scripts/sphinx-pre-install --no-virtualenv
Detected OS: Fedora release 36 (Thirty Six).
ERROR: please install "python-sphinx", otherwise, build won't work.
Warning: better to also install "sphinx_rtd_theme".
Warning: better to also install "texlive-ctex".
You should run:

sudo dnf install -y python3-sphinx python3-sphinx_rtd_theme texlive-ctex
note: If you want pdf, you need at least Sphinx 2.4.4.

Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 955.

It would be trivial to change the default to recommend the usage of the distro package. Yet:

1. Newer versions of Sphinx are two times slower than 2.4.x;

2. Recommending the same version everywhere ensures that the documentation will be compatible with Sphinx >= 2.4.

3. Older LTS distros may not have a Sphinx version compatible with the Kernel.

The hope is that, if building it would be fast enough, developers and maintainers would actually check if the build succeeds without warnings, as ultimately, the big noise is due to patches merged without being even build-tested.

Regards,
Mauro

Em Sun, 26 Jun 2022 08:55:24 +0100
Mauro Carvalho Chehab <[email protected]> escreveu:

Just sent a patch series addressing the existing issues against next-20220624:

	https://lore.kernel.org/linux-doc/[email protected]/T/#t

With that, the number of warnings will be reduced to only 3:

	$ ./scripts/documentation-file-ref-check 
	Documentation/dev-tools/kunit/run_wrapper.rst: Documentation/dev-tools/kunit/non_uml.rst
	Documentation/devicetree/bindings/regulator/siliconmitus,sm5703-regulator.yaml: Documentation/devicetree/bindings/mfd/siliconmitus,sm5703.yaml
	drivers/acpi/device_pm.c: Documentation/firmware-guide/acpi/low-power-probe.rst

Those are actually useful for doc discussions, as they show one of the issues we're currently having.

Sometimes, support for some functionality crosses multiple subsystems. On such cases, part of the support may end being merged on a future linux-next version - or even on a future Kernel version.

That seems to be the case of sm5703 support. The regulator bindings got already merged, but the mfd ones weren't.

So, there's not much that could be done to fix siliconmitus,sm5703-regulator.yaml broken reference to mfd/siliconmitus,sm5703.yaml.

The same seems to apply to low-power-probe.rst: the patch adding it was not merged yet at linux-next:

	https://lore.kernel.org/all/[email protected]/

but the patch using it at drivers/acpi/device_pm.c was merged.

Regards,
Mauro

Em Sun, 26 Jun 2022 08:55:24 +0100
Mauro Carvalho Chehab <[email protected]> escreveu:

...

...

...

Btw, once we fix the errors from the above checks, one of the things that could be done in order to avoid noisy doc builds would be to run this during normal Kernel build, if CONFIG_WERROR is set (and if .git is present at the source build dir):

	./scripts/documentation-file-ref-check --warn
	./scripts/get_abi.pl validate
	./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) 

aborting the build on such warnings.

On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it won't make much difference at the build time, and doing that would have avoided ~100 warnings during htmldocs build against current linux-next.

Regards,
Mauro

Couldn't we add this to the build bots and ask the authors to fix the commits?

Thanks,
Liam

Isn't for-next already integrated? Maybe not. I got a mail from Andrew telling me I had a kernel-doc copy-paste error.

Integration with build-bots would be great. I generally aim to add kernel-doc for all non-trivial code that I add. But during different patchset versions copy-paste errors do sometimes happen (e.g., for git grep/sed calls).

That last one is quite the command line...

There's reports for at least some of this for -next already, at least for newly introduced warnings. If we could get 0day or something else that picks stuff off the lists that would help a lot I think, as would getting it into the bots that people who like to fix up warnings tend to be using.

Em Mon, 27 Jun 2022 17:27:27 +0100
Mark Brown <[email protected]> escreveu:

Sent a patch series addressing those at linux-next:

	https://lore.kernel.org/linux-doc/[email protected]/T/#t

I submited a patch fixing one such issue:

	https://lore.kernel.org/all/1e92337c1ef74f5eb9e1c1871e20b858b490d269.1656235926.git.mchehab@kernel.org/ 

and Jonathan Cameron submitted a series addressing those:

	https://lore.kernel.org/linux-iio/[email protected]/T/#t 

Sent a patch series addressing all of those, also against linux-next:

	https://lore.kernel.org/linux-doc/[email protected]/T/#t

next-20220627 has currently:

	32 cross broken docs cross reference warnings;
	 4 duplicated ABI symbols;
	60 kernel-doc warnings.

After having those three series applied, the 3 above commands would produce just 3 warnings:

Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst
Warning: Documentation/devicetree/bindings/regulator/siliconmitus,sm5703-regulator.yaml references a file that doesn't exist: Documentation/devicetree/bindings/mfd/siliconmitus,sm5703.yaml
Warning: drivers/acpi/device_pm.c references a file that doesn't exist: Documentation/firmware-guide/acpi/low-power-probe.rst

Those 3 warnings depend on non-merged documents.

After those ~100 warnings that can be quickly checked with the 3 above command lines in a couple of seconds, there are the Sphinx errors. Reducing the build time (currently at 10 mins order) to something more palatable could help having more people checking them before submitting patches and waiting for 0day results.

True. Yet, placing it at docs/Makefile for the "make all" target should be trivial and will hopefully make more Kernel developers to check for some of the documentation issues.

The other two are already called, depending on some config vars:

	# Check for broken documentation file references
	ifeq ($(CONFIG_WARN_MISSING_DOCUMENTS),y)
	$(shell $(srctree)/scripts/documentation-file-ref-check --warn)
	endif

	# Check for broken ABI files
	ifeq ($(CONFIG_WARN_ABI_ERRORS),y)
	$(shell $(srctree)/scripts/get_abi.pl validate --dir $(srctree)/Documentation/ABI)
	endif

But they don't currently stop the build if CONFIG_WERROR is enabled.

Yes, newly-introduced warnings are monitored there, but still patches adding new warnings keep being merged, requiring extra periodic janitorial work in order to reduce the build noise that would otherwise increase with time.

Regards,
Mauro

I poked the zero-day bot owners. They are looking into it.

Reminder that the zero-day folks are open to suggestions on how to improve coverage. Just ping me or Dave Hansen and we can pass on requests.

-Tony

Those of us who do lots of docs builds see it a bit differently :)

I totally agree, which is why fixing warnings has been at the top of the priority list for a long time. That has mostly been done, with Mauro having taken on the bulk of that work.

It is hard, though, to maintain the warning-free condition. Warnings are easy to add and most developers don't check. Most documentation changes don't go through the docs tree, so I can't enforce that kind of check (and trying to force docs changes through my tree would be a cure far worse than the disease).

Making the docs build faster might make it easier to insist that more checking is done.

I'd be curious to know about how you got into that situation. Fedora handles this stuff pretty readily. I'll take another look at what we have there.

jon

At some point I just did "dnf install sphinx", and it got me a sphinx install.

So then the kernel doc build system sees "oh, you need a newer version of sphinx" or whatever, and suggests you do that horrible virtual python sandbox thing and do a random pip install in that, which I really don't want to do.

So I ignore it entirely, obviously.

End result: it took me a *loong* time to go "I wonder why modern Fedora ships with an ancient version of sphinx, since the modern version is apparently 4.x something".

And only at THAT point do you realize that there are three completely different projects called "sphinx".

There's the docbuild one that comes from python, which is the sphinx I want.

There's the search engine 'sphinx', which is what is packaged as "sphinx" in Fedora, and which the documentation code obviously refuses to use.

There's also CMU sphinx, which is a voice recognition system.

End result: building the docs can be really confusing if you have the wrong sphinx installed, and the "help message" is actually misleading and very unhelpful.

Linus

Cc: Mauro Carvalho Chehab <[email protected]>,
[email protected], Jonathan Corbet <[email protected]>, [email protected]
Cc: "Jonathan Corbet" <[email protected]>
Cc: "Mauro Carvalho Chehab" <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Mauro Carvalho Chehab <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]

Checking if Sphinx is available and read to run is tricky, and may involve installing several packages for the document build to happen.

There are two options to install Sphinx:

- via distro-provided packages;
- via pip, using virtualenv/venv.

Some recent discussions showed a couple of misleading instructions.

This series improves the Sphinx detection by:

- Fixing the logich with checks if a past venv is working and recommend just enabling it instead of installing a new venv;
- Detect if sphinx-build stopped working on a venv. This may happen during distribution updates;
- Move the PDF minimal version to be later, in order for it to be printed only after finishing the Sphinx version check;

Additionally, as now the Sphinx provided on almost all modern distros are above the minimal required version, place instructions about how to install Sphinx from the distro-provided packages after placing the instructions for installing it via venv.

This will hopefully help to have more developers checking documentation builds with

make htmldocs

Mauro Carvalho Chehab (4):
scripts: sphinx-pre-install: fix venv version check logic
scripts: sphinx-pre-install: report broken venv
scripts: sphinx-pre-install: check for PDF min version later on
scripts: sphinx-pre-install: provide both venv and package installs

 scripts/sphinx-pre-install | 74 ++++++++++++++++++++++++++++----------
 1 file changed, 55 insertions(+), 19 deletions(-)

2.36.1

The logic which checks if the venv version is good enough but was not activated is broken: it is checking against the wrong val, making it to recommend to re-create a venv every time. Fix it.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH 0/4] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

index f126ecbb0494..ae8c49734899 100755
@@ -741,7 +741,7 @@ sub recommend_sphinx_upgrade()
 
 	# Get the highest version from sphinx_*/bin/sphinx-build and the
 	# corresponding command to activate the venv/virtenv
-	$activate_cmd = get_virtenv();
+	($activate_cmd, $venv_ver) = get_virtenv();
 
 	# Store the highest version from Sphinx existing virtualenvs
 	if (($activate_cmd ne "") && ($venv_ver gt $cur_version)) {

@@ -759,10 +759,14 @@ sub recommend_sphinx_upgrade()
 	# Either there are already a virtual env or a new one should be created
 	$need_pip = 1;
 
+	return if (!$latest_avail_ver);
+
 	# Return if the reason is due to an upgrade or not
 	if ($latest_avail_ver lt $rec_version) {
 		$rec_sphinx_upgrade = 1;
 	}
+
+	return $latest_avail_ver;
 }
 
 #

@@ -820,7 +824,7 @@ sub recommend_sphinx_version($)
 	}
 
 	# Suggest newer versions if current ones are too old
-	if ($latest_avail_ver && $cur_version ge $min_version) {
+	if ($latest_avail_ver && $latest_avail_ver ge $min_version) {
 		# If there's a good enough version, ask the user to enable it
 		if ($latest_avail_ver ge $rec_version) {
 			printf "\nNeed to activate Sphinx (version $latest_avail_ver) on virtualenv with:\n";

@@ -897,7 +901,7 @@ sub check_needs()
 		}
 	}
 
-	recommend_sphinx_upgrade();
+	my $venv_ver = recommend_sphinx_upgrade();
 
 	my $virtualenv_cmd;
 

2.36.1

After distro upgrades, the directory names for python may change. On such case, the previously-created venv will be broken, and sphinx-build won't run.

Add a logic to report it.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH 0/4] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 6 ++++++
 1 file changed, 6 insertions(+)

index ae8c49734899..18537e5af692 100755
@@ -720,6 +720,12 @@ sub get_virtenv()
 		next if (! -f $sphinx_cmd);
 
 		my $ver = get_sphinx_version($sphinx_cmd);
+
+		if (!$ver) {
+			$f =~ s#/bin/activate##;
+			print("Warning: virtual envionment $f is not working.\nPython version upgrade? Remove it with:\n\n\trm -rf $f\n\n");
+		}
+
 		if ($need_sphinx && ($ver ge $min_version)) {
 			return ($f, $ver);
 		} elsif ($ver gt $cur_version) {

2.36.1

Better to add the PDF note late for venv recommendation.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH 0/4] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

index 18537e5af692..930a6d058c12 100755
@@ -785,12 +785,13 @@ sub recommend_sphinx_version($)
 {
 	my $virtualenv_cmd = shift;
 
-	if ($latest_avail_ver lt $min_pdf_version) {
-		print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
-	}
-
 	# Version is OK. Nothing to do.
-	return if ($cur_version && ($cur_version ge $rec_version));
+	if ($cur_version && ($cur_version ge $rec_version)) {
+		if ($cur_version lt $min_pdf_version) {
+			print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+		}
+		return;
+	};
 
 	if (!$need_sphinx) {
 		# sphinx-build is present and its version is >= $min_version

@@ -837,6 +838,10 @@ sub recommend_sphinx_version($)
 			printf "\t. $activate_cmd\n";
 			deactivate_help();
 
+			if ($latest_avail_ver lt $min_pdf_version) {
+				print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+			}
+
 			return;
 		}
 

2.36.1

As it is not a consensus about installing sphinx using venv, and modern distributions are now shipping with Sphinx versions above the minimal requirements to build the docs, provide both venv and package install commands by default.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH 0/4] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 43 ++++++++++++++++++++++++++++----------
 1 file changed, 32 insertions(+), 11 deletions(-)

index 930a6d058c12..27bac4fbe35b 100755
@@ -25,6 +25,7 @@ my $need_sphinx = 0;
 my $need_pip = 0;
 my $need_virtualenv = 0;
 my $rec_sphinx_upgrade = 0;
+my $verbose_warn_install = 1;
 my $install = "";
 my $virtenv_dir = "";
 my $python_cmd = "";

@@ -103,10 +104,12 @@ sub check_missing(%)
 			next;
 		}
 
-		if ($is_optional) {
-			print "Warning: better to also install \"$prog\".\n";
-		} else {
-			print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+		if ($verbose_warn_install) {
+			if ($is_optional) {
+				print "Warning: better to also install \"$prog\".\n";
+			} else {
+				print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+			}
 		}
 		if (defined($map{$prog})) {
 			$install .= " " . $map{$prog};

@@ -386,7 +389,8 @@ sub give_debian_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo apt-get install $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo apt-get install $install\n");
 }
 
 sub give_redhat_hints()

@@ -458,10 +462,12 @@ sub give_redhat_hints()
 
 	if (!$old) {
 		# dnf, for Fedora 18+
-		printf("You should run:\n\n\tsudo dnf install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo dnf install -y $install\n");
 	} else {
 		# yum, for RHEL (and clones) or Fedora version < 18
-		printf("You should run:\n\n\tsudo yum install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo yum install -y $install\n");
 	}
 }
 

@@ -509,7 +515,8 @@ sub give_opensuse_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo zypper install --no-recommends $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo zypper install --no-recommends $install\n");
 }
 
 sub give_mageia_hints()

@@ -553,7 +560,8 @@ sub give_mageia_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo $packager_cmd $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo $packager_cmd $install\n");
 }
 
 sub give_arch_linux_hints()

@@ -583,7 +591,8 @@ sub give_arch_linux_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo pacman -S $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo pacman -S $install\n");
 }
 
 sub give_gentoo_hints()

@@ -610,7 +619,8 @@ sub give_gentoo_hints()
 
 	return if (!$need && !$optional);
 
-	printf("You should run:\n\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n");
 
 	my $imagemagick = "media-gfx/imagemagick svg png";
 	my $cairo = "media-gfx/graphviz cairo pdf";

@@ -873,6 +883,16 @@ sub recommend_sphinx_version($)
 	printf "\t. $virtenv_dir/bin/activate\n";
 	printf "\tpip install -r $requirement_file\n";
 	deactivate_help();
+
+	printf "\nAnother option would be to install Sphinx package with:\n";
+
+	%missing = ();
+	$pdf = 0;
+	$optional = 0;
+	add_package("python-sphinx", 0);
+	check_python_module("sphinx_rtd_theme", 1);
+
+	check_distros();
 }
 
 sub check_needs()

@@ -945,6 +965,7 @@ sub check_needs()
 	check_program("latexmk", 2) if ($pdf);
 
 	# Do distro-specific checks and output distro-install commands
+	$verbose_warn_install = 0;
 	check_distros();
 
 	if (!$python_cmd) {

2.36.1

Checking if Sphinx is available and read to run is tricky, and may involve installing several packages for the document build to happen.

There are two options to install Sphinx:

- via distro-provided packages;
- via pip, using virtualenv/venv.

Some recent discussions showed a couple of misleading instructions.

This series improves the Sphinx detection by:

- Fixing the logich with checks if a past venv is working and recommend just enabling it instead of installing a new venv;
- Detect if sphinx-build stopped working on a venv. This may happen during distribution updates;
- Move the PDF minimal version to be later, in order for it to be printed only after finishing the Sphinx version check;

Additionally, as now the Sphinx provided on almost all modern distros are above the minimal required version, place instructions about how to install Sphinx from the distro-provided packages after placing the instructions for installing it via venv.

This will hopefully help to have more developers checking documentation builds with

make htmldocs

---

v2:
- Fix some issues at the logic that was recommending installing via distro-package manager;
- Add a notice about Sphinx 3.0+ false-positive warnings due to a Sphinx bug. People using a version different than the recommended one should know about that.

Mauro Carvalho Chehab (5):
scripts: sphinx-pre-install: fix venv version check logic
scripts: sphinx-pre-install: report broken venv
scripts: sphinx-pre-install: check for PDF min version later on
scripts: sphinx-pre-install: provide both venv and package installs
scripts: sphinx-pre-install: place a warning for Sphinx >= 3.0

 scripts/sphinx-pre-install | 90 +++++++++++++++++++++++++++++---------
 1 file changed, 69 insertions(+), 21 deletions(-)

2.36.1

The logic which checks if the venv version is good enough but was not activated is broken: it is checking against the wrong val, making it to recommend to re-create a venv every time. Fix it.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH v2 0/5] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

index f126ecbb0494..ae8c49734899 100755
@@ -741,7 +741,7 @@ sub recommend_sphinx_upgrade()
 
 	# Get the highest version from sphinx_*/bin/sphinx-build and the
 	# corresponding command to activate the venv/virtenv
-	$activate_cmd = get_virtenv();
+	($activate_cmd, $venv_ver) = get_virtenv();
 
 	# Store the highest version from Sphinx existing virtualenvs
 	if (($activate_cmd ne "") && ($venv_ver gt $cur_version)) {

@@ -759,10 +759,14 @@ sub recommend_sphinx_upgrade()
 	# Either there are already a virtual env or a new one should be created
 	$need_pip = 1;
 
+	return if (!$latest_avail_ver);
+
 	# Return if the reason is due to an upgrade or not
 	if ($latest_avail_ver lt $rec_version) {
 		$rec_sphinx_upgrade = 1;
 	}
+
+	return $latest_avail_ver;
 }
 
 #

@@ -820,7 +824,7 @@ sub recommend_sphinx_version($)
 	}
 
 	# Suggest newer versions if current ones are too old
-	if ($latest_avail_ver && $cur_version ge $min_version) {
+	if ($latest_avail_ver && $latest_avail_ver ge $min_version) {
 		# If there's a good enough version, ask the user to enable it
 		if ($latest_avail_ver ge $rec_version) {
 			printf "\nNeed to activate Sphinx (version $latest_avail_ver) on virtualenv with:\n";

@@ -897,7 +901,7 @@ sub check_needs()
 		}
 	}
 
-	recommend_sphinx_upgrade();
+	my $venv_ver = recommend_sphinx_upgrade();
 
 	my $virtualenv_cmd;
 

2.36.1

After distro upgrades, the directory names for python may change. On such case, the previously-created venv will be broken, and sphinx-build won't run.

Add a logic to report it.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH v2 0/5] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 6 ++++++
 1 file changed, 6 insertions(+)

index ae8c49734899..18537e5af692 100755
@@ -720,6 +720,12 @@ sub get_virtenv()
 		next if (! -f $sphinx_cmd);
 
 		my $ver = get_sphinx_version($sphinx_cmd);
+
+		if (!$ver) {
+			$f =~ s#/bin/activate##;
+			print("Warning: virtual envionment $f is not working.\nPython version upgrade? Remove it with:\n\n\trm -rf $f\n\n");
+		}
+
 		if ($need_sphinx && ($ver ge $min_version)) {
 			return ($f, $ver);
 		} elsif ($ver gt $cur_version) {

2.36.1

Better to add the PDF note late for venv recommendation.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH v2 0/5] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

index 18537e5af692..930a6d058c12 100755
@@ -785,12 +785,13 @@ sub recommend_sphinx_version($)
 {
 	my $virtualenv_cmd = shift;
 
-	if ($latest_avail_ver lt $min_pdf_version) {
-		print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
-	}
-
 	# Version is OK. Nothing to do.
-	return if ($cur_version && ($cur_version ge $rec_version));
+	if ($cur_version && ($cur_version ge $rec_version)) {
+		if ($cur_version lt $min_pdf_version) {
+			print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+		}
+		return;
+	};
 
 	if (!$need_sphinx) {
 		# sphinx-build is present and its version is >= $min_version

@@ -837,6 +838,10 @@ sub recommend_sphinx_version($)
 			printf "\t. $activate_cmd\n";
 			deactivate_help();
 
+			if ($latest_avail_ver lt $min_pdf_version) {
+				print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+			}
+
 			return;
 		}
 

2.36.1

As it is not a consensus about installing sphinx using venv, and modern distributions are now shipping with Sphinx versions above the minimal requirements to build the docs, provide both venv and package install commands by default.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH v2 0/5] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 55 +++++++++++++++++++++++++++++---------
 1 file changed, 42 insertions(+), 13 deletions(-)

index 930a6d058c12..106d75425d3f 100755
@@ -25,6 +25,7 @@ my $need_sphinx = 0;
 my $need_pip = 0;
 my $need_virtualenv = 0;
 my $rec_sphinx_upgrade = 0;
+my $verbose_warn_install = 1;
 my $install = "";
 my $virtenv_dir = "";
 my $python_cmd = "";

@@ -103,10 +104,12 @@ sub check_missing(%)
 			next;
 		}
 
-		if ($is_optional) {
-			print "Warning: better to also install \"$prog\".\n";
-		} else {
-			print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+		if ($verbose_warn_install) {
+			if ($is_optional) {
+				print "Warning: better to also install \"$prog\".\n";
+			} else {
+				print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+			}
 		}
 		if (defined($map{$prog})) {
 			$install .= " " . $map{$prog};

@@ -386,7 +389,8 @@ sub give_debian_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo apt-get install $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo apt-get install $install\n");
 }
 
 sub give_redhat_hints()

@@ -458,10 +462,12 @@ sub give_redhat_hints()
 
 	if (!$old) {
 		# dnf, for Fedora 18+
-		printf("You should run:\n\n\tsudo dnf install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo dnf install -y $install\n");
 	} else {
 		# yum, for RHEL (and clones) or Fedora version < 18
-		printf("You should run:\n\n\tsudo yum install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo yum install -y $install\n");
 	}
 }
 

@@ -509,7 +515,8 @@ sub give_opensuse_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo zypper install --no-recommends $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo zypper install --no-recommends $install\n");
 }
 
 sub give_mageia_hints()

@@ -553,7 +560,8 @@ sub give_mageia_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo $packager_cmd $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo $packager_cmd $install\n");
 }
 
 sub give_arch_linux_hints()

@@ -583,7 +591,8 @@ sub give_arch_linux_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo pacman -S $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo pacman -S $install\n");
 }
 
 sub give_gentoo_hints()

@@ -610,7 +619,8 @@ sub give_gentoo_hints()
 
 	return if (!$need && !$optional);
 
-	printf("You should run:\n\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n");
 
 	my $imagemagick = "media-gfx/imagemagick svg png";
 	my $cairo = "media-gfx/graphviz cairo pdf";

@@ -700,7 +710,7 @@ sub check_distros()
 
 sub deactivate_help()
 {
-	printf "\nIf you want to exit the virtualenv, you can use:\n";
+	printf "\n    If you want to exit the virtualenv, you can use:\n";
 	printf "\tdeactivate\n";
 }
 

@@ -863,7 +873,7 @@ sub recommend_sphinx_version($)
 			print "To upgrade Sphinx, use:\n\n";
 		}
 	} else {
-		print "Sphinx needs to be installed either as a package or via pip/pypi with:\n";
+		print "\nSphinx needs to be installed either:\n1) via pip/pypi with:\n\n";
 	}
 
 	$python_cmd = find_python_no_venv();

@@ -873,6 +883,25 @@ sub recommend_sphinx_version($)
 	printf "\t. $virtenv_dir/bin/activate\n";
 	printf "\tpip install -r $requirement_file\n";
 	deactivate_help();
+
+	printf "\n2) As a package with:\n";
+
+	my $old_need = $need;
+	my $old_optional = $optional;
+	%missing = ();
+	$pdf = 0;
+	$optional = 0;
+	$install = "";
+	$verbose_warn_install = 0;
+
+	add_package("python-sphinx", 0);
+	check_python_module("sphinx_rtd_theme", 1);
+
+	check_distros();
+
+	$need = $old_need;
+	$optional = $old_optional;
+
 }
 
 sub check_needs()

2.36.1

Newer versions of Sphinx have a known bug:

	https://github.com/sphinx-doc/sphinx/pull/8313

Those currently produces 11 false-positive warnings On Sphinx version 3.1+.

While Sphinx fix is not applied, let's warn adventurers that would be using Sphinx installed via distro packages.

---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH v2 0/5] at: https://lore.kernel.org/all/[email protected]/

 scripts/sphinx-pre-install | 4 ++++
 1 file changed, 4 insertions(+)

index 106d75425d3f..271c4eb1d702 100755
@@ -902,6 +902,10 @@ sub recommend_sphinx_version($)
 	$need = $old_need;
 	$optional = $old_optional;
 
+	printf "\n    Please note that Sphinx >= 3.0 will currently produce false-positive\n";
+	printf "   warning when the same name is used for more than one type (functions,\n";
+	printf "   structs, enums,...). This is known Sphinx bug. For more details, see:\n";
+	printf "\thttps://github.com/sphinx-doc/sphinx/pull/8313\n";
 }
 
 sub check_needs()

2.36.1

Hi Mauro,

So this is a meta-level feedback considering the most likely uses of sphinx-pre-install.

I think first-time users of sphinx-pre-install are more likely interested in getting ready for running "make html". They won't bother with "make pdfdocs". They won't likely be interested in virtualenv, either.

So I think it would be reasonable to change the default behavior of sphinx-pre-install.

I mean the reasonable behavior without any option would be that of when both --no-pdf and --no-virtualenv are given to the current version.

There are a few issues on --no-pdf.

It says imagemagick and gcc are necessary, but they are redundant in "make html", as far as I see.

Furthermore, it doesn't check dvipng and latex, which are used for generating math equation images in HTML.

Fedora, RHEL/CentOS, and openSUSE Leap provide helpful packages for installing math expression support.

    Fedora 36               python3-sphinx-latex (python3-sphinx depends on this)
    RHEL 9/CentOS stream 9  ditto
    openSUSE Leap 15.4      python3-Sphinx_4_2_0-latex

(python3-Sphinx_4_2_0 depends on this) or
python3-Sphinx-latex
(python3-Sphinx depends on this, version: 2.3.1)

Other distros, whose texlive packages are coarse grained, don't provide such helper packages.

Also, as mentioned previously, RHEL 9/CentOS stream9's texlive-xecjk doesn't work at the moment due to the lack of its dependency (texlive-ctex). I opened a bug ticket at RedHat bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=2086254

Unfortunately, I've heard no response yet.
If you know some means to boost its priority, please provide a comment or two to the ticket.

Until the issue can be resolved, "make pdfdocs" is limited to non-CJK build on CentOS stream 9, RHEL 9 and its clones if you must stick to distro packages. For non-CJK build to work, you must not have google-noto-sans-cjk-ttc-fonts.

openSUSE Leap does not support full CJK build of "make pdfdocs", either. Its Noto font packaging is peculiar and a similar named package of noto-sans-cjk-fonts installs a set of language-specific fonts, each of which doesn't qualify as a CJK font.

Seeing these problems of LTS distros, I'd suggest sphinx-pre-install --pdf would check packages for non-CJK builds.

Another option of --cjk would show you additional packages for full CJK build.

TL;DR, my suggestion of options and defaults to sphinx-pre-install:

--no-pdf (default): for htmldocs only
--no-virtualenv (default): distro Sphinx package
(mention --virtualenv if distro Sphinx package is too young)
--virtualenv: Sphinx by venv/virtualenv
--pdf: for pdfdocs
--no-cjk (default): don't bother with CJK pdfdocs --cjk: for CJK pdfdocs
(print warning if user's distro doesn't support CJK)

Thoughts?

Thanks, Akira

[CC: update address of ksummit]

[...]

These packages are supposed to cover LaTeX packages necessary for building LaTeX sources the version of Sphinx generates.

HOWEVER, in my test of openSUSE Leap 15.4, pythno3-Sphinx-4_2_0-latex does not cover texlive-tex-gyre, which is required since Sphinx 4.0.0.

Changelog of Sphinx 4.0.0 [1] says:

[1]: https://www.sphinx-doc.org/en/master/changes.html#release-4-0-0-released-may-09-2021

I'm thinking of opening a ticket at openSUSE's bugzilla.

Fedora 36's python3-sphinx-latex (for Sphinx 4.4.0) has texlive-collection-fontsrecommended and covers texlive-tex-gyre naturally.

Thanks, Akira

Em Wed, 6 Jul 2022 23:31:09 +0900
Akira Yokosawa <[email protected]> escreveu:

Yeah, I remember the script started using some meta-packages for LaTeX. This had to change a couple of times because distros that have dozens of packages for it were not installing everything on their meta packages. So, the packages install but they don't run as they depend on other non-installed packages.

That's basically why I added a list of *.sty inside the script and started using kpsewhich to double-check if all dependencies are there.

Regards,
Mauro

I think this makes sense. As far as I can tell, PDF builds are a relative rarity these days, and most people would rather not have to deal with virtualenv if they can avoid it. We should definitely emphasize native installs whenever that can work.

I'm planning to go ahead and apply Mauro's sphinx-pre-install patches since they make things better, but then we should look at these tweaks.

Thanks,

jon

Em Thu, 07 Jul 2022 12:45:14 -0600
Jonathan Corbet <[email protected]> escreveu:

IMO it makes sense to apply them, as they address some existing issues on it.

We can improve the script later on with Akira's comments and after having some discussions about what would be the default behavior that would fit better.

Regards,
Mauro

Em Tue, 5 Jul 2022 13:15:57 +0900
Akira Yokosawa <[email protected]> escreveu:

True, but, as you're pointing below, math expressions require LaTeX.

The idea of using --no-pdf is to setup an environment without LaTeX, meaning that math tags would only be partially parsed: basically, the output would be html with LaTeX-like math expressions (at least last time I tried).

Yes and no. The big issue with using distro packages is that it will produce 11 false-positive warnings due to duplicated C symbols. This will only be (hopefully) fixed on a later Sphinx 5.x (or 6.0). So, it would take more than 6 months to get rid of those.

Using 2.4.4, after the fixes I sent, plus the 3 fixes from IIO tree (yet to be merged on -next), there will be just 4 warnings.

So, IMO, for me, is still preferred to use 2.4.4 via venv.

With this series, both venv and non-venv settings will be shown by default, allowing the developer decide what he prefers.

Still, I'm not 100% sure if this is the best thing to do.

One alternative would be to run the script on an even more silent mode when called via makefile, in a way that it would, instead it will keep:

- not display anything if sphinx-build works;
- display enable/disable commands if venv is detected;

But, instead of showing install options, it would instead, print a message like:

Can't build the documentation. Please run:

./scripts/sphinx-pre-install <options)

Where options are:
--no-virtualenv - Recommend installing Sphinx instead of using a virtualenv
--no-pdf - don't check for dependencies required to build PDF docs
...

Another alternative would be to use --no-pdf for make htmldocs/epubdocs target, and a "--pdf" for make pdfdocs.

Well, gcc is not really necessary, but anyone using the Kernel tree very likely needs it. So, it doesn't hurt checking for it.

With regards to imagemagick, I'm not sure if, after your patches changing the way to build image, it is still needed. We need to review it.

Changing the dependency chain is a lot of work, as we need to retest everything on all supported platforms, using minimal install. Here, I'm using lxc download, as it usually gets something close to the minimal distro install.

Not checking for LaTeX is per design. It makes the install a lot havier than what's actually needed. dvipng is a new dependency after the changes on svg and dot conversions. Yeah, we need to test it on all distros and properly add it where needed.

yes, but this will install LaTeX. We don't want this for the minimal htmldocs build type, as math is used only on a handful set of documents, and most developers can live without that.

LTS distros like RHEL and SUSE are usually a lot more conservative and may not have everything. The building system needs to cope with that.

Even if they add now, RHEL 9.0 won't have it. So, the script would need to check if the distro supports it.

That doesn't sound right. We should probably fix conf.py to do the right thing. I mean, if this package is installed but texlive-xecjk, it should disable CJK fonts.

Makes sense.

Also makes sense. I don't think they're mutually exclusive.

The options make sense. Still I would discuss more about what would be the best default:

- Suggest both venv and no-venv;
- Ask the user to run the script;
- have different behaviors when called with make pdfdocs or make htmldocs / epubdocs.

Regards,
Mauro

[...]

Oh, that is your intention. :-/

When I saw this help text in "./scripts/sphinx-pre-install --help":

--no-pdf - don't check for dependencies required to build PDF docs

for the first time, my expectation was like this:

If I make "./scripts/sphinx-pre-install --no-pdf" happy, "make htmldocs" will complete without any warning on package requirements.

But I got this warning from "make htmldocs" among others:

WARNING: dvipng command 'dvipng' cannot be run (needed for math display), check the imgmath_dvipng setting

And my reaction to it was something like:

"What!!! Why sphinx-pre-install didn't complain?"
"OK. I'll ignore sphinx-pre-install. Let's see what I need to install."

The reason why this mismatch has never surfaced as a problem is, I suppose, because most people don't use --no-pdf at all.

However, in thinking of making --no-pdf the default, I thought --no-pdf should be more trustworthy.

If your goal is to provide LaTeX-free packages for "make htmldocs", the option should have been named --no-latex or similar (probably in addition to --no-pdf), no?

Well, arguing does not help. I stop here.

More constructive approach would be to make "make htmldocs" be truly latex-free. Then --no-pdf and --no-latex will be equivalent.

Fortunately, it is possible by using the mathjax extension instead of imgmath. With mathjax, rendering of math expressions is delegated to web browsers.

The resulting HTML will depend on mathjax code and math fonts from somewhere in the cloud (by default, https://cdnjs.cloudflare.com/ajax/libs/mathjax/...
or https://cdn.jsdelivr.net/npm/mathjax@3/... depending on the Sphinx version).

All you need is this one-liner:

--------

index 934727e23e0e..fe1084510329 100644

--- a/Documentation/conf.py

+++ b/Documentation/conf.py

@@ -106,7 +106,7 @@ else:

autosectionlabel_prefix_document = True

autosectionlabel_maxdepth = 2

-extensions.append("sphinx.ext.imgmath")

+extensions.append("sphinx.ext.mathjax")

# Add any paths that contain templates here, relative to this directory.

templates_path = ['_templates']
--------

In my tests, the mathjax extension works with all the versions of Sphinx I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4), and 5.0.2).
Note that math expressions should look much sharper (vector fonts) than those from imgmath (pixel images).
The time for a browser to complete the rendering might be longer than with imgmath, especially for pages with a lot of math expressions, though. (Yes, I see some of media documents have a lot of them.)

When you are detached from network connections, browsers will give up and show those expressions in mathjax source code.

Mauro, wouldn't this approach work for you?

Thanks, Akira

PS: Replies to the other topics will be sent separately.

Pulling in a bunch of JavaScript from the net while browsing the kernel docs is not an entirely pleasing solution either. But perhaps it's preferable to loading the system with Latex. I kind of wish we could just do without the fancy math processing entirely, but I lost that battle some time ago...:)

Mauro, have you looked at that option?

Thanks,

jon

Em Fri, 08 Jul 2022 08:02:52 -0600
Jonathan Corbet <[email protected]> escreveu:

If we could change conf.py to either load either one depending if it is building PDF or not would be the best. The technical problem is that the javascript solution is not compatible with PDF output.

The media uAPI book uses math, and also vm/memory-model.rst. The last one could easily use just a C code.

yet, it sounds funny to not have it used inside DRM. GPU processing does lots of math. i would expect to have math used there for things similar to the usages on media.

Trying to describe the colorspace conversion expressions there without math would be really painful:

https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/colorspaces-details.html?highlight=detailed+colorspace

Anyway, if you have some other solution, I'd gladly use something else.

Yes, but it got discarded due to pdf output. Not sure if this would work on epub either.

Hmm, if I understand what is written in the following page:
https://www.sphinx-doc.org/en/master/usage/extensions/math.html

, both imgmath and mathjax extensions are relevant only for HTML output.

It says:

Changed in version 1.8: Math support for non-HTML builders is integrated to sphinx-core. So mathbase extension is no longer needed.

When did you see the issue of "mathjax doesn't work for PDF output" ?

OK, mathjax has no chance, then...

Thanks, Akira

[-CC: ksummit-discuss]

For the record,

I tested mathjax and PDF output with Sphinx 1.7.9, whose latex mode can't handle nested tables.
I had no problem in building userspace-api.pdf and math expressions in it look perfect.

So I believe mathjax does not affect PDF output.

I'm not going to test earlier versions of Sphinx and I have no idea of what issue Mauro saw at the time, but it sounds to me the issue has been fixed since.

On the second thought, I think mathjax (latex-free "make htmldocs") is good enough for test build purposes. When javascript is disabled, math expressions are rendered in mathjax source.

As conf.py is programmable, it is possible to choose sphinx.ext.imgmath when dvipng is found on the build system.

As for sphinx-pre-install, what about adding an option

--no-js For those who disable javascript in their browser.

which provide the list of required packages for dvipng?

Thoughts?

Some comforting news for Jon:

With Sphinx >=4.0, mathjax is loaded only when a page with math expressions is opened. Sphinx is improving in this regard.

Thanks, Akira

Em Sat, 9 Jul 2022 08:01:02 +0900
Akira Yokosawa <[email protected]> escreveu:

Did you also test epubdocs?

It was an issue when we decided to use imgmath. If this got fixed for both supported non-html outputs, we can start using mathjax and get rid of installing latex and dvipng.

Good.

Hmm... are there a way to use it with javascript disabled? If so, maybe we can force it to always render math expressions during the build, instead or relying on javascript at exec time.

Not sure I like the idea. This would actually mean in practice that all developers that are currently doing doc builds will keep using imgmath, because they all have it already installed.

It is not that simple.

Sphinx has a configurable theme engine. On our builds, we're using since the beginning the RTD (readthedocs) theme as default, but recent versions default to classic if sphinx_rtd_theme package is not installed.

All themes I know that provide a search button use JS to implement such feature.

So, a "--no-js" won't provide a javascript-free build environment.

-

On a side discussion, should we keep recommending the install of sphinx_rtd_theme? It is not mandatory anymore to have it installed, and the theme is more a matter of personal preferences.

Also, when testing or modifying the docs, the theme doesn't really matter.

So, IMHO, we could stop recommending it.

Regards,
Mauro

No, I didn't.

Now I see that mathjax doesn't work with epubdocs.

I think I found a means to use mathjax only for htmldocs. It is mentioned in an issue at Sphinx:

    https://github.com/sphinx-doc/sphinx/issues/3740#issuecomment-385503305

Actually, it turns out that the mathjax extension is loaded by default since Sphinx 1.8. A simple assignment in conf.py can enable it:

html_math_renderer = 'mathjax'

As a matter of fact, it can be enabled via the SPHINXOPTS make variable.

Just try this against current docs-next:

make SPHINXDIRS=userspace-api SPHINXOPTS="-D html_math_renderer='mathjax'" htmldocs

As far as I know, ebook readers don't understand javascript and mathjax doesn't work. imgmath is mandatory for epub.

I'm suspecting now that pdfdocs had no issue with mathjax, but I'm not sure.

No chance. Javascript is a must for mathjax.

The purpose here is to get more developers who actually run "make htmldocs" on their (not limited to) documentation changes. Difference of math rendering doesn't matter much.

One possible improvement for imgmath is to use its SVG format option, which uses dvisvgm instead. It will provide scalable math expressions similar to those provided by mathjax. Of course you need properly working dvisvgm.

I see.

What about "--imgmath" ?

I agree.

Thanks, Akira

There are countless ways to install any software. So much for my entrée.

*

I don't know this script. Just what I've read in this thread and what I could extract from it in no more than ten minutes. I understand what it does - or rather I do just approximately. And alright! Why not help the Gentoos and Fedoras and Etc's. Noble.

But - wouldn't it be million times easier for all parties to base the support on a Dockerfile? Or a script producing the right Dockerfile possibly, considering matters like Java Script opt- ins and outs. If this is not a cheap pedantry in this context. I don't know.

These here bases look quite official:

https://hub.docker.com/u/sphinxdoc

(Not that I've used them or know them.)

The admins of whichever distribution will find their ways in the mesh, raw requirements suffice.

The non-admins may have problems with installing software and also running Docker, as that provides su powers in the straight scenario. pip comes as an alternative, if I'm not wrong. So maybe this pair is the golden arrow?

--thanks
T

Em Sat, 25 Jun 2022 11:11:40 -0700
Linus Torvalds <[email protected]> escreveu:

I solved almost all warnings that can be fixed against -next. Sending patches today.

However, there is an special case where warnings will be hit for perfectly fine code, when docs are built with Sphinx 3.0 and upper:

Documentation/driver-api/usb/usb:164: ./drivers/usb/core/message.c:967: WARNING: Duplicate C declaration, also defined at driver-api/usb/gadget:783. Declaration is '.. c:function:: int usb_string (struct usb_device *dev, int index, char *buf, size_t size)'. Documentation/driver-api/usb/usb.rst:967: WARNING: Duplicate C declaration, also defined at driver-api/usb/gadget:783. Declaration is '.. c:struct:: usb_string'.

In this specific case, there are both a struct and a function named "usb_string". So, the above is not an issue. While this is not fixed, there will be a 22 lines noise (11 two-line warnings) after my fixes[1].

The root cause is that Sphinx has a single namespace for symbols, no matter if the symbol is a function, a struct, an enum, etc. So, when the same name is declared on multiple places, like in the case of "usb_string", the cross-references will either point to the function or to the struct. While this is annoying, it is usually not a big deal, as they are typically placed at the same rst file.

While this problem affects *all* Sphinx versions so far, Sphinx 2.x.x doesn't warn when this happen, but newer versions complain about that. This is a well known issue and there are patches fixing it since Sphinx 3.1:

	https://github.com/sphinx-doc/sphinx/pull/8313

Those patches were expected to be merged on Sphinx 3.3, but ended not being merged so far, as applying them would break support for a Sphinx extension that it is used on several places.

---

[1] I got one case where the warning was actually reporting an issue:

Documentation/driver-api/hte/tegra194-hte:28: ./drivers/gpio/gpiolib.c:2464: WARNING: Duplicate C declaration, also defined at driver-api> Declaration is '.. c:function:: int gpiod_enable_hw_timestamp_ns (struct gpio_desc *desc, unsigned long flags)'. Documentation/driver-api/hte/tegra194-hte.rst:2464: WARNING: Duplicate C declaration, also defined at driver-api/gpio/index:2464. Declaration is '.. c:function:: int gpiod_enable_hw_timestamp_ns(struct gpio_desc *desc, unsigned long flags)'.

Basically drivers/gpio/gpiolib.c is included twice for the same symbols, one at Documentation/driver-api/gpio/index.rst, and then Documentation/driver-api/hte/tegra194-hte.rst includes it again for gpiod_enable_hw_timestamp_ns() and gpiod_disable_hw_timestamp_ns().

Regards,
Mauro

Is there a way to mark comments in the code as "design docs". And I don't mean the kernel-doc that is already above functions.

I have lots of comments about the design of complex code just above the code it is describing. Which is much more likely to get updated when the design changes or gets new features.

I found my design documents I have under Documentation to become grossly obsolete over time, where as the comments above the code I can keep up with.

Is there already some kind of mark up that we can add to the comments to state:

** .design.trace/ftrace.rst .ringbuffer. **

or something that can be extracted into another document? With the above example, I could have in Documentatino/trace/ftrace.rst a

.design .ringbuffer.

And the comment from the code would be extracted into the html documents that are created?

Or maybe it would be better to reference the code from the documentation?

.include kerne/trace/ring_buffer.c .ringbuffer.

That will look for a tag in kernel/trace/ring_buffer.c called ".ringbuffer" and pull in the comment into the document under Documentation.

Thoughts?

-- Steve

Hi Steven,

See the "DOC: overview" block in drivers/gpu/drm/drm_atomic_helper.c, and how it is integrated in Documentation/gpu/drm-kms-helpers.rst with

.. kernel-doc:: drivers/gpu/drm/drm_atomic_helper.c
:doc: overview

Is that what you're looking for ?

Regards,

Laurent Pinchart