docs: kernel-doc: doesn't suppress error/warnings

As discussed at:
https://lore.kernel.org/all/[email protected]/

Sphinx logger is set in a way that errors/warnings from kernel-doc are suppressed, except when V=1.

Finding the root cause is not trivial, as we need to go into the Sphinx overlays on the top of Python's default logger class. Instead, let's take the trivial approach: use logger class directly. As we can later revisit such decision, let's add a comment at the code.

Mauro Carvalho Chehab (1):
docs: kerneldoc.py: don't use Sphinx logger

 Documentation/sphinx/kerneldoc.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

2.49.0

Unfortunately, currently Sphinx logger is suppressing too much, not allowing warnings to be displayed. Disable it.

---
 Documentation/sphinx/kerneldoc.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

index b713a2c4a615..314479718a01 100644
@@ -311,7 +311,11 @@ def setup_kfiles(app):
     if kerneldoc_bin and kerneldoc_bin.endswith("kernel-doc.py"):
         print("Using Python kernel-doc")
         out_style = RestFormat()
-        kfiles = KernelFiles(out_style=out_style, logger=logger)
+
+        # Ideally, we should be using Sphinx logger here, but its filtering
+        # rules ending filtering out warnings and errors. So, let's use
+        # Python default logger instead.
+        kfiles = KernelFiles(out_style=out_style)
     else:
         print(f"Using {kerneldoc_bin}")
 

2.49.0

On linux-next-20250516, this gives me:

Cannot find file ../drivers/gpio/gpiolib-acpi.c
Cannot find file ../drivers/gpio/gpiolib-acpi.c
...
Sphinx parallel build error!

Versions
========

* Platform:         linux; (Linux-6.14.4-1-default-x86_64-with-glibc2.41)
* Python version:   3.13.3 (CPython)
* Sphinx version:   8.2.3

* Docutils version: 0.21.2
* Jinja2 version: 3.1.6
* Pygments version: 2.19.1

Last Messages
=============

reading sources... [ 75%]
translations/it_IT/subsystem-apis .. translations/zh_CN/admin-guide/mm/damon/reclaim

reading sources... [ 77%]
translations/zh_CN/admin-guide/mm/damon/start .. translations/zh_CN/core-api/symbol-namespaces

reading sources... [ 79%]
translations/zh_CN/core-api/this_cpu_ops .. translations/zh_CN/kernel-hacking/index

Loaded Extensions
=================

* sphinx.ext.mathjax (8.2.3)
* alabaster (1.0.0)
* sphinxcontrib.applehelp (2.0.0)
* sphinxcontrib.devhelp (1.0.6)
* sphinxcontrib.htmlhelp (2.1.0)
* sphinxcontrib.serializinghtml (1.1.10)
* sphinxcontrib.qthelp (2.0.0)
* kerneldoc (1.0)
* rstFlatTable (1.0)
* kernel_include (1.0)
* kfigure (1.0.0)
* sphinx.ext.ifconfig (8.2.3)
* automarkup (unknown version)
* maintainers_include (1.0)
* sphinx.ext.autosectionlabel (8.2.3)
* kernel_abi (1.0)
* kernel_feat (1.0)
* translations (unknown version)

Traceback
=========

File "/usr/lib/python3.13/site-packages/sphinx/util/parallel.py", line 137, in _join_one
raise SphinxParallelError(*result)
sphinx.errors.SphinxParallelError: KeyError: '../drivers/gpio/gpiolib-acpi.c'

and then it's finished (not a normal finish).
So IMHO this patch is not sufficient.

~Randy

Hi Randy,

Em Mon, 19 May 2025 22:55:08 -0700
Randy Dunlap <[email protected]> escreveu:

Well, on next-20250516:

	$ ls drivers/gpio/gpiolib-acpi.c
	ls: cannot access 'drivers/gpio/gpiolib-acpi.c': No such file or directory

Avoiding the script to abort is a matter of adding a try/except block at kerneldoc.py, but I'd say that, if an include file (or any other file needed for the build) is not found, "make" should abort anyway for the affected target (Kernel compilation or when doc building).

The only thing that bothers me is the "Sphinx parallel build error!" message. I'll check if there is a way to suppress it.

Regards,
Mauro

[...]

Interesting ...

So, it sounds to me you think you have the right to break Stephen's (and possibly other devs') workflow of test-building kernel docs, aren't you?

I don't buy such an argument.

An innocent typo in pathname somewhere in the doc tree deserves a friendly warning at most, not a fatal crash within Sphinx. That would need another run of "make htmldocs" after fixing the fatal error to see for other innocuous warnings.

And your change has no effect on exposing those innocuous warnings.

On current docs-next + your change above, running:

make cleandocs; make KERNELDOC=$PWD/scripts/kernel-doc.pl htmldocs

produces these 3 warnings:

----------------------------------------------------------------------
./drivers/gpu/drm/amd/include/amd_shared.h:369: warning: Incorrect use of kernel-doc format: * @DC_HDCP_LC_ENABLE_SW_FALLBACK If set, upon HDCP Locality Check FW
./drivers/gpu/drm/amd/include/amd_shared.h:369: warning: Incorrect use of kernel-doc format: * @DC_HDCP_LC_ENABLE_SW_FALLBACK If set, upon HDCP Locality Check FW
./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK' not described in enum 'DC_DEBUG_MASK'
----------------------------------------------------------------------

, while running:

make cleandocs; make htmldocs

or:

make cleandocs; make KERNELDOC=$PWD/scripts/kerneldoc htmldocs

doesn't produce them.

Sorry, but I believe you have run out of time.

And let me remind you of the expectation for backward-compatibility widely accepted our community.

Akira

Em Tue, 20 May 2025 17:23:30 +0900
Akira Yokosawa <[email protected]> escreveu:

If by "innocent typo" you mean removing/renaming/moving a file from the Kernel tree without updating Makefiles, or adding them with wrong names, then yes: I do think this should be fatal for the affected "make" targets. The quickest such build issues are caught, the best for everyone.

Yet, this is is a separate discussion: if there are bugs at the doc build system, they should be fixed. In this specific case, the missing file produces a KeyError when the messages that were supposed to be generated by it aren't stored on a dict. I sent already some patches addressing it:

	https://lore.kernel.org/linux-doc/[email protected]/T/#t

Plus the logger fix to prevent Sphinx to suppress warnings/errors:

	https://lore.kernel.org/linux-doc/[email protected]/T/#t

Testing here with next-20250516:

	$ ./scripts/kernel-doc.pl -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	$

	$ ./scripts/kernel-doc.py -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	$

Also, if I rename the value 'DC_HDCP_LC_ENABLE_SW_FALLBACK_XXX', both produces identical warnings:

	$ ./scripts/kernel-doc.pl -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK_XXX' not described in enum 'DC_DEBUG_MASK'
	./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Excess enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK' description in 'DC_DEBUG_MASK'

	$ ./scripts/kernel-doc.py -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	Warning: ./drivers/gpu/drm/amd/include/amd_shared.h:372 Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK_XXX' not described in enum 'DC_DEBUG_MASK'
	Warning: ./drivers/gpu/drm/amd/include/amd_shared.h:372 Excess enum value '%DC_HDCP_LC_ENABLE_SW_FALLBACK' description in 'DC_DEBUG_MASK'

So, it sounds to me that things are the way they're expected, after applying both series.

Regards,
Mauro

Hi Mauro,

I do appreciate your work on the document system. Particularly you have made the process much faster. However, see below.

They are found if you leave them as warnings and just continue on. I have been finding them by when the current python version just crashes (without any useful message before your current change) I rerun the htmldocs using the perl version to get the name of the missing file and then report it so it can be fixed. But the perl version continues on and produces all the other warnings so I can compare with the previous and only report new errors/warnigs. Sometime these reports are not foxed for days or weeks (or months).

So, please, now that you are producing more warnings, make a missing file just a warning (or at least continue on). I have enough to do every day without having to run "make htmldocs" more than once.

Cheers,
Stephen Rothwell

Em Tue, 20 May 2025 21:39:06 +1000
Stephen Rothwell <[email protected]> escreveu:

Hi Stephen,

The issues we're seeing weren't due to the conversion from perl to python, but, instead, to the new logic that calls the Python methods directly, instead of opening a shell to call kernel-doc.

Those caused some unintended changes, due to the way Sphinx works:

1. Sphinx has an overlay over the Python logger class. Such overlay has some logic to output colored messages, which is interesting. However, it also has a filtering logic which is affected by the flags passed when sphinx-build is called. That was actually suppressing not only verbose level logs, but also warnings and errors. The fix is a single-line change:

- kfiles = KernelFiles(out_style=out_style, logger=logger)
+ kfiles = KernelFiles(out_style=out_style)

The patch is at:

	https://lore.kernel.org/linux-doc/6b81b1aaa8446b4d850064dd38ffffa1a1cb6254.1747719873.git.mchehab+huawei@kernel.org/

2. Now that we're not opening a shell anymore, if kernel-doc.py dies, it could cause a fatal error on Sphinx. I sent today two patches fixing it:

2.1 Don't stop Documentation/sphinx/kerneldoc.py if a crash

       happens:

	https://lore.kernel.org/linux-doc/064bac2f462c13f56154891d8f3fb788db94f325.1747730982.git.mchehab+huawei@kernel.org/

2.2 Don't crash trying to retrieve kernel-doc messages from
a file that doesn't exist. This is patch 1/2 on this series:
https://lore.kernel.org/lkml/d97e86c7176f671405b4c15d75cb951349022a23.1747730982.git.mchehab+huawei@kernel.org/

OK. As I said, the crash was unintended. The idea was to preserve the old behavior as much as possible. Yet, it could probably be a good idea to have a way in the future to control the docs behavior for the cases where non-existing files are detected, allowing it to stop quickly on such cases, instead of taking several minutes to generate an html ouptut, as this could be useful for CI automation.

In order to simplify for Jon, I'll send those tree altogether(*).

(*) I'll add a 4th patch, adding scripts/lib/kdoc to linux-doc content. Without that, Lore doesn't store kernel-doc changes as part of docs. See:

	https://lore.kernel.org/linux-doc/d97e86c7176f671405b4c15d75cb951349022a23.1747730982.git.mchehab+huawei@kernel.org/

Regards,
Mauro