Some kernel-doc fixes

Hi Jon,

This small series contain 3 patches:
- patch 1 fixes PDF docs build, as reported by Akira; (I'm resending this one as-is from its v2)
- patch 2 addresses a complain from Jani about not being able of disabling "-q" flag when building docs with V=0;
- patch 3 addresses an issue indirectly reported by Jani that it the env vars that affects the wrapper aren't documented.

With regards to patch 2, docs build honours V=0 by adding a "-q" flag.

When V=1 is set, there are two effects in place:

1. sphix-build will be called without "-q";
2. Sphinx extensions will increase their verbosity levels.

Sometimes, it is desired to just remove "-q" without increasing extensions verbosity. That's what patch 2 does.

IMO, at least patch 1 should be merged during Kernel v6.21 development cycle.

Mauro Carvalho Chehab (3):
docs: kdoc: Fix pdfdocs build for tools
docs: sphinx-build-wrapper: allow -v override -q
tools: sphinx-build-wrapper: improve its help message

 tools/docs/sphinx-build-wrapper  | 42 +++++++++++++++++++++++++++-----
 tools/lib/python/kdoc/kdoc_re.py | 10 +++++---
 2 files changed, 43 insertions(+), 9 deletions(-)

2.52.0

the "\1" inside a docstring requires proper scaping to not be considered a hex character and break the build.

---
 tools/lib/python/kdoc/kdoc_re.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

index 2816bd9f90f8..0bf9e01cdc57 100644
@@ -228,14 +228,18 @@ class NestedMatch:
             yield line[t[0]:t[2]]
 
     def sub(self, regex, sub, line, count=0):
-        """
+        r"""
         This is similar to re.sub:
 
         It matches a regex that it is followed by a delimiter,
         replacing occurrences only if all delimiters are paired.
 
-        if r'\1' is used, it works just like re: it places there the
-        matched paired data with the delimiter stripped.
+        if the sub argument contains::
+
+            r'\1'
+
+        it will work just like re: it places there the matched paired data
+        with the delimiter stripped.
 
         If count is different than zero, it will replace at most count
         items.

2.52.0

Documentation builds were using "-q" for a long time, but sometimes it is nice to see the Sphinx progress, without increasing build verbosity - which would also turn on kernel-doc verbosity.

Instead of doing that, let's parse the sphinx-build already-existing -v: each time it is used, it increases the verbosity level.

With that, if the default is to use -q, a single -v will disable quiet mode. Passing more -v will keep increasing its verbosity.

---
 tools/docs/sphinx-build-wrapper | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

index 78ff7ac202ef..8080ace60680 100755
@@ -168,6 +168,7 @@ class SphinxBuilder:
         parser = argparse.ArgumentParser()
         parser.add_argument('-j', '--jobs', type=int)
         parser.add_argument('-q', '--quiet', action='store_true')
+        parser.add_argument('-v', '--verbose', default=0, action='count')
 
         #
         # Other sphinx-build arguments go as-is, so place them

@@ -179,10 +180,14 @@ class SphinxBuilder:
         # Build a list of sphinx args, honoring verbosity here if specified
         #
 
-        verbose = self.verbose
         sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
+
+        verbose = sphinx_args.verbose
+        if self.verbose:
+            verbose += 1
+
         if sphinx_args.quiet is True:
-            verbose = False
+            verbose = 0
 
         #
         # If the user explicitly sets "-j" at command line, use it.

@@ -195,8 +200,11 @@ class SphinxBuilder:
         else:
             self.n_jobs = None
 
-        if not verbose:
+        if verbose < 1:
             self.sphinxopts += ["-q"]
+        else:
+            for i in range(1, sphinx_args.verbose):
+                self.sphinxopts += ["-v"]
 
     def __init__(self, builddir, venv=None, verbose=False, n_jobs=None,
                  interactive=None):

2.52.0

Besides the parameters that are passed via command line arguments, the wrapper's behavior is affected by several environment variables.

Document that. While here, use __doc__ for its description.

---
 tools/docs/sphinx-build-wrapper | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

index 8080ace60680..b7c149dff06b 100755
@@ -814,20 +814,42 @@ def jobs_type(value):
     except ValueError:
         raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")  # pylint: disable=W0707
 
+EPILOG="""
+Besides the command line arguments, several environment variables affect its
+default behavior, meant to be used when called via Kernel Makefile:
+
+- KERNELVERSION:  Kernel major version
+- KERNELRELEASE:  Kernel release
+- KBUILD_VERBOSE: Contains the value of "make V=[0|1] variable.
+                  When V=0 (KBUILD_VERBOSE=0), sets verbose level to "-q".
+- SPHINXBUILD:    Documentation build tool (default: "sphinx-build").
+- SPHINXOPTS:     Extra options pased to SPHINXBUILD
+                  (default: "-j auto" and "-q" if KBUILD_VERBOSE=0).
+                  The "-v" flag can be used to increase verbosity.
+                  If V=0, the first "-v" will drop "-q".
+- PYTHON3:        Python command to run SPHINXBUILD
+- PDFLATEX:       LaTeX PDF engine. (default: "xelatex")
+- LATEXOPTS:      Optional set of command line arguments to the LaTeX engine
+- srctree:        Location of the Kernel root directory (default: ".").
+
+"""
+
 def main():
     """
     Main function. The only mandatory argument is the target. If not
     specified, the other arguments will use default values if not
     specified at os.environ.
     """
-    parser = argparse.ArgumentParser(description="Kernel documentation builder")
+    parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter,
+                                     description=__doc__,
+                                     epilog=EPILOG)
 
     parser.add_argument("target", choices=list(TARGETS.keys()),
                         help="Documentation target to build")
     parser.add_argument("--sphinxdirs", nargs="+",
                         help="Specific directories to build")
     parser.add_argument("--builddir", default="output",
-                        help="Sphinx configuration file")
+                        help="Sphinx configuration file (default: %(default)s)")
 
     parser.add_argument("--theme", help="Sphinx theme to use")
 

@@ -843,7 +865,7 @@ def main():
                         help="place build in verbose mode")
 
     parser.add_argument('-j', '--jobs', type=jobs_type,
-                        help="Sets number of jobs to use with sphinx-build")
+                        help="Sets number of jobs to use with sphinx-build(default: auto)")
 
     parser.add_argument('-i', '--interactive', action='store_true',
                         help="Change latex default to run in interactive mode")

2.52.0

6.21? I'm kind of assuming you mean the *next* cycle, which will be 7.0?

Thanks,

jon

I've applied this set, thanks.

jon

Hi Jon,

Thanks!

-

Jon,

I ended placing this series near the end of the /41 one mainly because the first patch conflicts there.

The conflict is trivial, to solve though, if you opt to merge from it.

IMO, it is worth merging this series before the merge window, as it will solve some troubles that will appear after Linus picks stuff from -next, related to some annotations Peter Zijlstra added on his tree, but it is up to you.

Yeah, it is a lot more complex than I would expect for a late -rc, so we could end just postpone it.

---

As a heads up, I'm working on a separate set of patches that, if things go well, we may end having a regression test for kernel-doc. When done, I'll be submitting in separate.

The idea is to have a YAML file with source code, KdocItem, man output and rst output, and a dynamic unit test to run them. I finished today to write a skeleton, but still requires polishing (*).

I'm thinking on modifying kernel-doc executable to be able to generate the content for such YAML file.

If things go well, we could have something like:

	$ tools/docs/kernel-doc --gen-yaml all_kdoc_tests.yaml .

To place all tests there. As we modify kernel-doc, we can run the code I'm currently writing. It would identify the differences for both ReST and man and report.

(*) I started working on it today.

The unittest generation part from YAML already works:

	$ tools/unittests/test_kdoc_from_yaml.py 
	Ran 3 tests in 0.004s

FAILED (failures=1)
test_kdoc_from_yaml:
KernelDocParser:
test_gen_function3: FAIL
KernelManOutput:
test_man_function3: OK
KernelRestOutput:
test_rst_function3: OK

Those 3 classes were auto-populated from yaml. Right now, only KernelDocParser test exectution was implemented.

Those three classes contain just the test functions (two are just stubs):

class KernelDocParser(KdocParser):
def run_parser_test(self, source, kdoc_item, exports, fname):
if isinstance(kdoc_item, dict):
kdoc_item = [kdoc_item]

if isinstance(exports, str):
exports=set([exports])
elif isinstance(exports, list):
exports=set(exports)

# For now, it is using here a function I wrote on a non-dynamic unit test self.run_test(source, kdoc_item, exports=exports, fname=fname)

class KernelManOutput(unittest.TestCase):
# TODO: for now, this is just a stub
def run_out_test(self, kdoc_item, out_type, data):
pass

class KernelRestOutput(unittest.TestCase):
# TODO: for now, this is just a stub
def run_out_test(self, kdoc_item, out_type, data):
pass

And It contains a separate class to add dynamic tests on those.

The only issue I need to handle on its current implementation for KernelDocParser (with the simple YAML test) is to solve whitespace issues:

FAIL: test_gen_function3 (test_kdoc_from_yaml.KernelDocParser.test_gen_function3)
Lambda-like function to run tests with provided vars
----------------------------------------------------------------------
Traceback (most recent call last):
File "/new_devel/docs/tools/unittests/test_kdoc_from_yaml.py", line 79, in test_method
self.run_parser_test(source, kdoc_item, exports, fname)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/new_devel/docs/tools/unittests/test_kdoc_from_yaml.py", line 54, in run_parser_test
self.run_test(source, kdoc_item, exports=exports, fname=fname) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/new_devel/docs/tools/unittests/test_kdoc_parser.py", line 84, in run_test
self.assertEqual(d[key], value, msg=f"at {key}")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'Description': 'Does nothing\n\n', 'Return': '\nalways return 0.\n'} != {'Description': 'Does nothing\n', 'Return': 'always return 0.\n'}
- {'Description': 'Does nothing\n\n', 'Return': '\nalways return 0.\n'}
? -- --

+ {'Description': 'Does nothing\n', 'Return': 'always return 0.\n'} : at sections

I'll probably add a regex to replace all whitespaces/new lines by a single one.

For man pages, I need to handle timestamps, but except for that and whitespace differences, both rst and man page functions should be straight forward: just call the output class passing the list of KdocItem objects.

I guess tomorrow I may have this fixed and the test functions for rst and man implemented.

Thanks,
Mauro

---

If you're curious enough, this is the test YAML file it is using to generate the 3 dynamic unit tests:

tests:
- name: function3
fname: function3.c
description: Just a simple example

source: |
/**
* function3: Exported function
* @arg1: @arg1 does nothing
*
* Does nothing
*
* return:
* always return 0.
*/
int function3(char *arg1) { return 0; };
EXPORT_SYMBOL(function3);

exports: function3
expected:
- kdoc_item:
name: function3
type: function
declaration_start_line: 1

sections:
Description: |
Does nothing

Return: |
always return 0.

parameterdescs:
arg1: |
@arg1 does nothing
parameterlist:
- arg1
parameterdesc_start_lines:
arg1: 2
parametertypes:
arg1: char *arg1

other_stuff:
func_macro: false
functiontype: int
purpose: Exported function
typedef: false

rst: |
.. c:function:: int function3 (char *arg1)

Exported function

.. container:: kernelindent

**Parameters**

``char *arg1``
**arg1** does nothing

**Description**

Does nothing

**Return**

always return 0.

man: |
.TH "function3" 9 "February 2026" "" "Kernel API Manual" .SH NAME
function3 \- Exported function
.SH SYNOPSIS
.B "int" function3
.BI "(char *arg1 " ");"
.SH ARGUMENTS
.IP "arg1" 12
\fIarg1\fP does nothing
.SH "DESCRIPTION"
Does nothing
.SH "RETURN"
always return 0.
.SH "SEE ALSO"
.PP
Kernel file \fBtest.c\fR

...

FWIW, I think it'll be painful to have the source and the expected result in the same YAML file, simply because all parts of this are fussy about whitespace and indentation. I'd put them all in separate files, with the YAML tying them together. Then you can also reuse a single source file with multiple tests with different parameters and different outputs. And you get editor syntax higlighting and other help for the individual files. And you can easily debug and compare the outputs with direct kernel-doc invocation. Etc.

In fact, this is exactly what I've done with Hawkmoth tests [1][2]. There's years of experience poured into this. I test everything through the parser directly, through the command-line, and through Sphinx.

BR,
Jani.

[1] https://github.com/jnikula/hawkmoth/blob/master/doc/developer/testing.rst
[2] https://github.com/jnikula/hawkmoth/tree/master/test

Jani Nikula, Intel

I considered that, but on the other hand, if one wants to test the intermediate internal KdocItem representation, having them altogether helps to see what happened.

True, but as I added support at kernel-doc to generate a yaml format, it should be easy to diff from what someone wrote and what kernel-doc actually produced - heh - sort of...

... pyyaml output is not user-friendly. Maybe there are ways to make it use a more compact/nicer notation, but digging into its documentation is not easy:

https://pyyaml.org/wiki/PyYAMLDocumentation

and it is not complete.

I'll take a look on it. Thanks for the hints!

Thanks,
Mauro

Side note, I really like using StrictYAML [1] with a schema for parsing YAML. It's maybe slower, and probably won't help with the output either, but the YAML type safety and validation are such nice features. (See the Norway Problem.)

BR,
Jani.

[1] https://hitchdev.com/strictyaml/

Jani Nikula, Intel