empty docs build with O= option
Akira Yokosawa
AppliedHi all,
Here is a follow-up to my message at:
https://lore.kernel.org/[email protected]/
Quoting relevant parts for your convenience:
Quoted 6 lines · Akira
As a matter of fact, I'm seeing weird performance regression of empty documentation builds when the O=<somedir> option is used.
It appeared in v6.15, which has your conversion of get_abi.pl into
get_abi.py. Will send a report once the time-consuming bisection completes.
Answer to Mauro's question at:
https://lore.kernel.org/[email protected]/
Quoted 3 lines · Mauro
Did you try with docs-next instead? I remember Jon caught one issue causing the doctree cache to expire. Can't remember if this was on 6.15 or at docs-next, but the fixes should be applied there.
Jon's complaint at https://lore.kernel.org/[email protected]/
was against Mauro's v2 series.
Jon has applied the v3 series with the issue fixed at:
https://lore.kernel.org/[email protected]/
, so the issue has never been in docs-next.
I'm not a python person and bisection is all I could do.
TL;DR
2nd run of "make O=$HOME/output htmldocs" takes much longer since kernel release v6.15.
2nd run is an empty build and should finish in a matter of 10 seconds.
"make htmldocs" without the O= option doesn't show this symptom.
Here is a summary table.
Note:
Numbers shown as (x) [y] mean 2nd "make O=$HOME/output htmldocs" run takes x seconds, while 2nd "make htmldocs" run takes y seconds (rounded up).
----------------------------------------
Sphinx 8.2.3 venv 8.1.3 7.2.6
------------------- --------- ----------
Distro f42 noble f42 noble
--------- --------- --------- ----------
Good (13) [13] (16) [16] ( 6) [ 6] ( 9) [ 9] v6.14
Bad --- (72) [ 6] (349) [11] v6.15
Bad --- (72) [ 6] (365) [11] v6.16-rc1 + [1]
Bad --- (72) [ 7] (342) [10] docs-next (2025.07.11) + [1]
Bad (77) [14] (82) [15] (73) [ 8] (347) [11] docs-next (2025.07.11) + [1,2,3]
--------- --------- --------- ----------[1]: cherry-pick 553ab30a1810 ("Documentation: nouveau: Update GSP message
queue kernel-doc reference"), to suppress noise of error handling within
Sphinx.
[2]: Mauro's ynl series v9 at https://lore.kernel.org/[email protected]/
(14/13 applied) [3]: Jon's kdoc series v2 at https://lore.kernel.org/[email protected]/ (12/12 as is)
f42: python 3.13.5, noble: 3.12.3
* Bisection result
Sphinx 8.2.3 (venv) on top of ubuntu:noble (python 3.12.3):
Last good commit:
(16) 6b48bea16848 ("scripts/get_abi.py: add support for symbol search")
First bad commit:
(48) 9d7ec8867960 ("docs: use get_abi.py for ABI generation")
Merged into Jon's docs-mw branch at:
(82) 1ce8294cfefb ("Merge branch 'mauro' into docs-mw")
Pre merge commits of 1ce8294cfefb:
(15) 2b087edf588c ("docs/zh_CN: Add secrets index Chinese translation")
(82) 1c7e66bc5d20 ("scripts/get_abi.pl: drop now obsoleted script")
Again, The numbers enclosed in () indicate the elapsed times (in seconds, rounded up) of 2nd "make O=$HOME/output htmldocs" runs.
Sphinx 8.2.3 takes longer than 8.1.3 for empty builds, but it is likely an independent issue on Sphinx side.
Mauro, I'd like you to have a look into this regression.
Thanks, Akira
Hi Akira,
Em Mon, 14 Jul 2025 19:42:49 +0900
Akira Yokosawa <[email protected]> escreveu:
Quoted 26 lines
Hi all,
Here is a follow-up to my message at:
https://lore.kernel.org/[email protected]/Quoting relevant parts for your convenience:
Quoted 6 lines · Akira
As a matter of fact, I'm seeing weird performance regression of empty documentation builds when the O=<somedir> option is used.
It appeared in v6.15, which has your conversion of get_abi.pl into
get_abi.py. Will send a report once the time-consuming bisection completes.Answer to Mauro's question at:
https://lore.kernel.org/[email protected]/Quoted 3 lines · Mauro
Did you try with docs-next instead? I remember Jon caught one issue causing the doctree cache to expire. Can't remember if this was on 6.15 or at docs-next, but the fixes should be applied there.
Jon's complaint at https://lore.kernel.org/[email protected]/
was against Mauro's v2 series.
Quoted 6 lines
Jon has applied the v3 series with the issue fixed at:
https://lore.kernel.org/[email protected]/, so the issue has never been in docs-next.
I'm not a python person and bisection is all I could do.
This is probably not a python issue itself, but instead, some weirdness at the way Sphinx cache is stored.
Quoted 58 lines
TL;DR
2nd run of "make O=$HOME/output htmldocs" takes much longer since kernel release v6.15.
2nd run is an empty build and should finish in a matter of 10 seconds.
"make htmldocs" without the O= option doesn't show this symptom.
Here is a summary table.
Note:
Numbers shown as (x) [y] mean 2nd "make O=$HOME/output htmldocs" run takes x seconds, while 2nd "make htmldocs" run takes y seconds (rounded up).----------------------------------------
Sphinx 8.2.3 venv 8.1.3 7.2.6------------------- --------- ---------- Distro f42 noble f42 noble --------- --------- --------- ---------- Good (13) [13] (16) [16] ( 6) [ 6] ( 9) [ 9] v6.14 Bad --- (72) [ 6] (349) [11] v6.15 Bad --- (72) [ 6] (365) [11] v6.16-rc1 + [1] Bad --- (72) [ 7] (342) [10] docs-next (2025.07.11) + [1] Bad (77) [14] (82) [15] (73) [ 8] (347) [11] docs-next (2025.07.11) + [1,2,3] --------- --------- --------- ----------[1]: cherry-pick
553ab30a1810("Documentation: nouveau: Update GSP message
queue kernel-doc reference"), to suppress noise of error handling within
Sphinx.
[2]: Mauro's ynl series v9 at https://lore.kernel.org/[email protected]/(14/13 applied) [3]: Jon's kdoc series v2 at https://lore.kernel.org/[email protected]/ (12/12 as is)f42: python 3.13.5, noble: 3.12.3
* Bisection result
Sphinx 8.2.3 (venv) on top of ubuntu:noble (python 3.12.3):
Last good commit:
(16)6b48bea16848("scripts/get_abi.py: add support for symbol search")First bad commit:
(48)9d7ec8867960("docs: useget_abi.pyfor ABI generation")Merged into Jon's docs-mw branch at:
(82)1ce8294cfefb("Merge branch 'mauro' into docs-mw")Pre merge commits of
1ce8294cfefb:
(15)2b087edf588c("docs/zh_CN: Add secrets index Chinese translation")
(82)1c7e66bc5d20("scripts/get_abi.pl: drop now obsoleted script")Again, The numbers enclosed in () indicate the elapsed times (in seconds, rounded up) of 2nd "make O=$HOME/output htmldocs" runs.
Quoted 6 lines
Sphinx 8.2.3 takes longer than 8.1.3 for empty builds, but it is likely an independent issue on Sphinx side.
Mauro, I'd like you to have a look into this regression.
Thanks, Akira
To debug it, I added the enclosed code to conf.py. See enclosed. The
logic there uses this from https://www.sphinx-doc.org/en/master/extdev/event_callbacks.html:
env-get-outdated(app, env, added, changed, removed)
Emitted when the environment determines which source files have changed and should be re-read. added, changed and removed are sets of docnames that the environment has determined. You can return a list of docnames to re-read in addition to these.
Assuming that the issue is during ABI generation, we can check with that what is the root case.
Running the second time without O= doeesn't require rebuild ABI:
$ time make SPHINXDIRS=admin-guide htmldocs
Python version: 3.13.5
Docutils version: 0.21.2
make[3]: Nothing to be done for 'html'.
Using alabaster theme
source directory: admin-guide
Using Python kernel-doc [OUTDATED]
Added: set()
Changed: {'gpio/obsolete', 'gpio/index'}
Removed: set()
All docs count: 385
Found docs count: 385
/root/Documentation/admin-guide/gpio/index.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev' [toc.not_readable] /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev_v1' [toc.not_readable] /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/sysfs' [toc.not_readable]
real 0m1,489s
user 0m1,305s
sys 0m0,263s
Here, it seems that the root cause is because of a non-resolved dependency at gpio/obsolete.
Now, running the second time with O=, we have:
$ make clean; time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs
... [OUTDATED]
Added: set()
Changed: {'abi-stable', 'abi-testing-files', 'abi-removed', 'gpio/index', 'abi-obsolete-files', 'abi-testing', 'abi-stable-files', 'abi-obsolete', 'gpio/obsolete', 'abi-removed-files', 'abi'}
Removed: set()
All docs count: 385
Found docs count: 385
Basically, Sphinx thinks that abi* files had some changes, but only when O= is used.
At Documentation/sphinx/kernel_abi.py, the only part we touch the dependency chain is here:
if f != old_f:
# Add the file to Sphinx build dependencies env.note_dependency(os.path.abspath(f))
If we comment such code or change the logic to:
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index db6f0380de94..9f2198eb7af1 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -146,8 +146,9 @@ class KernelCmd(Directive):
n += 1
if f != old_f:
- # Add the file to Sphinx build dependencies
- env.note_dependency(os.path.abspath(f))
+ # Add the file to Sphinx build dependencies if the file exists
+ if os.path.isfile(f):+ env.note_dependency(os.path.abspath(f))
old_f = f
We get a quick build again:
[OUTDATED]
Added: set()
Changed: {'gpio/index', 'gpio/obsolete'}
Removed: set()
All docs count: 385
Found docs count: 385
/root/Documentation/admin-guide/gpio/index.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev' [toc.not_readable] /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev_v1' [toc.not_readable] /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/sysfs' [toc.not_readable] make[1]: Leaving directory '/tmp/foo'
real 0m1,444s user 0m1,267s sys 0m0,252s
However, that's because it won't add any dependency at all when O= is used. The problem seems to be related to relative/absolute patches when adding such dependencies.
Let me try to write a proper fix.
Jon,
Btw, should we add something like the code below to be optionally used at conf.py when we need to identify cache issues?
Regards,
Mauro
---
This is what I added at the end of conf.py, at setup(app):
Thanks,
Mauro