Add Kconfig pages and cross-references to Documentation
Subsystems (3)
cover Add Kconfig pages and cross-references to Documentation
This series adds Kconfig pages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).
There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):
(On an XPS 13 9300)
Before:
real 6m43.576s
user 23m32.611s
sys 1m48.220s
After:
real 11m56.845s
user 47m40.528s
sys 2m27.382s
There are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.
---
Nícolas F. R. A. Prado (2):
docs: Add documentation generation for Kconfig symbols
docs: automarkup: Cross-reference CONFIG_ symbols
Documentation/.gitignore | 2 + Documentation/Config/index.rst | 17 ++ Documentation/Makefile | 12 +- Documentation/kbuild/index.rst | 2 + Documentation/sphinx/automarkup.py | 36 +++- scripts/kconfig2rst.py | 336 +++++++++++++++++++++++++++++++++++++ 6 files changed, 403 insertions(+), 2 deletions(-)
---
base-commit: 405e2241def89c88f008dcb899eb5b6d4be8b43c
change-id: 20250403-kconfig-docs-19d90ba266dd
Best regards,
Signature · 1 line
Nícolas F. R. A. Prado <[email protected]>
RFC PATCH 1/2 docs: Add documentation generation for Kconfig symbols
Add the contents of all Kconfig files to the Documentation to both increase their visibility and allow for cross-referencing throughout the documentation. In order to achieve this:
* Add a new script 'kconfig2rst' that converts a Kconfig file into a reStructuredText document.
* Add an extra step to the documentation building that runs the script for every Kconfig in the source tree, generating a documentation page for each one.
* Add a new "Kconfig symbols" page in the documentation, that is listed on the "Kernel Build System" page, which contains an index of all Kconfig files and their Kconfig symbols, linking to the corresponding pages.
The generated documentation pages have the config symbols as sections with labels that can be referenced from anywhere in the documentation. The exceptions are configs that appear multiple times. Those don't get labels, as that would generate 'duplicate label' warnings from sphinx. To allow this, a list of configs that appear more than once is embedded in the kconfig2rst script. When a config appears more than once in the same Kconfig file, a count is appended in the section to prevent sphinx's auto-labeling to cause the same warning.
The paths in 'source' directives in the Kconfig files are turned into links to the generated documentation page to allow for navigation to included Kconfig files.
Config symbols on 'depends'/'select'/etc lines are prepended by 'CONFIG_' to allow them to be cross-referenced by automarkup, though no cross-references are created in this commit.
--- Documentation/.gitignore | 2 + Documentation/Config/index.rst | 17 +++ Documentation/Makefile | 12 +- Documentation/kbuild/index.rst | 2 + scripts/kconfig2rst.py | 336 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 368 insertions(+), 1 deletion(-)
Documentation/.gitignore +2 -0
@@ -1,3 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only output *.pyc +Config/ +!Config/index.rst
Documentation/Config/index.rst new file +17 -0
@@ -0,0 +1,17 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Kconfig symbols +=============== + +.. toctree:: + :glob: + :maxdepth: 2 + + * + */* + */*/* + */*/*/* + */*/*/*/* + */*/*/*/*/* + */*/*/*/*/*/*
Documentation/Makefile +11 -1
@@ -115,7 +115,7 @@ $(YNL_INDEX): $(YNL_RST_FILES) $(YNL_RST_DIR)/%.rst: $(YNL_YAML_DIR)/%.yaml $(YNL_TOOL) $(Q)$(YNL_TOOL) -i $< -o $@ -htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX) +htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX) kconfigdocs htmldocs: @$(srctree)/scripts/sphinx-pre-install --version-check
@@ -182,9 +182,19 @@ endif # HAVE_SPHINX refcheckdocs: $(Q)cd $(srctree);scripts/documentation-file-ref-check +KCONFIG_DOC_DIR=$(srctree)/Documentation/Config +KCONFIGS := $(shell find $(srctree) -name Kconfig -type f) +KCONFIGS_RST := $(patsubst %, $(KCONFIG_DOC_DIR)/%.rst, $(KCONFIGS)) + +$(KCONFIGS_RST): $(KCONFIGS) + $(Q)cd $(srctree); $(foreach var,$^,$(shell mkdir -p $(KCONFIG_DOC_DIR)/$(shell dirname $(var)); scripts/kconfig2rst.py $(var) >$(KCONFIG_DOC_DIR)/$(var).rst)) + +kconfigdocs: $(KCONFIGS_RST) + cleandocs: $(Q)rm -f $(YNL_INDEX) $(YNL_RST_FILES) $(Q)rm -rf $(BUILDDIR) + $(Q)rm -rf $(filter-out %index.rst,$(wildcard $(KCONFIG_DOC_DIR)/*)) $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media clean dochelp:
Documentation/kbuild/index.rst +2 -0
@@ -15,6 +15,8 @@ Kernel Build System makefiles modules + /Config/index + headers_install issues
scripts/kconfig2rst.py new file +336 -0
336 lines not on this page · raw message
Signature · 1 line
2.49.0
kernel test robot
build botRe: [PATCH RFC 1/2] docs: Add documentation generation for Kconfig symbols
kernel test robot noticed the following build warnings:
Hi Nícolas,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on 405e2241def89c88f008dcb899eb5b6d4be8b43c]
url: https://github.com/intel-lab-lkp/linux/commits/N-colas-F-R-A-Prado/docs-Add-documentation-generation-for-Kconfig-symbols/20250404-220651 base: 405e2241def89c88f008dcb899eb5b6d4be8b43c patch link: https://lore.kernel.org/r/20250404-kconfig-docs-v1-1-4c3155d4ba44%40collabora.com patch subject: [PATCH RFC 1/2] docs: Add documentation generation for Kconfig symbols config: arc-randconfig-002-20250405 (https://download.01.org/0day-ci/archive/20250405/[email protected]/config) compiler: arc-linux-gcc (GCC) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250405/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> Documentation/Config/index.rst: warning: ignored by one of the .gitignore files
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH RFC 1/2] docs: Add documentation generation for Kconfig symbols
In reply to Nícolas F. R. A. PradoEm Fri, 04 Apr 2025 10:02:52 -0400
Nícolas F. R. A. Prado <[email protected]> escreveu:
Quoted 22 lines
Add the contents of all
Kconfigfiles to the Documentation to both increase their visibility and allow for cross-referencing throughout the documentation. In order to achieve this:
* Add a new script 'kconfig2rst' that converts aKconfigfile into a reStructuredText document.
* Add an extra step to the documentation building that runs the script for everyKconfigin the source tree, generating a documentation page for each one.
* Add a new "Kconfigsymbols" page in the documentation, that is listed on the "Kernel Build System" page, which contains an index of allKconfigfiles and theirKconfigsymbols, linking to the corresponding pages.The generated documentation pages have the config symbols as sections with labels that can be referenced from anywhere in the documentation. The exceptions are configs that appear multiple times. Those don't get labels, as that would generate 'duplicate label' warnings from sphinx. To allow this, a list of configs that appear more than once is embedded in the kconfig2rst script. When a config appears more than once in the same
Kconfigfile, a count is appended in the section to prevent sphinx's auto-labeling to cause the same warning.
Quoted 7 lines
The paths in 'source' directives in the
Kconfigfiles are turned into links to the generated documentation page to allow for navigation to includedKconfigfiles.Config symbols on 'depends'/'select'/etc lines are prepended by '
CONFIG_' to allow them to be cross-referenced by automarkup, though no cross-references are created in this commit.
Despite the huge increase on the time to produce documentation, I'm not sure how worth is to have it, as there are already cross-reference services doing something somewhat similar, like:
Yet, I didn't test this series yet. So, not sure yet about its value.
Anyway, it follows some comments about the current implementation.
After addressed on a v2, I intend to test and see how it behaves.
Quoted 33 lines
Signed-off-by Nícolas F. R. A. Prado <[email protected]>--- Documentation/.gitignore | 2 + Documentation/Config/index.rst | 17 +++ Documentation/Makefile | 12 +- Documentation/kbuild/index.rst | 2 + scripts/kconfig2rst.py | 336 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 368 insertions(+), 1 deletion(-)Documentation/.gitignore +2
-0@@ -1,3 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only output *.pyc +Config/ +!Config/index.rstDocumentation/Config/index.rst new file +7
-0@@ -0,0 +1,17 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Kconfig symbols +=============== + +.. toctree::
Documentation/Config/index.rst line 17
Quoted 10 lines
Documentation/Config/index.rst new file +10
-0+ :glob: + :maxdepth: 2 + + * + */* + */*/* + */*/*/* + */*/*/*/* + */*/*/*/*/* + */*/*/*/*/*/*
That sounds weird, hard to maintain and probably slow.
Better to have a Sphinx extension instead, with a decent implementation of glob. The Python's one is slow, on my tests with the Kernel tree. I worked on something that worked fine for kernel-doc.py:
https://lore.kernel.org/linux-doc/12a54f1b8f4afd2e70a87195a2aa34f96d736b77.1740387599.git.mchehab+huawei@kernel.org/
Perhaps this script could import the class from it, once such series gets merged. It could make sense to split it on a separate file if we're going to re-use its code.
Quoted 58 lines
Documentation/Makefile +11
-1@@ -115,7 +115,7 @@ $(YNL_INDEX): $(YNL_RST_FILES) $(YNL_RST_DIR)/%.rst: $(YNL_YAML_DIR)/%.yaml $(YNL_TOOL) $(Q)$(YNL_TOOL) -i $< -o $@ -htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX) +htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX) kconfigdocs htmldocs: @$(srctree)/scripts/sphinx-pre-install --version-check@@ -182,9 +182,19 @@ endif # HAVE_SPHINX refcheckdocs: $(Q)cd $(srctree);scripts/documentation-file-ref-check +KCONFIG_DOC_DIR=$(srctree)/Documentation/Config +KCONFIGS := $(shell find $(srctree) -name Kconfig -type f) +KCONFIGS_RST := $(patsubst %, $(KCONFIG_DOC_DIR)/%.rst, $(KCONFIGS)) + +$(KCONFIGS_RST): $(KCONFIGS) + $(Q)cd $(srctree); $(foreach var,$^,$(shell mkdir -p $(KCONFIG_DOC_DIR)/$(shell dirname $(var)); scripts/kconfig2rst.py $(var) >$(KCONFIG_DOC_DIR)/$(var).rst)) + +kconfigdocs: $(KCONFIGS_RST) + cleandocs: $(Q)rm -f $(YNL_INDEX) $(YNL_RST_FILES) $(Q)rm -rf $(BUILDDIR) + $(Q)rm -rf $(filter-out %index.rst,$(wildcard $(KCONFIG_DOC_DIR)/*)) $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media clean dochelp:Documentation/kbuild/index.rst +2
-0@@ -15,6 +15,8 @@ Kernel Build System makefiles modules + /Config/index + headers_install issuesscripts/kconfig2rst.py new file +6
-0@@ -0,0 +1,336 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Collabora Ltd + +import sys +import re
scripts/kconfig2rst.py line 15
Quoted 10 lines
scripts/kconfig2rst.py new file +9
-0+import os + +import argparse + +BASE_PATH_DEFAULT = "Documentation/Config/" +CFG_LEN = 60 +RE_indentation = r"^[ \t]*" +in_help_txt = False +help_txt = ""
Better to follow Python's standards and keep all constants on uppercase.
I would place the main code inside a class, with the non-const data inside the class, as it makes the code cleaner and helps using it as both a Sphinx extension and as a standalone command (which is useful for testing it).
Quoted 192 lines
scripts/kconfig2rst.py new file +192
-0+ +# These configs appear more than once, thus we don't generate labels or xrefs to +# them to avoid duplicate label warnings from Sphinx +REPEATED_CONFIGS = [ + "32BIT", + "4KSTACKS", + "64BIT", + "A", + "ADVANCED_OPTIONS", + "ALPHA_LEGACY_START_ADDRESS", + "ARCH_AIROHA", + "ARCH_ALPINE", + "ARCH_BCM2835", + "ARCH_BCM_IPROC", + "ARCH_BRCMSTB", + "ARCH_DEFAULT_CRASH_DUMP", + "ARCH_FLATMEM_ENABLE", + "ARCH_FORCE_MAX_ORDER", + "ARCH_HAS_ADD_PAGES", + "ARCH_HAS_CACHE_LINE_SIZE", + "ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION", + "ARCH_HAS_ILOG2_U32", + "ARCH_HAS_ILOG2_U64", + "ARCH_HIBERNATION_HEADER", + "ARCH_HIBERNATION_POSSIBLE", + "ARCH_HISI", + "ARCH_MAY_HAVE_PC_FDC", + "ARCH_MEMORY_PROBE", + "ARCH_MMAP_RND_BITS_MAX", + "ARCH_MMAP_RND_BITS_MIN", + "ARCH_MMAP_RND_COMPAT_BITS_MAX", + "ARCH_MMAP_RND_COMPAT_BITS_MIN", + "ARCH_MTD_XIP", + "ARCH_OMAP", + "ARCH_PKEY_BITS", + "ARCH_PROC_KCORE_TEXT", + "ARCH_R9A07G043", + "ARCH_RENESAS", + "ARCH_ROCKCHIP", + "ARCH_SELECT_MEMORY_MODEL", + "ARCH_SELECTS_CRASH_DUMP", + "ARCH_SELECTS_KEXEC_FILE", + "ARCH_SPARSEMEM_DEFAULT", + "ARCH_SPARSEMEM_ENABLE", + "ARCH_SUNXI", + "ARCH_SUPPORTS_CRASH_DUMP", + "ARCH_SUPPORTS_CRASH_HOTPLUG", + "ARCH_SUPPORTS_KEXEC", + "ARCH_SUPPORTS_KEXEC_FILE", + "ARCH_SUPPORTS_KEXEC_JUMP", + "ARCH_SUPPORTS_KEXEC_PURGATORY", + "ARCH_SUPPORTS_KEXEC_SIG", + "ARCH_SUPPORTS_UPROBES", + "ARCH_SUSPEND_POSSIBLE", + "ARCH_UNIPHIER", + "ARCH_VIRT", + "AUDIT_ARCH", + "B", + "BCH_CONST_M", + "BCH_CONST_T", + "BUILTIN_DTB", + "BUILTIN_DTB_NAME", + "C", + "CC_HAVE_STACKPROTECTOR_TLS", + "CHOICE_B", + "CHOICE_C", + "CMDLINE", + "CMDLINE_BOOL", + "CMDLINE_EXTEND", + "CMDLINE_FORCE", + "CMDLINE_FROM_BOOTLOADER", + "CMDLINE_OVERRIDE", + "CMM", + "COMPAT", + "COMPAT_VDSO", + "CORE", + "CORE_BELL_A", + "CORE_BELL_A_ADVANCED", + "CPU_BIG_ENDIAN", + "CPU_HAS_FPU", + "CPU_HAS_PREFETCH", + "CPU_LITTLE_ENDIAN", + "CRYPTO_CHACHA20_NEON", + "CRYPTO_JITTERENTROPY_MEMORY_BLOCKS", + "CRYPTO_JITTERENTROPY_MEMORY_BLOCKSIZE", + "CRYPTO_JITTERENTROPY_OSR", + "CRYPTO_JITTERENTROPY_TESTINTERFACE", + "CRYPTO_NHPOLY1305_NEON", + "DEBUG_ENTRY", + "DMA_NONCOHERENT", + "DMI", + "DRAM_BASE", + "DUMMY", + "DUMMY_CONSOLE", + "EARLY_PRINTK", + "EFI", + "EFI_STUB", + "FIT_IMAGE_FDT_EPM5", + "FIX_EARLYCON_MEM", + "FPU", + "GENERIC_BUG", + "GENERIC_BUG_RELATIVE_POINTERS", + "GENERIC_CALIBRATE_DELAY", + "GENERIC_CSUM", + "GENERIC_HWEIGHT", + "GENERIC_ISA_DMA", + "GENERIC_LOCKBREAK", + "HAS_IOMEM", + "HAVE_SMP", + "HAVE_TCM", + "HEARTBEAT", + "HIGHMEM", + "HOTPLUG_CPU", + "HW_PERF_EVENTS", + "HZ", + "HZ_100", + "HZ_1000", + "HZ_1024", + "HZ_128", + "HZ_250", + "HZ_256", + "ILLEGAL_POINTER_VALUE", + "IRQSTACKS", + "ISA", + "ISA_DMA_API", + "KASAN_SHADOW_OFFSET", + "KERNEL_MODE_NEON", + "KERNEL_START", + "KERNEL_START_BOOL", + "KUSER_HELPERS", + "KVM", + "KVM_GUEST", + "L1_CACHE_SHIFT", + "LEDS_EXPRESSWIRE", + "LOCKDEP_SUPPORT", + "LOWMEM_SIZE", + "LOWMEM_SIZE_BOOL", + "MACH_LOONGSON32", + "MACH_LOONGSON64", + "MACH_TX49XX", + "MAGIC_SYSRQ", + "MATH_EMULATION", + "MCOUNT", + "MMU", + "NODES_SHIFT", + "NO_IOPORT_MAP", + "NR_CPUS", + "NR_CPUS_DEFAULT", + "NR_CPUS_RANGE_END", + "NUMA", + "PAGE_OFFSET", + "PANIC_TIMEOUT", + "PARAVIRT", + "PARAVIRT_SPINLOCKS", + "PARAVIRT_TIME_ACCOUNTING", + "PFAULT", + "PGTABLE_LEVELS", + "PHYSICAL_ALIGN", + "PHYSICAL_START", + "PID_IN_CONTEXTIDR", + "PM", + "POWERPC64_CPU", + "PRINT_STACK_DEPTH", + "RANDOMIZE_BASE", + "RANDOMIZE_BASE_MAX_OFFSET", + "RELOCATABLE", + "SBUS", + "SCHED_CLUSTER", + "SCHED_HRTICK", + "SCHED_MC", + "SCHED_OMIT_FRAME_POINTER", + "SCHED_SMT", + "SERIAL_CONSOLE", + "SMP", + "STACKPROTECTOR_PER_TASK", + "STACKTRACE_SUPPORT", + "SWAP_IO_SPACE", + "SYS_SUPPORTS_APM_EMULATION", + "SYS_SUPPORTS_NUMA", + "SYS_SUPPORTS_SMP", + "TASK_SIZE", + "TASK_SIZE_BOOL", + "TCP_CONG_CUBIC", + "TIME_LOW_RES", + "UNWINDER_FRAME_POINTER", + "UNWINDER_GUESS", + "UNWINDER_ORC", + "USE_OF", + "VMSPLIT_1G", + "VMSPLIT_2G", + "VMSPLIT_3G", + "VMSPLIT_3G_OPT",
scripts/kconfig2rst.py line 217
Quoted 10 lines
scripts/kconfig2rst.py new file +10
-0+ "X", + "X86_32", + "X86_64", + "XEN", + "XEN_DOM0", + "XIP_KERNEL", + "XIP_PHYS_ADDR", + "ARCH_BCM", + "VIRTUALIZATION", +]
Maintaining this sounds a nightmare, as new (eventually duplicated) symbols may happen anytime.
The best here sounds to do something similar to what I did with get_abi.py: parse them all altogether, dynamically detecting duplication. IMO, it also makes sense to have dereference pages for such duplicated symbols pointing to all occurrences of them.
Quoted 21 lines
scripts/kconfig2rst.py new file +21
-0+ + +def print_title(title): + heading = "=" * len(title) + print(heading) + print(title) + print(heading) + print() + + +parser = argparse.ArgumentParser( + prog="kconfig2rst", description="Convert a Kconfig file into ReStructuredText" +) + +parser.add_argument("kconfig", help="Path to input Kconfig file") +parser.add_argument( + "--base-doc-path", + default=BASE_PATH_DEFAULT, + help="Base path of generated rST files for usage in 'source' links", +) +args = parser.parse_args()
scripts/kconfig2rst.py line 248
Quoted 10 lines
scripts/kconfig2rst.py new file +10
-0+ +print_title(args.kconfig) + +line_accum = "" +continued_line = False + +repeated_config_count = {} + +with open(args.kconfig) as f: + for il in f:
This won't handle directories. Better to use my glob function.
Also, calling lines as as "il" sounds weird for me. I would just call it "line".
scripts/kconfig2rst.py line 250
Quoted 2 lines
scripts/kconfig2rst.py new file +2
-0+ # If line ends with \, accumulate it and handle full line + if re.search(r"\\\n$", il):
Better to use endswith("\\\n"),, as it is faster. We can also use removesuffix(), as I guess the minimal Python version is now 3.9.
Quoted 7 lines
scripts/kconfig2rst.py new file +7
-0+ continued_line = True + line_accum += il[:-2] # accumulate without backslash and newline + continue + + if continued_line: + continued_line = False + l = line_accum + il
scripts/kconfig2rst.py line 267
Quoted 10 lines
scripts/kconfig2rst.py new file +10
-0+ line_accum = "" + else: + l = il + + if in_help_txt: + if l == "\n": + help_txt += l + continue + if first_line_help_txt: + help_txt_indentation = re.match(RE_indentation, l).group(0).expandtabs()
Please compile all regular expressions, to make it faster.
Quoted 8 lines
scripts/kconfig2rst.py new file +8
-0+ first_line_help_txt = False + # Consider any line with same or more indentation as part of help text + if ( + help_txt_indentation + in re.match(RE_indentation, l).group(0).expandtabs() + ): + help_txt += l + continue
scripts/kconfig2rst.py line 285
Quoted 10 lines
scripts/kconfig2rst.py new file +10
-0+ else: + in_help_txt = False + print(help_txt) + help_txt = "" + else: + l = re.sub(r"[*]", r"\*", l) # Escape asterisks + + if re.match(r"^[ \t]*#.*", l): + # Skip comments + continue
I would strip comments first, as I guess Kconfig syntaxe allow to use comments after any texts, like:
config SYMBOL # some comment
scripts/kconfig2rst.py line 293
Quoted 8 lines
scripts/kconfig2rst.py new file +8
-0+ + if re.match(r"^[ \t]*help", l): + in_help_txt = True + first_line_help_txt = True + print("* help::\n") + continue + + m = re.match("^[ \t]*(menu)?config (?P<cfgname>[A-Za-z0-9_]+)", l)
Better to accept multiple spaces after config, as it would be valid to have:
config FOO
Quoted 8 lines
scripts/kconfig2rst.py new file +8
-0+ if m: + section_name = f"\nCONFIG_{m.group('cfgname')}" + underline = f"\n{'='*CFG_LEN}\n" + if m.group("cfgname") in REPEATED_CONFIGS: + repeated_config_count[m.group("cfgname")] = ( + repeated_config_count.get(m.group("cfgname"), 0) + 1 + ) + if repeated_config_count[m.group("cfgname")] > 1:
scripts/kconfig2rst.py line 311
Quoted 10 lines
scripts/kconfig2rst.py new file +10
-0+ section_name += f"({repeated_config_count[m.group('cfgname')]})" + print(section_name + underline) + else: + print(f"\n.. _CONFIG_{m.group('cfgname')}:\n\n" + section_name + underline) + continue + + m = re.match( + r"^[ \t]*(def_bool|def_tristate|depends on|select|range|visible if|imply|default|prompt|bool|tristate|string|hex|int|modules)( \"(.*)\")?(?P<expr> [^\"]*)?", + l, + )
I would place the valid matches on an array and do a join to create the compiled regex to match them. This would make easier to maintain as Kconfig syntax add more notations.
Quoted 7 lines
scripts/kconfig2rst.py new file +7
-0+ if m: + expr = m.group('expr') if m.group('expr') else '' + not_expr = l + if expr: + expr = re.sub(r'[A-Z0-9_]{2,}', rf" CONFIG_\g<0> ", expr) + not_expr = l[:m.start('expr')] + print("* " + not_expr.lstrip() + expr.rstrip())
scripts/kconfig2rst.py line 328
Quoted 10 lines
scripts/kconfig2rst.py new file +10
-0+ continue + + m = re.match(r'^[ \t]*source "(.*)"', l) + if m: + # Format Kconfig file paths as Documentation/... so they can be turned + # into links by the automarkup plugin + print(f"\nsource {args.base_doc_path + m.group(1)}.rst\n") + continue + + m = re.match(r"[^ \t]*choice|endchoice|comment|menu|endmenu|if|endif", l)
Same here.
Quoted 9 lines
scripts/kconfig2rst.py new file +8
-0+ if m: + print("\n" + l.strip() + "\n") + continue + + print(l.strip()) + +if help_txt: + print(help_txt) # Flush any pending help text
RFC PATCH 2/2 docs: automarkup: Cross-reference CONFIG_ symbols
In reply to Nícolas F. R. A. PradoNow that Kconfig symbols are part of the Documentation and have sections that can be referenced, extend automarkup to automatically cross-reference CONFIG_ strings to those sections.
--- Documentation/sphinx/automarkup.py | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-)
Documentation/sphinx/automarkup.py +35 -1
@@ -47,6 +47,8 @@ RE_abi_symbol = re.compile(r'(\b/(sys|config|proc)/[\w\-/]+)') RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$') +RE_kconfig = re.compile(r'CONFIG_[A-Za-z0-9_]+') + # # Reserved C words that we should skip when cross-referencing #
@@ -88,7 +90,8 @@ def markup_refs(docname, app, node): RE_union: markup_c_ref, RE_enum: markup_c_ref, RE_typedef: markup_c_ref, - RE_git: markup_git} + RE_git: markup_git, + RE_kconfig: markup_kconfig_ref} match_iterators = [regex.finditer(t) for regex in markup_func] #
@@ -260,6 +263,37 @@ def markup_doc_ref(docname, app, match): else: return nodes.Text(match.group(0)) +# +# Try to replace a kernel config reference of the form CONFIG_... with a +# cross reference to that kconfig's section +# +def markup_kconfig_ref(docname, app, match): + stddom = app.env.domains['std'] + # + # Go through the dance of getting an xref out of the std domain + # + target = match.group(0).lower() + xref = None + pxref = addnodes.pending_xref('', refdomain = 'std', reftype = 'ref', + reftarget = target, modname = None, + classname = None, refexplicit = False) + # + # XXX The Latex builder will throw NoUri exceptions here, + # work around that by ignoring them. + # + try: + xref = stddom.resolve_xref(app.env, docname, app.builder, 'ref', + target, pxref, None) + except NoUri: + xref = None + # + # Return the xref if we got it; otherwise just return the plain text. + # + if xref: + return xref + else: + return nodes.Text(match.group(0)) + # # Try to replace a documentation reference for ABI symbols and files # with a cross reference to that page
Signature · 1 line
2.49.0
Jonathan Corbet
maintainerQuoted 18 lines · Nícolas F. R. A. Prado
This series adds
Kconfigpages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):
(On an XPS 13 9300)
Before:
real 6m43.576s
user 23m32.611s
sys 1m48.220sAfter:
Quoted 7 lines
real 11m56.845s
user 47m40.528s
sys 2m27.382sThere are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.
I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong.
I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case.
I'll try to look closer, but I'll remain a bit distracted for a little while yet.
Thanks,
jon
Quoted 28 lines · Jonathan Corbet
Quoted 18 lines · Nícolas F. R. A. Prado
This series adds
Kconfigpages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):
(On an XPS 13 9300)
Before:
real 6m43.576s
user 23m32.611s
sys 1m48.220sAfter:
Quoted 7 lines
real 11m56.845s
user 47m40.528s
sys 2m27.382sThere are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.
Quoted 5 lines
I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong.
I have not yet. Thought I'd get some feedback before looking into the performance. But I agree with the sentiment.
Quoted 4 lines
I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case.
That is what is currently done for the ABI, AFAIK, so definitely seems doable.
The key difference between the ABI approach and this here, is that my goal was to reflect the Kconfig file hierarchy in the Documentation. So each Kconfig file gets its own documentation page, while the ABI approach collects the contents of all ABI files into just a few documentation pages (stable, testing, etc). (So there's a non-constant number of .rst files, which means they have to be generated and can't be a sphinx plugin in this approach).
I went for this approach because the filesystem hierarchy seemed the most logical way to group the Kconfig symbols. Also Kconfig files have directives like 'menu' that should be present in the documentation in the same order they appear in the file to fully describe dependencies of the symbols, and having all of that in the same page seems like it would be confusing. But given the potential benefits it's worth a try for sure.
Now that I think about it, seems quite likely that a lot of the time spent comes from creating a subshell and running the script for every Kconfig file. So making a single script or sphinx extension that itself handles iterating over all the files would likely greatly reduce the run time. I'll test that.
Thanks,
Nícolas
Quoted 7 lines
I'll try to look closer, but I'll remain a bit distracted for a little while yet.
Thanks,
jon
Em Fri, 4 Apr 2025 12:24:27 -0400
Nícolas F. R. A. Prado <[email protected]> escreveu:
Quoted 29 lines · Jonathan Corbet
Quoted 28 lines
Quoted 25 lines · Nícolas F. R. A. Prado
This series adds
Kconfigpages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):
(On an XPS 13 9300)
Before:
real 6m43.576s
user 23m32.611s
sys 1m48.220sAfter:
real 11m56.845s
user 47m40.528s
sys 2m27.382sThere are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.
Quoted 8 lines
Quoted 5 lines
I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong.
I have not yet. Thought I'd get some feedback before looking into the performance. But I agree with the sentiment.
My feeling is that the issue is using :glob" and a lot of wildcards inside Sphinx. Instead, you should use something similar to what I've done to get *.[ch] for the new kernel-doc.py implementation.
Placing it as an extension on a similar way to what i did with get_abi.py would likely help as well.
Quoted 5 lines
Quoted 3 lines
I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case.
That is what is currently done for the ABI, AFAIK, so definitely seems doable.
Yes, doing that via an extension is doable. If done right, it can also be fast.
Quoted 6 lines
The key difference between the ABI approach and this here, is that my goal was to reflect the
Kconfigfile hierarchy in the Documentation. So eachKconfigfile gets its own documentation page, while the ABI approach collects the contents of all ABI files into just a few documentation pages (stable, testing, etc). (So there's a non-constant number of .rst files, which means they have to be generated and can't be a sphinx plugin in this approach).
Actually, get-api.py (the new version, merged for 6.15) generates a dict just once. Then, Sphinx rst files filters part of the doc, but I see your
point: for every entry, we would need a .rst file if we follow the same approach.
That's said, it may have a way to tell Sphinx to threat Kconfig files on a similar way it handles ".txt" and ".rst" files. Something like the extension to handle markdown works:
https://www.sphinx-doc.org/en/master/usage/markdown.html
Another alternative would be to use:
https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-include_patterns
but this would require Sphinx 5.1, which is above our current minimal version. That's said, nothing prevents to only enable generating such documentatation if the Sphinx version supports it.
Quoted 22 lines
I went for this approach because the filesystem hierarchy seemed the most logical way to group the
Kconfigsymbols. AlsoKconfigfiles have directives like 'menu' that should be present in the documentation in the same order they appear in the file to fully describe dependencies of the symbols, and having all of that in the same page seems like it would be confusing. But given the potential benefits it's worth a try for sure.Now that I think about it, seems quite likely that a lot of the time spent comes from creating a subshell and running the script for every
Kconfigfile. So making a single script or sphinx extension that itself handles iterating over all the files would likely greatly reduce the run time. I'll test that.Thanks,
NícolasQuoted 7 lines
I'll try to look closer, but I'll remain a bit distracted for a little while yet.
Thanks,
jon