Skip to contentSkip to conversation

Add Kconfig pages and cross-references to Documentation

RFC PATCH In review Nícolas F. R. A. Prado 4 participants 8 messages Apr 4, 2025 – Apr 7, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Add Kconfig pages and cross-references to Documentation

This series adds Kconfig pages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).

There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):

(On an XPS 13 9300)

Before:

real 6m43.576s
user 23m32.611s
sys 1m48.220s

After:

real 11m56.845s
user 47m40.528s
sys 2m27.382s

There are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.

---
Nícolas F. R. A. Prado (2):
docs: Add documentation generation for Kconfig symbols
docs: automarkup: Cross-reference CONFIG_ symbols

 Documentation/.gitignore           |   2 +
 Documentation/Config/index.rst     |  17 ++
 Documentation/Makefile             |  12 +-
 Documentation/kbuild/index.rst     |   2 +
 Documentation/sphinx/automarkup.py |  36 +++-
 scripts/kconfig2rst.py             | 336 +++++++++++++++++++++++++++++++++++++
 6 files changed, 403 insertions(+), 2 deletions(-)

---
base-commit: 405e2241def89c88f008dcb899eb5b6d4be8b43c
change-id: 20250403-kconfig-docs-19d90ba266dd

Best regards,

Signature · 1 line
Nícolas F. R. A. Prado <[email protected]>
#1 lore raw

RFC PATCH 1/2 docs: Add documentation generation for Kconfig symbols

Add the contents of all Kconfig files to the Documentation to both increase their visibility and allow for cross-referencing throughout the documentation. In order to achieve this:
* Add a new script 'kconfig2rst' that converts a Kconfig file into a reStructuredText document.
* Add an extra step to the documentation building that runs the script for every Kconfig in the source tree, generating a documentation page for each one.
* Add a new "Kconfig symbols" page in the documentation, that is listed on the "Kernel Build System" page, which contains an index of all Kconfig files and their Kconfig symbols, linking to the corresponding pages.

The generated documentation pages have the config symbols as sections with labels that can be referenced from anywhere in the documentation. The exceptions are configs that appear multiple times. Those don't get labels, as that would generate 'duplicate label' warnings from sphinx. To allow this, a list of configs that appear more than once is embedded in the kconfig2rst script. When a config appears more than once in the same Kconfig file, a count is appended in the section to prevent sphinx's auto-labeling to cause the same warning.

The paths in 'source' directives in the Kconfig files are turned into links to the generated documentation page to allow for navigation to included Kconfig files.

Config symbols on 'depends'/'select'/etc lines are prepended by 'CONFIG_' to allow them to be cross-referenced by automarkup, though no cross-references are created in this commit.

---
 Documentation/.gitignore       |   2 +
 Documentation/Config/index.rst |  17 +++
 Documentation/Makefile         |  12 +-
 Documentation/kbuild/index.rst |   2 +
 scripts/kconfig2rst.py         | 336 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 368 insertions(+), 1 deletion(-)
Documentation/.gitignore +2 -0
index d6dc7c9b8e25020f1f3b28811df2291c38695d5f..2fc70a398dc874fcb83834cb6337f602c64a070a 100644
@@ -1,3 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 output
 *.pyc
+Config/
+!Config/index.rst
Documentation/Config/index.rst new file +17 -0
new file mode 100644
index 0000000000000000000000000000000000000000..2abaa9844dd2a9f57bed0a8d050da3538865b1a5
@@ -0,0 +1,17 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+Kconfig symbols
+===============
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   *
+   */*
+   */*/*
+   */*/*/*
+   */*/*/*/*
+   */*/*/*/*/*
+   */*/*/*/*/*/*
Documentation/Makefile +11 -1
index 63094646df2890a788542a273e4a828a844b2932..74ebc5303b47f0837a9ab31d39b5464af5f17995 100644
@@ -115,7 +115,7 @@ $(YNL_INDEX): $(YNL_RST_FILES)
 $(YNL_RST_DIR)/%.rst: $(YNL_YAML_DIR)/%.yaml $(YNL_TOOL)
 	$(Q)$(YNL_TOOL) -i $< -o $@
 
-htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX)
+htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX) kconfigdocs
 
 htmldocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
@@ -182,9 +182,19 @@ endif # HAVE_SPHINX
 refcheckdocs:
 	$(Q)cd $(srctree);scripts/documentation-file-ref-check
 
+KCONFIG_DOC_DIR=$(srctree)/Documentation/Config
+KCONFIGS := $(shell find $(srctree) -name Kconfig -type f)
+KCONFIGS_RST := $(patsubst %, $(KCONFIG_DOC_DIR)/%.rst, $(KCONFIGS))
+
+$(KCONFIGS_RST): $(KCONFIGS)
+	$(Q)cd $(srctree); $(foreach var,$^,$(shell mkdir -p $(KCONFIG_DOC_DIR)/$(shell dirname $(var)); scripts/kconfig2rst.py $(var) >$(KCONFIG_DOC_DIR)/$(var).rst))
+
+kconfigdocs: $(KCONFIGS_RST)
+
 cleandocs:
 	$(Q)rm -f $(YNL_INDEX) $(YNL_RST_FILES)
 	$(Q)rm -rf $(BUILDDIR)
+	$(Q)rm -rf $(filter-out %index.rst,$(wildcard $(KCONFIG_DOC_DIR)/*))
 	$(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media clean
 
 dochelp:
Documentation/kbuild/index.rst +2 -0
index 3731ab22bfe745c5c51963cffe58fb652dadf88c..47a1d9753a9fb7b55b8a7141da8123ca97b15cfb 100644
@@ -15,6 +15,8 @@ Kernel Build System
     makefiles
     modules
 
+    /Config/index
+
     headers_install
 
     issues
scripts/kconfig2rst.py new file +336 -0

336 lines not on this page · raw message

Signature · 1 line
2.49.0
#2 lore raw

Re: [PATCH RFC 1/2] docs: Add documentation generation for Kconfig symbols

kernel test robot noticed the following build warnings:

Hi Nícolas,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on 405e2241def89c88f008dcb899eb5b6d4be8b43c]

url:    https://github.com/intel-lab-lkp/linux/commits/N-colas-F-R-A-Prado/docs-Add-documentation-generation-for-Kconfig-symbols/20250404-220651
base:   405e2241def89c88f008dcb899eb5b6d4be8b43c
patch link:    https://lore.kernel.org/r/20250404-kconfig-docs-v1-1-4c3155d4ba44%40collabora.com
patch subject: [PATCH RFC 1/2] docs: Add documentation generation for Kconfig symbols
config: arc-randconfig-002-20250405 (https://download.01.org/0day-ci/archive/20250405/[email protected]/config)
compiler: arc-linux-gcc (GCC) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250405/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> Documentation/Config/index.rst: warning: ignored by one of the .gitignore files
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#3 lore raw

Re: [PATCH RFC 1/2] docs: Add documentation generation for Kconfig symbols

In reply to Nícolas F. R. A. Prado

Em Fri, 04 Apr 2025 10:02:52 -0400
Nícolas F. R. A. Prado <[email protected]> escreveu:

Quoted 22 lines

Add the contents of all Kconfig files to the Documentation to both increase their visibility and allow for cross-referencing throughout the documentation. In order to achieve this:
* Add a new script 'kconfig2rst' that converts a Kconfig file into a reStructuredText document.
* Add an extra step to the documentation building that runs the script for every Kconfig in the source tree, generating a documentation page for each one.
* Add a new "Kconfig symbols" page in the documentation, that is listed on the "Kernel Build System" page, which contains an index of all Kconfig files and their Kconfig symbols, linking to the corresponding pages.

The generated documentation pages have the config symbols as sections with labels that can be referenced from anywhere in the documentation. The exceptions are configs that appear multiple times. Those don't get labels, as that would generate 'duplicate label' warnings from sphinx. To allow this, a list of configs that appear more than once is embedded in the kconfig2rst script. When a config appears more than once in the same Kconfig file, a count is appended in the section to prevent sphinx's auto-labeling to cause the same warning.

Quoted 7 lines

The paths in 'source' directives in the Kconfig files are turned into links to the generated documentation page to allow for navigation to included Kconfig files.

Config symbols on 'depends'/'select'/etc lines are prepended by 'CONFIG_' to allow them to be cross-referenced by automarkup, though no cross-references are created in this commit.

Despite the huge increase on the time to produce documentation, I'm not sure how worth is to have it, as there are already cross-reference services doing something somewhat similar, like:

https://elixir.bootlin.com

Yet, I didn't test this series yet. So, not sure yet about its value.

Anyway, it follows some comments about the current implementation.

After addressed on a v2, I intend to test and see how it behaves.

Quoted 33 lines
---
 Documentation/.gitignore       |   2 +
 Documentation/Config/index.rst |  17 +++
 Documentation/Makefile         |  12 +-
 Documentation/kbuild/index.rst |   2 +
 scripts/kconfig2rst.py         | 336 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 368 insertions(+), 1 deletion(-)
Documentation/.gitignore +2 -0
index d6dc7c9b8e25020f1f3b28811df2291c38695d5f..2fc70a398dc874fcb83834cb6337f602c64a070a 100644
@@ -1,3 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 output
 *.pyc
+Config/
+!Config/index.rst
Documentation/Config/index.rst new file +7 -0
new file mode 100644
index 0000000000000000000000000000000000000000..2abaa9844dd2a9f57bed0a8d050da3538865b1a5
@@ -0,0 +1,17 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+Kconfig symbols
+===============
+
+.. toctree::

Documentation/Config/index.rst line 17

Quoted 10 lines
Documentation/Config/index.rst new file +10 -0
+   :glob:
+   :maxdepth: 2
+
+   *
+   */*
+   */*/*
+   */*/*/*
+   */*/*/*/*
+   */*/*/*/*/*
+   */*/*/*/*/*/*

That sounds weird, hard to maintain and probably slow.

Better to have a Sphinx extension instead, with a decent implementation of glob. The Python's one is slow, on my tests with the Kernel tree. I worked on something that worked fine for kernel-doc.py:

	https://lore.kernel.org/linux-doc/12a54f1b8f4afd2e70a87195a2aa34f96d736b77.1740387599.git.mchehab+huawei@kernel.org/

Perhaps this script could import the class from it, once such series gets merged. It could make sense to split it on a separate file if we're going to re-use its code.

Quoted 58 lines
Documentation/Makefile +11 -1
index 63094646df2890a788542a273e4a828a844b2932..74ebc5303b47f0837a9ab31d39b5464af5f17995 100644
@@ -115,7 +115,7 @@ $(YNL_INDEX): $(YNL_RST_FILES)
 $(YNL_RST_DIR)/%.rst: $(YNL_YAML_DIR)/%.yaml $(YNL_TOOL)
 	$(Q)$(YNL_TOOL) -i $< -o $@
 
-htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX)
+htmldocs texinfodocs latexdocs epubdocs xmldocs: $(YNL_INDEX) kconfigdocs
 
 htmldocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
@@ -182,9 +182,19 @@ endif # HAVE_SPHINX
 refcheckdocs:
 	$(Q)cd $(srctree);scripts/documentation-file-ref-check
 
+KCONFIG_DOC_DIR=$(srctree)/Documentation/Config
+KCONFIGS := $(shell find $(srctree) -name Kconfig -type f)
+KCONFIGS_RST := $(patsubst %, $(KCONFIG_DOC_DIR)/%.rst, $(KCONFIGS))
+
+$(KCONFIGS_RST): $(KCONFIGS)
+	$(Q)cd $(srctree); $(foreach var,$^,$(shell mkdir -p $(KCONFIG_DOC_DIR)/$(shell dirname $(var)); scripts/kconfig2rst.py $(var) >$(KCONFIG_DOC_DIR)/$(var).rst))
+
+kconfigdocs: $(KCONFIGS_RST)
+
 cleandocs:
 	$(Q)rm -f $(YNL_INDEX) $(YNL_RST_FILES)
 	$(Q)rm -rf $(BUILDDIR)
+	$(Q)rm -rf $(filter-out %index.rst,$(wildcard $(KCONFIG_DOC_DIR)/*))
 	$(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media clean
 
 dochelp:
Documentation/kbuild/index.rst +2 -0
index 3731ab22bfe745c5c51963cffe58fb652dadf88c..47a1d9753a9fb7b55b8a7141da8123ca97b15cfb 100644
@@ -15,6 +15,8 @@ Kernel Build System
     makefiles
     modules
 
+    /Config/index
+
     headers_install
 
     issues
scripts/kconfig2rst.py new file +6 -0
new file mode 100755
index 0000000000000000000000000000000000000000..5af073a1c669ac43c95bb7af00099dcd9473a6ae
@@ -0,0 +1,336 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+# Copyright 2025 Collabora Ltd
+
+import sys
+import re

scripts/kconfig2rst.py line 15

Quoted 10 lines
scripts/kconfig2rst.py new file +9 -0
+import os
+
+import argparse
+
+BASE_PATH_DEFAULT = "Documentation/Config/"
+CFG_LEN = 60
 
+RE_indentation = r"^[ \t]*"
+in_help_txt = False
+help_txt = ""

Better to follow Python's standards and keep all constants on uppercase.

I would place the main code inside a class, with the non-const data inside the class, as it makes the code cleaner and helps using it as both a Sphinx extension and as a standalone command (which is useful for testing it).

Quoted 192 lines
scripts/kconfig2rst.py new file +192 -0
+
+# These configs appear more than once, thus we don't generate labels or xrefs to
+# them to avoid duplicate label warnings from Sphinx
+REPEATED_CONFIGS = [
+    "32BIT",
+    "4KSTACKS",
+    "64BIT",
+    "A",
+    "ADVANCED_OPTIONS",
+    "ALPHA_LEGACY_START_ADDRESS",
+    "ARCH_AIROHA",
+    "ARCH_ALPINE",
+    "ARCH_BCM2835",
+    "ARCH_BCM_IPROC",
+    "ARCH_BRCMSTB",
+    "ARCH_DEFAULT_CRASH_DUMP",
+    "ARCH_FLATMEM_ENABLE",
+    "ARCH_FORCE_MAX_ORDER",
+    "ARCH_HAS_ADD_PAGES",
+    "ARCH_HAS_CACHE_LINE_SIZE",
+    "ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION",
+    "ARCH_HAS_ILOG2_U32",
+    "ARCH_HAS_ILOG2_U64",
+    "ARCH_HIBERNATION_HEADER",
+    "ARCH_HIBERNATION_POSSIBLE",
+    "ARCH_HISI",
+    "ARCH_MAY_HAVE_PC_FDC",
+    "ARCH_MEMORY_PROBE",
+    "ARCH_MMAP_RND_BITS_MAX",
+    "ARCH_MMAP_RND_BITS_MIN",
+    "ARCH_MMAP_RND_COMPAT_BITS_MAX",
+    "ARCH_MMAP_RND_COMPAT_BITS_MIN",
+    "ARCH_MTD_XIP",
+    "ARCH_OMAP",
+    "ARCH_PKEY_BITS",
+    "ARCH_PROC_KCORE_TEXT",
+    "ARCH_R9A07G043",
+    "ARCH_RENESAS",
+    "ARCH_ROCKCHIP",
+    "ARCH_SELECT_MEMORY_MODEL",
+    "ARCH_SELECTS_CRASH_DUMP",
+    "ARCH_SELECTS_KEXEC_FILE",
+    "ARCH_SPARSEMEM_DEFAULT",
+    "ARCH_SPARSEMEM_ENABLE",
+    "ARCH_SUNXI",
+    "ARCH_SUPPORTS_CRASH_DUMP",
+    "ARCH_SUPPORTS_CRASH_HOTPLUG",
+    "ARCH_SUPPORTS_KEXEC",
+    "ARCH_SUPPORTS_KEXEC_FILE",
+    "ARCH_SUPPORTS_KEXEC_JUMP",
+    "ARCH_SUPPORTS_KEXEC_PURGATORY",
+    "ARCH_SUPPORTS_KEXEC_SIG",
+    "ARCH_SUPPORTS_UPROBES",
+    "ARCH_SUSPEND_POSSIBLE",
+    "ARCH_UNIPHIER",
+    "ARCH_VIRT",
+    "AUDIT_ARCH",
+    "B",
+    "BCH_CONST_M",
+    "BCH_CONST_T",
+    "BUILTIN_DTB",
+    "BUILTIN_DTB_NAME",
+    "C",
+    "CC_HAVE_STACKPROTECTOR_TLS",
+    "CHOICE_B",
+    "CHOICE_C",
+    "CMDLINE",
+    "CMDLINE_BOOL",
+    "CMDLINE_EXTEND",
+    "CMDLINE_FORCE",
+    "CMDLINE_FROM_BOOTLOADER",
+    "CMDLINE_OVERRIDE",
+    "CMM",
+    "COMPAT",
+    "COMPAT_VDSO",
+    "CORE",
+    "CORE_BELL_A",
+    "CORE_BELL_A_ADVANCED",
+    "CPU_BIG_ENDIAN",
+    "CPU_HAS_FPU",
+    "CPU_HAS_PREFETCH",
+    "CPU_LITTLE_ENDIAN",
+    "CRYPTO_CHACHA20_NEON",
+    "CRYPTO_JITTERENTROPY_MEMORY_BLOCKS",
+    "CRYPTO_JITTERENTROPY_MEMORY_BLOCKSIZE",
+    "CRYPTO_JITTERENTROPY_OSR",
+    "CRYPTO_JITTERENTROPY_TESTINTERFACE",
+    "CRYPTO_NHPOLY1305_NEON",
+    "DEBUG_ENTRY",
+    "DMA_NONCOHERENT",
+    "DMI",
+    "DRAM_BASE",
+    "DUMMY",
+    "DUMMY_CONSOLE",
+    "EARLY_PRINTK",
+    "EFI",
+    "EFI_STUB",
+    "FIT_IMAGE_FDT_EPM5",
+    "FIX_EARLYCON_MEM",
+    "FPU",
+    "GENERIC_BUG",
+    "GENERIC_BUG_RELATIVE_POINTERS",
+    "GENERIC_CALIBRATE_DELAY",
+    "GENERIC_CSUM",
+    "GENERIC_HWEIGHT",
+    "GENERIC_ISA_DMA",
+    "GENERIC_LOCKBREAK",
+    "HAS_IOMEM",
+    "HAVE_SMP",
+    "HAVE_TCM",
+    "HEARTBEAT",
+    "HIGHMEM",
+    "HOTPLUG_CPU",
+    "HW_PERF_EVENTS",
+    "HZ",
+    "HZ_100",
+    "HZ_1000",
+    "HZ_1024",
+    "HZ_128",
+    "HZ_250",
+    "HZ_256",
+    "ILLEGAL_POINTER_VALUE",
+    "IRQSTACKS",
+    "ISA",
+    "ISA_DMA_API",
+    "KASAN_SHADOW_OFFSET",
+    "KERNEL_MODE_NEON",
+    "KERNEL_START",
+    "KERNEL_START_BOOL",
+    "KUSER_HELPERS",
+    "KVM",
+    "KVM_GUEST",
+    "L1_CACHE_SHIFT",
+    "LEDS_EXPRESSWIRE",
+    "LOCKDEP_SUPPORT",
+    "LOWMEM_SIZE",
+    "LOWMEM_SIZE_BOOL",
+    "MACH_LOONGSON32",
+    "MACH_LOONGSON64",
+    "MACH_TX49XX",
+    "MAGIC_SYSRQ",
+    "MATH_EMULATION",
+    "MCOUNT",
+    "MMU",
+    "NODES_SHIFT",
+    "NO_IOPORT_MAP",
+    "NR_CPUS",
+    "NR_CPUS_DEFAULT",
+    "NR_CPUS_RANGE_END",
+    "NUMA",
+    "PAGE_OFFSET",
+    "PANIC_TIMEOUT",
+    "PARAVIRT",
+    "PARAVIRT_SPINLOCKS",
+    "PARAVIRT_TIME_ACCOUNTING",
+    "PFAULT",
+    "PGTABLE_LEVELS",
+    "PHYSICAL_ALIGN",
+    "PHYSICAL_START",
+    "PID_IN_CONTEXTIDR",
+    "PM",
+    "POWERPC64_CPU",
+    "PRINT_STACK_DEPTH",
+    "RANDOMIZE_BASE",
+    "RANDOMIZE_BASE_MAX_OFFSET",
+    "RELOCATABLE",
+    "SBUS",
+    "SCHED_CLUSTER",
+    "SCHED_HRTICK",
+    "SCHED_MC",
+    "SCHED_OMIT_FRAME_POINTER",
+    "SCHED_SMT",
+    "SERIAL_CONSOLE",
+    "SMP",
+    "STACKPROTECTOR_PER_TASK",
+    "STACKTRACE_SUPPORT",
+    "SWAP_IO_SPACE",
+    "SYS_SUPPORTS_APM_EMULATION",
+    "SYS_SUPPORTS_NUMA",
+    "SYS_SUPPORTS_SMP",
+    "TASK_SIZE",
+    "TASK_SIZE_BOOL",
+    "TCP_CONG_CUBIC",
+    "TIME_LOW_RES",
+    "UNWINDER_FRAME_POINTER",
+    "UNWINDER_GUESS",
+    "UNWINDER_ORC",
+    "USE_OF",
+    "VMSPLIT_1G",
+    "VMSPLIT_2G",
+    "VMSPLIT_3G",
+    "VMSPLIT_3G_OPT",

scripts/kconfig2rst.py line 217

Quoted 10 lines
scripts/kconfig2rst.py new file +10 -0
+    "X",
+    "X86_32",
+    "X86_64",
+    "XEN",
+    "XEN_DOM0",
+    "XIP_KERNEL",
+    "XIP_PHYS_ADDR",
+    "ARCH_BCM",
+    "VIRTUALIZATION",
+]

Maintaining this sounds a nightmare, as new (eventually duplicated) symbols may happen anytime.

The best here sounds to do something similar to what I did with get_abi.py: parse them all altogether, dynamically detecting duplication. IMO, it also makes sense to have dereference pages for such duplicated symbols pointing to all occurrences of them.

Quoted 21 lines
scripts/kconfig2rst.py new file +21 -0
+
+
+def print_title(title):
+    heading = "=" * len(title)
+    print(heading)
+    print(title)
+    print(heading)
+    print()
+
+
+parser = argparse.ArgumentParser(
+    prog="kconfig2rst", description="Convert a Kconfig file into ReStructuredText"
+)
+
+parser.add_argument("kconfig", help="Path to input Kconfig file")
+parser.add_argument(
+    "--base-doc-path",
+    default=BASE_PATH_DEFAULT,
+    help="Base path of generated rST files for usage in 'source' links",
+)
+args = parser.parse_args()

scripts/kconfig2rst.py line 248

Quoted 10 lines
scripts/kconfig2rst.py new file +10 -0
+
+print_title(args.kconfig)
+
+line_accum = ""
+continued_line = False
+
+repeated_config_count = {}
+
+with open(args.kconfig) as f:
+    for il in f:

This won't handle directories. Better to use my glob function.

Also, calling lines as as "il" sounds weird for me. I would just call it "line".

scripts/kconfig2rst.py line 250

Quoted 2 lines
scripts/kconfig2rst.py new file +2 -0
+        # If line ends with \, accumulate it and handle full line
+        if re.search(r"\\\n$", il):

Better to use endswith("\\\n"),, as it is faster. We can also use removesuffix(), as I guess the minimal Python version is now 3.9.

Quoted 7 lines
scripts/kconfig2rst.py new file +7 -0
+            continued_line = True
+            line_accum += il[:-2]  # accumulate without backslash and newline
+            continue
+
+        if continued_line:
+            continued_line = False
+            l = line_accum + il

scripts/kconfig2rst.py line 267

Quoted 10 lines
scripts/kconfig2rst.py new file +10 -0
+            line_accum = ""
+        else:
+            l = il
+
+        if in_help_txt:
+            if l == "\n":
+                help_txt += l
+                continue
+            if first_line_help_txt:
+                help_txt_indentation = re.match(RE_indentation, l).group(0).expandtabs()

Please compile all regular expressions, to make it faster.

Quoted 8 lines
scripts/kconfig2rst.py new file +8 -0
+                first_line_help_txt = False
+            # Consider any line with same or more indentation as part of help text
+            if (
+                help_txt_indentation
+                in re.match(RE_indentation, l).group(0).expandtabs()
+            ):
+                help_txt += l
+                continue

scripts/kconfig2rst.py line 285

Quoted 10 lines
scripts/kconfig2rst.py new file +10 -0
+            else:
+                in_help_txt = False
+                print(help_txt)
+                help_txt = ""
+        else:
+            l = re.sub(r"[*]", r"\*", l)  # Escape asterisks
+
+        if re.match(r"^[ \t]*#.*", l):
+            # Skip comments
+            continue

I would strip comments first, as I guess Kconfig syntaxe allow to use comments after any texts, like:

config SYMBOL # some comment

scripts/kconfig2rst.py line 293

Quoted 8 lines
scripts/kconfig2rst.py new file +8 -0
+
+        if re.match(r"^[ \t]*help", l):
+            in_help_txt = True
+            first_line_help_txt = True
+            print("* help::\n")
+            continue
+
+        m = re.match("^[ \t]*(menu)?config (?P<cfgname>[A-Za-z0-9_]+)", l)

Better to accept multiple spaces after config, as it would be valid to have:

config FOO

Quoted 8 lines
scripts/kconfig2rst.py new file +8 -0
+        if m:
+            section_name = f"\nCONFIG_{m.group('cfgname')}"
+            underline = f"\n{'='*CFG_LEN}\n"
+            if m.group("cfgname") in REPEATED_CONFIGS:
+                repeated_config_count[m.group("cfgname")] = (
+                    repeated_config_count.get(m.group("cfgname"), 0) + 1
+                )
+                if repeated_config_count[m.group("cfgname")] > 1:

scripts/kconfig2rst.py line 311

Quoted 10 lines
scripts/kconfig2rst.py new file +10 -0
+                    section_name += f"({repeated_config_count[m.group('cfgname')]})"
+                print(section_name + underline)
+            else:
+                print(f"\n.. _CONFIG_{m.group('cfgname')}:\n\n" + section_name + underline)
+            continue
+
+        m = re.match(
+            r"^[ \t]*(def_bool|def_tristate|depends on|select|range|visible if|imply|default|prompt|bool|tristate|string|hex|int|modules)( \"(.*)\")?(?P<expr> [^\"]*)?",
+            l,
+        )

I would place the valid matches on an array and do a join to create the compiled regex to match them. This would make easier to maintain as Kconfig syntax add more notations.

Quoted 7 lines
scripts/kconfig2rst.py new file +7 -0
+        if m:
+            expr = m.group('expr') if m.group('expr') else ''
+            not_expr = l
+            if expr:
+                expr = re.sub(r'[A-Z0-9_]{2,}', rf" CONFIG_\g<0> ", expr)
+                not_expr = l[:m.start('expr')]
+            print("* " + not_expr.lstrip() + expr.rstrip())

scripts/kconfig2rst.py line 328

Quoted 10 lines
scripts/kconfig2rst.py new file +10 -0
+            continue
+
+        m = re.match(r'^[ \t]*source "(.*)"', l)
+        if m:
+            # Format Kconfig file paths as Documentation/... so they can be turned
+            # into links by the automarkup plugin
+            print(f"\nsource {args.base_doc_path + m.group(1)}.rst\n")
+            continue
+
+        m = re.match(r"[^ \t]*choice|endchoice|comment|menu|endmenu|if|endif", l)

Same here.

Quoted 9 lines
scripts/kconfig2rst.py new file +8 -0
+        if m:
+            print("\n" + l.strip() + "\n")
+            continue
+
+        print(l.strip())
+
+if help_txt:
+    print(help_txt)  # Flush any pending help text
#4 lore raw

RFC PATCH 2/2 docs: automarkup: Cross-reference CONFIG_ symbols

In reply to Nícolas F. R. A. Prado

Now that Kconfig symbols are part of the Documentation and have sections that can be referenced, extend automarkup to automatically cross-reference CONFIG_ strings to those sections.

---
 Documentation/sphinx/automarkup.py | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)
Documentation/sphinx/automarkup.py +35 -1
index ecf54d22e9dc6ab459a91fde580c1cf161f054ed..dd940baca480634180a58c209aaed0cd1a16319a 100644
@@ -47,6 +47,8 @@ RE_abi_symbol = re.compile(r'(\b/(sys|config|proc)/[\w\-/]+)')
 
 RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
 
+RE_kconfig = re.compile(r'CONFIG_[A-Za-z0-9_]+')
+
 #
 # Reserved C words that we should skip when cross-referencing
 #
@@ -88,7 +90,8 @@ def markup_refs(docname, app, node):
                            RE_union: markup_c_ref,
                            RE_enum: markup_c_ref,
                            RE_typedef: markup_c_ref,
-                           RE_git: markup_git}
+                           RE_git: markup_git,
+                           RE_kconfig: markup_kconfig_ref}
 
     match_iterators = [regex.finditer(t) for regex in markup_func]
     #
@@ -260,6 +263,37 @@ def markup_doc_ref(docname, app, match):
     else:
         return nodes.Text(match.group(0))
 
+#
+# Try to replace a kernel config reference of the form CONFIG_... with a
+# cross reference to that kconfig's section
+#
+def markup_kconfig_ref(docname, app, match):
+    stddom = app.env.domains['std']
+    #
+    # Go through the dance of getting an xref out of the std domain
+    #
+    target = match.group(0).lower()
+    xref = None
+    pxref = addnodes.pending_xref('', refdomain = 'std', reftype = 'ref',
+                                  reftarget = target, modname = None,
+                                  classname = None, refexplicit = False)
+    #
+    # XXX The Latex builder will throw NoUri exceptions here,
+    # work around that by ignoring them.
+    #
+    try:
+        xref = stddom.resolve_xref(app.env, docname, app.builder, 'ref',
+                                   target, pxref, None)
+    except NoUri:
+        xref = None
+    #
+    # Return the xref if we got it; otherwise just return the plain text.
+    #
+    if xref:
+        return xref
+    else:
+        return nodes.Text(match.group(0))
+
 #
 # Try to replace a documentation reference for ABI symbols and files
 # with a cross reference to that page
Signature · 1 line
2.49.0
maintainer
#5 lore raw
In reply to Nícolas F. R. A. Prado
Quoted 18 lines · Nícolas F. R. A. Prado

This series adds Kconfig pages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).

There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):

(On an XPS 13 9300)

Before:

real 6m43.576s
user 23m32.611s
sys 1m48.220s

After:

Quoted 7 lines

real 11m56.845s
user 47m40.528s
sys 2m27.382s

There are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.

I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong.

I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case.

I'll try to look closer, but I'll remain a bit distracted for a little while yet.

Thanks,

jon

#6 lore raw
Quoted 28 lines · Jonathan Corbet
Quoted 18 lines · Nícolas F. R. A. Prado

This series adds Kconfig pages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).

There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):

(On an XPS 13 9300)

Before:

real 6m43.576s
user 23m32.611s
sys 1m48.220s

After:

Quoted 7 lines

real 11m56.845s
user 47m40.528s
sys 2m27.382s

There are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.

Quoted 5 lines

I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong.

I have not yet. Thought I'd get some feedback before looking into the performance. But I agree with the sentiment.

Quoted 4 lines

I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case.

That is what is currently done for the ABI, AFAIK, so definitely seems doable.

The key difference between the ABI approach and this here, is that my goal was to reflect the Kconfig file hierarchy in the Documentation. So each Kconfig file gets its own documentation page, while the ABI approach collects the contents of all ABI files into just a few documentation pages (stable, testing, etc). (So there's a non-constant number of .rst files, which means they have to be generated and can't be a sphinx plugin in this approach).

I went for this approach because the filesystem hierarchy seemed the most logical way to group the Kconfig symbols. Also Kconfig files have directives like 'menu' that should be present in the documentation in the same order they appear in the file to fully describe dependencies of the symbols, and having all of that in the same page seems like it would be confusing. But given the potential benefits it's worth a try for sure.

Now that I think about it, seems quite likely that a lot of the time spent comes from creating a subshell and running the script for every Kconfig file. So making a single script or sphinx extension that itself handles iterating over all the files would likely greatly reduce the run time. I'll test that.

Thanks,
Nícolas

Quoted 7 lines

I'll try to look closer, but I'll remain a bit distracted for a little while yet.

Thanks,

jon

#7 lore raw

Em Fri, 4 Apr 2025 12:24:27 -0400
Nícolas F. R. A. Prado <[email protected]> escreveu:

Quoted 29 lines · Jonathan Corbet
Quoted 28 lines
Quoted 25 lines · Nícolas F. R. A. Prado

This series adds Kconfig pages (patch 1) to the Documentation, and automarkups CONFIG_* text as cross-references to those pages (patch 2).

There is a huge change in build time with this series, so we'd either have to so some optimization and/or put this behind a flag in make so it is only generated when desired (for instance for the online documentation):

(On an XPS 13 9300)

Before:

real 6m43.576s
user 23m32.611s
sys 1m48.220s

After:

real 11m56.845s
user 47m40.528s
sys 2m27.382s

There are also some issues that were solved in ad-hoc ways (eg the sphinx warnings due to repeated Kconfigs, by embedding the list of repeated configs in the script). Hence the RFC.

Quoted 8 lines
Quoted 5 lines

I'm still digging out from LSFMM, so have only glanced at this ... I can see the appeal of doing this, but nearly doubling the docs build time really isn't going to fly. Have you looked to see what is taking all of that time? The idea that it takes as long to process KConfig entries as it does to build the entire rest of the docs seems ... a bit wrong.

I have not yet. Thought I'd get some feedback before looking into the performance. But I agree with the sentiment.

My feeling is that the issue is using :glob" and a lot of wildcards inside Sphinx. Instead, you should use something similar to what I've done to get *.[ch] for the new kernel-doc.py implementation.

Placing it as an extension on a similar way to what i did with get_abi.py would likely help as well.

Quoted 5 lines
Quoted 3 lines

I wonder what it would take to create a Sphinx extension that would simply walk the source tree and slurp up the KConfig entries directly? That would be nicer than adding a separate script in any case.

That is what is currently done for the ABI, AFAIK, so definitely seems doable.

Yes, doing that via an extension is doable. If done right, it can also be fast.

Quoted 6 lines

The key difference between the ABI approach and this here, is that my goal was to reflect the Kconfig file hierarchy in the Documentation. So each Kconfig file gets its own documentation page, while the ABI approach collects the contents of all ABI files into just a few documentation pages (stable, testing, etc). (So there's a non-constant number of .rst files, which means they have to be generated and can't be a sphinx plugin in this approach).

Actually, get-api.py (the new version, merged for 6.15) generates a dict just once. Then, Sphinx rst files filters part of the doc, but I see your
point: for every entry, we would need a .rst file if we follow the same approach.

That's said, it may have a way to tell Sphinx to threat Kconfig files on a similar way it handles ".txt" and ".rst" files. Something like the extension to handle markdown works:

https://www.sphinx-doc.org/en/master/usage/markdown.html

Another alternative would be to use:

https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-include_patterns

but this would require Sphinx 5.1, which is above our current minimal version. That's said, nothing prevents to only enable generating such documentatation if the Sphinx version supports it.

Quoted 22 lines

I went for this approach because the filesystem hierarchy seemed the most logical way to group the Kconfig symbols. Also Kconfig files have directives like 'menu' that should be present in the documentation in the same order they appear in the file to fully describe dependencies of the symbols, and having all of that in the same page seems like it would be confusing. But given the potential benefits it's worth a try for sure.

Now that I think about it, seems quite likely that a lot of the time spent comes from creating a subshell and running the script for every Kconfig file. So making a single script or sphinx extension that itself handles iterating over all the files would likely greatly reduce the run time. I'll test that.

Thanks,
Nícolas

Quoted 7 lines

I'll try to look closer, but I'll remain a bit distracted for a little while yet.

Thanks,

jon