Skip to contentSkip to conversation

doc: generate CVE catalog documentation

PATCH In review Sachin Sant 4 participants 9 messages Apr 28 – May 7

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/2 doc: generate CVE catalog documentation

Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.

The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cve.rst) containing:
* Total CVE count
* All CVEs sorted in descending order (newest first)
* Table of CVEs:
- CVE ID
- Test name (Cross-references to test catalog entries)
- Integrates CVE catalog into main documentation index

---
V4 changes:
- Simplified the CVE table (id, test name)
- Removed individual CVE pages
- v3 link https://lore.kernel.org/ltp/[email protected]/T/#u

V3 changes:
- CVEs sorted in descending order
- append test name to CVE id : CVE (Test Name)
- Separate page for CVE catalog
- Link cve testcases to Test catalog entry
- v2 link https://lore.kernel.org/ltp/[email protected]/

V2 changes:
- Replace Fixes tag by Closes
- V1 link https://lore.kernel.org/ltp/[email protected]/T/#u

---
 doc/Makefile              |  2 +-
 doc/conf.py               | 91 +++++++++++++++++++++++++++++++++++++++
 doc/index.rst             |  4 ++
 doc/users/cve_catalog.rst |  6 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 doc/users/cve_catalog.rst
doc/Makefile +1 -1
index 3123b1cd7..e99cbe666 100644
@@ -30,7 +30,7 @@ spelling:
 	$(RUN_VENV); sphinx-build -b spelling -d build/doctree . build/spelling
 
 clean:
-	rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \
+	rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \
 		${abs_top_builddir}/metadata/ltp.json
 
 distclean: clean
doc/conf.py +91 -0

107 lines not on this page · raw message

doc/index.rst +4 -0
index 496a12f80..733495f51 100644
@@ -12,6 +12,7 @@
    users/testers_guide
    users/supported_systems
    users/stats
+   users/cve_catalog
    users/test_catalog
 
 .. toctree::
@@ -58,6 +59,9 @@ For users
 :doc:`users/stats`
    Some LTP statistics
 
+:doc:`users/cve_catalog`
+   LTP reproducers for known CVEs
+
 :doc:`users/test_catalog`
    The LTP test catalog
 
doc/users/cve_catalog.rst new file +6 -0
new file mode 100644
index 000000000..5a5b9b54a
@@ -0,0 +1,6 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+CVE catalog
+===========
+
+.. include:: ../_static/cve.rst
Signature · 5 lines
2.39.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

PATCH 2/2 doc: Rename statistics page to 'Supported syscalls'

Rename the documentation page from 'Statistics' to 'Supported syscalls' to better reflect its content, which focuses on syscall coverage.

- Renamed doc/users/stats.rst to doc/users/supported_syscalls.rst
- Updated page title and description
- Updated references in doc/index.rst
- Updated spelling exclusion pattern in doc/conf.py
- Renamed function definition from generate_syscalls_stats to generate_supported_syscalls
- Updated app.connect() call to use the new function name

---
 doc/conf.py                      | 13 +++++++------
 doc/index.rst                    |  6 +++---
 doc/users/stats.rst              |  9 ---------
 doc/users/supported_syscalls.rst |  9 +++++++++
 4 files changed, 19 insertions(+), 18 deletions(-)
 delete mode 100644 doc/users/stats.rst
 create mode 100644 doc/users/supported_syscalls.rst
doc/conf.py +7 -6
index d692638a0..a4fdf1436 100644
@@ -55,7 +55,7 @@ manpages_url = 'https://man7.org/linux/man-pages/man{section}/{page}.{section}.h
 
 spelling_lang = "en_US"
 spelling_warning = True
-spelling_exclude_patterns = ['users/stats.rst']
+spelling_exclude_patterns = ['users/supported_syscalls.rst']
 spelling_word_list_filename = "spelling_wordlist"
 
 # -- Options for HTML output -------------------------------------------------
@@ -65,11 +65,12 @@ html_theme = 'sphinx_rtd_theme'
 html_static_path = ['_static']
 
 
-def generate_syscalls_stats(_):
+def generate_supported_syscalls(_):
     """
-    Generate statistics for syscalls. We fetch the syscalls list from the kernel
-    sources, then we compare it with testcases/kernel/syscalls folder and
-    generate a file that is included in the statistics documentation section.
+    Generate supported syscalls documentation. We fetch the syscalls list from
+    the kernel sources, then we compare it with testcases/kernel/syscalls folder
+    and generate a file that is included in the supported syscalls documentation
+    section.
     """
     output = '_static/syscalls.rst'
 
@@ -632,6 +633,6 @@ def setup(app):
     customizations.
     """
     app.add_css_file('custom.css')
-    app.connect('builder-inited', generate_syscalls_stats)
+    app.connect('builder-inited', generate_supported_syscalls)
     app.connect('builder-inited', generate_cve_catalog)
     app.connect('builder-inited', generate_test_catalog)
doc/index.rst +3 -3
index 733495f51..f80ca4be1 100644
@@ -11,7 +11,7 @@
    users/setup_tests
    users/testers_guide
    users/supported_systems
-   users/stats
+   users/supported_syscalls
    users/cve_catalog
    users/test_catalog
 
@@ -56,8 +56,8 @@ For users
 :doc:`users/supported_systems`
    A list of supported technologies by the LTP framework
 
-:doc:`users/stats`
-   Some LTP statistics
+:doc:`users/supported_syscalls`
+   Syscalls supported by LTP tests
 
 :doc:`users/cve_catalog`
    LTP reproducers for known CVEs
doc/users/stats.rst deleted +0 -9
deleted file mode 100644
index 7073442aa..000000000
@@ -1,9 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0-or-later
-
-Statistics
-==========
-
-In this section we collect some statistics related to the current state of
-LTP tests.
-
-.. include:: ../_static/syscalls.rst
doc/users/supported_syscalls.rst new file +9 -0
new file mode 100644
index 000000000..6914852c6
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+Supported syscalls
+==================
+
+In this section we collect information about syscalls that are currently
+tested by LTP.
+
+.. include:: ../_static/syscalls.rst
Signature · 5 lines
2.39.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by LTP AI Reviewer on v4 1/2
#3 lore raw
In reply to Sachin Sant

Hi Sachin,

Quoted 2 lines · Sachin Sant

doc: generate CVE catalog documentation
doc: Rename statistics page to 'Supported syscalls'

Both patches look good. The CVE catalog generator correctly handles missing runtest/cve and metadata files with warnings rather than hard failures. The rename is consistent — spelling exclusion, function name, app.connect() call, and index references are all updated in lockstep.

---
Note:

Our agent completed the review of the patch. The full review can be found at: https://github.com/linux-test-project/ltp-agent/actions/runs/25066131650

The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
In reply to Sachin Sant

Hi!

Quoted 12 lines

Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.

The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cve.rst) containing:
* Total CVE count
* All CVEs sorted in descending order (newest first)
* Table of CVEs:
- CVE ID
- Test name (Cross-references to test catalog entries)
- Integrates CVE catalog into main documentation index

First of all, this is very cool idea, thanks for doing this!

Quoted 34 lines

---
V4 changes:
- Simplified the CVE table (id, test name)
- Removed individual CVE pages
- v3 link https://lore.kernel.org/ltp/[email protected]/T/#u

V3 changes:
- CVEs sorted in descending order
- append test name to CVE id : CVE (Test Name)
- Separate page for CVE catalog
- Link cve testcases to Test catalog entry
- v2 link https://lore.kernel.org/ltp/[email protected]/

V2 changes:
- Replace Fixes tag by Closes
- V1 link https://lore.kernel.org/ltp/[email protected]/T/#u

---
 doc/Makefile              |  2 +-
 doc/conf.py               | 91 +++++++++++++++++++++++++++++++++++++++
 doc/index.rst             |  4 ++
 doc/users/cve_catalog.rst |  6 +++
 4 files changed, 102 insertions(+), 1 deletion(-)
 create mode 100644 doc/users/cve_catalog.rst
doc/Makefile +0 -0
index 3123b1cd7..e99cbe666 100644

doc/Makefile spelling: · line 34

Quoted 6 lines
doc/Makefile +1 -1
@@ -30,7 +30,7 @@ spelling:
 	$(RUN_VENV); sphinx-build -b spelling -d build/doctree . build/spelling
 
 clean:
-	rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \
+	rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \
 		${abs_top_builddir}/metadata/ltp.json

This shouldn't be added here. the ltp.json is cleaned up by the metadata/Makefile.

Quoted 24 lines
doc/Makefile +0 -0
 distclean: clean
doc/conf.py +9 -0
index 63d09352e..d692638a0 100644
@@ -30,6 +30,15 @@ extensions = [
     'sphinx.ext.extlinks',
 ]
 
+# Configure autosectionlabel to prefix labels with document name
+# This prevents duplicate labels when same test name appears in multiple files
+autosectionlabel_prefix_document = True
+# Only create labels for sections with unique names
+autosectionlabel_maxdepth = 2
+
+# Suppress duplicate label warnings for kernel-doc generated content
+suppress_warnings = ['autosectionlabel.*']
+
 exclude_patterns = ["html*", '_static*', '.venv*']
 extlinks = {
     'repo': (f'{ltp_repo}/%s', '%s'),
@@ -535,6 +544,87 @@ def generate_test_catalog(_):
     with open(output, 'w+', encoding='utf-8') as new_tests:
         new_tests.write('\n'.join(text))

doc/conf.py def generate_test_catalog(_): · line 555

Quoted 10 lines
doc/conf.py +9 -0
 
+def generate_cve_catalog(_):
+    """
+    Generate CVE catalog in a single file. Parse runtest/cve file and
+    generate documentation with links to CVE databases and test sources.
+    Similar to test_catalog, creates a single _static/cve.rst file with
+    all CVE information.
+    """
+    output = '_static/cve.rst'
+    runtest_cve = '../runtest/cve'

I do not like much that we depend on the cve runtest file, we want to get rid of runtest files eventually and depend only on the ltp.json.

Quoted 16 lines
doc/conf.py +16 -0
+    metadata_file = '../metadata/ltp.json'
+
+    # Load metadata to check which tests exist in the catalog
+    metadata = None
+    try:
+        with open(metadata_file, 'r', encoding='utf-8') as data:
+            metadata = json.load(data)
+    except FileNotFoundError:
+        logger = sphinx.util.logging.getLogger(__name__)
+        msg = f"Can't find metadata file ({metadata_file})"
+        logger.warning(msg)
+
+    # Parse runtest/cve file
+    cve_data = {}
+
+    try:

doc/conf.py def generate_test_catalog(_): · line 581

Quoted 10 lines
doc/conf.py +10 -0
+        with open(runtest_cve, 'r', encoding='utf-8') as f:
+            for line in f:
+                line = line.strip()
+                if not line or line.startswith('#'):
+                    continue
+
+                parts = line.split(None, 2)
+                if len(parts) >= 2:
+                    cve_id = parts[0].upper()
+                    test_name = parts[1]

Moreover it's not guaranteed at all that this is a test name. It's whatever needs to be excuted to run the test.

Quoted 33 lines
doc/conf.py +33 -0
+
+                    cve_data[cve_id] = {
+                        'cve_id': cve_id,
+                        'test_name': test_name,
+                    }
+    except FileNotFoundError:
+        logger = sphinx.util.logging.getLogger(__name__)
+        msg = f"Can't find runtest/cve file ({runtest_cve})"
+        logger.warning(msg)
+        return
+
+    # Generate single CVE catalog file
+    total_cves = len(cve_data)
+    text = [
+        '.. warning::',
+        '    The following CVE catalog has been generated from the',
+        '    runtest/cve file and includes all CVE reproducers in LTP.',
+        '',
+        f'LTP includes reproducers for {total_cves} known CVEs.',
+        '',
+        '.. list-table::',
+        '   :header-rows: 1',
+        '   :widths: 40 60',
+        '',
+        '   * - CVE ID',
+        '     - Test Name',
+    ]
+
+    # Add CVEs in descending order (newest first)
+    for cve_id, cve_info in sorted(cve_data.items(), reverse=True):
+        test_name = cve_info["test_name"]
+
+        # Only create cross-reference if test exists in metadata

doc/conf.py def generate_test_catalog(_): · line 624

Quoted 10 lines
doc/conf.py +10 -0
+        if metadata and test_name in metadata.get('tests', {}):
+            test_anchor = f"users/test_catalog:{test_name}"
+            test_link = f":ref:`{test_name} <{test_anchor}>`"
+        else:
+            test_link = f"``{test_name}``"
+
+        text.extend([
+            f'   * - {cve_id}',
+            f'     - {test_link}',
+        ])

Can we please instead iterate over the "tests" in the metadata here and collect all tests that have "CVE" key in "tags"?

That should be more straightforward since we do not need to parse two files.

Quoted 51 lines
doc/conf.py +3 -0
+    with open(output, 'w+', encoding='utf-8') as cve_catalog:
+        cve_catalog.write('\n'.join(text))
 
 def setup(app):
     """
@@ -543,4 +633,5 @@ def setup(app):
     """
     app.add_css_file('custom.css')
     app.connect('builder-inited', generate_syscalls_stats)
+    app.connect('builder-inited', generate_cve_catalog)
     app.connect('builder-inited', generate_test_catalog)
doc/index.rst +4 -0
index 496a12f80..733495f51 100644
@@ -12,6 +12,7 @@
    users/testers_guide
    users/supported_systems
    users/stats
+   users/cve_catalog
    users/test_catalog
 
 .. toctree::
@@ -58,6 +59,9 @@ For users
 :doc:`users/stats`
    Some LTP statistics
 
+:doc:`users/cve_catalog`
+   LTP reproducers for known CVEs
+
 :doc:`users/test_catalog`
    The LTP test catalog
 
doc/users/cve_catalog.rst new file +6 -0
new file mode 100644
index 000000000..5a5b9b54a
@@ -0,0 +1,6 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+CVE catalog
+===========
+
+.. include:: ../_static/cve.rst
Signature · 5 lines
2.39.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw
Quoted 6 lines · Cyril Hrubis

Hi!

doc/Makefile spelling: · line 34

Quoted 3 lines
doc/Makefile +1 -1
-	rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \
+	rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \
  		${abs_top_builddir}/metadata/ltp.json

This shouldn't be added here. the ltp.json is cleaned up by the metadata/Makefile.

My bad, probably an oversight on my part. Will remove.

Quoted 9 lines

doc/conf.py def generate_test_catalog(_): · line 555

Quoted 7 lines
doc/conf.py +7 -0
+    Generate CVE catalog in a single file. Parse runtest/cve file and
+    generate documentation with links to CVE databases and test sources.
+    Similar to test_catalog, creates a single _static/cve.rst file with
+    all CVE information.
+    """
+    output = '_static/cve.rst'
+    runtest_cve = '../runtest/cve'

I do not like much that we depend on the cve runtest file, we want to get rid of runtest files eventually and depend only on the ltp.json.

Ah I see. Thanks for the review. Will remove the dependency on runtest/cve.

Quoted 9 lines

doc/conf.py def generate_test_catalog(_): · line 624

Quoted 4 lines
doc/conf.py +4 -0
+        text.extend([
+            f'   * - {cve_id}',
+            f'     - {test_link}',
+        ])

Can we please instead iterate over the "tests" in the metadata here and collect all tests that have "CVE" key in "tags"?

That should be more straightforward since we do not need to parse two files.

Sure, will rewrite the logic based on ltp.json metadata. Will send an updated version of the patch.

Signature · 7 lines
Thanks
- Sachin



-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw
Quoted 4 lines · Sachin Sant
Quoted 1 line · Cyril Hrubis

Hi!

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

-    rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \
+    rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \
          ${abs_top_builddir}/metadata/ltp.json

This shouldn't be added here. the ltp.json is cleaned up by the metadata/Makefile.

My bad, probably an oversight on my part. Will remove.

Actually this change (remove ltp.json) is not introduced by my patch. This was already present and was added via commit 86de077fb128.

Signature · 7 lines
Thanks
- Sachin



-- 
Mailing list info: https://lists.linux.it/listinfo/ltp