doc: generate CVE catalog documentation
PATCH v4 1/2 doc: generate CVE catalog documentation
Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.
The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cve.rst) containing:
* Total CVE count
* All CVEs sorted in descending order (newest first)
* Table of CVEs:
- CVE ID
- Test name (Cross-references to test catalog entries)
- Integrates CVE catalog into main documentation index
---
V4 changes:
- Simplified the CVE table (id, test name)
- Removed individual CVE pages
- v3 link https://lore.kernel.org/ltp/[email protected]/T/#u
V3 changes:
- CVEs sorted in descending order
- append test name to CVE id : CVE (Test Name)
- Separate page for CVE catalog
- Link cve testcases to Test catalog entry
- v2 link https://lore.kernel.org/ltp/[email protected]/
V2 changes:
- Replace Fixes tag by Closes
- V1 link https://lore.kernel.org/ltp/[email protected]/T/#u
--- doc/Makefile | 2 +- doc/conf.py | 91 +++++++++++++++++++++++++++++++++++++++ doc/index.rst | 4 ++ doc/users/cve_catalog.rst | 6 +++ 4 files changed, 102 insertions(+), 1 deletion(-) create mode 100644 doc/users/cve_catalog.rst
doc/Makefile +1 -1
@@ -30,7 +30,7 @@ spelling: $(RUN_VENV); sphinx-build -b spelling -d build/doctree . build/spelling clean: - rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \ + rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \ ${abs_top_builddir}/metadata/ltp.json distclean: clean
doc/conf.py +91 -0
107 lines not on this page · raw message
doc/index.rst +4 -0
@@ -12,6 +12,7 @@ users/testers_guide users/supported_systems users/stats + users/cve_catalog users/test_catalog .. toctree::
@@ -58,6 +59,9 @@ For users :doc:`users/stats` Some LTP statistics +:doc:`users/cve_catalog` + LTP reproducers for known CVEs + :doc:`users/test_catalog` The LTP test catalog
doc/users/cve_catalog.rst new file +6 -0
@@ -0,0 +1,6 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +CVE catalog +=========== + +.. include:: ../_static/cve.rst
Signature · 5 lines
2.39.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH 2/2 doc: Rename statistics page to 'Supported syscalls'
Rename the documentation page from 'Statistics' to 'Supported syscalls' to better reflect its content, which focuses on syscall coverage.
- Renamed doc/users/stats.rst to doc/users/supported_syscalls.rst
- Updated page title and description
- Updated references in doc/index.rst
- Updated spelling exclusion pattern in doc/conf.py
- Renamed function definition from generate_syscalls_stats to generate_supported_syscalls
- Updated app.connect() call to use the new function name
--- doc/conf.py | 13 +++++++------ doc/index.rst | 6 +++--- doc/users/stats.rst | 9 --------- doc/users/supported_syscalls.rst | 9 +++++++++ 4 files changed, 19 insertions(+), 18 deletions(-) delete mode 100644 doc/users/stats.rst create mode 100644 doc/users/supported_syscalls.rst
doc/conf.py +7 -6
@@ -55,7 +55,7 @@ manpages_url = 'https://man7.org/linux/man-pages/man{section}/{page}.{section}.h spelling_lang = "en_US" spelling_warning = True -spelling_exclude_patterns = ['users/stats.rst'] +spelling_exclude_patterns = ['users/supported_syscalls.rst'] spelling_word_list_filename = "spelling_wordlist" # -- Options for HTML output -------------------------------------------------
@@ -65,11 +65,12 @@ html_theme = 'sphinx_rtd_theme' html_static_path = ['_static'] -def generate_syscalls_stats(_): +def generate_supported_syscalls(_): """ - Generate statistics for syscalls. We fetch the syscalls list from the kernel - sources, then we compare it with testcases/kernel/syscalls folder and - generate a file that is included in the statistics documentation section. + Generate supported syscalls documentation. We fetch the syscalls list from + the kernel sources, then we compare it with testcases/kernel/syscalls folder + and generate a file that is included in the supported syscalls documentation + section. """ output = '_static/syscalls.rst'
@@ -632,6 +633,6 @@ def setup(app): customizations. """ app.add_css_file('custom.css') - app.connect('builder-inited', generate_syscalls_stats) + app.connect('builder-inited', generate_supported_syscalls) app.connect('builder-inited', generate_cve_catalog) app.connect('builder-inited', generate_test_catalog)
doc/index.rst +3 -3
@@ -11,7 +11,7 @@ users/setup_tests users/testers_guide users/supported_systems - users/stats + users/supported_syscalls users/cve_catalog users/test_catalog
@@ -56,8 +56,8 @@ For users :doc:`users/supported_systems` A list of supported technologies by the LTP framework -:doc:`users/stats` - Some LTP statistics +:doc:`users/supported_syscalls` + Syscalls supported by LTP tests :doc:`users/cve_catalog` LTP reproducers for known CVEs
doc/users/stats.rst deleted +0 -9
@@ -1,9 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0-or-later - -Statistics -========== - -In this section we collect some statistics related to the current state of -LTP tests. - -.. include:: ../_static/syscalls.rst
doc/users/supported_syscalls.rst new file +9 -0
@@ -0,0 +1,9 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Supported syscalls +================== + +In this section we collect information about syscalls that are currently +tested by LTP. + +.. include:: ../_static/syscalls.rst
Signature · 5 lines
2.39.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 2/2] doc: Rename statistics page to 'Supported syscalls'
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Sachin,
Quoted 2 lines · Sachin Sant
doc: generate CVE catalog documentation
doc: Rename statistics page to 'Supported syscalls'
Both patches look good. The CVE catalog generator correctly handles missing runtest/cve and metadata files with warnings rather than hard failures. The rename is consistent — spelling exclusion, function name, app.connect() call, and index references are all updated in lockstep.
---
Note:
Our agent completed the review of the patch. The full review can be found at: https://github.com/linux-test-project/ltp-agent/actions/runs/25066131650
The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.
Regards,
LTP AI Reviewer
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 12 lines
Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.
The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cve.rst) containing:
* Total CVE count
* All CVEs sorted in descending order (newest first)
* Table of CVEs:
- CVE ID
- Test name (Cross-references to test catalog entries)
- Integrates CVE catalog into main documentation index
First of all, this is very cool idea, thanks for doing this!
Quoted 34 lines
Cc Andrea Cervesato <[email protected]>Cc Petr Vorel <[email protected]>Signed-off-by Sachin Sant <[email protected]>---
V4 changes:
- Simplified the CVE table (id, test name)
- Removed individual CVE pages
- v3 link https://lore.kernel.org/ltp/[email protected]/T/#uV3 changes:
- CVEs sorted in descending order
- append test name to CVE id : CVE (Test Name)
- Separate page for CVE catalog
- Link cve testcases to Test catalog entry
- v2 link https://lore.kernel.org/ltp/[email protected]/V2 changes:
- Replace Fixes tag by Closes
- V1 link https://lore.kernel.org/ltp/[email protected]/T/#u--- doc/Makefile | 2 +- doc/conf.py | 91 +++++++++++++++++++++++++++++++++++++++ doc/index.rst | 4 ++ doc/users/cve_catalog.rst | 6 +++ 4 files changed, 102 insertions(+), 1 deletion(-) create mode 100644 doc/users/cve_catalog.rstdoc/Makefile +0
-0
doc/Makefile spelling: · line 34
Quoted 6 lines
doc/Makefile +1
-1@@ -30,7 +30,7 @@ spelling: $(RUN_VENV); sphinx-build -b spelling -d build/doctree . build/spelling clean: - rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \ + rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \ ${abs_top_builddir}/metadata/ltp.json
This shouldn't be added here. the ltp.json is cleaned up by the metadata/Makefile.
Quoted 24 lines
doc/Makefile +0
-0distclean: cleandoc/conf.py +9
-0@@ -30,6 +30,15 @@ extensions = [ 'sphinx.ext.extlinks', ] +# Configure autosectionlabel to prefix labels with document name +# This prevents duplicate labels when same test name appears in multiple files +autosectionlabel_prefix_document = True +# Only create labels for sections with unique names +autosectionlabel_maxdepth = 2 + +# Suppress duplicate label warnings for kernel-doc generated content +suppress_warnings = ['autosectionlabel.*'] + exclude_patterns = ["html*", '_static*', '.venv*'] extlinks = { 'repo': (f'{ltp_repo}/%s', '%s'),@@ -535,6 +544,87 @@ def generate_test_catalog(_): with open(output, 'w+', encoding='utf-8') as new_tests: new_tests.write('\n'.join(text))
doc/conf.py def generate_test_catalog(_): · line 555
Quoted 10 lines
doc/conf.py +9
-0+def generate_cve_catalog(_): + """ + Generate CVE catalog in a single file. Parse runtest/cve file and + generate documentation with links to CVE databases and test sources. + Similar to test_catalog, creates a single _static/cve.rst file with + all CVE information. + """ + output = '_static/cve.rst' + runtest_cve = '../runtest/cve'
I do not like much that we depend on the cve runtest file, we want to get rid of runtest files eventually and depend only on the ltp.json.
Quoted 16 lines
doc/conf.py +16
-0+ metadata_file = '../metadata/ltp.json' + + # Load metadata to check which tests exist in the catalog + metadata = None + try: + with open(metadata_file, 'r', encoding='utf-8') as data: + metadata = json.load(data) + except FileNotFoundError: + logger = sphinx.util.logging.getLogger(__name__) + msg = f"Can't find metadata file ({metadata_file})" + logger.warning(msg) + + # Parse runtest/cve file + cve_data = {} + + try:
doc/conf.py def generate_test_catalog(_): · line 581
Quoted 10 lines
doc/conf.py +10
-0+ with open(runtest_cve, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if not line or line.startswith('#'): + continue + + parts = line.split(None, 2) + if len(parts) >= 2: + cve_id = parts[0].upper() + test_name = parts[1]
Moreover it's not guaranteed at all that this is a test name. It's whatever needs to be excuted to run the test.
Quoted 33 lines
doc/conf.py +33
-0+ + cve_data[cve_id] = { + 'cve_id': cve_id, + 'test_name': test_name, + } + except FileNotFoundError: + logger = sphinx.util.logging.getLogger(__name__) + msg = f"Can't find runtest/cve file ({runtest_cve})" + logger.warning(msg) + return + + # Generate single CVE catalog file + total_cves = len(cve_data) + text = [ + '.. warning::', + ' The following CVE catalog has been generated from the', + ' runtest/cve file and includes all CVE reproducers in LTP.', + '', + f'LTP includes reproducers for {total_cves} known CVEs.', + '', + '.. list-table::', + ' :header-rows: 1', + ' :widths: 40 60', + '', + ' * - CVE ID', + ' - Test Name', + ] + + # Add CVEs in descending order (newest first) + for cve_id, cve_info in sorted(cve_data.items(), reverse=True): + test_name = cve_info["test_name"] + + # Only create cross-reference if test exists in metadata
doc/conf.py def generate_test_catalog(_): · line 624
Quoted 10 lines
doc/conf.py +10
-0+ if metadata and test_name in metadata.get('tests', {}): + test_anchor = f"users/test_catalog:{test_name}" + test_link = f":ref:`{test_name} <{test_anchor}>`" + else: + test_link = f"``{test_name}``" + + text.extend([ + f' * - {cve_id}', + f' - {test_link}', + ])
Can we please instead iterate over the "tests" in the metadata here and collect all tests that have "CVE" key in "tags"?
That should be more straightforward since we do not need to parse two files.
Quoted 51 lines
doc/conf.py +3
-0+ with open(output, 'w+', encoding='utf-8') as cve_catalog: + cve_catalog.write('\n'.join(text)) def setup(app): """@@ -543,4 +633,5 @@ def setup(app): """ app.add_css_file('custom.css') app.connect('builder-inited', generate_syscalls_stats) + app.connect('builder-inited', generate_cve_catalog) app.connect('builder-inited', generate_test_catalog)doc/index.rst +4
-0@@ -12,6 +12,7 @@ users/testers_guide users/supported_systems users/stats + users/cve_catalog users/test_catalog .. toctree::@@ -58,6 +59,9 @@ For users :doc:`users/stats` Some LTP statistics +:doc:`users/cve_catalog` + LTP reproducers for known CVEs + :doc:`users/test_catalog` The LTP test catalogdoc/users/cve_catalog.rst new file +6
-0@@ -0,0 +1,6 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +CVE catalog +=========== + +.. include:: ../_static/cve.rstSignature · 5 lines
2.39.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 6 lines · Cyril Hrubis
Hi!
doc/Makefile spelling: · line 34
Quoted 3 lines
doc/Makefile +1
-1- rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \ + rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cve.rst syscalls.tbl \ ${abs_top_builddir}/metadata/ltp.jsonThis shouldn't be added here. the ltp.json is cleaned up by the metadata/Makefile.
My bad, probably an oversight on my part. Will remove.
Quoted 9 lines
doc/conf.py def generate_test_catalog(_): · line 555
Quoted 7 lines
doc/conf.py +7
-0+ Generate CVE catalog in a single file. Parse runtest/cve file and + generate documentation with links to CVE databases and test sources. + Similar to test_catalog, creates a single _static/cve.rst file with + all CVE information. + """ + output = '_static/cve.rst' + runtest_cve = '../runtest/cve'I do not like much that we depend on the cve runtest file, we want to get rid of runtest files eventually and depend only on the ltp.json.
Ah I see. Thanks for the review. Will remove the dependency on runtest/cve.
Quoted 9 lines
doc/conf.py def generate_test_catalog(_): · line 624
Quoted 4 lines
doc/conf.py +4
-0+ text.extend([ + f' * - {cve_id}', + f' - {test_link}', + ])Can we please instead iterate over the "tests" in the metadata here and collect all tests that have "CVE" key in "tags"?
That should be more straightforward since we do not need to parse two files.
Sure, will rewrite the logic based on ltp.json metadata. Will send an updated version of the patch.
Signature · 7 lines
Thanks - Sachin -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 4 lines · Sachin Sant
Quoted 1 line · Cyril Hrubis
Hi!
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
- rm -rf html/ build/
_static/syscalls.rst_static/tests.rstsyscalls.tbl \
+ rm -rf html/ build/_static/syscalls.rst_static/tests.rst_static/cve.rstsyscalls.tbl \
${abs_top_builddir}/metadata/ltp.jsonThis shouldn't be added here. the ltp.json is cleaned up by the metadata/Makefile.
My bad, probably an oversight on my part. Will remove.
Actually this change (remove ltp.json) is not introduced by my patch. This was already present and was added via commit 86de077fb128.
Signature · 7 lines
Thanks - Sachin -- Mailing list info: https://lists.linux.it/listinfo/ltp