doc: generate CVE catalog documentation
PATCH v3 1/2 doc: generate CVE catalog documentation
Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.
The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cves.rst) containing:
* Total CVE count
* All CVEs sorted in descending order (newest first)
* For each CVE:
- Links to CVE MITRE database
- Cross-references to test catalog entries
- Test command details and options
- Vulnerability description
- Integrates CVE catalog into main documentation index
---
V3 changes:
- CVEs sorted in descending order
- append test name to CVE id : CVE (Test Name)
- Separate page for CVE catalog
- Link cve testcases to Test catalog entry
- v2 link https://lore.kernel.org/ltp/[email protected]/
V2 changes:
- Replace Fixes tag by Closes
- V1 link https://lore.kernel.org/ltp/[email protected]/T/#u
--- doc/Makefile | 2 +- doc/conf.py | 127 ++++++++++++++++++++++++++++++++++++++ doc/index.rst | 4 ++ doc/users/cve_catalog.rst | 6 ++ 4 files changed, 138 insertions(+), 1 deletion(-) create mode 100644 doc/users/cve_catalog.rst
doc/Makefile +1 -1
@@ -30,7 +30,7 @@ spelling: $(RUN_VENV); sphinx-build -b spelling -d build/doctree . build/spelling clean: - rm -rf html/ build/ _static/syscalls.rst _static/tests.rst syscalls.tbl \ + rm -rf html/ build/ _static/syscalls.rst _static/tests.rst _static/cves.rst syscalls.tbl \ ${abs_top_builddir}/metadata/ltp.json distclean: clean
doc/conf.py +127 -0
143 lines not on this page · raw message
doc/index.rst +4 -0
@@ -12,6 +12,7 @@ users/testers_guide users/supported_systems users/stats + users/cve_catalog users/test_catalog .. toctree::
@@ -58,6 +59,9 @@ For users :doc:`users/stats` Some LTP statistics +:doc:`users/cve_catalog` + LTP reproducers for known CVEs + :doc:`users/test_catalog` The LTP test catalog
doc/users/cve_catalog.rst new file +6 -0
@@ -0,0 +1,6 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +CVE Catalog +=========== + +.. include:: ../_static/cves.rst
Signature · 5 lines
2.39.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH 2/2 doc: Rename statistics page to 'Supported syscalls'
Rename the documentation page from 'Statistics' to 'Supported syscalls' to better reflect its content, which focuses on syscall coverage.
- Renamed doc/users/stats.rst to doc/users/supported_syscalls.rst
- Updated page title and description
- Updated references in doc/index.rst
- Updated spelling exclusion pattern in doc/conf.py
- Renamed function definition from generate_syscalls_stats to generate_supported_syscalls
- Updated app.connect() call to use the new function name
--- doc/conf.py | 13 +++++++------ doc/index.rst | 6 +++--- doc/users/stats.rst | 9 --------- doc/users/supported_syscalls.rst | 9 +++++++++ 4 files changed, 19 insertions(+), 18 deletions(-) delete mode 100644 doc/users/stats.rst create mode 100644 doc/users/supported_syscalls.rst
doc/conf.py +7 -6
@@ -56,7 +56,7 @@ manpages_url = 'https://man7.org/linux/man-pages/man{section}/{page}.{section}.h spelling_lang = "en_US" spelling_warning = True -spelling_exclude_patterns = ['users/stats.rst'] +spelling_exclude_patterns = ['users/supported_syscalls.rst'] spelling_word_list_filename = "spelling_wordlist" # -- Options for HTML output -------------------------------------------------
@@ -66,11 +66,12 @@ html_theme = 'sphinx_rtd_theme' html_static_path = ['_static'] -def generate_syscalls_stats(_): +def generate_supported_syscalls(_): """ - Generate statistics for syscalls. We fetch the syscalls list from the kernel - sources, then we compare it with testcases/kernel/syscalls folder and - generate a file that is included in the statistics documentation section. + Generate supported syscalls documentation. We fetch the syscalls list from + the kernel sources, then we compare it with testcases/kernel/syscalls folder + and generate a file that is included in the supported syscalls documentation + section. """ output = '_static/syscalls.rst'
@@ -668,6 +669,6 @@ def setup(app): customizations. """ app.add_css_file('custom.css') - app.connect('builder-inited', generate_syscalls_stats) + app.connect('builder-inited', generate_supported_syscalls) app.connect('builder-inited', generate_cve_catalog) app.connect('builder-inited', generate_test_catalog)
doc/index.rst +3 -3
@@ -11,7 +11,7 @@ users/setup_tests users/testers_guide users/supported_systems - users/stats + users/supported_syscalls users/cve_catalog users/test_catalog
@@ -56,8 +56,8 @@ For users :doc:`users/supported_systems` A list of supported technologies by the LTP framework -:doc:`users/stats` - Some LTP statistics +:doc:`users/supported_syscalls` + Syscalls supported by LTP tests :doc:`users/cve_catalog` LTP reproducers for known CVEs
doc/users/stats.rst deleted +0 -9
@@ -1,9 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0-or-later - -Statistics -========== - -In this section we collect some statistics related to the current state of -LTP tests. - -.. include:: ../_static/syscalls.rst
doc/users/supported_syscalls.rst new file +9 -0
@@ -0,0 +1,9 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Supported syscalls +================== + +In this section we collect information about syscalls that are currently +tested by LTP. + +.. include:: ../_static/syscalls.rst
Signature · 5 lines
2.39.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH 2/2] doc: Rename statistics page to 'Supported syscalls'
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Sachin,
Quoted 2 lines · Sachin Sant
doc: generate CVE catalog documentation
doc: Rename statistics page to 'Supported syscalls'
Both patches look good to me.
One minor note for future reference: the suppress_warnings entry added in patch 1 silences all autosectionlabel warnings globally, not just those from kernel-doc content as the comment implies. This is pre-existing context and does not block this series.
---
Note:
Our agent completed the review of the patch.
The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.
Regards,
LTP AI Reviewer
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
The idea is good, a few comments on the implementation tho:
Quoted 6 lines
Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.
The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cves.rst) containing:
Please use just 'cve.rst' instead of 'cves.rst'. The plural is not providing any info here.
doc/conf.py def generate_test_catalog(_): · line 560
Quoted 13 lines
doc/conf.py +13
-0+def generate_cve_catalog(_): + """ + Generate CVE catalog in a single file. Parse runtest/cve file and + generate documentation with links to CVE databases and test sources. + Similar to test_catalog, creates a single _static/cves.rst file with + all CVE information. + """ + output = '_static/cves.rst' + runtest_cve = '../runtest/cve' + + # Parse runtest/cve file + cve_data = {} + cve_pattern = re.compile(r'^(cve-(\d{4})-\d+)\s+(\S+)(?:\s+(.*))?$')
Do we need a regexp for this? The runtest file is well defining the following structure:
<cve-id> <binary> <arg0> <arg1> ..
We only need to split lines which are not starting with # char, considering space as separator.
Quoted 90 lines
doc/conf.py +90
-0+ + try: + with open(runtest_cve, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if not line or line.startswith('#'): + continue + + match = cve_pattern.match(line) + if match: + cve_id = match.group(1).upper() + year = match.group(2) + test_name = match.group(3) + options = match.group(4) if match.group(4) else '' + + cve_data[cve_id] = { + 'cve_id': cve_id, + 'year': year, + 'test_name': test_name, + 'options': options, + } + except FileNotFoundError: + logger = sphinx.util.logging.getLogger(__name__) + msg = f"Can't find runtest/cve file ({runtest_cve})" + logger.warning(msg) + return + + # Generate single CVE catalog file + total_cves = len(cve_data) + text = [ + '.. warning::', + ' The following CVE catalog has been generated from the', + ' runtest/cve file and includes all CVE reproducers in LTP.', + '', + f'LTP includes reproducers for {total_cves} known CVEs. These ' + 'tests help verify', + 'that systems are patched against known vulnerabilities.', + '', + ] + + # Load metadata to check which tests have documentation + metadata = None + metadata_file = '../metadata/ltp.json' + try: + with open(metadata_file, 'r', encoding='utf-8') as data: + metadata = json.load(data) + except FileNotFoundError: + pass + + # Add CVEs in descending order (newest first) + for cve_id, cve_info in sorted(cve_data.items(), reverse=True): + cve_url = f"https://cve.mitre.org/cgi-bin/cvename.cgi?name={cve_id}" + test_name = cve_info["test_name"] + + # Only create cross-reference if test exists in metadata + if metadata and test_name in metadata.get('tests', {}): + # Create anchor using the correct document path prefix + test_anchor = f"users/test_catalog:{test_name}" + test_link = f":ref:`{test_name} <{test_anchor}>`" + else: + # If test not in metadata, just use plain text formatting + test_link = f"``{test_name}``" + + # Create section header with CVE ID and test name + section_title = f'{cve_id} ({test_name})' + text.extend([ + section_title, + len(section_title) * '-', + '', + f'**CVE Reference:** `{cve_id} <{cve_url}>`_', + '', + f'**Test Name:** {test_link}', + '', + ]) + + if cve_info['options']: + text.extend([ + f'**Test Options:** ``{cve_info["options"]}``', + '', + ]) + + # Build test command on a single line to avoid RST formatting issues + test_cmd = f'``{test_name}' + if cve_info['options']: + test_cmd += f' {cve_info["options"]}' + test_cmd += '``' + + text.extend([ + f'This test reproduces the vulnerability described in {cve_id}.', + 'The test verifies that the system is properly patched against',
doc/conf.py def generate_test_catalog(_): · line 660
Quoted 10 lines
doc/conf.py +10
-0+ 'this known security vulnerability.', + '', + f'* **CVE Year:** {cve_info["year"]}', + f'* **Test Command:** {test_cmd}', + '', + '.. raw:: html', + '', + ' <hr>', + '', + ])
All this text is redundant and occupying space for no reason. Also I'm not sure about this approach, now we have tests list duplication inside the documentation.
What about generating only a simple reference table? There are only 3 information we need: CVE ID, test binary, year.
| ID | Test name | Year |
-------------------------
| .. | .. | .. |
etc..
There's no need to have a direct link to the CVE, since we already pointing CVE to the Test catalog anyway.
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi,
...
Quoted 6 lines
Quoted 3 lines
The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cves.rst) containing:Please use just '
cve.rst' instead of 'cves.rst'. The plural is not providing any info here.
+1
Quoted 17 lines
doc/conf.py def generate_test_catalog(_): · line 560
Quoted 13 lines
doc/conf.py +13
-0+def generate_cve_catalog(_): + """ + Generate CVE catalog in a single file. Parse runtest/cve file and + generate documentation with links to CVE databases and test sources. + Similar to test_catalog, creates a single _static/cves.rst file with + all CVE information. + """ + output = '_static/cves.rst' + runtest_cve = '../runtest/cve' + + # Parse runtest/cve file + cve_data = {} + cve_pattern = re.compile(r'^(cve-(\d{4})-\d+)\s+(\S+)(?:\s+(.*))?$')Do we need a regexp for this? The runtest file is well defining the following structure:
Quoted 4 lines
<cve-id> <binary> <arg0> <arg1> ..
We only need to split lines which are not starting with # char, considering space as separator.
+1
...
Quoted 6 lines
What about generating only a simple reference table? There are only 3 information we need: CVE ID, test binary, year.
| ID | Test name | Year |
-------------------------
| .. | .. | .. |
Do we need year in a column at all? If we like separating by year, we can make more tables each being just for a particular year.
Quoted 4 lines
etc..
There's no need to have a direct link to the CVE, since we already pointing CVE to the Test catalog anyway.
OTOH it would not harm (people interested at CVE will have look mostly on this page, that would save them one click.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 10 lines
Quoted 8 lines
Add a Sphinx builder hook to parse runtest/cve and generate a comprehensive CVE catalog in a single documentation file.
The implementation:
- Parses runtest/cve to extract CVE IDs, test names, and options
- Generates a single CVE catalog file (_static/cves.rst) containing:Please use just '
cve.rst' instead of 'cves.rst'. The plural is not providing any info here.
Sure, will change the name.
Quoted 7 lines
Quoted 7 lines
doc/conf.py +7
-0+def generate_cve_catalog(_): + """ + Generate CVE catalog in a single file. Parse runtest/cve file and + generate documentation with links to CVE databases and test sources. + Similar to test_catalog, creates a single _static/cves.rst file with + all CVE information. + """
Quoted 8 lines
doc/conf.py def generate_test_catalog(_): · line 560
Quoted 6 lines
doc/conf.py +6
-0+ output = '_static/cves.rst' + runtest_cve = '../runtest/cve' + + # Parse runtest/cve file + cve_data = {} + cve_pattern = re.compile(r'^(cve-(\d{4})-\d+)\s+(\S+)(?:\s+(.*))?$')Do we need a regexp for this? The runtest file is well defining the following structure:
Yeah, we may not need a regexp here.
Quoted 5 lines
<cve-id> <binary> <arg0> <arg1> ..We only need to split lines which are not starting with # char, considering space as separator.
got it. Will include this change in the next version.
Quoted 18 lines
Quoted 4 lines
doc/conf.py +4
-0+ + text.extend([ + f'This test reproduces the vulnerability described in {cve_id}.', + 'The test verifies that the system is properly patched against',doc/conf.py def generate_test_catalog(_): · line 660
Quoted 10 lines
doc/conf.py +10
-0+ 'this known security vulnerability.', + '', + f'* **CVE Year:** {cve_info["year"]}', + f'* **Test Command:** {test_cmd}', + '', + '.. raw:: html', + '', + ' <hr>', + '', + ])All this text is redundant and occupying space for no reason. Also I'm not sure about this approach, now we have tests list duplication inside the documentation.
Quoted 8 lines
What about generating only a simple reference table? There are only 3 information we need: CVE ID, test binary, year.
| ID | Test name | Year |
-------------------------
| .. | .. | .. |etc..
Okay. Do we need year in the column since the CVE name also has year. For cve.rst, I implemented the same logic as tests.rst.
A separate entry for CVE catalog.
Once a user clicks on this option it will expand with a list of individual CVE pages in the left pane. Each entry follows the following format
<CVE ID> ( <Test Name> )
Based on review comments how should the page look like?
A separate entry for CVE catalog.
Once a user clicks on this option it will display the CVE list in tabular format sorted by year. The table will follow (assuming Year to be retained)
| ID | Test name | Year |
-------------------------
| .. | .. | .. |
Test name will point to an entry in Test catalog.
No individual pages required for each CVE.
Can you confirm?
Quoted 1 line
There's no need to have a direct link to the CVE, since we already pointing
Okay. I can make this change. I followed the earlier
Thanks for the review.
Signature · 7 lines
Thanks - Sachin -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 1 line
Okay. Do we need year in the column since the CVE name also has year.
No we dont need the year as Petr mentioned.
King regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp