Skip to contentSkip to conversation

doc: generate CVE reproducer statistics

PATCH In review Sachin Sant 3 participants 4 messages Apr 23

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 doc: generate CVE reproducer statistics

Add a Sphinx builder hook to parse runtest/cve, collect CVE reproducer metadata, and generate a documentation page with per-year counts and links to CVE entries and test sources.

Also include the generated CVE reproducer statistics page from doc/users/stats.rst.

---
V2 changes:
- Replace Fixes tag by Closes
- V1 link https://lore.kernel.org/ltp/[email protected]/T/#u

---
 doc/conf.py         | 169 ++++++++++++++++++++++++++++++++++++++++++++
 doc/users/stats.rst |   2 +
 2 files changed, 171 insertions(+)
doc/conf.py +169 -0
index 63d09352e..19e744d79 100644
@@ -535,6 +535,174 @@ def generate_test_catalog(_):
     with open(output, 'w+', encoding='utf-8') as new_tests:
         new_tests.write('\n'.join(text))
 
+def generate_cve_stats(_):
+    """
+    Generate statistics for CVE reproducers. Parse runtest/cve file,
+    scan testcases directories, and generate documentation with links
+    to CVE databases and test sources.
+    """
+    output = '_static/cve_reproducers.rst'
+    runtest_cve = '../runtest/cve'
+
+    text = [
+        'CVE Reproducers\n',
+        '---------------\n\n',
+        'LTP includes reproducers for known CVEs. These tests help verify\n',
+        'that systems are patched against known vulnerabilities.\n\n',
+    ]
+
+    # Parse runtest/cve file
+    cve_data = {}
+    cve_pattern = re.compile(r'^(cve-(\d{4})-\d+)\s+(\S+)(?:\s+(.*))?$')
+
+    try:
+        with open(runtest_cve, 'r', encoding='utf-8') as f:
+            for line in f:
+                line = line.strip()
+                # Skip comments and empty lines
+                if not line or line.startswith('#'):
+                    continue
+
+                match = cve_pattern.match(line)
+                if match:
+                    cve_id = match.group(1).upper()
+                    year = match.group(2)
+                    test_name = match.group(3)
+                    options = match.group(4) if match.group(4) else ''
+
+                    cve_data[cve_id] = {
+                        'cve_id': cve_id,
+                        'year': year,
+                        'test_name': test_name,
+                        'options': options,
+                        'source_path': None,
+                        'description': ''
+                    }
+    except FileNotFoundError:
+        logger = sphinx.util.logging.getLogger(__name__)
+        msg = f"Can't find runtest/cve file ({runtest_cve})"
+        logger.warning(msg)
+
+        with open(output, 'w+', encoding='utf-8') as stats:
+            stats.write(f".. warning::\n\n    {msg}")
+        return
+
+    # Scan for CVE test source files
+    testcases_dirs = [
+        '../testcases/cve',
+        '../testcases/kernel/syscalls',
+        '../testcases/kernel/mem',
+        '../testcases/kernel/pty',
+    ]
+
+    # Build a mapping of all potential source files
+    # Key: base filename without extension, Value: full path
+    source_files = {}
+    for base_dir in testcases_dirs:
+        if not os.path.exists(base_dir):
+            continue
+
+        for dirpath, _, files in os.walk(base_dir):
+            for filename in files:
+                if filename.endswith('.c') or filename.endswith('.sh'):
+                    # Store base name without extension
+                    base_name = filename.rsplit('.', 1)[0]
+                    rel_path = os.path.join(
+                        dirpath.replace('../', ''),
+                        filename
+                    )
+                    source_files[base_name] = rel_path
+
+    # Match CVE entries with source files
+    for cve_id, cve_info in cve_data.items():
+        test_name = cve_info['test_name']
+        source_path = None
+
+        if test_name in source_files:
+            source_path = source_files[test_name]
+        elif cve_id.lower() in source_files:
+            source_path = source_files[cve_id.lower()]
+
+        if source_path:
+            cve_info['source_path'] = source_path
+
+            try:
+                src_file = os.path.join('..', source_path)
+                with open(src_file, 'r', encoding='utf-8') as src:
+                    in_comment = False
+                    desc_lines = []
+                    for line in src:
+                        line = line.strip()
+                        if line.startswith('/*\\'):
+                            in_comment = True
+                            continue
+                        if in_comment:
+                            if line.endswith('*/'):
+                                break
+                            if line.startswith('*'):
+                                desc_line = line[1:].strip()
+                                if desc_line:
+                                    desc_lines.append(desc_line)
+
+                    if desc_lines:
+                        # Use first non-empty line as description
+                        cve_info['description'] = desc_lines[0]
+            except (IOError, UnicodeDecodeError):
+                pass
+
+    # Generate statistics
+    total_cves = len(cve_data)
+    cves_by_year = {}
+    for cve_info in cve_data.values():
+        year = cve_info['year']
+        cves_by_year[year] = cves_by_year.get(year, 0) + 1
+
+    text.append(f'* **Total CVEs tested:** {total_cves}\n')
+    text.append('* **CVEs by year:**\n\n')
+
+    for year in sorted(cves_by_year.keys()):
+        text.append(f'  * {year}: {cves_by_year[year]} CVEs\n')
+
+    text.append('\n')
+
+    # Generate CVE table
+    text.extend([
+        'CVE List\n',
+        '~~~~~~~~\n\n',
+        '.. list-table::\n',
+        '   :header-rows: 1\n',
+        '   :widths: 20 25 10 45\n\n',
+        '   * - CVE ID\n',
+        '     - Test Name\n',
+        '     - Year\n',
+        '     - Description\n',
+    ])
+
+    # Sort CVEs by ID (chronologically)
+    for cve_id in sorted(cve_data.keys()):
+        cve_info = cve_data[cve_id]
+
+        cve_url = f"https://cve.mitre.org/cgi-bin/cvename.cgi?name={cve_id}"
+        cve_link = f"`{cve_id} <{cve_url}>`_"
+
+        if cve_info['source_path']:
+            test_url = f"{ltp_repo_base_url}/{cve_info['source_path']}"
+            test_link = f"`{cve_info['test_name']} <{test_url}>`__"
+        else:
+            test_link = cve_info['test_name']
+
+        description = cve_info['description'] if cve_info['description'] else 'CVE reproducer test'
+
+        text.extend([
+            f"   * - {cve_link}\n",
+            f"     - {test_link}\n",
+            f"     - {cve_info['year']}\n",
+            f"     - {description}\n",
+        ])
+
+    with open(output, 'w+', encoding='utf-8') as stats:
+        stats.writelines(text)
+
 
 def setup(app):
     """
@@ -543,4 +711,5 @@ def setup(app):
     """
     app.add_css_file('custom.css')
     app.connect('builder-inited', generate_syscalls_stats)
+    app.connect('builder-inited', generate_cve_stats)
     app.connect('builder-inited', generate_test_catalog)
doc/users/stats.rst +2 -0
index 7073442aa..9ecf806bb 100644
@@ -7,3 +7,5 @@ In this section we collect some statistics related to the current state of
 LTP tests.
 
 .. include:: ../_static/syscalls.rst
+
+.. include:: ../_static/cve_reproducers.rst
Signature · 5 lines
2.39.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

Hi Sachin,

[ Cc Andrea ]

Quoted 6 lines

Add a Sphinx builder hook to parse runtest/cve, collect CVE reproducer metadata, and generate a documentation page with per-year counts and links to CVE entries and test sources.

Also include the generated CVE reproducer statistics page from doc/users/stats.rst.

Thanks for implementing this.

I'd prefer this to be on a separate page named "CVE Reproducers" (OT: I'd also prefer to rename "Statistics" to "Supported syscalls", but that's separate thing.)

More notes:

* I suppose it shouldn't be too hard to do it similarly as what we do in the test catalog [1], have each CVE id + optional binary name in parenthesis in the menu on the left:

	CVE-2025-38236
	...
	CVE-2016-5195 (dirtyc0w)
	...

Or, we could have tables by years (i.e. left menu only years with counted number of tests in parenthesis)

	2025 (2)
	2023 (3)

I don't have a strong preference, which one is better.

* Sort by CVE ID descending (newest CVE on the top, people are mostly interested at current problems than about history).
* Maybe link to the file should be linked to the test catalog page (which has also link to the test source?), e.g. [2] (link could be relative).
* The description is mostly useless (often just CVE ID or "CVE reproducer test"), link to the test catalog would help.
* Year is useless (visible from CVE ID)
* Some of the tests don't have link now, e.g. dirtyc0w, snd_timer01.

Kind regards,
Petr

[1] https://linux-test-project.readthedocs.io/en/latest/users/test_catalog.html
[2] https://linux-test-project.readthedocs.io/en/latest/users/test_catalog.html#cve-2016-7042

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
Quoted 5 lines · Petr Vorel

Thanks for implementing this.

I'd prefer this to be on a separate page named "CVE Reproducers" (OT: I'd also prefer to rename "Statistics" to "Supported syscalls", but that's separate thing.)

Sure, will attempt to implement it.

Quoted 11 lines

More notes:

* I suppose it shouldn't be too hard to do it similarly as what we do in the test catalog [1], have each CVE id + optional binary name in parenthesis in the menu on the left:

	CVE-2025-38236
	...
	CVE-2016-5195 (dirtyc0w)
	...

Yes, agree. I will use this format.
CVE id + binary name in parenthesis sounds better  as compared to below scheme. Will wait for additional feedback.

Quoted 10 lines

Or, we could have tables by years (i.e. left menu only years with counted number of tests in parenthesis)

	2025 (2)
	2023 (3)

I don't have a strong preference, which one is better.

* Sort by CVE ID descending (newest CVE on the top, people are mostly interested at current problems than about history).

Will implement this logic.

Quoted 2 lines

* Maybe link to the file should be linked to the test catalog page (which has also link to the test source?), e.g. [2] (link could be relative).

Sounds logical. Will make the change.

Quoted 3 lines

* The description is mostly useless (often just CVE ID or "CVE reproducer test"), link to the test catalog would help.
* Year is useless (visible from CVE ID)

Will drop these columns.

Quoted 1 line

* Some of the tests don't have link now, e.g. dirtyc0w, snd_timer01.

Will fix it.
Thanks for the review.

Signature · 7 lines
Thanks
- Sachin



-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by LTP AI Reviewer on v2 1/1
#3 lore raw
In reply to Sachin Sant

Hi Sachin,

Quoted 1 line · Sachin Sant

doc: generate CVE reproducer statistics

---
Note:

Our agent completed the review of the patch. The full review can be found at: https://github.com/linux-test-project/ltp-agent/actions/runs/24837980800

The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp