Skip to contentSkip to conversation

support/scripts/pkg-stats: fix CVE identification (branch yem/pkgstat-cve)

PATCH In review Yann E. MORIN 2 participants 5 messages Feb 25, 2024 – Feb 28, 2024

lore
Share & tools
lore
#0 lore raw

cover support/scripts/pkg-stats: fix CVE identification (branch yem/pkgstat-cve)

Hello All!

Since we switched to the FKIE CVE feed (as the NVD JSON feed are oing extinct soon-ish), we missed the fact that the new feeds did not have the same layout as the former ones, breaking CVE reporting.

As a consequence of having no CVE reported, w emissed the fact that the CVEs were not sorted in the new feeds, while they were in the formers.

This series fixes both issus, and introduces a minor fixup for a python deprecation warning.

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (3):
support/scripts/pkg-stats: fix datetime deprecation warning support/scripts/pkg-stats: fkie json format support/scripts/pkg-stats: sort CVEs in HTML output

 support/scripts/cve.py    | 26 +++++++++++++++++---------
 support/scripts/pkg-stats |  6 +++---
 2 files changed, 20 insertions(+), 12 deletions(-)

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#1 lore raw

PATCH 1/3 support/scripts/pkg-stats: fix datetime deprecation warning

Abide by the warning reported with python 3.12:

.../support/scripts/pkg-stats:1289: DeprecationWarning:
datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).

---
 support/scripts/pkg-stats | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/pkg-stats +1 -1
index 7835cb3d62..00d1eef09a 100755
@@ -1286,7 +1286,7 @@ def __main__():
         package_list = set([v["name"] for v in show_info_js.values() if 'name' in v])
     else:
         package_list = None
-    date = datetime.datetime.utcnow()
+    date = datetime.datetime.now(datetime.UTC)
     commit = subprocess.check_output(['git', '-C', brpath,
                                       'rev-parse',
                                       'HEAD']).splitlines()[0].decode()
Signature · 6 lines
2.43.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

PATCH 2/3 support/scripts/pkg-stats: fkie json format

In reply to Yann E. MORIN

Commit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for the JSON files) missed the fact that the layout of the FKIE data files are different from the NVD ones.

Most differences are relatively trivial fields renaming, and those are easily spotted in this patch.

There is however one key difference in the layout of the configurations. Where the NVD had "configurations" as an object with a "nodes" key, the FKIE has a "configurations" as a list of objects with a single "nodes" key; i.e. it is one-level deeper.

Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/scripts/cve.py | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
support/scripts/cve.py +10 -9
index f12a8048cd..747ad881c9 100755
@@ -128,7 +128,7 @@ class CVE:
             filename = CVE.download_nvd_year(nvd_dir, year)
             try:
                 uncompressed = subprocess.check_output(["xz", "-d", "-c", filename])
-                content = ijson.items(uncompressed, 'CVE_Items.item')
+                content = ijson.items(uncompressed, 'cve_items.item')
             except:  # noqa: E722
                 print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
                 raise
@@ -155,11 +155,11 @@ class CVE:
             for parsed_node in self.parse_node(child):
                 yield parsed_node
 
-        for cpe in node.get('cpe_match', ()):
+        for cpe in node.get('cpeMatch', ()):
             if not cpe['vulnerable']:
                 return
-            product = cpe_product(cpe['cpe23Uri'])
-            version = cpe_version(cpe['cpe23Uri'])
+            product = cpe_product(cpe['criteria'])
+            version = cpe_version(cpe['criteria'])
             # ignore when product is '-', which means N/A
             if product == '-':
                 return
@@ -191,7 +191,7 @@ class CVE:
                     v_end = cpe['versionEndExcluding']
 
             yield {
-                'id': cpe['cpe23Uri'],
+                'id': cpe['criteria'],
                 'v_start': v_start,
                 'op_start': op_start,
                 'v_end': v_end,
@@ -199,14 +199,15 @@ class CVE:
             }
 
     def each_cpe(self):
-        for node in self.nvd_cve['configurations']['nodes']:
-            for cpe in self.parse_node(node):
-                yield cpe
+        for nodes in self.nvd_cve.get('configurations', []):
+            for node in nodes['nodes']:
+                for cpe in self.parse_node(node):
+                    yield cpe
 
     @property
     def identifier(self):
         """The CVE unique identifier"""
-        return self.nvd_cve['cve']['CVE_data_meta']['ID']
+        return self.nvd_cve['id']
 
     @property
     def affected_products(self):
Signature · 6 lines
2.43.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

PATCH 3/3 support/scripts/pkg-stats: sort CVEs in HTML output

In reply to Yann E. MORIN

While the old NVD JSON feed provided data files where the CVEs were sorted by ID, the new feed from FKIE does not have sorted CVEs.

Add a method to sort a list of CVE IDs (i.e. CVE ID strings, not CVE objects!), and use that when emiting the HTML output.

The JSON output need not be sorted, because it is supposed to be used for post-processing, and we do not care about the ordering there; a consumer interested in sorting should sort on their side.

Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/scripts/cve.py    | 7 +++++++
 support/scripts/pkg-stats | 4 ++--
 2 files changed, 9 insertions(+), 2 deletions(-)
support/scripts/cve.py +7 -0
index 747ad881c9..1a3c307e12 100755
@@ -117,6 +117,13 @@ class CVE:
         open(path_metaf, "w").write(page_meta.text)
         return path_jsonf_xz
 
+    @staticmethod
+    def sort_id(cve_ids):
+        def cve_key(cve_id):
+            year, id_ = cve_id.split('-')[1:]
+            return (int(year), int(id_))
+        return sorted(cve_ids, key=cve_key)
+
     @classmethod
     def read_nvd_dir(cls, nvd_dir):
         """
support/scripts/pkg-stats +2 -2
index 00d1eef09a..7eefd92a22 100755
@@ -1051,9 +1051,9 @@ def dump_html_pkg(f, pkg):
         f.write(f' <div onclick="expandField(\'{data_field_id}\')" \
         class="see-more centered cve_ignored">see all ({cve_total}) &#9662;</div>\n')
     if pkg.is_status_error("cve"):
-        for cve in pkg.cves:
+        for cve in cvecheck.CVE.sort_id(pkg.cves):
             f.write(f'   <a href="https://security-tracker.debian.org/tracker/{cve}">{cve}</a><br/>\n')
-        for cve in pkg.unsure_cves:
+        for cve in cvecheck.CVE.sort_id(pkg.unsure_cves):
             f.write(f'   <a href="https://security-tracker.debian.org/tracker/{cve}">{cve} <i>(unsure)</i></a><br/>\n')
     elif pkg.is_status_na("cve"):
         f.write(f"""    {pkg.status['cve'][1]}""")
Signature · 6 lines
2.43.2

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw
In reply to Yann E. MORIN
Quoted 6 lines · Yann E. MORIN

Hello All!

Since we switched to the FKIE CVE feed (as the NVD JSON feed are oing extinct soon-ish), we missed the fact that the new feeds did not have the same layout as the former ones, breaking CVE reporting.

Quoted 8 lines

As a consequence of having no CVE reported, w emissed the fact that the CVEs were not sorted in the new feeds, while they were in the formers.

This series fixes both issus, and introduces a minor fixup for a python deprecation warning.

Regards,
Yann E. MORIN.

Series applied to master, thanks!

When I was working on it, I thought something needed to change in parse_node
itself (because that was done in the other NVDv2 series), but apparently not.

That said, we currently do make some sweeping assumptions about how a
configuration is specified....

Regards,
Arnout

Quoted 12 lines

----------------------------------------------------------------
Yann E. MORIN (3):
support/scripts/pkg-stats: fix datetime deprecation warning support/scripts/pkg-stats: fkie json format support/scripts/pkg-stats: sort CVEs in HTML output

  support/scripts/cve.py    | 26 +++++++++++++++++---------
  support/scripts/pkg-stats |  6 +++---
  2 files changed, 20 insertions(+), 12 deletions(-)
Quoted 7 lines

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot