support/scripts/pkg-stats: fix CVE identification (branch yem/pkgstat-cve)
cover support/scripts/pkg-stats: fix CVE identification (branch yem/pkgstat-cve)
Hello All!
Since we switched to the FKIE CVE feed (as the NVD JSON feed are oing extinct soon-ish), we missed the fact that the new feeds did not have the same layout as the former ones, breaking CVE reporting.
As a consequence of having no CVE reported, w emissed the fact that the CVEs were not sorted in the new feeds, while they were in the formers.
This series fixes both issus, and introduces a minor fixup for a python deprecation warning.
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Yann E. MORIN (3):
support/scripts/pkg-stats: fix datetime deprecation warning support/scripts/pkg-stats: fkie json format support/scripts/pkg-stats: sort CVEs in HTML output
support/scripts/cve.py | 26 +++++++++++++++++--------- support/scripts/pkg-stats | 6 +++--- 2 files changed, 20 insertions(+), 12 deletions(-)
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 1/3 support/scripts/pkg-stats: fix datetime deprecation warning
Abide by the warning reported with python 3.12:
.../support/scripts/pkg-stats:1289: DeprecationWarning:
datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
--- support/scripts/pkg-stats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/pkg-stats +1 -1
@@ -1286,7 +1286,7 @@ def __main__(): package_list = set([v["name"] for v in show_info_js.values() if 'name' in v]) else: package_list = None - date = datetime.datetime.utcnow() + date = datetime.datetime.now(datetime.UTC) commit = subprocess.check_output(['git', '-C', brpath, 'rev-parse', 'HEAD']).splitlines()[0].decode()
Signature · 6 lines
2.43.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/3 support/scripts/pkg-stats: fkie json format
In reply to Yann E. MORINCommit 22b69455526f (support/scripts/cve.py: switch from NVD to FKIE for the JSON files) missed the fact that the layout of the FKIE data files are different from the NVD ones.
Most differences are relatively trivial fields renaming, and those are easily spotted in this patch.
There is however one key difference in the layout of the configurations. Where the NVD had "configurations" as an object with a "nodes" key, the FKIE has a "configurations" as a list of objects with a single "nodes" key; i.e. it is one-level deeper.
--- support/scripts/cve.py | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
support/scripts/cve.py +10 -9
@@ -128,7 +128,7 @@ class CVE: filename = CVE.download_nvd_year(nvd_dir, year) try: uncompressed = subprocess.check_output(["xz", "-d", "-c", filename]) - content = ijson.items(uncompressed, 'CVE_Items.item') + content = ijson.items(uncompressed, 'cve_items.item') except: # noqa: E722 print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename) raise
@@ -155,11 +155,11 @@ class CVE: for parsed_node in self.parse_node(child): yield parsed_node - for cpe in node.get('cpe_match', ()): + for cpe in node.get('cpeMatch', ()): if not cpe['vulnerable']: return - product = cpe_product(cpe['cpe23Uri']) - version = cpe_version(cpe['cpe23Uri']) + product = cpe_product(cpe['criteria']) + version = cpe_version(cpe['criteria']) # ignore when product is '-', which means N/A if product == '-': return
@@ -191,7 +191,7 @@ class CVE: v_end = cpe['versionEndExcluding'] yield { - 'id': cpe['cpe23Uri'], + 'id': cpe['criteria'], 'v_start': v_start, 'op_start': op_start, 'v_end': v_end,
@@ -199,14 +199,15 @@ class CVE: } def each_cpe(self): - for node in self.nvd_cve['configurations']['nodes']: - for cpe in self.parse_node(node): - yield cpe + for nodes in self.nvd_cve.get('configurations', []): + for node in nodes['nodes']: + for cpe in self.parse_node(node): + yield cpe @property def identifier(self): """The CVE unique identifier""" - return self.nvd_cve['cve']['CVE_data_meta']['ID'] + return self.nvd_cve['id'] @property def affected_products(self):
Signature · 6 lines
2.43.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 3/3 support/scripts/pkg-stats: sort CVEs in HTML output
In reply to Yann E. MORINWhile the old NVD JSON feed provided data files where the CVEs were sorted by ID, the new feed from FKIE does not have sorted CVEs.
Add a method to sort a list of CVE IDs (i.e. CVE ID strings, not CVE objects!), and use that when emiting the HTML output.
The JSON output need not be sorted, because it is supposed to be used for post-processing, and we do not care about the ordering there; a consumer interested in sorting should sort on their side.
--- support/scripts/cve.py | 7 +++++++ support/scripts/pkg-stats | 4 ++-- 2 files changed, 9 insertions(+), 2 deletions(-)
support/scripts/cve.py +7 -0
@@ -117,6 +117,13 @@ class CVE: open(path_metaf, "w").write(page_meta.text) return path_jsonf_xz + @staticmethod + def sort_id(cve_ids): + def cve_key(cve_id): + year, id_ = cve_id.split('-')[1:] + return (int(year), int(id_)) + return sorted(cve_ids, key=cve_key) + @classmethod def read_nvd_dir(cls, nvd_dir): """
support/scripts/pkg-stats +2 -2
@@ -1051,9 +1051,9 @@ def dump_html_pkg(f, pkg): f.write(f' <div onclick="expandField(\'{data_field_id}\')" \ class="see-more centered cve_ignored">see all ({cve_total}) ▾</div>\n') if pkg.is_status_error("cve"): - for cve in pkg.cves: + for cve in cvecheck.CVE.sort_id(pkg.cves): f.write(f' <a href="https://security-tracker.debian.org/tracker/{cve}">{cve}</a><br/>\n') - for cve in pkg.unsure_cves: + for cve in cvecheck.CVE.sort_id(pkg.unsure_cves): f.write(f' <a href="https://security-tracker.debian.org/tracker/{cve}">{cve} <i>(unsure)</i></a><br/>\n') elif pkg.is_status_na("cve"): f.write(f""" {pkg.status['cve'][1]}""")
Signature · 6 lines
2.43.2 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 6 lines · Yann E. MORIN
Hello All!
Since we switched to the FKIE CVE feed (as the NVD JSON feed are oing extinct soon-ish), we missed the fact that the new feeds did not have the same layout as the former ones, breaking CVE reporting.
Quoted 8 lines
As a consequence of having no CVE reported, w emissed the fact that the CVEs were not sorted in the new feeds, while they were in the formers.
This series fixes both issus, and introduces a minor fixup for a python deprecation warning.
Regards,
Yann E. MORIN.
Series applied to master, thanks!
When I was working on it, I thought something needed to change in parse_node
itself (because that was done in the other NVDv2 series), but apparently not.
That said, we currently do make some sweeping assumptions about how a
configuration is specified....
Regards,
Arnout
Quoted 12 lines
----------------------------------------------------------------
Yann E. MORIN (3):
support/scripts/pkg-stats: fix datetime deprecation warning support/scripts/pkg-stats: fkie json format support/scripts/pkg-stats: sort CVEs in HTML outputsupport/scripts/cve.py | 26 +++++++++++++++++--------- support/scripts/pkg-stats | 6 +++--- 2 files changed, 20 insertions(+), 12 deletions(-)
Quoted 7 lines
--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot