cve-check: encode affected product/vendor in CVE_STATUS
PATCH 1/34 cve-check: encode affected product/vendor in CVE_STATUS
From: Marta Rybczynska <[email protected]>
CVE_STATUS contains assesment of a given CVE, but until now it didn't have include the affected vendor/product. In the case of a global system include, that CVE_STATUS was visible in all recipes.
This patch allows encoding of affected product/vendor to each CVE_STATUS assessment, also for groups. We can then filter them later and use only CVEs that correspond to the recipe.
This is going to be used in meta/conf/distro/include/cve-extra-exclusions.inc and similar places.
Backport Changes:
- Discarded the changes to meta/lib/oe/spdx30_tasks.py, as the commit history for this file diverges from the base commit itself (9c9b9545049a in the scarthgap branch).
- Additionally, the changes do not introduce any major features and are primarily focused on code restructuring.
--- meta/classes/cve-check.bbclass | 24 ++++++++++----------- meta/lib/oe/cve_check.py | 39 ++++++++++++++++++++++++++-------- 2 files changed, 42 insertions(+), 21 deletions(-)
meta/classes/cve-check.bbclass +12 -12
@@ -316,8 +316,8 @@ def check_cves(d, patched_cves): # Convert CVE_STATUS into ignored CVEs and check validity cve_ignore = [] for cve in (d.getVarFlags("CVE_STATUS") or {}): - decoded_status, _, _ = decode_cve_status(d, cve) - if decoded_status == "Ignored": + decoded_status = decode_cve_status(d, cve) + if 'mapping' in decoded_status and decoded_status['mapping'] == "Ignored": cve_ignore.append(cve) import sqlite3
@@ -500,11 +500,11 @@ def cve_write_data_text(d, patched, unpatched, ignored, cve_data): write_string += "PACKAGE VERSION: %s%s\n" % (d.getVar("EXTENDPE"), d.getVar("PV")) write_string += "CVE: %s\n" % cve write_string += "CVE STATUS: %s\n" % status - _, detail, description = decode_cve_status(d, cve) - if detail: - write_string += "CVE DETAIL: %s\n" % detail - if description: - write_string += "CVE DESCRIPTION: %s\n" % description + status_details = decode_cve_status(d, cve) + if 'detail' in status_details: + write_string += "CVE DETAIL: %s\n" % status_details['detail'] + if 'description' in status_details: + write_string += "CVE DESCRIPTION: %s\n" % status_details['description'] write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"] write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"]
@@ -632,11 +632,11 @@ def cve_write_data_json(d, patched, unpatched, ignored, cve_data, cve_status): "status" : status, "link": issue_link } - _, detail, description = decode_cve_status(d, cve) - if detail: - cve_item["detail"] = detail - if description: - cve_item["description"] = description + status_details = decode_cve_status(d, cve) + if 'detail' in status_details: + cve_item["detail"] = status_details['detail'] + if 'description' in status_details: + cve_item["description"] = status_details['description'] cve_list.append(cve_item) package_data["issue"] = cve_list
meta/lib/oe/cve_check.py +30 -9
@@ -132,8 +132,8 @@ def get_patched_cves(d): # Search for additional patched CVEs for cve in (d.getVarFlags("CVE_STATUS") or {}): - decoded_status, _, _ = decode_cve_status(d, cve) - if decoded_status == "Patched": + decoded_status = decode_cve_status(d, cve) + if 'mapping' in decoded_status and decoded_status['mapping'] == "Patched": bb.debug(2, "CVE %s is additionally patched" % cve) patched_cves.add(cve)
@@ -227,22 +227,43 @@ def convert_cve_version(version): def decode_cve_status(d, cve): """ - Convert CVE_STATUS into status, detail and description. + Convert CVE_STATUS into status, vendor, product, detail and description. """ status = d.getVarFlag("CVE_STATUS", cve) if not status: - return ("", "", "") + return {} + + status_split = status.split(':', 5) + status_out = {} + status_out["detail"] = status_split[0] + product = "*" + vendor = "*" + description = "" + if len(status_split) >= 4 and status_split[1].strip() == "cpe": + # Both vendor and product are mandatory if cpe: present, the syntax is then: + # detail: cpe:vendor:product:description + vendor = status_split[2].strip() + product = status_split[3].strip() + description = status_split[4].strip() + elif len(status_split) >= 2 and status_split[1].strip() == "cpe": + # Malformed CPE + bb.warn('Invalid CPE information for CVE_STATUS[%s] = "%s", not setting CPE' % (detail, cve, status)) + else: + # Other case: no CPE, the syntax is then: + # detail: description + description = status_split[len(status_split)-1].strip() if (len(status_split) > 1) else "" - status_split = status.split(':', 1) - detail = status_split[0] - description = status_split[1].strip() if (len(status_split) > 1) else "" + status_out["vendor"] = vendor + status_out["product"] = product + status_out["description"] = description - status_mapping = d.getVarFlag("CVE_CHECK_STATUSMAP", detail) + status_mapping = d.getVarFlag("CVE_CHECK_STATUSMAP", status_out['detail']) if status_mapping is None: bb.warn('Invalid detail "%s" for CVE_STATUS[%s] = "%s", fallback to Unpatched' % (detail, cve, status)) status_mapping = "Unpatched" + status_out["mapping"] = status_mapping - return (status_mapping, detail, description) + return status_out def extend_cve_status(d): # do this only once in case multiple classes use this
PATCH 2/34 cve_check: Update selftest with new status detail
From: Samantha Jalabert <[email protected]>
--- meta/lib/oeqa/selftest/cases/cve_check.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/cve_check.py +2 -1
@@ -217,9 +217,10 @@ CVE_CHECK_REPORT_PATCHED = "1" # m4 CVE should not be in logrotate self.assertNotIn("CVE-2008-1687", found_cves) # logrotate has both Patched and Ignored CVEs + detail = "version-not-in-range" self.assertIn("CVE-2011-1098", found_cves) self.assertEqual(found_cves["CVE-2011-1098"]["status"], "Patched") - self.assertEqual(len(found_cves["CVE-2011-1098"]["detail"]), 0) + self.assertEqual(found_cves["CVE-2011-1098"]["detail"], detail) self.assertEqual(len(found_cves["CVE-2011-1098"]["description"]), 0) detail = "not-applicable-platform" description = "CVE is debian, gentoo or SUSE specific on the way logrotate was installed/used"
PATCH 3/34 cve-check: annotate CVEs during analysis
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Add status information for each CVE under analysis.
Previously the information passed between different function of the cve-check class included only tables of patched, unpatched, ignored vulnerabilities and the general status of the recipe.
The VEX work requires more information, and we need to pass them between different functions, so that it can be enriched as the analysis progresses. Instead of multiple tables, use a single one with annotations for each CVE encountered. For example, a patched CVE will have:
{"abbrev-status": "Patched", "status": "version-not-in-range"}
abbrev-status contains the general status (Patched, Unpatched, Ignored and Unknown that will be added in the VEX code) status contains more detailed information that can come from CVE_STATUS and the analysis.
Additional fields of the annotation include for example the name of the patch file fixing a given CVE.
We also use the annotation in CVE_STATUS to filter out entries that do not apply to the given recipe
Backport Changes:
- Cherry-picking this patch, which precedes commit [358dbfcd80ae] in master. Since commit [358dbfcd80ae] was already cherry-picked earlier in scarthgap, adjusted the changes accordingly to avoid conflicts.
--- meta/classes/cve-check.bbclass | 214 +++++++++++++++++---------------- meta/lib/oe/cve_check.py | 35 +++++- 2 files changed, 142 insertions(+), 107 deletions(-)
meta/classes/cve-check.bbclass +113 -101
@@ -176,10 +176,10 @@ python do_cve_check () { patched_cves = get_patched_cves(d) except FileNotFoundError: bb.fatal("Failure in searching patches") - ignored, patched, unpatched, status = check_cves(d, patched_cves) - if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status): - cve_data = get_cve_info(d, patched + unpatched + ignored) - cve_write_data(d, patched, unpatched, ignored, cve_data, status) + cve_data, status = check_cves(d, patched_cves) + if len(cve_data) or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status): + get_cve_info(d, cve_data) + cve_write_data(d, cve_data, status) else: bb.note("No CVE database found, skipping CVE check")
@@ -287,7 +287,51 @@ ROOTFS_POSTPROCESS_COMMAND:prepend = "${@'cve_check_write_rootfs_manifest ' if d do_rootfs[recrdeptask] += "${@'do_cve_check' if d.getVar('CVE_CHECK_CREATE_MANIFEST') == '1' else ''}" do_populate_sdk[recrdeptask] += "${@'do_cve_check' if d.getVar('CVE_CHECK_CREATE_MANIFEST') == '1' else ''}" -def check_cves(d, patched_cves): +def cve_is_ignored(d, cve_data, cve): + if cve not in cve_data: + return False + if cve_data[cve]['abbrev-status'] == "Ignored": + return True + return False + +def cve_is_patched(d, cve_data, cve): + if cve not in cve_data: + return False + if cve_data[cve]['abbrev-status'] == "Patched": + return True + return False + +def cve_update(d, cve_data, cve, entry): + # If no entry, just add it + if cve not in cve_data: + cve_data[cve] = entry + return + # If we are updating, there might be change in the status + bb.debug("Trying CVE entry update for %s from %s to %s" % (cve, cve_data[cve]['abbrev-status'], entry['abbrev-status'])) + if cve_data[cve]['abbrev-status'] == "Unknown": + cve_data[cve] = entry + return + if cve_data[cve]['abbrev-status'] == entry['abbrev-status']: + return + # Update like in {'abbrev-status': 'Patched', 'status': 'version-not-in-range'} to {'abbrev-status': 'Unpatched', 'status': 'version-in-range'} + if entry['abbrev-status'] == "Unpatched" and cve_data[cve]['abbrev-status'] == "Patched": + if entry['status'] == "version-in-range" and cve_data[cve]['status'] == "version-not-in-range": + # New result from the scan, vulnerable + cve_data[cve] = entry + bb.debug("CVE entry %s update from Patched to Unpatched from the scan result" % cve) + return + if entry['abbrev-status'] == "Patched" and cve_data[cve]['abbrev-status'] == "Unpatched": + if entry['status'] == "version-not-in-range" and cve_data[cve]['status'] == "version-in-range": + # Range does not match the scan, but we already have a vulnerable match, ignore + bb.debug("CVE entry %s update from Patched to Unpatched from the scan result - not applying" % cve) + return + # If we have an "Ignored", it has a priority + if cve_data[cve]['abbrev-status'] == "Ignored": + bb.debug("CVE %s not updating because Ignored" % cve) + return + bb.warn("Unhandled CVE entry update for %s from %s to %s" % (cve, cve_data[cve], entry)) + +def check_cves(d, cve_data): """ Connect to the NVD database and find unpatched cves. """
@@ -297,28 +341,19 @@ def check_cves(d, patched_cves): real_pv = d.getVar("PV") suffix = d.getVar("CVE_VERSION_SUFFIX") - cves_unpatched = [] - cves_ignored = [] cves_status = [] cves_in_recipe = False # CVE_PRODUCT can contain more than one product (eg. curl/libcurl) products = d.getVar("CVE_PRODUCT").split() # If this has been unset then we're not scanning for CVEs here (for example, image recipes) if not products: - return ([], [], [], []) + return ([], []) pv = d.getVar("CVE_VERSION").split("+git")[0] # If the recipe has been skipped/ignored we return empty lists if pn in d.getVar("CVE_CHECK_SKIP_RECIPE").split(): bb.note("Recipe has been skipped by cve-check") - return ([], [], [], []) - - # Convert CVE_STATUS into ignored CVEs and check validity - cve_ignore = [] - for cve in (d.getVarFlags("CVE_STATUS") or {}): - decoded_status = decode_cve_status(d, cve) - if 'mapping' in decoded_status and decoded_status['mapping'] == "Ignored": - cve_ignore.append(cve) + return ([], []) import sqlite3 db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro")
@@ -337,11 +372,10 @@ def check_cves(d, patched_cves): for cverow in cve_cursor: cve = cverow[0] - if cve in cve_ignore: + if cve_is_ignored(d, cve_data, cve): bb.note("%s-%s ignores %s" % (product, pv, cve)) - cves_ignored.append(cve) continue - elif cve in patched_cves: + elif cve_is_patched(d, cve_data, cve): bb.note("%s has been patched" % (cve)) continue # Write status once only for each product
@@ -357,7 +391,7 @@ def check_cves(d, patched_cves): for row in product_cursor: (_, _, _, version_start, operator_start, version_end, operator_end) = row #bb.debug(2, "Evaluating row " + str(row)) - if cve in cve_ignore: + if cve_is_ignored(d, cve_data, cve): ignored = True version_start = convert_cve_version(version_start)
@@ -396,16 +430,16 @@ def check_cves(d, patched_cves): if vulnerable: if ignored: bb.note("%s is ignored in %s-%s" % (cve, pn, real_pv)) - cves_ignored.append(cve) + cve_update(d, cve_data, cve, {"abbrev-status": "Ignored"}) else: bb.note("%s-%s is vulnerable to %s" % (pn, real_pv, cve)) - cves_unpatched.append(cve) + cve_update(d, cve_data, cve, {"abbrev-status": "Unpatched", "status": "version-in-range"}) break product_cursor.close() if not vulnerable: bb.note("%s-%s is not vulnerable to %s" % (pn, real_pv, cve)) - patched_cves.add(cve) + cve_update(d, cve_data, cve, {"abbrev-status": "Patched", "status": "version-not-in-range"}) cve_cursor.close() if not cves_in_product:
@@ -413,49 +447,46 @@ def check_cves(d, patched_cves): cves_status.append([product, False]) conn.close() - diff_ignore = list(set(cve_ignore) - set(cves_ignored)) - if diff_ignore: - oe.qa.handle_error("cve_status_not_in_db", "Found CVE (%s) with CVE_STATUS set that are not found in database for this component" % " ".join(diff_ignore), d) if not cves_in_recipe: bb.note("No CVE records for products in recipe %s" % (pn)) - return (list(cves_ignored), list(patched_cves), cves_unpatched, cves_status) + return (cve_data, cves_status) -def get_cve_info(d, cves): +def get_cve_info(d, cve_data): """ Get CVE information from the database. """ import sqlite3 - cve_data = {} db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro") conn = sqlite3.connect(db_file, uri=True) - for cve in cves: + for cve in cve_data: cursor = conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)) for row in cursor: - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["scorev4"] = row[4] - cve_data[row[0]]["modified"] = row[5] - cve_data[row[0]]["vector"] = row[6] - cve_data[row[0]]["vectorString"] = row[7] + # The CVE itdelf has been added already + if row[0] not in cve_data: + bb.note("CVE record %s not present" % row[0]) + continue + #cve_data[row[0]] = {} + cve_data[row[0]]["NVD-summary"] = row[1] + cve_data[row[0]]["NVD-scorev2"] = row[2] + cve_data[row[0]]["NVD-scorev3"] = row[3] + cve_data[row[0]]["NVD-scorev4"] = row[4] + cve_data[row[0]]["NVD-modified"] = row[5] + cve_data[row[0]]["NVD-vector"] = row[6] + cve_data[row[0]]["NVD-vectorString"] = row[7] cursor.close() conn.close() - return cve_data -def cve_write_data_text(d, patched, unpatched, ignored, cve_data): +def cve_write_data_text(d, cve_data): """ Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and CVE manifest if enabled. """ - from oe.cve_check import decode_cve_status - cve_file = d.getVar("CVE_CHECK_LOG") fdir_name = d.getVar("FILE_DIRNAME") layer = fdir_name.split("/")[-3]
@@ -472,7 +503,7 @@ def cve_write_data_text(d, patched, unpatched, ignored, cve_data): return # Early exit, the text format does not report packages without CVEs - if not patched+unpatched+ignored: + if not len(cve_data): return nvd_link = "https://nvd.nist.gov/vuln/detail/"
@@ -481,37 +512,30 @@ def cve_write_data_text(d, patched, unpatched, ignored, cve_data): bb.utils.mkdirhier(os.path.dirname(cve_file)) for cve in sorted(cve_data): - is_patched = cve in patched - is_ignored = cve in ignored - - status = "Unpatched" - if (is_patched or is_ignored) and not report_all: + if not report_all and (cve_data[cve]["abbrev-status"] == "Patched" or cve_data[cve]["abbrev-status"] == "Ignored"): continue - if is_ignored: - status = "Ignored" - elif is_patched: - status = "Patched" - else: - # default value of status is Unpatched - unpatched_cves.append(cve) - write_string += "LAYER: %s\n" % layer write_string += "PACKAGE NAME: %s\n" % d.getVar("PN") write_string += "PACKAGE VERSION: %s%s\n" % (d.getVar("EXTENDPE"), d.getVar("PV")) write_string += "CVE: %s\n" % cve - write_string += "CVE STATUS: %s\n" % status - status_details = decode_cve_status(d, cve) - if 'detail' in status_details: - write_string += "CVE DETAIL: %s\n" % status_details['detail'] - if 'description' in status_details: - write_string += "CVE DESCRIPTION: %s\n" % status_details['description'] - write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] - write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"] - write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"] - write_string += "CVSS v4 BASE SCORE: %s\n" % cve_data[cve]["scorev4"] - write_string += "VECTOR: %s\n" % cve_data[cve]["vector"] - write_string += "VECTORSTRING: %s\n" % cve_data[cve]["vectorString"] + write_string += "CVE STATUS: %s\n" % cve_data[cve]["abbrev-status"] + + if 'status' in cve_data[cve]: + write_string += "CVE DETAIL: %s\n" % cve_data[cve]["status"] + if 'justification' in cve_data[cve]: + write_string += "CVE DESCRIPTION: %s\n" % cve_data[cve]["justification"] + + if "NVD-summary" in cve_data[cve]: + write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["NVD-summary"] + write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["NVD-scorev2"] + write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["NVD-scorev3"] + write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["NVD-scorev4"] + write_string += "VECTOR: %s\n" % cve_data[cve]["NVD-vector"] + write_string += "VECTORSTRING: %s\n" % cve_data[cve]["NVD-vectorString"] + write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve) + if cve_data[cve]["abbrev-status"] == "Unpatched": + unpatched_cves.append(cve) if unpatched_cves and d.getVar("CVE_CHECK_SHOW_WARNINGS") == "1": bb.warn("Found unpatched CVE (%s), for more information check %s" % (" ".join(unpatched_cves),cve_file))
@@ -563,13 +587,11 @@ def cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_fi with open(index_path, "a+") as f: f.write("%s\n" % fragment_path) -def cve_write_data_json(d, patched, unpatched, ignored, cve_data, cve_status): +def cve_write_data_json(d, cve_data, cve_status): """ Prepare CVE data for the JSON format, then write it. """ - from oe.cve_check import decode_cve_status - output = {"version":"1", "package": []} nvd_link = "https://nvd.nist.gov/vuln/detail/"
@@ -587,8 +609,6 @@ def cve_write_data_json(d, patched, unpatched, ignored, cve_data, cve_status): if include_layers and layer not in include_layers: return - unpatched_cves = [] - product_data = [] for s in cve_status: p = {"product": s[0], "cvesInRecord": "Yes"}
@@ -603,40 +623,32 @@ def cve_write_data_json(d, patched, unpatched, ignored, cve_data, cve_status): "version" : package_version, "products": product_data } + cve_list = [] for cve in sorted(cve_data): - is_patched = cve in patched - is_ignored = cve in ignored - status = "Unpatched" - if (is_patched or is_ignored) and not report_all: + if not report_all and (cve_data[cve]["abbrev-status"] == "Patched" or cve_data[cve]["abbrev-status"] == "Ignored"): continue - if is_ignored: - status = "Ignored" - elif is_patched: - status = "Patched" - else: - # default value of status is Unpatched - unpatched_cves.append(cve) - issue_link = "%s%s" % (nvd_link, cve) cve_item = { "id" : cve, - "summary" : cve_data[cve]["summary"], - "scorev2" : cve_data[cve]["scorev2"], - "scorev3" : cve_data[cve]["scorev3"], - "scorev4" : cve_data[cve]["scorev4"], - "vector" : cve_data[cve]["vector"], - "vectorString" : cve_data[cve]["vectorString"], - "status" : status, - "link": issue_link + "status" : cve_data[cve]["abbrev-status"], + "link": issue_link, } - status_details = decode_cve_status(d, cve) - if 'detail' in status_details: - cve_item["detail"] = status_details['detail'] - if 'description' in status_details: - cve_item["description"] = status_details['description'] + if 'NVD-summary' in cve_data[cve]: + cve_item["summary"] = cve_data[cve]["NVD-summary"] + cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"] + cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"] + cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"] + cve_item["vector"] = cve_data[cve]["NVD-vector"] + cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"] + if 'status' in cve_data[cve]: + cve_item["detail"] = cve_data[cve]["status"] + if 'justification' in cve_data[cve]: + cve_item["description"] = cve_data[cve]["justification"] + if 'resource' in cve_data[cve]: + cve_item["patch-file"] = cve_data[cve]["resource"] cve_list.append(cve_item) package_data["issue"] = cve_list
@@ -648,12 +660,12 @@ def cve_write_data_json(d, patched, unpatched, ignored, cve_data, cve_status): cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file) -def cve_write_data(d, patched, unpatched, ignored, cve_data, status): +def cve_write_data(d, cve_data, status): """ Write CVE data in each enabled format. """ if d.getVar("CVE_CHECK_FORMAT_TEXT") == "1": - cve_write_data_text(d, patched, unpatched, ignored, cve_data) + cve_write_data_text(d, cve_data) if d.getVar("CVE_CHECK_FORMAT_JSON") == "1": - cve_write_data_json(d, patched, unpatched, ignored, cve_data, status) + cve_write_data_json(d, cve_data, status)
meta/lib/oe/cve_check.py +29 -6
@@ -88,7 +88,7 @@ def get_patched_cves(d): # (cve_match regular expression) cve_file_name_match = re.compile(r".*(CVE-\d{4}-\d+)", re.IGNORECASE) - patched_cves = set() + patched_cves = {} patches = oe.patch.src_patches(d) bb.debug(2, "Scanning %d patches for CVEs" % len(patches)) for url in patches:
@@ -98,7 +98,7 @@ def get_patched_cves(d): fname_match = cve_file_name_match.search(patch_file) if fname_match: cve = fname_match.group(1).upper() - patched_cves.add(cve) + patched_cves[cve] = {"abbrev-status": "Patched", "status": "fix-file-included", "resource": patch_file} bb.debug(2, "Found %s from patch file name %s" % (cve, patch_file)) # Remote patches won't be present and compressed patches won't be
@@ -124,7 +124,7 @@ def get_patched_cves(d): cves = patch_text[match.start()+5:match.end()] for cve in cves.split(): bb.debug(2, "Patch %s solves %s" % (patch_file, cve)) - patched_cves.add(cve) + patched_cves[cve] = {"abbrev-status": "Patched", "status": "fix-file-included", "resource": patch_file} text_match = True if not fname_match and not text_match:
@@ -133,9 +133,15 @@ def get_patched_cves(d): # Search for additional patched CVEs for cve in (d.getVarFlags("CVE_STATUS") or {}): decoded_status = decode_cve_status(d, cve) - if 'mapping' in decoded_status and decoded_status['mapping'] == "Patched": - bb.debug(2, "CVE %s is additionally patched" % cve) - patched_cves.add(cve) + products = d.getVar("CVE_PRODUCT") + if has_cve_product_match(decoded_status, products) == True: + patched_cves[cve] = { + "abbrev-status": decoded_status["mapping"], + "status": decoded_status["detail"], + "justification": decoded_status["description"], + "affected-vendor": decoded_status["vendor"], + "affected-product": decoded_status["product"] + } return patched_cves
@@ -286,3 +292,20 @@ def extend_cve_status(d): d.setVarFlag("CVE_STATUS", cve, d.getVarFlag(cve_status_group, "status")) else: bb.warn("CVE_STATUS_GROUPS contains undefined variable %s" % cve_status_group) + +def has_cve_product_match(detailed_status, products): + """ + Check product/vendor match between detailed_status from decode_cve_status and a string of + products (like from CVE_PRODUCT) + """ + for product in products.split(): + vendor = "*" + if ":" in product: + vendor, product = product.split(":", 1) + + if (vendor == detailed_status["vendor"] or detailed_status["vendor"] == "*") and \ + (product == detailed_status["product"] or detailed_status["product"] == "*"): + return True + + #if no match, return False + return False
PATCH 4/34 cve-check-map: add new statuses
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Add 'fix-file-included', 'version-not-in-range' and 'version-in-range' generated by the cve-check.
'fix-file-included' means that a fix file for the CVE has been located.
'version-not-in-range' means that the product version has been found outside of the vulnerable range.
'version-in-range' means that the product version has been found inside of the vulnerable range.
--- meta/conf/cve-check-map.conf | 9 +++++++++ 1 file changed, 9 insertions(+)
meta/conf/cve-check-map.conf +9 -0
@@ -8,11 +8,17 @@ CVE_CHECK_STATUSMAP[backported-patch] = "Patched" CVE_CHECK_STATUSMAP[cpe-stable-backport] = "Patched" # use when NVD DB does not mention correct version or does not mention any verion at all CVE_CHECK_STATUSMAP[fixed-version] = "Patched" +# use when a fix file has been included (set automatically) +CVE_CHECK_STATUSMAP[fix-file-included] = "Patched" +# do not use directly: automatic scan reports version number NOT in the vulnerable range (set automatically) +CVE_CHECK_STATUSMAP[version-not-in-range] = "Patched" # used internally by this class if CVE vulnerability is detected which is not marked as fixed or ignored CVE_CHECK_STATUSMAP[unpatched] = "Unpatched" # use when CVE is confirmed by upstream but fix is still not available CVE_CHECK_STATUSMAP[vulnerable-investigating] = "Unpatched" +# do not use directly: automatic scan reports version number IS in the vulnerable range (set automatically) +CVE_CHECK_STATUSMAP[version-in-range] = "Unpatched" # used for migration from old concept, do not use for new vulnerabilities CVE_CHECK_STATUSMAP[ignored] = "Ignored"
@@ -26,3 +32,6 @@ CVE_CHECK_STATUSMAP[not-applicable-config] = "Ignored" CVE_CHECK_STATUSMAP[not-applicable-platform] = "Ignored" # use when upstream acknowledged the vulnerability but does not plan to fix it CVE_CHECK_STATUSMAP[upstream-wontfix] = "Ignored" + +# use when it is impossible to conclude if the vulnerability is present or not +CVE_CHECK_STATUSMAP[unknown] = "Unknown"
PATCH 5/34 selftest: add test_product_match
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
CVECheck.test_product_match tests has_cve_product_match()
--- meta/lib/oeqa/selftest/cases/cve_check.py | 48 +++++++++++++++++++++++ 1 file changed, 48 insertions(+)
meta/lib/oeqa/selftest/cases/cve_check.py +48 -0
@@ -72,6 +72,54 @@ class CVECheck(OESelftestTestCase): self.assertEqual(convert_cve_version("6.2_rc8"), "6.2-rc8") self.assertEqual(convert_cve_version("6.2_rc31"), "6.2-rc31") + def test_product_match(self): + from oe.cve_check import has_cve_product_match + + status = {} + status["detail"] = "ignored" + status["vendor"] = "*" + status["product"] = "*" + status["description"] = "" + status["mapping"] = "" + + self.assertEqual(has_cve_product_match(status, "some_vendor:some_product"), True) + self.assertEqual(has_cve_product_match(status, "*:*"), True) + self.assertEqual(has_cve_product_match(status, "some_product"), True) + self.assertEqual(has_cve_product_match(status, "glibc"), True) + self.assertEqual(has_cve_product_match(status, "glibca"), True) + self.assertEqual(has_cve_product_match(status, "aglibc"), True) + self.assertEqual(has_cve_product_match(status, "*"), True) + self.assertEqual(has_cve_product_match(status, "aglibc glibc test:test"), True) + + status["product"] = "glibc" + self.assertEqual(has_cve_product_match(status, "some_vendor:some_product"), False) + # The CPE in the recipe must be defined, no * accepted + self.assertEqual(has_cve_product_match(status, "*:*"), False) + self.assertEqual(has_cve_product_match(status, "*"), False) + self.assertEqual(has_cve_product_match(status, "some_product"), False) + self.assertEqual(has_cve_product_match(status, "glibc"), True) + self.assertEqual(has_cve_product_match(status, "glibca"), False) + self.assertEqual(has_cve_product_match(status, "aglibc"), False) + self.assertEqual(has_cve_product_match(status, "some_vendor:glibc"), True) + self.assertEqual(has_cve_product_match(status, "some_vendor:glibc test"), True) + self.assertEqual(has_cve_product_match(status, "test some_vendor:glibc"), True) + + status["vendor"] = "glibca" + status["product"] = "glibc" + self.assertEqual(has_cve_product_match(status, "some_vendor:some_product"), False) + # The CPE in the recipe must be defined, no * accepted + self.assertEqual(has_cve_product_match(status, "*:*"), False) + self.assertEqual(has_cve_product_match(status, "*"), False) + self.assertEqual(has_cve_product_match(status, "some_product"), False) + self.assertEqual(has_cve_product_match(status, "glibc"), False) + self.assertEqual(has_cve_product_match(status, "glibca"), False) + self.assertEqual(has_cve_product_match(status, "aglibc"), False) + self.assertEqual(has_cve_product_match(status, "some_vendor:glibc"), False) + self.assertEqual(has_cve_product_match(status, "glibca:glibc"), True) + self.assertEqual(has_cve_product_match(status, "test:test glibca:glibc"), True) + self.assertEqual(has_cve_product_match(status, "test glibca:glibc"), True) + self.assertEqual(has_cve_product_match(status, "glibca:glibc test"), True) + def test_recipe_report_json(self): config = """
PATCH 6/34 cve-check: remove the TEXT format support
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Remove the TEXT format support, as the JSON format offers more functions. Users who do automation should have migrated already.
Support of both formats makes the code more complex than necessary.
Users can convert JSON files to TEXT files with cve-json-to-text.py in scripts/
Backport Changes:
- The changes from commit [81e702c85c62] have been discarded.
--- meta/classes/cve-check.bbclass | 118 +-------------------------------- 1 file changed, 1 insertion(+), 117 deletions(-)
meta/classes/cve-check.bbclass +1 -117
@@ -36,20 +36,15 @@ CVE_CHECK_DB_DIR ?= "${STAGING_DIR}/CVE_CHECK" CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_FILE_LOCK ?= "${CVE_CHECK_DB_FILE}.lock" -CVE_CHECK_LOG ?= "${T}/cve.log" -CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check" CVE_CHECK_SUMMARY_DIR ?= "${LOG_DIR}/cve" CVE_CHECK_SUMMARY_FILE_NAME ?= "cve-summary" -CVE_CHECK_SUMMARY_FILE ?= "${CVE_CHECK_SUMMARY_DIR}/${CVE_CHECK_SUMMARY_FILE_NAME}" CVE_CHECK_SUMMARY_FILE_NAME_JSON = "cve-summary.json" CVE_CHECK_SUMMARY_INDEX_PATH = "${CVE_CHECK_SUMMARY_DIR}/cve-summary-index.txt" CVE_CHECK_LOG_JSON ?= "${T}/cve.json" CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve" -CVE_CHECK_RECIPE_FILE ?= "${CVE_CHECK_DIR}/${PN}" CVE_CHECK_RECIPE_FILE_JSON ?= "${CVE_CHECK_DIR}/${PN}_cve.json" -CVE_CHECK_MANIFEST ?= "${IMGDEPLOYDIR}/${IMAGE_NAME}.cve" CVE_CHECK_MANIFEST_JSON_SUFFIX ?= "json" CVE_CHECK_MANIFEST_JSON ?= "${IMGDEPLOYDIR}/${IMAGE_NAME}.${CVE_CHECK_MANIFEST_JSON_SUFFIX}" CVE_CHECK_COPY_FILES ??= "1"
@@ -60,9 +55,6 @@ CVE_CHECK_REPORT_PATCHED ??= "1" CVE_CHECK_SHOW_WARNINGS ??= "1" -# Provide text output -CVE_CHECK_FORMAT_TEXT ??= "1" - # Provide JSON output CVE_CHECK_FORMAT_JSON ??= "1"
@@ -139,20 +131,11 @@ python cve_save_summary_handler () { import datetime from oe.cve_check import update_symlinks - cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE") - cve_summary_name = d.getVar("CVE_CHECK_SUMMARY_FILE_NAME") cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR") bb.utils.mkdirhier(cvelogpath) timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S') - cve_summary_file = os.path.join(cvelogpath, "%s-%s.txt" % (cve_summary_name, timestamp)) - - if os.path.exists(cve_tmp_file): - shutil.copyfile(cve_tmp_file, cve_summary_file) - cvefile_link = os.path.join(cvelogpath, cve_summary_name) - update_symlinks(cve_summary_file, cvefile_link) - bb.plain("Complete CVE report summary created at: %s" % cvefile_link) if d.getVar("CVE_CHECK_FORMAT_JSON") == "1": json_summary_link_name = os.path.join(cvelogpath, d.getVar("CVE_CHECK_SUMMARY_FILE_NAME_JSON"))
@@ -195,7 +178,6 @@ python cve_check_cleanup () { """ Delete the file used to gather all the CVE information. """ - bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE")) bb.utils.remove(e.data.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")) }
@@ -213,9 +195,6 @@ python cve_check_write_rootfs_manifest () { from oe.cve_check import cve_check_merge_jsons, update_symlinks if d.getVar("CVE_CHECK_COPY_FILES") == "1": - deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE") - if os.path.exists(deploy_file): - bb.utils.remove(deploy_file) deploy_file_json = d.getVar("CVE_CHECK_RECIPE_FILE_JSON") if os.path.exists(deploy_file_json): bb.utils.remove(deploy_file_json)
@@ -235,19 +214,13 @@ python cve_check_write_rootfs_manifest () { json_data = {"version":"1", "package": []} text_data = "" enable_json = d.getVar("CVE_CHECK_FORMAT_JSON") == "1" - enable_text = d.getVar("CVE_CHECK_FORMAT_TEXT") == "1" save_pn = d.getVar("PN") for pkg in recipies: - # To be able to use the CVE_CHECK_RECIPE_FILE variable we have to evaluate + # To be able to use the CVE_CHECK_RECIPE_FILE_JSON variable we have to evaluate # it with the different PN names set each time. d.setVar("PN", pkg) - if enable_text: - pkgfilepath = d.getVar("CVE_CHECK_RECIPE_FILE") - if os.path.exists(pkgfilepath): - with open(pkgfilepath) as pfile: - text_data += pfile.read() if enable_json: pkgfilepath = d.getVar("CVE_CHECK_RECIPE_FILE_JSON")
@@ -258,17 +231,6 @@ python cve_check_write_rootfs_manifest () { d.setVar("PN", save_pn) - if enable_text: - manifest_name = d.getVar("CVE_CHECK_MANIFEST") - - with open(manifest_name, "w") as f: - f.write(text_data) - - if link_name: - link_path = os.path.join(deploy_dir, "%s.cve" % link_name) - update_symlinks(manifest_name, link_path) - bb.plain("Image CVE report stored in: %s" % manifest_name) - if enable_json: manifest_name_suffix = d.getVar("CVE_CHECK_MANIFEST_JSON_SUFFIX") manifest_name = d.getVar("CVE_CHECK_MANIFEST_JSON")
@@ -481,82 +443,6 @@ def get_cve_info(d, cve_data): cursor.close() conn.close() -def cve_write_data_text(d, cve_data): - """ - Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and - CVE manifest if enabled. - """ - - cve_file = d.getVar("CVE_CHECK_LOG") - fdir_name = d.getVar("FILE_DIRNAME") - layer = fdir_name.split("/")[-3] - - include_layers = d.getVar("CVE_CHECK_LAYER_INCLUDELIST").split() - exclude_layers = d.getVar("CVE_CHECK_LAYER_EXCLUDELIST").split() - - report_all = d.getVar("CVE_CHECK_REPORT_PATCHED") == "1" - - if exclude_layers and layer in exclude_layers: - return - - if include_layers and layer not in include_layers: - return - - # Early exit, the text format does not report packages without CVEs - if not len(cve_data): - return - - nvd_link = "https://nvd.nist.gov/vuln/detail/" - write_string = "" - unpatched_cves = [] - bb.utils.mkdirhier(os.path.dirname(cve_file)) - - for cve in sorted(cve_data): - if not report_all and (cve_data[cve]["abbrev-status"] == "Patched" or cve_data[cve]["abbrev-status"] == "Ignored"): - continue - write_string += "LAYER: %s\n" % layer - write_string += "PACKAGE NAME: %s\n" % d.getVar("PN") - write_string += "PACKAGE VERSION: %s%s\n" % (d.getVar("EXTENDPE"), d.getVar("PV")) - write_string += "CVE: %s\n" % cve - write_string += "CVE STATUS: %s\n" % cve_data[cve]["abbrev-status"] - - if 'status' in cve_data[cve]: - write_string += "CVE DETAIL: %s\n" % cve_data[cve]["status"] - if 'justification' in cve_data[cve]: - write_string += "CVE DESCRIPTION: %s\n" % cve_data[cve]["justification"] - - if "NVD-summary" in cve_data[cve]: - write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["NVD-summary"] - write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["NVD-scorev2"] - write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["NVD-scorev3"] - write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["NVD-scorev4"] - write_string += "VECTOR: %s\n" % cve_data[cve]["NVD-vector"] - write_string += "VECTORSTRING: %s\n" % cve_data[cve]["NVD-vectorString"] - - write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve) - if cve_data[cve]["abbrev-status"] == "Unpatched": - unpatched_cves.append(cve) - - if unpatched_cves and d.getVar("CVE_CHECK_SHOW_WARNINGS") == "1": - bb.warn("Found unpatched CVE (%s), for more information check %s" % (" ".join(unpatched_cves),cve_file)) - - with open(cve_file, "w") as f: - bb.note("Writing file %s with CVE information" % cve_file) - f.write(write_string) - - if d.getVar("CVE_CHECK_COPY_FILES") == "1": - deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE") - bb.utils.mkdirhier(os.path.dirname(deploy_file)) - with open(deploy_file, "w") as f: - f.write(write_string) - - if d.getVar("CVE_CHECK_CREATE_MANIFEST") == "1": - cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR") - bb.utils.mkdirhier(cvelogpath) - - with open(d.getVar("CVE_CHECK_TMP_FILE"), "a") as f: - f.write("%s" % write_string) - def cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file): """ Write CVE information in the JSON format: to WORKDIR; and to
@@ -665,7 +551,5 @@ def cve_write_data(d, cve_data, status): Write CVE data in each enabled format. """ - if d.getVar("CVE_CHECK_FORMAT_TEXT") == "1": - cve_write_data_text(d, cve_data) if d.getVar("CVE_CHECK_FORMAT_JSON") == "1": cve_write_data_json(d, cve_data, status)
PATCH 7/34 cve-check-update-nvd2-native: Incremement DL_DIR database location
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Richard Purdie <[email protected]>
We're seeing a lot of sqlite database corruption issues in our automated testing. It is unclear why this is happening. There were process imrpovements implemented in master and it is unclear if older releases are somehow making those changes ineffective or if the problem is elsewhere.
By changing the location in DL_DIR, we split the two sets of accesses to be separate and can isolate whether the master changes really did improve things or not. If successful, we may consider backporting those changes to the stable releases.
--- meta/recipes-core/meta/cve-update-nvd2-native.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +1 -1
@@ -34,7 +34,7 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" # Number of attempts for each http query to nvd server before giving up CVE_DB_UPDATE_ATTEMPTS ?= "5" -CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}" +CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
PATCH 8/34 cve-check: add field "modified" to JSON report
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Katawann <[email protected]>
Added the "modified" field to the JSON export in the cve-check.class. This field captures the last modification date of each CVE, providing more detailed information on changes and updates within the exported data.
--- meta/classes/cve-check.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes/cve-check.bbclass +1 -0
@@ -527,6 +527,7 @@ def cve_write_data_json(d, cve_data, cve_status): cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"] cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"] cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"] + cve_item["modified"] = cve_data[cve]["NVD-modified"] cve_item["vector"] = cve_data[cve]["NVD-vector"] cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"] if 'status' in cve_data[cve]:
PATCH 9/34 cve-check: do not skip cve status description after :
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
Correct maxsplit parameter from 5 to 4 to not drop text if description contains ":".
Example:
Quoted 1 line
Quoted 1 line
Quoted 1 line
"detail: cpe:vendor:product:description:cont".split(':', 5)
['detail', ' cpe', 'vendor', 'product', 'description', 'xxx']
Quoted 1 line
Quoted 1 line
Quoted 1 line
"detail: cpe:vendor:product:description:cont".split(':', 4)
['detail', ' cpe', 'vendor', 'product', 'description:xxx']
--- meta/lib/oe/cve_check.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/cve_check.py +1 -1
@@ -239,7 +239,7 @@ def decode_cve_status(d, cve): if not status: return {} - status_split = status.split(':', 5) + status_split = status.split(':', 4) status_out = {} status_out["detail"] = status_split[0] product = "*"
PATCH 10/34 cve-check: fix malformed cve status description with : characters
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
When CPE is not provided and character ":" is in cve status description, current code takes only last part of split function.
This works only if there is no ":" in description, otherwise it drops the other split parts.
Do a new split of the original string to take the whole description unchanged. This fixes following entries from world build of poky+meta-oe+meta-python:
tiff-4.6.0-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2015-7313 CVE_STATUS: fixed-version: Tested with check from https://security-tracker.debian.org/tracker/CVE-2015-7313 and already 4.3.0 doesn't have the issue description: //security-tracker.debian.org/tracker/CVE-2015-7313 and already 4.3.0 doesn't have the issue corrected: Tested with check from https://security-tracker.debian.org/tracker/CVE-2015-7313 and already 4.3.0 doesn't have the issue
gnupg-2.5.0-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2022-3219 CVE_STATUS: upstream-wontfix: Upstream doesn't seem to be keen on merging the proposed commit - https://dev.gnupg.org/T5993 description: //dev.gnupg.org/T5993 corrected: Upstream doesn't seem to be keen on merging the proposed commit - https://dev.gnupg.org/T5993
libyaml-0.2.5-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2024-35325 CVE_STATUS: upstream-wontfix: Upstream thinks this is a misuse (or wrong use) of the libyaml API - https://github.com/yaml/libyaml/issues/303 description: //github.com/yaml/libyaml/issues/303 corrected: Upstream thinks this is a misuse (or wrong use) of the libyaml API - https://github.com/yaml/libyaml/issues/303
libyaml-0.2.5-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2024-35326 CVE_STATUS: upstream-wontfix: Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302 description: //github.com/yaml/libyaml/issues/302 corrected: Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302
libyaml-0.2.5-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2024-35328 CVE_STATUS: upstream-wontfix: Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302 description: //github.com/yaml/libyaml/issues/302 corrected: Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302
cpio-2.15-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2023-7216 CVE_STATUS: disputed: intended behaviour, see https://lists.gnu.org/archive/html/bug-cpio/2024-03/msg00000.html description: //lists.gnu.org/archive/html/bug-cpio/2024-03/msg00000.html corrected: intended behaviour, see https://lists.gnu.org/archive/html/bug-cpio/2024-03/msg00000.html
openssh-9.9p1-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2023-51767 CVE_STATUS: upstream-wontfix: It was demonstrated on modified sshd and does not exist in upstream openssh https://bugzilla.mindrot.org/show_bug.cgi?id=3656#c1. description: //bugzilla.mindrot.org/show_bug.cgi?id=3656#c1. corrected: It was demonstrated on modified sshd and does not exist in upstream openssh https://bugzilla.mindrot.org/show_bug.cgi?id=3656#c1.
cups-2.4.10-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2021-25317 CVE_STATUS: not-applicable-config: This concerns /var/log/cups having lp ownership, our /var/log/cups is root:root, so this doesn't apply. description: root, so this doesn't apply. corrected: This concerns /var/log/cups having lp ownership, our /var/log/cups is root:root, so this doesn't apply.
unzip-1_6.0-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2008-0888 CVE_STATUS: fixed-version: Patch from https://bugzilla.redhat.com/attachment.cgi?id=293893&action=diff applied to 6.0 source description: //bugzilla.redhat.com/attachment.cgi?id=293893&action=diff applied to 6.0 source corrected: Patch from https://bugzilla.redhat.com/attachment.cgi?id=293893&action=diff applied to 6.0 source
syslog-ng-4.7.0-r0 do_cve_check: CVE_STATUS with 6 parts for CVE-2022-38725 CVE_STATUS: cpe-incorrect: cve-check wrongly matches cpe:2.3:a:oneidentity:syslog-ng:*:*:*:*:premium:*:*:* < 7.0.32 description: syslog-ng:*:*:*:*:premium:*:*:* < 7.0.32 corrected: cve-check wrongly matches cpe:2.3:a:oneidentity:syslog-ng:*:*:*:*:premium:*:*:* < 7.0.32
--- meta/lib/oe/cve_check.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/cve_check.py +1 -1
@@ -257,7 +257,7 @@ def decode_cve_status(d, cve): else: # Other case: no CPE, the syntax is then: # detail: description - description = status_split[len(status_split)-1].strip() if (len(status_split) > 1) else "" + description = status.split(':', 1)[1].strip() if (len(status_split) > 1) else "" status_out["vendor"] = vendor status_out["product"] = product
PATCH 11/34 cve-check: restore CVE_CHECK_SHOW_WARNINGS functionality
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
Commit 05ef4f2a7b225c8d230eaca8d333ffb921729d79 removed this functionality by accident. It was implemented in text exporter, while it should have been a global feature independent on exporter type to avoid such accidental deletion.
--- meta/classes/cve-check.bbclass | 5 +++++ 1 file changed, 5 insertions(+)
meta/classes/cve-check.bbclass +5 -0
@@ -413,6 +413,11 @@ def check_cves(d, cve_data): if not cves_in_recipe: bb.note("No CVE records for products in recipe %s" % (pn)) + if d.getVar("CVE_CHECK_SHOW_WARNINGS") == "1": + unpatched_cves = [cve for cve in cve_data if cve_data[cve]["abbrev-status"] == "Unpatched"] + if unpatched_cves: + bb.warn("Found unpatched CVE (%s)" % " ".join(unpatched_cves)) + return (cve_data, cves_status) def get_cve_info(d, cve_data):
PATCH 12/34 cve-check: fix cvesInRecord
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
Currently flag cvesInRecord is set to false if all CVEs are ignored or patched. This is inconsistent as it shows false if a CVE was fixed via patch and true if this CVE was fixed by upgrade. In both cases the CVE is valid and was fixed.
As I understand this flag, it should say if any CVE exists for particular component's product (regardless of how this CVE is handled) and can be used to validate if a product is correctly set.
Note that skipping ignored CVEs may make sense in some cases, as ignored may mean that NVD DB is wrong, but in many cases it is ignored for other reasons. Further patch can be done to evaluate ignore subtype but that would be against my understanding of this flag as described above.
--- meta/classes/cve-check.bbclass | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
meta/classes/cve-check.bbclass +6 -5
@@ -334,17 +334,18 @@ def check_cves(d, cve_data): for cverow in cve_cursor: cve = cverow[0] + # Write status once only for each product + if not cves_in_product: + cves_status.append([product, True]) + cves_in_product = True + cves_in_recipe = True + if cve_is_ignored(d, cve_data, cve): bb.note("%s-%s ignores %s" % (product, pv, cve)) continue elif cve_is_patched(d, cve_data, cve): bb.note("%s has been patched" % (cve)) continue - # Write status once only for each product - if not cves_in_product: - cves_status.append([product, True]) - cves_in_product = True - cves_in_recipe = True vulnerable = False ignored = False
PATCH 13/34 cve-check: Fix errors in log lines
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Colin McAllister <[email protected]>
Two warning lines in cve_check.py reference a variable that doesn't exist. These would cause a runtime error if the conditions they are hidden in were to be entered. The log lines have been updated to no longer reference an undefined variable.
--- meta/lib/oe/cve_check.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
meta/lib/oe/cve_check.py +10 -3
@@ -253,7 +253,10 @@ def decode_cve_status(d, cve): description = status_split[4].strip() elif len(status_split) >= 2 and status_split[1].strip() == "cpe": # Malformed CPE - bb.warn('Invalid CPE information for CVE_STATUS[%s] = "%s", not setting CPE' % (detail, cve, status)) + bb.warn( + 'Invalid CPE information for CVE_STATUS[%s] = "%s", not setting CPE' + % (cve, status) + ) else: # Other case: no CPE, the syntax is then: # detail: description
@@ -263,9 +266,13 @@ def decode_cve_status(d, cve): status_out["product"] = product status_out["description"] = description - status_mapping = d.getVarFlag("CVE_CHECK_STATUSMAP", status_out['detail']) + detail = status_out["detail"] + status_mapping = d.getVarFlag("CVE_CHECK_STATUSMAP", detail) if status_mapping is None: - bb.warn('Invalid detail "%s" for CVE_STATUS[%s] = "%s", fallback to Unpatched' % (detail, cve, status)) + bb.warn( + 'Invalid detail "%s" for CVE_STATUS[%s] = "%s", fallback to Unpatched' + % (detail, cve, status) + ) status_mapping = "Unpatched" status_out["mapping"] = status_mapping
PATCH 14/34 cve-check: Rework patch parsing
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Colin McAllister <[email protected]>
The cve_check functionality to parse CVE IDs from the patch filename and patch contents have been reworked to improve parsing and also utilize tests. This ensures that the parsing works as intended.
Additionally, the new patched_cves dict has a few issues I tried to fix as well. If multiple patch files exist for a single CVE ID, only the last one will show up with the "resource" key. The value for the "resource" key has been updated to hold a list and return all patch files associated with a given CVE ID. Also, at the end of get_patch_cves, CVE_STATUS can overwrite an existing entry in the dict. This could cause an issue, for example, if a CVE has been addressed via a patch, but a CVE_STATUS line also exists that ignores the given CVE ID. A warning has been added if this ever happens.
--- meta/lib/oe/cve_check.py | 166 ++++++++++++------ meta/lib/oeqa/selftest/cases/cve_check.py | 205 ++++++++++++++++++++++ 2 files changed, 317 insertions(+), 54 deletions(-)
meta/lib/oe/cve_check.py +112 -54
@@ -5,9 +5,11 @@ # import collections -import re -import itertools import functools +import itertools +import os.path +import re +import oe.patch _Version = collections.namedtuple( "_Version", ["release", "patch_l", "pre_l", "pre_v"]
@@ -71,76 +73,132 @@ def _cmpkey(release, patch_l, pre_l, pre_v): return _release, _patch, _pre -def get_patched_cves(d): +def parse_cve_from_filename(patch_filename): """ - Get patches that solve CVEs using the "CVE: " tag. + Parses CVE ID from the filename + + Matches the last "CVE-YYYY-ID" in the file name, also if written + in lowercase. Possible to have multiple CVE IDs in a single + file name, but only the last one will be detected from the file name. + + Returns the last CVE ID foudn in the filename. If no CVE ID is found + an empty string is returned. """ + cve_file_name_match = re.compile(r".*(CVE-\d{4}-\d{4,})", re.IGNORECASE) - import re - import oe.patch + # Check patch file name for CVE ID + fname_match = cve_file_name_match.search(patch_filename) + return fname_match.group(1).upper() if fname_match else "" - cve_match = re.compile(r"CVE:( CVE-\d{4}-\d+)+") - # Matches the last "CVE-YYYY-ID" in the file name, also if written - # in lowercase. Possible to have multiple CVE IDs in a single - # file name, but only the last one will be detected from the file name. - # However, patch files contents addressing multiple CVE IDs are supported - # (cve_match regular expression) - cve_file_name_match = re.compile(r".*(CVE-\d{4}-\d+)", re.IGNORECASE) +def parse_cves_from_patch_contents(patch_contents): + """ + Parses CVE IDs from patch contents + Matches all CVE IDs contained on a line that starts with "CVE: ". Any + delimiter (',', '&', "and", etc.) can be used without any issues. Multiple + "CVE:" lines can also exist. + + Returns a set of all CVE IDs found in the patch contents. + """ + cve_ids = set() + cve_match = re.compile(r"CVE-\d{4}-\d{4,}") + # Search for one or more "CVE: " lines + for line in patch_contents.split("\n"): + if not line.startswith("CVE:"): + continue + cve_ids.update(cve_match.findall(line)) + return cve_ids + + +def parse_cves_from_patch_file(patch_file): + """ + Parses CVE IDs associated with a particular patch file, using both the filename + and patch contents. + + Returns a set of all CVE IDs found in the patch filename and contents. + """ + cve_ids = set() + filename_cve = parse_cve_from_filename(patch_file) + if filename_cve: + bb.debug(2, "Found %s from patch file name %s" % (filename_cve, patch_file)) + cve_ids.add(parse_cve_from_filename(patch_file)) + + # Remote patches won't be present and compressed patches won't be + # unpacked, so say we're not scanning them + if not os.path.isfile(patch_file): + bb.note("%s is remote or compressed, not scanning content" % patch_file) + return cve_ids + + with open(patch_file, "r", encoding="utf-8") as f: + try: + patch_text = f.read() + except UnicodeDecodeError: + bb.debug( + 1, + "Failed to read patch %s using UTF-8 encoding" + " trying with iso8859-1" % patch_file, + ) + f.close() + with open(patch_file, "r", encoding="iso8859-1") as f: + patch_text = f.read() + + cve_ids.update(parse_cves_from_patch_contents(patch_text)) + + if not cve_ids: + bb.debug(2, "Patch %s doesn't solve CVEs" % patch_file) + else: + bb.debug(2, "Patch %s solves %s" % (patch_file, ", ".join(sorted(cve_ids)))) + + return cve_ids + + +def get_patched_cves(d): + """ + Determines the CVE IDs that have been solved by either patches incuded within + SRC_URI or by setting CVE_STATUS. + + Returns a dictionary with the CVE IDs as keys and an associated dictonary of + relevant metadata as the value. + """ patched_cves = {} patches = oe.patch.src_patches(d) bb.debug(2, "Scanning %d patches for CVEs" % len(patches)) + + # Check each patch file for url in patches: patch_file = bb.fetch.decodeurl(url)[2] - - # Check patch file name for CVE ID - fname_match = cve_file_name_match.search(patch_file) - if fname_match: - cve = fname_match.group(1).upper() - patched_cves[cve] = {"abbrev-status": "Patched", "status": "fix-file-included", "resource": patch_file} - bb.debug(2, "Found %s from patch file name %s" % (cve, patch_file)) - - # Remote patches won't be present and compressed patches won't be - # unpacked, so say we're not scanning them - if not os.path.isfile(patch_file): - bb.note("%s is remote or compressed, not scanning content" % patch_file) - continue - - with open(patch_file, "r", encoding="utf-8") as f: - try: - patch_text = f.read() - except UnicodeDecodeError: - bb.debug(1, "Failed to read patch %s using UTF-8 encoding" - " trying with iso8859-1" % patch_file) - f.close() - with open(patch_file, "r", encoding="iso8859-1") as f: - patch_text = f.read() - - # Search for one or more "CVE: " lines - text_match = False - for match in cve_match.finditer(patch_text): - # Get only the CVEs without the "CVE: " tag - cves = patch_text[match.start()+5:match.end()] - for cve in cves.split(): - bb.debug(2, "Patch %s solves %s" % (patch_file, cve)) - patched_cves[cve] = {"abbrev-status": "Patched", "status": "fix-file-included", "resource": patch_file} - text_match = True - - if not fname_match and not text_match: - bb.debug(2, "Patch %s doesn't solve CVEs" % patch_file) + for cve_id in parse_cves_from_patch_file(patch_file): + if cve_id not in patched_cves: + { + "abbrev-status": "Patched", + "status": "fix-file-included", + "resource": [patch_file], + } + else: + patched_cves[cve_id]["resource"].append(patch_file) # Search for additional patched CVEs - for cve in (d.getVarFlags("CVE_STATUS") or {}): - decoded_status = decode_cve_status(d, cve) + for cve_id in d.getVarFlags("CVE_STATUS") or {}: + decoded_status = decode_cve_status(d, cve_id) products = d.getVar("CVE_PRODUCT") - if has_cve_product_match(decoded_status, products) == True: - patched_cves[cve] = { + if has_cve_product_match(decoded_status, products): + if cve_id in patched_cves: + bb.warn( + 'CVE_STATUS[%s] = "%s" is overwriting previous status of "%s: %s"' + % ( + cve_id, + d.getVarFlag("CVE_STATUS", cve_id), + patched_cves[cve_id]["abbrev-status"], + patched_cves[cve_id]["status"], + ) + ) + patched_cves[cve_id] = { "abbrev-status": decoded_status["mapping"], "status": decoded_status["detail"], "justification": decoded_status["description"], "affected-vendor": decoded_status["vendor"], - "affected-product": decoded_status["product"] + "affected-product": decoded_status["product"], } return patched_cves
meta/lib/oeqa/selftest/cases/cve_check.py +205 -0
211 lines not on this page · raw message
PATCH 15/34 meta/lib/oe/cve_check.py: fix patched_cves not updated
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: hongxu <[email protected]>
Due to commit [cve-check: Rework patch parsing] applied, it missed to update patched_cves dictionary if cve_id not in patched_cves
--- meta/lib/oe/cve_check.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/cve_check.py +1 -1
@@ -170,7 +170,7 @@ def get_patched_cves(d): patch_file = bb.fetch.decodeurl(url)[2] for cve_id in parse_cves_from_patch_file(patch_file): if cve_id not in patched_cves: - { + patched_cves[cve_id] = { "abbrev-status": "Patched", "status": "fix-file-included", "resource": [patch_file],
PATCH 16/34 cve-check: allow feed choice
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Allow choice of one of three feeds and update task dependencies accordingly. All feeds contain data from NVD and are stored in different files.
Set the NVD_DB_VERSION variable to choose feed:
NVD2 (default) - the NVD feed with API version 2
NVD1 - the NVD JSON feed (deprecated)
FKIE - the FKIE-CAD feed reconstruction
In case of malformed database feed name, we default to NVD2 and show an error.
--- meta/classes/cve-check.bbclass | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
meta/classes/cve-check.bbclass +12 -2
@@ -31,7 +31,12 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" -CVE_CHECK_DB_FILENAME ?= "nvdcve_2-2.db" +# Possible database sources: NVD1, NVD2, FKIE +NVD_DB_VERSION ?= "NVD2" + +# Use different file names for each database source, as they synchronize at different moments, so may be slightly different +CVE_CHECK_DB_FILENAME ?= "${@'nvdcve_2-2.db' if d.getVar('NVD_DB_VERSION') == 'NVD2' else 'nvdcve_1-3.db' if d.getVar('NVD_DB_VERSION') == 'NVD1' else 'nvdfkie_1-1.db'}" +CVE_CHECK_DB_FETCHER ?= "${@'cve-update-nvd2-native' if d.getVar('NVD_DB_VERSION') == 'NVD2' else 'cve-update-db-native'}" CVE_CHECK_DB_DIR ?= "${STAGING_DIR}/CVE_CHECK" CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_FILE_LOCK ?= "${CVE_CHECK_DB_FILE}.lock"
@@ -101,6 +106,11 @@ CVE_VERSION_SUFFIX ??= "" python () { from oe.cve_check import extend_cve_status extend_cve_status(d) + + nvd_database_type = d.getVar("NVD_DB_VERSION") + if nvd_database_type not in ("NVD1", "NVD2", "FKIE"): + bb.erroronce("Malformed NVD_DB_VERSION, must be one of: NVD1, NVD2, FKIE. Defaulting to NVD2") + d.setVar("NVD_DB_VERSION", "NVD2") } def generate_json_report(d, out_path, link_path):
@@ -171,7 +181,7 @@ python do_cve_check () { addtask cve_check before do_build do_cve_check[vardeps] += "CVE_STATUS CVE_CHECK_STATUSMAP" -do_cve_check[depends] = "cve-update-nvd2-native:do_unpack" +do_cve_check[depends] = "${CVE_CHECK_DB_FETCHER}:do_unpack" do_cve_check[nostamp] = "1" python cve_check_cleanup () {
PATCH 17/34 cve-update-db-native: restore
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Restore cve-update-db from kirkstone
Use cve-update-db-native.bb from OE 8c10f4a4dc12f65212576e6e568fa4369014aaa0
--- .../recipes-core/meta/cve-update-db-native.bb | 291 ++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb
meta/recipes-core/meta/cve-update-db-native.bb new file +291 -0
@@ -0,0 +1,291 @@ +SUMMARY = "Updates the NVD CVE database" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +inherit native + +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install +deltask do_populate_sysroot + +NVDCVE_URL ?= "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-" +# CVE database update interval, in seconds. By default: once a day (24*60*60). +# Use 0 to force the update +# Use a negative value to skip the update +CVE_DB_UPDATE_INTERVAL ?= "86400" + +# Timeout for blocking socket operations, such as the connection attempt. +CVE_SOCKET_TIMEOUT ?= "60" + +CVE_DB_TEMP_FILE ?= "${CVE_CHECK_DB_DIR}/temp_nvdcve_1.1.db" + +python () { + if not bb.data.inherits_class("cve-check", d): + raise bb.parse.SkipRecipe("Skip recipe when cve-check class is not loaded.") +} + +python do_fetch() { + """ + Update NVD database with json data feed + """ + import bb.utils + import bb.progress + import shutil + + bb.utils.export_proxies(d) + + db_file = d.getVar("CVE_CHECK_DB_FILE") + db_dir = os.path.dirname(db_file) + db_tmp_file = d.getVar("CVE_DB_TEMP_FILE") + + cleanup_db_download(db_file, db_tmp_file) + + # The NVD database changes once a day, so no need to update more frequently + # Allow the user to force-update + try: + import time + update_interval = int(d.getVar("CVE_DB_UPDATE_INTERVAL")) + if update_interval < 0: + bb.note("CVE database update skipped") + return + if time.time() - os.path.getmtime(db_file) < update_interval: + bb.debug(2, "Recently updated, skipping") + return + + except OSError: + pass + + bb.utils.mkdirhier(db_dir) + if os.path.exists(db_file): + shutil.copy2(db_file, db_tmp_file) + + if update_db_file(db_tmp_file, d) == True: + # Update downloaded correctly, can swap files + shutil.move(db_tmp_file, db_file) + else: + # Update failed, do not modify the database + bb.note("CVE database update failed") + os.remove(db_tmp_file) +} + +do_fetch[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}" +do_fetch[file-checksums] = "" +do_fetch[vardeps] = "" + +def cleanup_db_download(db_file, db_tmp_file): + """ + Cleanup the download space from possible failed downloads + """ + + # Clean up the updates done on the main file + # Remove it only if a journal file exists - it means a complete re-download + if os.path.exists("{0}-journal".format(db_file)): + # If a journal is present the last update might have been interrupted. In that case, + # just wipe any leftovers and force the DB to be recreated. + os.remove("{0}-journal".format(db_file)) + + if os.path.exists(db_file): + os.remove(db_file) + + # Clean-up the temporary file downloads, we can remove both journal + # and the temporary database + if os.path.exists("{0}-journal".format(db_tmp_file)): + # If a journal is present the last update might have been interrupted. In that case, + # just wipe any leftovers and force the DB to be recreated. + os.remove("{0}-journal".format(db_tmp_file)) + + if os.path.exists(db_tmp_file): + os.remove(db_tmp_file) + +def update_db_file(db_tmp_file, d): + """ + Update the given database file + """ + import bb.utils, bb.progress + from datetime import date + import urllib, gzip, sqlite3 + + YEAR_START = 2002 + cve_socket_timeout = int(d.getVar("CVE_SOCKET_TIMEOUT")) + + # Connect to database + conn = sqlite3.connect(db_tmp_file) + initialize_db(conn) + + with bb.progress.ProgressHandler(d) as ph, open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f: + total_years = date.today().year + 1 - YEAR_START + for i, year in enumerate(range(YEAR_START, date.today().year + 1)): + bb.debug(2, "Updating %d" % year) + ph.update((float(i + 1) / total_years) * 100) + year_url = (d.getVar('NVDCVE_URL')) + str(year) + meta_url = year_url + ".meta" + json_url = year_url + ".json.gz" + + # Retrieve meta last modified date + try: + response = urllib.request.urlopen(meta_url, timeout=cve_socket_timeout) + except urllib.error.URLError as e: + cve_f.write('Warning: CVE db update error, Unable to fetch CVE data.\n\n') + bb.warn("Failed to fetch CVE data (%s)" % e) + import socket + result = socket.getaddrinfo("nvd.nist.gov", 443, proto=socket.IPPROTO_TCP) + bb.warn("Host IPs are %s" % (", ".join(t[4][0] for t in result))) + return False + + if response: + for l in response.read().decode("utf-8").splitlines(): + key, value = l.split(":", 1) + if key == "lastModifiedDate": + last_modified = value + break + else: + bb.warn("Cannot parse CVE metadata, update failed") + return False + + # Compare with current db last modified date + cursor = conn.execute("select DATE from META where YEAR = ?", (year,)) + meta = cursor.fetchone() + cursor.close() + + if not meta or meta[0] != last_modified: + bb.debug(2, "Updating entries") + # Clear products table entries corresponding to current year + conn.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)).close() + + # Update db with current year json file + try: + response = urllib.request.urlopen(json_url, timeout=cve_socket_timeout) + if response: + update_db(conn, gzip.decompress(response.read()).decode('utf-8')) + conn.execute("insert or replace into META values (?, ?)", [year, last_modified]).close() + except urllib.error.URLError as e: + cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return False + else: + bb.debug(2, "Already up to date (last modified %s)" % last_modified) + # Update success, set the date to cve_check file. + if year == date.today().year: + cve_f.write('CVE database update : %s\n\n' % date.today()) + + conn.commit() + conn.close() + return True + +def initialize_db(conn): + with conn: + c = conn.cursor() + + c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ + SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ + VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ + VERSION_END TEXT, OPERATOR_END TEXT)") + c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_ID_IDX on PRODUCTS(ID);") + + c.close() + +def parse_node_and_insert(conn, node, cveId): + # Parse children node if needed + for child in node.get('children', ()): + parse_node_and_insert(conn, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe.get('cpe23Uri') + if not cpe23: + return + cpe23 = cpe23.split(':') + if len(cpe23) < 6: + return + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if cpe23[6] == '*' or cpe23[6] == '-': + version_suffix = "" + else: + version_suffix = "_" + cpe23[6] + + if version != '*' and version != '-': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version + version_suffix, '=', '', ''] + elif version == '-': + # no version information is available + yield [cveId, vendor, product, version, '', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: + op_start = '>=' + v_start = cpe['versionStartIncluding'] + + if 'versionStartExcluding' in cpe: + op_start = '>' + v_start = cpe['versionStartExcluding'] + + if 'versionEndIncluding' in cpe: + op_end = '<=' + v_end = cpe['versionEndIncluding'] + + if 'versionEndExcluding' in cpe: + op_end = '<' + v_end = cpe['versionEndExcluding'] + + if op_start or op_end or v_start or v_end: + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + else: + # This is no version information, expressed differently. + # Save processing by representing as -. + yield [cveId, vendor, product, '-', '', '', ''] + + conn.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()).close() + +def update_db(conn, jsondata): + import json + root = json.loads(jsondata) + + for elt in root['CVE_Items']: + if not elt['impact']: + continue + + accessVector = None + cveId = elt['cve']['CVE_data_meta']['ID'] + cveDesc = elt['cve']['description']['description_data'][0]['value'] + date = elt['lastModifiedDate'] + try: + accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector'] + cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore'] + except KeyError: + cvssv2 = 0.0 + try: + accessVector = accessVector or elt['impact']['baseMetricV3']['cvssV3']['attackVector'] + cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore'] + except KeyError: + accessVector = accessVector or "UNKNOWN" + cvssv3 = 0.0 + + conn.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", + [cveId, cveDesc, cvssv2, cvssv3, date, accessVector]).close() + + configurations = elt['configurations']['nodes'] + for config in configurations: + parse_node_and_insert(conn, config, cveId) + + +do_fetch[nostamp] = "1" + +EXCLUDE_FROM_WORLD = "1"
PATCH 18/34 cve-update-db-native: update structure
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Update the database structure and tasks to fit the current YP master. This means:
- add the unpack task
- update the database structure (CVSS, vector string)
- use the temporary database in the same directory as the download
However, the old feed does not include CVSS4
--- .../recipes-core/meta/cve-update-db-native.bb | 28 ++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +21 -7
@@ -5,7 +5,6 @@ INHIBIT_DEFAULT_DEPS = "1" inherit native -deltask do_unpack deltask do_patch deltask do_configure deltask do_compile
@@ -21,7 +20,10 @@ CVE_DB_UPDATE_INTERVAL ?= "86400" # Timeout for blocking socket operations, such as the connection attempt. CVE_SOCKET_TIMEOUT ?= "60" -CVE_DB_TEMP_FILE ?= "${CVE_CHECK_DB_DIR}/temp_nvdcve_1.1.db" +CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}" +CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" + +CVE_DB_TEMP_FILE ?= "${CVE_CHECK_DB_DLDIR_FILE}.tmp" python () { if not bb.data.inherits_class("cve-check", d):
@@ -38,7 +40,7 @@ python do_fetch() { bb.utils.export_proxies(d) - db_file = d.getVar("CVE_CHECK_DB_FILE") + db_file = d.getVar("CVE_CHECK_DB_DLDIR_FILE") db_dir = os.path.dirname(db_file) db_tmp_file = d.getVar("CVE_DB_TEMP_FILE")
@@ -72,10 +74,16 @@ python do_fetch() { os.remove(db_tmp_file) } -do_fetch[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}" +do_fetch[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK}" do_fetch[file-checksums] = "" do_fetch[vardeps] = "" +python do_unpack() { + import shutil + shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE")) +} +do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}" + def cleanup_db_download(db_file, db_tmp_file): """ Cleanup the download space from possible failed downloads
@@ -183,7 +191,7 @@ def initialize_db(conn): c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ - SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + SCOREV2 TEXT, SCOREV3 TEXT, SCOREV4 TEXT, MODIFIED INTEGER, VECTOR TEXT, VECTORSTRING TEXT)") c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \
@@ -263,23 +271,29 @@ def update_db(conn, jsondata): continue accessVector = None + vectorString = None + cvssv2 = 0.0 + cvssv3 = 0.0 + cvssv4 = 0.0 cveId = elt['cve']['CVE_data_meta']['ID'] cveDesc = elt['cve']['description']['description_data'][0]['value'] date = elt['lastModifiedDate'] try: accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector'] + vectorString = elt['impact']['baseMetricV2']['cvssV2']['vectorString'] cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore'] except KeyError: cvssv2 = 0.0 try: accessVector = accessVector or elt['impact']['baseMetricV3']['cvssV3']['attackVector'] + vectorString = vectorString or elt['impact']['baseMetricV3']['cvssV3']['vectorString'] cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore'] except KeyError: accessVector = accessVector or "UNKNOWN" cvssv3 = 0.0 - conn.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", - [cveId, cveDesc, cvssv2, cvssv3, date, accessVector]).close() + conn.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?, ?, ?)", + [cveId, cveDesc, cvssv2, cvssv3, cvssv4, date, accessVector, vectorString]).close() configurations = elt['configurations']['nodes'] for config in configurations:
PATCH 19/34 cve-update-db-native: add the fkie source
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Add support for FKIE-CAD reconstruction of NVD feed from
https://github.com/fkie-cad/nvd-json-data-feeds
We download this feed directly from github releases.
--- .../recipes-core/meta/cve-update-db-native.bb | 126 ++++++++++++++++-- 1 file changed, 113 insertions(+), 13 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +113 -13
@@ -12,6 +12,8 @@ deltask do_install deltask do_populate_sysroot NVDCVE_URL ?= "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-" +FKIE_URL ?= "https://github.com/fkie-cad/nvd-json-data-feeds/releases/latest/download/CVE-" + # CVE database update interval, in seconds. By default: once a day (24*60*60). # Use 0 to force the update # Use a negative value to skip the update
@@ -109,6 +111,30 @@ def cleanup_db_download(db_file, db_tmp_file): if os.path.exists(db_tmp_file): os.remove(db_tmp_file) +def db_file_names(d, year, is_nvd): + if is_nvd: + year_url = d.getVar('NVDCVE_URL') + str(year) + meta_url = year_url + ".meta" + json_url = year_url + ".json.gz" + return json_url, meta_url + year_url = d.getVar('FKIE_URL') + str(year) + meta_url = year_url + ".meta" + json_url = year_url + ".json.xz" + return json_url, meta_url + +def host_db_name(d, is_nvd): + if is_nvd: + return "nvd.nist.gov" + return "github.com" + +def db_decompress(d, data, is_nvd): + import gzip, lzma + + if is_nvd: + return gzip.decompress(data).decode('utf-8') + # otherwise + return lzma.decompress(data) + def update_db_file(db_tmp_file, d): """ Update the given database file
@@ -119,6 +145,7 @@ def update_db_file(db_tmp_file, d): YEAR_START = 2002 cve_socket_timeout = int(d.getVar("CVE_SOCKET_TIMEOUT")) + is_nvd = d.getVar("NVD_DB_VERSION") == "NVD1" # Connect to database conn = sqlite3.connect(db_tmp_file)
@@ -129,9 +156,7 @@ def update_db_file(db_tmp_file, d): for i, year in enumerate(range(YEAR_START, date.today().year + 1)): bb.debug(2, "Updating %d" % year) ph.update((float(i + 1) / total_years) * 100) - year_url = (d.getVar('NVDCVE_URL')) + str(year) - meta_url = year_url + ".meta" - json_url = year_url + ".json.gz" + json_url, meta_url = db_file_names(d, year, is_nvd) # Retrieve meta last modified date try:
@@ -140,7 +165,7 @@ def update_db_file(db_tmp_file, d): cve_f.write('Warning: CVE db update error, Unable to fetch CVE data.\n\n') bb.warn("Failed to fetch CVE data (%s)" % e) import socket - result = socket.getaddrinfo("nvd.nist.gov", 443, proto=socket.IPPROTO_TCP) + result = socket.getaddrinfo(host_db_name(d, is_nvd), 443, proto=socket.IPPROTO_TCP) bb.warn("Host IPs are %s" % (", ".join(t[4][0] for t in result))) return False
@@ -168,7 +193,7 @@ def update_db_file(db_tmp_file, d): try: response = urllib.request.urlopen(json_url, timeout=cve_socket_timeout) if response: - update_db(conn, gzip.decompress(response.read()).decode('utf-8')) + update_db(d, conn, db_decompress(d, response.read(), is_nvd)) conn.execute("insert or replace into META values (?, ?)", [year, last_modified]).close() except urllib.error.URLError as e: cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n')
@@ -200,16 +225,22 @@ def initialize_db(conn): c.close() -def parse_node_and_insert(conn, node, cveId): +def parse_node_and_insert(conn, node, cveId, is_nvd): # Parse children node if needed for child in node.get('children', ()): - parse_node_and_insert(conn, child, cveId) + parse_node_and_insert(conn, child, cveId, is_nvd) + + def cpe_generator(is_nvd): + match_string = "cpeMatch" + cpe_string = 'criteria' + if is_nvd: + match_string = "cpe_match" + cpe_string = 'cpe23Uri' - def cpe_generator(): - for cpe in node.get('cpe_match', ()): + for cpe in node.get(match_string, ()): if not cpe['vulnerable']: return - cpe23 = cpe.get('cpe23Uri') + cpe23 = cpe.get(cpe_string) if not cpe23: return cpe23 = cpe23.split(':')
@@ -260,9 +291,9 @@ def parse_node_and_insert(conn, node, cveId): # Save processing by representing as -. yield [cveId, vendor, product, '-', '', '', ''] - conn.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()).close() + conn.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator(is_nvd)).close() -def update_db(conn, jsondata): +def update_db_nvdjson(conn, jsondata): import json root = json.loads(jsondata)
@@ -297,8 +328,77 @@ def update_db(conn, jsondata): configurations = elt['configurations']['nodes'] for config in configurations: - parse_node_and_insert(conn, config, cveId) + parse_node_and_insert(conn, config, cveId, True) + +def update_db_fkie(conn, jsondata): + import json + root = json.loads(jsondata) + + for elt in root['cve_items']: + if not 'vulnStatus' in elt or elt['vulnStatus'] == 'Rejected': + continue + + if not 'configurations' in elt: + continue + + accessVector = None + vectorString = None + cvssv2 = 0.0 + cvssv3 = 0.0 + cvssv4 = 0.0 + cveId = elt['id'] + cveDesc = elt['descriptions'][0]['value'] + date = elt['lastModified'] + try: + for m in elt['metrics']['cvssMetricV2']: + if m['type'] == 'Primary': + accessVector = m['cvssData']['accessVector'] + vectorString = m['cvssData']['vectorString'] + cvssv2 = m['cvssData']['baseScore'] + except KeyError: + cvssv2 = 0.0 + try: + for m in elt['metrics']['cvssMetricV30']: + if m['type'] == 'Primary': + accessVector = m['cvssData']['accessVector'] + vectorString = m['cvssData']['vectorString'] + cvssv3 = m['cvssData']['baseScore'] + except KeyError: + accessVector = accessVector or "UNKNOWN" + cvssv3 = 0.0 + try: + for m in elt['metrics']['cvssMetricV31']: + if m['type'] == 'Primary': + accessVector = m['cvssData']['accessVector'] + vectorString = m['cvssData']['vectorString'] + cvssv3 = m['cvssData']['baseScore'] + except KeyError: + accessVector = accessVector or "UNKNOWN" + cvssv3 = 0.0 + try: + for m in elt['metrics']['cvssMetricV40']: + if m['type'] == 'Primary': + accessVector = m['cvssData']['accessVector'] + vectorString = m['cvssData']['vectorString'] + cvssv4 = m['cvssData']['baseScore'] + except KeyError: + accessVector = accessVector or "UNKNOWN" + cvssv4 = 0.0 + conn.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?, ?, ?)", + [cveId, cveDesc, cvssv2, cvssv3, cvssv4, date, accessVector, vectorString]).close() + + for config in elt['configurations']: + # This is suboptimal as it doesn't handle AND/OR and negate, but is better than nothing + for node in config["nodes"]: + parse_node_and_insert(conn, node, cveId, False) + + +def update_db(d, conn, jsondata): + if (d.getVar("NVD_DB_VERSION") == "FKIE"): + return update_db_fkie(conn, jsondata) + else: + return update_db_nvdjson(conn, jsondata) do_fetch[nostamp] = "1"
PATCH 20/34 cve-check: change the default feed
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Marta Rybczynska <[email protected]>
Move to the FKIE feed by default, as it is showing better stability than NVD2. Content of the feed should be the same.
--- meta/classes/cve-check.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/cve-check.bbclass +1 -1
@@ -32,7 +32,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" # Possible database sources: NVD1, NVD2, FKIE -NVD_DB_VERSION ?= "NVD2" +NVD_DB_VERSION ?= "FKIE" # Use different file names for each database source, as they synchronize at different moments, so may be slightly different CVE_CHECK_DB_FILENAME ?= "${@'nvdcve_2-2.db' if d.getVar('NVD_DB_VERSION') == 'NVD2' else 'nvdcve_1-3.db' if d.getVar('NVD_DB_VERSION') == 'NVD1' else 'nvdfkie_1-1.db'}"
PATCH 21/34 cve-check: fix debug message
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Daniel Turull <[email protected]>
Debug level was not added as a parameter, causing a warning.
--- meta/classes/cve-check.bbclass | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/classes/cve-check.bbclass +4 -4
@@ -279,7 +279,7 @@ def cve_update(d, cve_data, cve, entry): cve_data[cve] = entry return # If we are updating, there might be change in the status - bb.debug("Trying CVE entry update for %s from %s to %s" % (cve, cve_data[cve]['abbrev-status'], entry['abbrev-status'])) + bb.debug(1, "Trying CVE entry update for %s from %s to %s" % (cve, cve_data[cve]['abbrev-status'], entry['abbrev-status'])) if cve_data[cve]['abbrev-status'] == "Unknown": cve_data[cve] = entry return
@@ -290,16 +290,16 @@ def cve_update(d, cve_data, cve, entry): if entry['status'] == "version-in-range" and cve_data[cve]['status'] == "version-not-in-range": # New result from the scan, vulnerable cve_data[cve] = entry - bb.debug("CVE entry %s update from Patched to Unpatched from the scan result" % cve) + bb.debug(1, "CVE entry %s update from Patched to Unpatched from the scan result" % cve) return if entry['abbrev-status'] == "Patched" and cve_data[cve]['abbrev-status'] == "Unpatched": if entry['status'] == "version-not-in-range" and cve_data[cve]['status'] == "version-in-range": # Range does not match the scan, but we already have a vulnerable match, ignore - bb.debug("CVE entry %s update from Patched to Unpatched from the scan result - not applying" % cve) + bb.debug(1, "CVE entry %s update from Patched to Unpatched from the scan result - not applying" % cve) return # If we have an "Ignored", it has a priority if cve_data[cve]['abbrev-status'] == "Ignored": - bb.debug("CVE %s not updating because Ignored" % cve) + bb.debug(1, "CVE %s not updating because Ignored" % cve) return bb.warn("Unhandled CVE entry update for %s from %s to %s" % (cve, cve_data[cve], entry))
PATCH 22/34 spdx30: Allow VEX Justification to be configurable
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Joshua Watt <[email protected]>
Instead of hard coding the VEX justifications for "Ignored" CVE status, add a map that configures what justification should be used for each status.
This allows other justifications to be easily added, and also ensures that status fields added externally (by downstream) can set an appropriate justification if necessary.
--- meta/conf/cve-check-map.conf | 4 ++++ meta/lib/oe/spdx30_tasks.py | 33 ++++++++++++++++----------------- 2 files changed, 20 insertions(+), 17 deletions(-)
meta/conf/cve-check-map.conf +4 -0
@@ -28,8 +28,12 @@ CVE_CHECK_STATUSMAP[cpe-incorrect] = "Ignored" CVE_CHECK_STATUSMAP[disputed] = "Ignored" # use when vulnerability depends on build or runtime configuration which is not used CVE_CHECK_STATUSMAP[not-applicable-config] = "Ignored" +CVE_CHECK_VEX_JUSTIFICATION[not-applicable-config] = "vulnerableCodeNotPresent" + # use when vulnerability affects other platform (e.g. Windows or Debian) CVE_CHECK_STATUSMAP[not-applicable-platform] = "Ignored" +CVE_CHECK_VEX_JUSTIFICATION[not-applicable-platform] = "vulnerableCodeNotPresent" + # use when upstream acknowledged the vulnerability but does not plan to fix it CVE_CHECK_STATUSMAP[upstream-wontfix] = "Ignored"
meta/lib/oe/spdx30_tasks.py +16 -17
@@ -719,24 +719,23 @@ def create_spdx(d): impact_statement=description, ) - if detail in ( - "ignored", - "cpe-incorrect", - "disputed", - "upstream-wontfix", - ): - # VEX doesn't have justifications for this - pass - elif detail in ( - "not-applicable-config", - "not-applicable-platform", - ): - for v in spdx_vex: - v.security_justificationType = ( - oe.spdx30.security_VexJustificationType.vulnerableCodeNotPresent + vex_just_type = d.getVarFlag( + "CVE_CHECK_VEX_JUSTIFICATION", detail + ) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" ) - else: - bb.fatal(f"Unknown detail '{detail}' for ignored {cve}") + + for v in spdx_vex: + v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + else: bb.fatal(f"Unknown {cve} status '{status}'")
PATCH 23/34 cve-update-db-native: fix fetcher for CVEs missing nodes
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
As of now, update of CVE DB from FKIE source (which is the defailt) fails with following error:
File: '<build>/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 393, function: update_db_fkie
0389: [cveId, cveDesc, cvssv2, cvssv3, cvssv4, date, accessVector, vectorString]).close()
0390:
0391: for config in elt['configurations']:
0392: # This is suboptimal as it doesn't handle AND/OR and negate, but is better than nothing
*** 0393: for node in config["nodes"]:
0394: parse_node_and_insert(conn, node, cveId, False)
0395:
0396:def update_db(d, conn, jsondata):
0397: if (d.getVar("NVD_DB_VERSION") == "FKIE"):Exception: KeyError: 'nodes'
Entry for new CVE-2025-32915 is broken.
--- meta/recipes-core/meta/cve-update-db-native.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/meta/cve-update-db-native.bb +1 -1
@@ -390,7 +390,7 @@ def update_db_fkie(conn, jsondata): for config in elt['configurations']: # This is suboptimal as it doesn't handle AND/OR and negate, but is better than nothing - for node in config["nodes"]: + for node in config.get("nodes") or []: parse_node_and_insert(conn, node, cveId, False)
PATCH 24/34 cve-update-db-native: Use a local copy of the database during builds
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
OE-Core rev: 03596904392d257572a905a182b92c780d636744
This seems to be misimplemented when re-adding update from nvd1 feed. Use file in temporary directory instead of downloads directory for update process.
--- meta/recipes-core/meta/cve-update-db-native.bb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +3 -3
@@ -24,8 +24,7 @@ CVE_SOCKET_TIMEOUT ?= "60" CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" - -CVE_DB_TEMP_FILE ?= "${CVE_CHECK_DB_DLDIR_FILE}.tmp" +CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp" python () { if not bb.data.inherits_class("cve-check", d):
@@ -44,7 +43,7 @@ python do_fetch() { db_file = d.getVar("CVE_CHECK_DB_DLDIR_FILE") db_dir = os.path.dirname(db_file) - db_tmp_file = d.getVar("CVE_DB_TEMP_FILE") + db_tmp_file = d.getVar("CVE_CHECK_DB_TEMP_FILE") cleanup_db_download(db_file, db_tmp_file)
@@ -64,6 +63,7 @@ python do_fetch() { pass bb.utils.mkdirhier(db_dir) + bb.utils.mkdirhier(os.path.dirname(db_tmp_file)) if os.path.exists(db_file): shutil.copy2(db_file, db_tmp_file)
PATCH 25/34 cve-update-db-native: Handle BB_NO_NETWORK and missing db
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
OE-Core rev: 337c0806d2784d74bee8d6420fb8b4d48795d5fa
This commit was not applied on nvd1/fkie fetcher.
--- meta/recipes-core/meta/cve-update-db-native.bb | 5 +++++ 1 file changed, 5 insertions(+)
meta/recipes-core/meta/cve-update-db-native.bb +5 -0
@@ -54,6 +54,8 @@ python do_fetch() { update_interval = int(d.getVar("CVE_DB_UPDATE_INTERVAL")) if update_interval < 0: bb.note("CVE database update skipped") + if not os.path.exists(db_file): + bb.error("CVE database %s not present, database fetch/update skipped" % db_file) return if time.time() - os.path.getmtime(db_file) < update_interval: bb.debug(2, "Recently updated, skipping")
@@ -62,6 +64,9 @@ python do_fetch() { except OSError: pass + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): + bb.error("BB_NO_NETWORK attempted to disable fetch, this recipe uses CVE_DB_UPDATE_INTERVAL to control download, set to '-1' to disable fetch or update") + bb.utils.mkdirhier(db_dir) bb.utils.mkdirhier(os.path.dirname(db_tmp_file)) if os.path.exists(db_file):
PATCH 26/34 cve-update-db-native: log a little more
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
OE-Core rev: b64a869b9c5e1d504f1011da16b5c5ff721afbf0
This commit was not applied on nvd1/fkie fetcher.
--- meta/recipes-core/meta/cve-update-db-native.bb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +4 -4
@@ -58,7 +58,7 @@ python do_fetch() { bb.error("CVE database %s not present, database fetch/update skipped" % db_file) return if time.time() - os.path.getmtime(db_file) < update_interval: - bb.debug(2, "Recently updated, skipping") + bb.note("CVE database recently updated, skipping") return except OSError:
@@ -77,7 +77,7 @@ python do_fetch() { shutil.move(db_tmp_file, db_file) else: # Update failed, do not modify the database - bb.note("CVE database update failed") + bb.warn("CVE database update failed") os.remove(db_tmp_file) }
@@ -159,7 +159,7 @@ def update_db_file(db_tmp_file, d): with bb.progress.ProgressHandler(d) as ph, open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f: total_years = date.today().year + 1 - YEAR_START for i, year in enumerate(range(YEAR_START, date.today().year + 1)): - bb.debug(2, "Updating %d" % year) + bb.note("Updating %d" % year) ph.update((float(i + 1) / total_years) * 100) json_url, meta_url = db_file_names(d, year, is_nvd)
@@ -190,7 +190,7 @@ def update_db_file(db_tmp_file, d): cursor.close() if not meta or meta[0] != last_modified: - bb.debug(2, "Updating entries") + bb.note("Updating entries") # Clear products table entries corresponding to current year conn.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)).close()
PATCH 27/34 cve-update: decrease update interval to 23 hours
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
If the job runs every day at the same time, it usually updates only every second day, because it takes non-0 time for DB update and set the timestamp. So it does not take full 24-hours from time when the DB was updated until the next job starts.
--- meta/recipes-core/meta/cve-update-db-native.bb | 4 ++-- meta/recipes-core/meta/cve-update-nvd2-native.bb | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +2 -2
@@ -14,10 +14,10 @@ deltask do_populate_sysroot NVDCVE_URL ?= "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-" FKIE_URL ?= "https://github.com/fkie-cad/nvd-json-data-feeds/releases/latest/download/CVE-" -# CVE database update interval, in seconds. By default: once a day (24*60*60). +# CVE database update interval, in seconds. By default: once a day (23*60*60). # Use 0 to force the update # Use a negative value to skip the update -CVE_DB_UPDATE_INTERVAL ?= "86400" +CVE_DB_UPDATE_INTERVAL ?= "82800" # Timeout for blocking socket operations, such as the connection attempt. CVE_SOCKET_TIMEOUT ?= "60"
meta/recipes-core/meta/cve-update-nvd2-native.bb +2 -2
@@ -20,10 +20,10 @@ NVDCVE_URL ?= "https://services.nvd.nist.gov/rest/json/cves/2.0" # then setting this to get higher rate limits. NVDCVE_API_KEY ?= "" -# CVE database update interval, in seconds. By default: once a day (24*60*60). +# CVE database update interval, in seconds. By default: once a day (23*60*60). # Use 0 to force the update # Use a negative value to skip the update -CVE_DB_UPDATE_INTERVAL ?= "86400" +CVE_DB_UPDATE_INTERVAL ?= "82800" # CVE database incremental update age threshold, in seconds. If the database is # older than this threshold, do a full re-download, else, do an incremental
PATCH 28/34 cve-update: remove cleanup of db_file in downloads
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
Since the code was changed to update the DB in temporary file, code cleaning the final file in downloads is never executed. Remove it.
Since the code always removes both files in temporary directory, remove also comment which is trying to differentiate this code from code just removed.
--- meta/recipes-core/meta/cve-update-db-native.bb | 17 ++--------------- .../recipes-core/meta/cve-update-nvd2-native.bb | 17 ++--------------- 2 files changed, 4 insertions(+), 30 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +2 -15
@@ -45,7 +45,7 @@ python do_fetch() { db_dir = os.path.dirname(db_file) db_tmp_file = d.getVar("CVE_CHECK_DB_TEMP_FILE") - cleanup_db_download(db_file, db_tmp_file) + cleanup_db_download(db_tmp_file) # The NVD database changes once a day, so no need to update more frequently # Allow the user to force-update
@@ -91,28 +91,15 @@ python do_unpack() { } do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}" -def cleanup_db_download(db_file, db_tmp_file): +def cleanup_db_download(db_tmp_file): """ Cleanup the download space from possible failed downloads """ - # Clean up the updates done on the main file - # Remove it only if a journal file exists - it means a complete re-download - if os.path.exists("{0}-journal".format(db_file)): - # If a journal is present the last update might have been interrupted. In that case, - # just wipe any leftovers and force the DB to be recreated. - os.remove("{0}-journal".format(db_file)) - - if os.path.exists(db_file): - os.remove(db_file) - # Clean-up the temporary file downloads, we can remove both journal # and the temporary database if os.path.exists("{0}-journal".format(db_tmp_file)): - # If a journal is present the last update might have been interrupted. In that case, - # just wipe any leftovers and force the DB to be recreated. os.remove("{0}-journal".format(db_tmp_file)) - if os.path.exists(db_tmp_file): os.remove(db_tmp_file)
meta/recipes-core/meta/cve-update-nvd2-native.bb +2 -15
@@ -57,7 +57,7 @@ python do_fetch() { db_dir = os.path.dirname(db_file) db_tmp_file = d.getVar("CVE_CHECK_DB_TEMP_FILE") - cleanup_db_download(db_file, db_tmp_file) + cleanup_db_download(db_tmp_file) # By default let's update the whole database (since time 0) database_time = 0
@@ -106,28 +106,15 @@ python do_unpack() { } do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}" -def cleanup_db_download(db_file, db_tmp_file): +def cleanup_db_download(db_tmp_file): """ Cleanup the download space from possible failed downloads """ - # Clean up the updates done on the main file - # Remove it only if a journal file exists - it means a complete re-download - if os.path.exists("{0}-journal".format(db_file)): - # If a journal is present the last update might have been interrupted. In that case, - # just wipe any leftovers and force the DB to be recreated. - os.remove("{0}-journal".format(db_file)) - - if os.path.exists(db_file): - os.remove(db_file) - # Clean-up the temporary file downloads, we can remove both journal # and the temporary database if os.path.exists("{0}-journal".format(db_tmp_file)): - # If a journal is present the last update might have been interrupted. In that case, - # just wipe any leftovers and force the DB to be recreated. os.remove("{0}-journal".format(db_tmp_file)) - if os.path.exists(db_tmp_file): os.remove(db_tmp_file)
PATCH 29/34 cve-update-db-native: Fix FKIE CVE accessVector parsing
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Jonathan Schnitzler <[email protected]>
Use "attackVector" for CVSS >= 3 as it only CVSS v2 uses "accessVector".
--- meta/recipes-core/meta/cve-update-db-native.bb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +3 -3
@@ -352,7 +352,7 @@ def update_db_fkie(conn, jsondata): try: for m in elt['metrics']['cvssMetricV30']: if m['type'] == 'Primary': - accessVector = m['cvssData']['accessVector'] + accessVector = m['cvssData']['attackVector'] vectorString = m['cvssData']['vectorString'] cvssv3 = m['cvssData']['baseScore'] except KeyError:
@@ -361,7 +361,7 @@ def update_db_fkie(conn, jsondata): try: for m in elt['metrics']['cvssMetricV31']: if m['type'] == 'Primary': - accessVector = m['cvssData']['accessVector'] + accessVector = m['cvssData']['attackVector'] vectorString = m['cvssData']['vectorString'] cvssv3 = m['cvssData']['baseScore'] except KeyError:
@@ -370,7 +370,7 @@ def update_db_fkie(conn, jsondata): try: for m in elt['metrics']['cvssMetricV40']: if m['type'] == 'Primary': - accessVector = m['cvssData']['accessVector'] + accessVector = m['cvssData']['attackVector'] vectorString = m['cvssData']['vectorString'] cvssv4 = m['cvssData']['baseScore'] except KeyError:
PATCH 30/34 cve-update-db-native: FKIE CVE parsing: Use Secondary metric
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Jonathan Schnitzler <[email protected]>
If there is no primary metric use the Secondary one.
--- .../recipes-core/meta/cve-update-db-native.bb | 53 ++++++++++++------- 1 file changed, 33 insertions(+), 20 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +33 -20
@@ -322,6 +322,15 @@ def update_db_nvdjson(conn, jsondata): for config in configurations: parse_node_and_insert(conn, config, cveId, True) +def get_metric_entry(metric): + primaries = [c for c in metric if c['type'] == "Primary"] + secondaries = [c for c in metric if c['type'] == "Secondary"] + if len(primaries) > 0: + return primaries[0] + elif len(secondaries)>0: + return secondaries[0] + return None + def update_db_fkie(conn, jsondata): import json root = json.loads(jsondata)
@@ -342,37 +351,41 @@ def update_db_fkie(conn, jsondata): cveDesc = elt['descriptions'][0]['value'] date = elt['lastModified'] try: - for m in elt['metrics']['cvssMetricV2']: - if m['type'] == 'Primary': - accessVector = m['cvssData']['accessVector'] - vectorString = m['cvssData']['vectorString'] - cvssv2 = m['cvssData']['baseScore'] + if 'cvssMetricV2' in elt['metrics']: + entry = get_metric_entry(elt['metrics']['cvssMetricV2']) + if entry: + accessVector = entry['cvssData']['accessVector'] + vectorString = entry['cvssData']['vectorString'] + cvssv2 = entry['cvssData']['baseScore'] except KeyError: cvssv2 = 0.0 try: - for m in elt['metrics']['cvssMetricV30']: - if m['type'] == 'Primary': - accessVector = m['cvssData']['attackVector'] - vectorString = m['cvssData']['vectorString'] - cvssv3 = m['cvssData']['baseScore'] + if 'cvssMetricV30' in elt['metrics']: + entry = get_metric_entry(elt['metrics']['cvssMetricV30']) + if entry: + accessVector = entry['cvssData']['attackVector'] + vectorString = entry['cvssData']['vectorString'] + cvssv3 = entry['cvssData']['baseScore'] except KeyError: accessVector = accessVector or "UNKNOWN" cvssv3 = 0.0 try: - for m in elt['metrics']['cvssMetricV31']: - if m['type'] == 'Primary': - accessVector = m['cvssData']['attackVector'] - vectorString = m['cvssData']['vectorString'] - cvssv3 = m['cvssData']['baseScore'] + if 'cvssMetricV31' in elt['metrics']: + entry = get_metric_entry(elt['metrics']['cvssMetricV31']) + if entry: + accessVector = entry['cvssData']['attackVector'] + vectorString = entry['cvssData']['vectorString'] + cvssv3 = entry['cvssData']['baseScore'] except KeyError: accessVector = accessVector or "UNKNOWN" cvssv3 = 0.0 try: - for m in elt['metrics']['cvssMetricV40']: - if m['type'] == 'Primary': - accessVector = m['cvssData']['attackVector'] - vectorString = m['cvssData']['vectorString'] - cvssv4 = m['cvssData']['baseScore'] + if 'cvssMetricV40' in elt['metrics']: + entry = get_metric_entry(elt['metrics']['cvssMetricV40']) + if entry: + accessVector = entry['cvssData']['attackVector'] + vectorString = entry['cvssData']['vectorString'] + cvssv4 = entry['cvssData']['baseScore'] except KeyError: accessVector = accessVector or "UNKNOWN" cvssv4 = 0.0
PATCH 31/34 cve-update: log timestamps and add force update for future time
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Peter Marko <[email protected]>
CVE update is currently not working properly on autobuilder. This improves logging for problem analysis.
Future time is something which could be reason for current autobuilder problems since the DB was not updated for more than 3 months by now.
--- meta/recipes-core/meta/cve-update-db-native.bb | 7 ++++++- meta/recipes-core/meta/cve-update-nvd2-native.bb | 9 +++++++-- 2 files changed, 13 insertions(+), 3 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +6 -1
@@ -57,7 +57,12 @@ python do_fetch() { if not os.path.exists(db_file): bb.error("CVE database %s not present, database fetch/update skipped" % db_file) return - if time.time() - os.path.getmtime(db_file) < update_interval: + curr_time = time.time() + database_time = os.path.getmtime(db_file) + bb.note("Current time: %s; DB time: %s" % (time.ctime(curr_time), time.ctime(database_time))) + if curr_time < database_time: + bb.warn("Database time is in the future, force DB update") + elif curr_time - database_time < update_interval: bb.note("CVE database recently updated, skipping") return
meta/recipes-core/meta/cve-update-nvd2-native.bb +7 -2
@@ -71,10 +71,15 @@ python do_fetch() { if not os.path.exists(db_file): bb.error("CVE database %s not present, database fetch/update skipped" % db_file) return - if time.time() - os.path.getmtime(db_file) < update_interval: + curr_time = time.time() + database_time = os.path.getmtime(db_file) + bb.note("Current time: %s; DB time: %s" % (time.ctime(curr_time), time.ctime(database_time))) + if curr_time < database_time: + bb.warn("Database time is in the future, force DB update") + database_time = 0 + elif curr_time - database_time < update_interval: bb.note("CVE database recently updated, skipping") return - database_time = os.path.getmtime(db_file) except OSError: pass
PATCH 32/34 cve-update-db-native: pycodestyle fixes
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Niko Mauno <[email protected]>
Fixes following pycodestyle complaints:
cve-update-db-native.bb:80:39: E712 comparison to True should be 'if cond is True:' or 'if cond:' cve-update-db-native.bb:128:20: E401 multiple imports on one line cve-update-db-native.bb:130:18: E401 multiple imports on one line cve-update-db-native.bb:171:21: E741 ambiguous variable name 'l' cve-update-db-native.bb:335:26: E225 missing whitespace around operator cve-update-db-native.bb:344:12: E713 test for membership should be 'not in' cve-update-db-native.bb:347:12: E713 test for membership should be 'not in'
Also leaves out a redundant 'gzip' import in update_db_file().
--- meta/recipes-core/meta/cve-update-db-native.bb | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +10 -8
@@ -77,7 +77,7 @@ python do_fetch() { if os.path.exists(db_file): shutil.copy2(db_file, db_tmp_file) - if update_db_file(db_tmp_file, d) == True: + if update_db_file(db_tmp_file, d): # Update downloaded correctly, can swap files shutil.move(db_tmp_file, db_file) else:
@@ -136,9 +136,11 @@ def update_db_file(db_tmp_file, d): """ Update the given database file """ - import bb.utils, bb.progress + import bb.progress + import bb.utils from datetime import date - import urllib, gzip, sqlite3 + import sqlite3 + import urllib YEAR_START = 2002 cve_socket_timeout = int(d.getVar("CVE_SOCKET_TIMEOUT"))
@@ -167,8 +169,8 @@ def update_db_file(db_tmp_file, d): return False if response: - for l in response.read().decode("utf-8").splitlines(): - key, value = l.split(":", 1) + for line in response.read().decode("utf-8").splitlines(): + key, value = line.split(":", 1) if key == "lastModifiedDate": last_modified = value break
@@ -332,7 +334,7 @@ def get_metric_entry(metric): secondaries = [c for c in metric if c['type'] == "Secondary"] if len(primaries) > 0: return primaries[0] - elif len(secondaries)>0: + elif len(secondaries) > 0: return secondaries[0] return None
@@ -341,10 +343,10 @@ def update_db_fkie(conn, jsondata): root = json.loads(jsondata) for elt in root['cve_items']: - if not 'vulnStatus' in elt or elt['vulnStatus'] == 'Rejected': + if 'vulnStatus' not in elt or elt['vulnStatus'] == 'Rejected': continue - if not 'configurations' in elt: + if 'configurations' not in elt: continue accessVector = None
PATCH 33/34 cve-update-nvd2-native: pycodestyle fixes
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Niko Mauno <[email protected]>
Fixes following pycodestyle complaints:
cve-update-nvd2-native.bb:95:54: E712 comparison to True should be 'if cond is True:' or 'if cond:' cve-update-nvd2-native.bb:127:15: E211 whitespace before '(' cve-update-nvd2-native.bb:127:17: E201 whitespace after '(' cve-update-nvd2-native.bb:127:19: E201 whitespace after '(' cve-update-nvd2-native.bb:127:44: E202 whitespace before ')' cve-update-nvd2-native.bb:127:46: E203 whitespace before ',' cve-update-nvd2-native.bb:174:20: E401 multiple imports on one line cve-update-nvd2-native.bb:183:29: E203 whitespace before ':' cve-update-nvd2-native.bb:236:16: E111 indentation is not a multiple of 4 cve-update-nvd2-native.bb:241:16: E111 indentation is not a multiple of 4 cve-update-nvd2-native.bb:336:39: E222 multiple spaces after operator
--- meta/recipes-core/meta/cve-update-nvd2-native.bb | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +8 -7
@@ -92,7 +92,7 @@ python do_fetch() { if os.path.exists(db_file): shutil.copy2(db_file, db_tmp_file) - if update_db_file(db_tmp_file, d, database_time) == True: + if update_db_file(db_tmp_file, d, database_time): # Update downloaded correctly, can swap files shutil.move(db_tmp_file, db_file) else:
@@ -124,7 +124,7 @@ def cleanup_db_download(db_tmp_file): os.remove(db_tmp_file) def nvd_request_wait(attempt, min_wait): - return min ( ( (2 * attempt) + min_wait ) , 30) + return min(((2 * attempt) + min_wait), 30) def nvd_request_next(url, attempts, api_key, args, min_wait): """
@@ -171,7 +171,8 @@ def update_db_file(db_tmp_file, d, database_time): """ Update the given database file """ - import bb.utils, bb.progress + import bb.progress + import bb.utils import datetime import sqlite3 import json
@@ -180,7 +181,7 @@ def update_db_file(db_tmp_file, d, database_time): conn = sqlite3.connect(db_tmp_file) initialize_db(conn) - req_args = {'startIndex' : 0} + req_args = {'startIndex': 0} incr_update_threshold = int(d.getVar("CVE_DB_INCR_UPDATE_AGE_THRES")) if database_time != 0:
@@ -233,12 +234,12 @@ def update_db_file(db_tmp_file, d, database_time): per_page = data["resultsPerPage"] bb.note("Got %d entries" % per_page) for cve in data["vulnerabilities"]: - update_db(conn, cve) + update_db(conn, cve) index += per_page ph.update((float(index) / (total+1)) * 100) if index >= total: - break + break # Recommended by NVD time.sleep(wait_time)
@@ -333,7 +334,7 @@ def update_db(conn, elt): accessVector = None vectorString = None cveId = elt['cve']['id'] - if elt['cve'].get('vulnStatus') == "Rejected": + if elt['cve'].get('vulnStatus') == "Rejected": c = conn.cursor() c.execute("delete from PRODUCTS where ID = ?;", [cveId]) c.execute("delete from NVD where ID = ?;", [cveId])
PATCH 34/34 cve-update: Avoid NFS caching issues
In reply to Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)From: Paul Barker <[email protected]>
When moving the updated CVE database file to the downloads directory, ensure that it has a different inode number to the previous version of this file.
We have seen "sqlite3.DatabaseError: database disk image is malformed" exceptions on our autobuilder when trying to read the CVE database in do_cve_check tasks. The context here is that the downloads directory (where the updated database file is copied to) is shared between workers as an NFS mount. Different autobuilder workers were seeing different checksums for the database file, which indicates that a mix of both new and stale data was being read. Forcing each new version of the database file to have a different inode number will prevent stale data from being read from local caches.
This should fix [YOCTO #16086].
--- meta/recipes-core/meta/cve-update-db-native.bb | 9 +++++++-- meta/recipes-core/meta/cve-update-nvd2-native.bb | 9 +++++++-- 2 files changed, 14 insertions(+), 4 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +7 -2
@@ -78,8 +78,13 @@ python do_fetch() { shutil.copy2(db_file, db_tmp_file) if update_db_file(db_tmp_file, d): - # Update downloaded correctly, can swap files - shutil.move(db_tmp_file, db_file) + # Update downloaded correctly, we can swap files. To avoid potential + # NFS caching issues, ensure that the destination file has a new inode + # number. We do this in two steps as the downloads directory may be on + # a different filesystem to tmpdir we're working in. + new_file = "%s.new" % (db_file) + shutil.move(db_tmp_file, new_file) + os.rename(new_file, db_file) else: # Update failed, do not modify the database bb.warn("CVE database update failed")
meta/recipes-core/meta/cve-update-nvd2-native.bb +7 -2
@@ -93,8 +93,13 @@ python do_fetch() { shutil.copy2(db_file, db_tmp_file) if update_db_file(db_tmp_file, d, database_time): - # Update downloaded correctly, can swap files - shutil.move(db_tmp_file, db_file) + # Update downloaded correctly, we can swap files. To avoid potential + # NFS caching issues, ensure that the destination file has a new inode + # number. We do this in two steps as the downloads directory may be on + # a different filesystem to tmpdir we're working in. + new_file = "%s.new" % (db_file) + shutil.move(db_tmp_file, new_file) + os.rename(new_file, db_file) else: # Update failed, do not modify the database bb.warn("CVE database update failed")
Quoted 20 lines · Het Patel via lists.openembedded.org
From: Marta Rybczynska <[email protected]>
CVE_STATUScontains assesment of a given CVE, but until now it didn't have include the affected vendor/product. In the case of a global system include, thatCVE_STATUSwas visible in all recipes.This patch allows encoding of affected product/vendor to each
CVE_STATUSassessment, also for groups. We can then filter them later and use only CVEs that correspond to the recipe.This is going to be used in meta/conf/distro/include/cve-extra-exclusions.inc and similar places.
Backport Changes:
- Discarded the changes tometa/lib/oe/spdx30_tasks.py, as the commit history for this file diverges from the base commit itself (9c9b9545049ain the scarthgap branch).
- Additionally, the changes do not introduce any major features and are primarily focused on code restructuring.
Quoted 4 lines
Signed-off-by Marta Rybczynska <[email protected]>Signed-off-by Richard Purdie <[email protected]>(cherry picked from commit abca80a716e92fc18d3085aba1a15f4bac72379c)Signed-off-by Het Patel <[email protected]>
Hi,
When sending a long list of backport patches like this, please include a cover letter explaining the benefit you see to having these on the stable branch and include some test results.
Have you confirmed that all the patches in your series are also on the whinlatter branch as well as master?
Best regards,
Signature · 1 line
Paul Barker
Hi Paul,
Yes, all patches in your series are present on both the whinlatter branch and master. These patches have been cherry‑picked directly from the master branch. Their primary purpose is to migrate the CVE reporting files in the scarthgap branch so that they align with the master implementation.
I have attached a comparison of the CVE reports generated on the scarthgap branch before and after the migration. As shown, several additional fields are included in the post‑migration report, such as "patch-file", and "detail": "version-not-in-range". With these changes, the CVE report format in scarthgap now closely matches the format used in the master branch.
Please let me know if you have any questions or need further clarification.
Kind regards,
Het Patel.
[cid:9392158e-5418-49ae-ac27-40de739c7c02]
________________________________
From: Paul Barker
Sent: Monday, February 23, 2026 3:16 PM
To: Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco); [email protected]
Cc: xe-linux-external(mailer list); Viral Chavda (vchavda); Yoann Congal
Subject: Re: [OE-core] [openembedded-core] [scarthgap] [PATCH v1 01/34] cve-check: encode affected product/vendor in CVE_STATUS
Quoted 20 lines · Het Patel via lists.openembedded.org
From: Marta Rybczynska <[email protected]>
CVE_STATUScontains assesment of a given CVE, but until now it didn't have include the affected vendor/product. In the case of a global system include, thatCVE_STATUSwas visible in all recipes.This patch allows encoding of affected product/vendor to each
CVE_STATUSassessment, also for groups. We can then filter them later and use only CVEs that correspond to the recipe.This is going to be used in meta/conf/distro/include/cve-extra-exclusions.inc and similar places.
Backport Changes:
- Discarded the changes tometa/lib/oe/spdx30_tasks.py, as the commit history for this file diverges from the base commit itself (9c9b9545049ain the scarthgap branch).
- Additionally, the changes do not introduce any major features and are primarily focused on code restructuring.
Quoted 4 lines
Signed-off-by Marta Rybczynska <[email protected]>Signed-off-by Richard Purdie <[email protected]>(cherry picked from commit abca80a716e92fc18d3085aba1a15f4bac72379c)Signed-off-by Het Patel <[email protected]>
Hi,
When sending a long list of backport patches like this, please include a cover letter explaining the benefit you see to having these on the stable branch and include some test results.
Have you confirmed that all the patches in your series are also on the whinlatter branch as well as master?
Best regards,
--
Paul Barker
Gentle reminder.
________________________________
From: Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco) <[email protected]>
Sent: Monday, February 23, 2026 6:01 PM
To: Paul Barker <[email protected]>; [email protected] <[email protected]>
Cc: xe-linux-external(mailer list) <[email protected]>; Viral Chavda (vchavda) <[email protected]>; Yoann Congal <[email protected]>
Subject: Re: [OE-core] [openembedded-core] [scarthgap] [PATCH v1 01/34] cve-check: encode affected product/vendor in CVE_STATUS
Hi Paul,
Yes, all patches in your series are present on both the whinlatter branch and master. These patches have been cherry‑picked directly from the master branch. Their primary purpose is to migrate the CVE reporting files in the scarthgap branch so that they align with the master implementation.
I have attached a comparison of the CVE reports generated on the scarthgap branch before and after the migration. As shown, several additional fields are included in the post‑migration report, such as "patch-file", and "detail": "version-not-in-range". With these changes, the CVE report format in scarthgap now closely matches the format used in the master branch.
Please let me know if you have any questions or need further clarification.
Kind regards,
Het Patel.
[cid:9392158e-5418-49ae-ac27-40de739c7c02]
________________________________
From: Paul Barker
Sent: Monday, February 23, 2026 3:16 PM
To: Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco); [email protected]
Cc: xe-linux-external(mailer list); Viral Chavda (vchavda); Yoann Congal
Subject: Re: [OE-core] [openembedded-core] [scarthgap] [PATCH v1 01/34] cve-check: encode affected product/vendor in CVE_STATUS
Quoted 20 lines · Het Patel via lists.openembedded.org
From: Marta Rybczynska <[email protected]>
CVE_STATUScontains assesment of a given CVE, but until now it didn't have include the affected vendor/product. In the case of a global system include, thatCVE_STATUSwas visible in all recipes.This patch allows encoding of affected product/vendor to each
CVE_STATUSassessment, also for groups. We can then filter them later and use only CVEs that correspond to the recipe.This is going to be used in meta/conf/distro/include/cve-extra-exclusions.inc and similar places.
Backport Changes:
- Discarded the changes tometa/lib/oe/spdx30_tasks.py, as the commit history for this file diverges from the base commit itself (9c9b9545049ain the scarthgap branch).
- Additionally, the changes do not introduce any major features and are primarily focused on code restructuring.
Quoted 4 lines
Signed-off-by Marta Rybczynska <[email protected]>Signed-off-by Richard Purdie <[email protected]>(cherry picked from commit abca80a716e92fc18d3085aba1a15f4bac72379c)Signed-off-by Het Patel <[email protected]>
Hi,
When sending a long list of backport patches like this, please include a cover letter explaining the benefit you see to having these on the stable branch and include some test results.
Have you confirmed that all the patches in your series are also on the whinlatter branch as well as master?
Best regards,
--
Paul Barker
Quoted 1 line · Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco)
Gentle reminder.
Hello,
Sorry but I have to reject the series.
The LTS policy is restricted to bugfixes and security updates. A significant portion of these patches introduce new features or refactoring, which increases regression risk.
Please split the series and resubmit only the bugfixes if that makes sense for you.
Regards,
Quoted 67 lines
________________________________
From: Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco) <[email protected]>
Sent: Monday, February 23, 2026 6:01 PM
To: Paul Barker <[email protected]>; [email protected] <[email protected]>
Cc: xe-linux-external(mailer list) <[email protected]>; Viral Chavda (vchavda) <[email protected]>; Yoann Congal <[email protected]>
Subject: Re: [OE-core] [openembedded-core] [scarthgap] [PATCH v1 01/34] cve-check: encode affected product/vendor inCVE_STATUSHi Paul,
Yes, all patches in your series are present on both the whinlatter branch and master. These patches have been cherry‑picked directly from the master branch. Their primary purpose is to migrate the CVE reporting files in the scarthgap branch so that they align with the master implementation.
I have attached a comparison of the CVE reports generated on the scarthgap branch before and after the migration. As shown, several additional fields are included in the post‑migration report, such as "patch-file", and "detail": "version-not-in-range". With these changes, the CVE report format in scarthgap now closely matches the format used in the master branch.
Please let me know if you have any questions or need further clarification.
Kind regards,
Het Patel.[cid:9392158e-5418-49ae-ac27-40de739c7c02]
________________________________
From: Paul Barker
Sent: Monday, February 23, 2026 3:16 PM
To: Het Patel -X (hetpat - E INFOCHIPS PRIVATE LIMITED at Cisco); [email protected]
Cc: xe-linux-external(mailer list); Viral Chavda (vchavda); Yoann Congal
Subject: Re: [OE-core] [openembedded-core] [scarthgap] [PATCH v1 01/34] cve-check: encode affected product/vendor inCVE_STATUSQuoted 20 lines · Het Patel via lists.openembedded.org
From: Marta Rybczynska <[email protected]>
CVE_STATUScontains assesment of a given CVE, but until now it didn't have include the affected vendor/product. In the case of a global system include, thatCVE_STATUSwas visible in all recipes.This patch allows encoding of affected product/vendor to each
CVE_STATUSassessment, also for groups. We can then filter them later and use only CVEs that correspond to the recipe.This is going to be used in meta/conf/distro/include/cve-extra-exclusions.inc and similar places.
Backport Changes:
- Discarded the changes tometa/lib/oe/spdx30_tasks.py, as the commit history for this file diverges from the base commit itself (9c9b9545049ain the scarthgap branch).
- Additionally, the changes do not introduce any major features and are primarily focused on code restructuring.Quoted 4 lines
Signed-off-by Marta Rybczynska <[email protected]>Signed-off-by Richard Purdie <[email protected]>(cherry picked from commit abca80a716e92fc18d3085aba1a15f4bac72379c)Signed-off-by Het Patel <[email protected]>Hi,
When sending a long list of backport patches like this, please include a cover letter explaining the benefit you see to having these on the stable branch and include some test results.
Have you confirmed that all the patches in your series are also on the whinlatter branch as well as master?
Best regards,
--
Paul Barker
Signature · 2 lines
Yoann Congal Smile ECS