cve-check: Add versioned CVSS vector strings
PATCH 1/1 cve-check: Add versioned CVSS vector strings
Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.
This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".
---
This is an alternative patch to "cve-update-nvd2-native: Update vector logic" where each versioned vector string is attempted to be included. This does introduce a breaking API change, where vectorString has been removed and replaced with the versioned vectorString variables. I tried to update all references to the current use of vector strings, but I was a little confused looking at meta/lib/oe/spdx30.py. I'm not sure if this change will affect that file. I did turn on SPDX generation and ran another build. I looked at the outputs, but didn't see any vector string output.
meta/classes/cve-check.bbclass | 20 +++++---- meta/classes/vex.bbclass | 4 +- .../meta/cve-update-nvd2-native.bb | 42 ++++++++++++++----- scripts/cve-json-to-text.py | 8 +++- 4 files changed, 52 insertions(+), 22 deletions(-)
meta/classes/cve-check.bbclass +12 -8
@@ -31,7 +31,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" -CVE_CHECK_DB_FILENAME ?= "nvdcve_2-2.db" +CVE_CHECK_DB_FILENAME ?= "nvdcve_2-3.db" CVE_CHECK_DB_DIR ?= "${STAGING_DIR}/CVE_CHECK" CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_FILE_LOCK ?= "${CVE_CHECK_DB_FILE}.lock"
@@ -449,12 +449,14 @@ def get_cve_info(d, cve_data): continue #cve_data[row[0]] = {} cve_data[row[0]]["NVD-summary"] = row[1] - cve_data[row[0]]["NVD-scorev2"] = row[2] - cve_data[row[0]]["NVD-scorev3"] = row[3] - cve_data[row[0]]["NVD-scorev4"] = row[4] - cve_data[row[0]]["NVD-modified"] = row[5] - cve_data[row[0]]["NVD-vector"] = row[6] - cve_data[row[0]]["NVD-vectorString"] = row[7] + cve_data[row[0]]["NVD-vectorStringV2"] = row[2] + cve_data[row[0]]["NVD-scorev2"] = row[3] + cve_data[row[0]]["NVD-vectorStringV3"] = row[4] + cve_data[row[0]]["NVD-scorev3"] = row[5] + cve_data[row[0]]["NVD-vectorStringV4"] = row[6] + cve_data[row[0]]["NVD-scorev4"] = row[7] + cve_data[row[0]]["NVD-modified"] = row[8] + cve_data[row[0]]["NVD-vector"] = row[9] cursor.close() conn.close()
@@ -539,12 +541,14 @@ def cve_write_data_json(d, cve_data, cve_status): } if 'NVD-summary' in cve_data[cve]: cve_item["summary"] = cve_data[cve]["NVD-summary"] + cve_item["vectorStringV2"] = cve_data[cve]["NVD-vectorStringV2"] cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"] + cve_item["vectorStringV3"] = cve_data[cve]["NVD-vectorStringV3"] cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"] + cve_item["vectorStringV4"] = cve_data[cve]["NVD-vectorStringV4"] cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"] cve_item["modified"] = cve_data[cve]["NVD-modified"] cve_item["vector"] = cve_data[cve]["NVD-vector"] - cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"] if 'status' in cve_data[cve]: cve_item["detail"] = cve_data[cve]["status"] if 'justification' in cve_data[cve]:
meta/classes/vex.bbclass +3 -1
@@ -280,11 +280,13 @@ def cve_write_data_json(d, cve_data, cve_status): } if 'NVD-summary' in cve_data[cve]: cve_item["summary"] = cve_data[cve]["NVD-summary"] + cve_item["vectorStringV2"] = cve_data[cve]["NVD-vectorStringV2"] cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"] + cve_item["vectorStringV3"] = cve_data[cve]["NVD-vectorStringV3"] cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"] + cve_item["vectorStringV4"] = cve_data[cve]["NVD-vectorStringV4"] cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"] cve_item["vector"] = cve_data[cve]["NVD-vector"] - cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"] if 'status' in cve_data[cve]: cve_item["detail"] = cve_data[cve]["status"] if 'justification' in cve_data[cve]:
meta/recipes-core/meta/cve-update-nvd2-native.bb +31 -11
81 lines not on this page · raw message
scripts/cve-json-to-text.py +6 -2
@@ -121,16 +121,20 @@ def process_data(filename, data): lines += "CVE DESCRIPTION: %s\n" % issue["description"] if "summary" in issue: lines += "CVE SUMMARY: %s\n" % issue["summary"] + if "vectorStringV2" in issue: + lines += "VECTORSTRING v2: %s\n" % issue["vectorStringV2"] if "scorev2" in issue: lines += "CVSS v2 BASE SCORE: %s\n" % issue["scorev2"] + if "vectorStringV3" in issue: + lines += "VECTORSTRING v3: %s\n" % issue["vectorStringV3"] if "scorev3" in issue: lines += "CVSS v3 BASE SCORE: %s\n" % issue["scorev3"] + if "vectorStringV4" in issue: + lines += "VECTORSTRING v4: %s\n" % issue["vectorStringV4"] if "scorev4" in issue: lines += "CVSS v4 BASE SCORE: %s\n" % issue["scorev4"] if "vector" in issue: lines += "VECTOR: %s\n" % issue["vector"] - if "vectorString" in issue: - lines += "VECTORSTRING: %s\n" % issue["vectorString"] lines += "MORE INFORMATION: https://nvd.nist.gov/vuln/detail/%s\n" % issue["id"] lines += "\n"
Signature · 1 line
2.34.1
Quoted 10 lines · Colin McAllister via lists.openembedded.org
Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.
This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".
Signed-off-by Colin McAllister <[email protected]>
Hi Colin,
Thanks for your new patch. As for last week, it seems to be triggering some issues on the autobuilder:
ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_unpack(d)
0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
0101:do_fetch[vardeps] = ""
0102:
0103:python do_unpack() {
0104: import shutil
*** 0105: shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE"))
0106:}
0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}"
0108:
0109:def cleanup_db_download(db_file, db_tmp_file):
File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
0260:
0261: if not follow_symlinks and _islink(src):
0262: os.symlink(os.readlink(src), dst)
0263: else: *** 0264: with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
0265: # macOS
0266: if _HAS_FCOPYFILE:
0267: try:
0268: _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-3.db'
https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/stdio
Is this something you can fix ?
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
This is due to current NVD DB stability issues.
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database update failed This can happen only if API calls to NVD DB fail.
So either valkyrie infrastructure needs to increase the retry settings to 20+ (via CVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our infra), or this change needs to wait until NVD DB infra is fixed (which can take a loooong time). Of course the increase of timeout may mean the update job may take 3-4 hours more... But once it completes, the DB file will be cached and all should return to normal.
Peter
Quoted 72 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Mathieu Dubois-Briand via lists.openembedded.org
Sent: Monday, December 2, 2024 12:34
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vector stringsQuoted 10 lines · Colin McAllister via lists.openembedded.org
Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.
This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".
Signed-off-by Colin McAllister <[email protected]>Hi Colin,
Thanks for your new patch. As for last week, it seems to be triggering some issues on the autobuilder:
ERROR: cve-update-nvd2-native-1.0-r0
do_unpack: Error executing a python function inexec_func_python()autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_unpack(d) 0003: File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes- core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack 0101:do_fetch[vardeps] = "" 0102: 0103:python do_unpack() { 0104: import shutil *** 0105: shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE")) 0106:} 0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}" 0108: 0109:def cleanup_db_download(db_file, db_tmp_file): File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile 0260: 0261: if not follow_symlinks and _islink(src): 0262: os.symlink(os.readlink(src), dst) 0263: else:*** 0264: with
open(src, 'rb')as fsrc,open(dst, 'wb')as fdst:
0265: # macOS
0266: if_HAS_FCOPYFILE:
0267: try:
0268:_fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-3.db'https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st
dio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st
dioIs this something you can fix ?
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Hello,
Would the tests be able to be run again? I think the API appears to be more stable now. If the tests are still failing, I can rebase this patch on top of a
commit that changes the retry attempts to a higher number and send in both patches.
Besides the issue where the database fails to rebuild due to the NIST API's stability, is there any other feedback or are there any concerns with this patch?
Thanks,
Colin
Quoted 95 lines · Peter
This is due to current NVD DB stability issues.
WARNING: cve-update-nvd2-native-1.0-r0do_fetch: CVE database update failed This can happen only if API calls to NVD DB fail.So either valkyrie infrastructure needs to increase the retry settings to 20+
(viaCVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our infra),
or this change needs to wait until NVD DB infra is fixed (which can take a loooong time).
Of course the increase of timeout may mean the update job may take 3-4 hours more...
But once it completes, the DB file will be cached and all should return to normal.Peter
Quoted 8 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Mathieu Dubois-Briand via lists.openembedded.org
Sent: Monday, December 2, 2024 12:34
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vectorstrings
Quoted 14 lines
Quoted 10 lines · Colin McAllister via lists.openembedded.org
Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.
This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".
Signed-off-by Colin McAllister <[email protected]>Quoted 8 lines
Hi Colin,
Thanks for your new patch. As for last week, it seems to be triggering some issues on the autobuilder:
ERROR: cve-update-nvd2-native-1.0-r0
do_unpack: Error executing a python function inexec_func_python()autogenerated:
The stack trace of python calls that resulted in this exception/failurewas:
Quoted 22 lines
File: '
exec_func_python()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_unpack(d) 0003: File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes- core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack 0101:do_fetch[vardeps] = "" 0102: 0103:python do_unpack() { 0104: import shutil *** 0105: shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE")) 0106:} 0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}" 0108: 0109:def cleanup_db_download(db_file, db_tmp_file): File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile 0260: 0261: if not follow_symlinks and _islink(src): 0262: os.symlink(os.readlink(src), dst) 0263: else:Quoted 5 lines
*** 0264: with
open(src, 'rb')as fsrc,open(dst, 'wb')as fdst:
0265: # macOS
0266: if_HAS_FCOPYFILE:
0267: try:
0268: _fastcopy_fcopyfile(fsrc, fdst,posix._COPYFILE_DATA)
Quoted 5 lines
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-3.db'https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st
Quoted 2 lines
dio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st
Quoted 8 lines
dio
Is this something you can fix ?
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com