Skip to contentSkip to conversation

cve-check: Add versioned CVSS vector strings

PATCH In review Colin McAllister 3 participants 4 messages Nov 30, 2024 – Dec 9, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cve-check: Add versioned CVSS vector strings

Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.

This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".

---
This is an alternative patch to "cve-update-nvd2-native: Update vector logic" where each versioned vector string is attempted to be included. This does introduce a breaking API change, where vectorString has been removed and replaced with the versioned vectorString variables. I tried to update all references to the current use of vector strings, but I was a little confused looking at meta/lib/oe/spdx30.py. I'm not sure if this change will affect that file. I did turn on SPDX generation and ran another build. I looked at the outputs, but didn't see any vector string output.

 meta/classes/cve-check.bbclass                | 20 +++++----
 meta/classes/vex.bbclass                      |  4 +-
 .../meta/cve-update-nvd2-native.bb            | 42 ++++++++++++++-----
 scripts/cve-json-to-text.py                   |  8 +++-
 4 files changed, 52 insertions(+), 22 deletions(-)
meta/classes/cve-check.bbclass +12 -8
index 6e10dd915a..eacfa7b041 100644
@@ -31,7 +31,7 @@
 CVE_PRODUCT ??= "${BPN}"
 CVE_VERSION ??= "${PV}"
 
-CVE_CHECK_DB_FILENAME ?= "nvdcve_2-2.db"
+CVE_CHECK_DB_FILENAME ?= "nvdcve_2-3.db"
 CVE_CHECK_DB_DIR ?= "${STAGING_DIR}/CVE_CHECK"
 CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/${CVE_CHECK_DB_FILENAME}"
 CVE_CHECK_DB_FILE_LOCK ?= "${CVE_CHECK_DB_FILE}.lock"
@@ -449,12 +449,14 @@ def get_cve_info(d, cve_data):
                 continue
             #cve_data[row[0]] = {}
             cve_data[row[0]]["NVD-summary"] = row[1]
-            cve_data[row[0]]["NVD-scorev2"] = row[2]
-            cve_data[row[0]]["NVD-scorev3"] = row[3]
-            cve_data[row[0]]["NVD-scorev4"] = row[4]
-            cve_data[row[0]]["NVD-modified"] = row[5]
-            cve_data[row[0]]["NVD-vector"] = row[6]
-            cve_data[row[0]]["NVD-vectorString"] = row[7]
+            cve_data[row[0]]["NVD-vectorStringV2"] = row[2]
+            cve_data[row[0]]["NVD-scorev2"] = row[3]
+            cve_data[row[0]]["NVD-vectorStringV3"] = row[4]
+            cve_data[row[0]]["NVD-scorev3"] = row[5]
+            cve_data[row[0]]["NVD-vectorStringV4"] = row[6]
+            cve_data[row[0]]["NVD-scorev4"] = row[7]
+            cve_data[row[0]]["NVD-modified"] = row[8]
+            cve_data[row[0]]["NVD-vector"] = row[9]
         cursor.close()
     conn.close()
 
@@ -539,12 +541,14 @@ def cve_write_data_json(d, cve_data, cve_status):
         }
         if 'NVD-summary' in cve_data[cve]:
             cve_item["summary"] = cve_data[cve]["NVD-summary"]
+            cve_item["vectorStringV2"] = cve_data[cve]["NVD-vectorStringV2"]
             cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"]
+            cve_item["vectorStringV3"] = cve_data[cve]["NVD-vectorStringV3"]
             cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"]
+            cve_item["vectorStringV4"] = cve_data[cve]["NVD-vectorStringV4"]
             cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"]
             cve_item["modified"] = cve_data[cve]["NVD-modified"]
             cve_item["vector"] = cve_data[cve]["NVD-vector"]
-            cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"]
         if 'status' in cve_data[cve]:
             cve_item["detail"] = cve_data[cve]["status"]
         if 'justification' in cve_data[cve]:
meta/classes/vex.bbclass +3 -1
index 01d4e52051..5e0131788d 100644
@@ -280,11 +280,13 @@ def cve_write_data_json(d, cve_data, cve_status):
         }
         if 'NVD-summary' in cve_data[cve]:
             cve_item["summary"] = cve_data[cve]["NVD-summary"]
+            cve_item["vectorStringV2"] = cve_data[cve]["NVD-vectorStringV2"]
             cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"]
+            cve_item["vectorStringV3"] = cve_data[cve]["NVD-vectorStringV3"]
             cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"]
+            cve_item["vectorStringV4"] = cve_data[cve]["NVD-vectorStringV4"]
             cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"]
             cve_item["vector"] = cve_data[cve]["NVD-vector"]
-            cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"]
         if 'status' in cve_data[cve]:
             cve_item["detail"] = cve_data[cve]["status"]
         if 'justification' in cve_data[cve]:
meta/recipes-core/meta/cve-update-nvd2-native.bb +31 -11

81 lines not on this page · raw message

scripts/cve-json-to-text.py +6 -2
index 87a5669987..3bab2426d6 100755
@@ -121,16 +121,20 @@ def process_data(filename, data):
                 lines += "CVE DESCRIPTION: %s\n" % issue["description"]
             if "summary" in issue:
                 lines += "CVE SUMMARY: %s\n" % issue["summary"]
+            if "vectorStringV2" in issue:
+                lines += "VECTORSTRING v2: %s\n" % issue["vectorStringV2"]
             if "scorev2" in issue:
                 lines += "CVSS v2 BASE SCORE: %s\n" % issue["scorev2"]
+            if "vectorStringV3" in issue:
+                lines += "VECTORSTRING v3: %s\n" % issue["vectorStringV3"]
             if "scorev3" in issue:
                 lines += "CVSS v3 BASE SCORE: %s\n" % issue["scorev3"]
+            if "vectorStringV4" in issue:
+                lines += "VECTORSTRING v4: %s\n" % issue["vectorStringV4"]
             if "scorev4" in issue:
                 lines += "CVSS v4 BASE SCORE: %s\n" % issue["scorev4"]
             if "vector" in issue:
                 lines += "VECTOR: %s\n" % issue["vector"]
-            if "vectorString" in issue:
-                lines += "VECTORSTRING: %s\n" % issue["vectorString"]
             lines += "MORE INFORMATION: https://nvd.nist.gov/vuln/detail/%s\n" % issue["id"]
             lines += "\n"
 
Signature · 1 line
2.34.1
#1 lore raw
Quoted 10 lines · Colin McAllister via lists.openembedded.org

Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.

This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".

Hi Colin,

Thanks for your new patch. As for last week, it seems to be triggering some issues on the autobuilder:

ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_unpack(d)
     0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
     0101:do_fetch[vardeps] = ""
     0102:
     0103:python do_unpack() {
     0104:    import shutil
 *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE"))
     0106:}
     0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}"
     0108:
     0109:def cleanup_db_download(db_file, db_tmp_file):
File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
     0260:
     0261:    if not follow_symlinks and _islink(src):
     0262:        os.symlink(os.readlink(src), dst)
     0263:    else:

*** 0264: with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
0265: # macOS
0266: if _HAS_FCOPYFILE:
0267: try:
0268: _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-3.db'

https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/stdio

Is this something you can fix ?

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

This is due to current NVD DB stability issues.
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database update failed This can happen only if API calls to NVD DB fail.

So either valkyrie infrastructure needs to increase the retry settings to 20+ (via CVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our infra), or this change needs to wait until NVD DB infra is fixed (which can take a loooong time). Of course the increase of timeout may mean the update job may take 3-4 hours more... But once it completes, the DB file will be cached and all should return to normal.

Peter

Quoted 72 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Mathieu Dubois-Briand via lists.openembedded.org
Sent: Monday, December 2, 2024 12:34
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vector strings

Quoted 10 lines · Colin McAllister via lists.openembedded.org

Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.

This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".

Hi Colin,

Thanks for your new patch. As for last week, it seems to be triggering some issues on the autobuilder:

ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_unpack(d)
     0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-
core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
     0101:do_fetch[vardeps] = ""
     0102:
     0103:python do_unpack() {
     0104:    import shutil
 *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"),
d.getVar("CVE_CHECK_DB_FILE"))
     0106:}
     0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK}
${CVE_CHECK_DB_FILE_LOCK}"
     0108:
     0109:def cleanup_db_download(db_file, db_tmp_file):
File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
     0260:
     0261:    if not follow_symlinks and _islink(src):
     0262:        os.symlink(os.readlink(src), dst)
     0263:    else:

*** 0264: with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
0265: # macOS
0266: if _HAS_FCOPYFILE:
0267: try:
0268: _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2- 3.db'

https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st
dio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st
dio

Is this something you can fix ?

--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

author
#3 lore raw

Hello,

Would the tests be able to be run again? I think the API appears to be more stable now. If the tests are still failing, I can rebase this patch on top of a
commit that changes the retry attempts to a higher number and send in both patches.

Besides the issue where the database fails to rebuild due to the NIST API's stability, is there any other feedback or are there any concerns with this patch?

Thanks,
Colin

Quoted 95 lines · Peter

This is due to current NVD DB stability issues.
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database update failed This can happen only if API calls to NVD DB fail.

So either valkyrie infrastructure needs to increase the retry settings to 20+
(via CVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our infra),
or this change needs to wait until NVD DB infra is fixed (which can take a loooong time).
Of course the increase of timeout may mean the update job may take 3-4 hours more...
But once it completes, the DB file will be cached and all should return to normal.

Peter

Quoted 8 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Mathieu Dubois-Briand via lists.openembedded.org
Sent: Monday, December 2, 2024 12:34
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vector

strings

Quoted 14 lines
Quoted 10 lines · Colin McAllister via lists.openembedded.org

Currently, cve-check includes a vector string for each CVE included in the issue list for each package. This vector string is the lowest CVSS version that's available. For example, if a CVE has both a v2 and v3.1 vector strint, the v2 vector string is only included.

This patch adds each supported vector string (v2, v3, and v4). For v3, v3.1 is preferred over v3. If a vector string is not available for a given verison, the string will default to "UNKNOWN".

Quoted 8 lines

Hi Colin,

Thanks for your new patch. As for last week, it seems to be triggering some issues on the autobuilder:

ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure

was:

Quoted 22 lines

File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_unpack(d)
     0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-
core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
     0101:do_fetch[vardeps] = ""
     0102:
     0103:python do_unpack() {
     0104:    import shutil
 *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"),
d.getVar("CVE_CHECK_DB_FILE"))
     0106:}
     0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK}
${CVE_CHECK_DB_FILE_LOCK}"
     0108:
     0109:def cleanup_db_download(db_file, db_tmp_file):
File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
     0260:
     0261:    if not follow_symlinks and _islink(src):
     0262:        os.symlink(os.readlink(src), dst)
     0263:    else:
Quoted 5 lines

*** 0264: with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
0265: # macOS
0266: if _HAS_FCOPYFILE:
0267: try:
0268: _fastcopy_fcopyfile(fsrc, fdst,

posix._COPYFILE_DATA)

Quoted 5 lines

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2- 3.db'

https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st

Quoted 2 lines

dio

https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st

Quoted 8 lines

dio

Is this something you can fix ?

--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com