Skip to contentSkip to conversation

cve-update-nvd2-native: allow setting resultsPerPage

PATCH In review Awais B 2 participants 2 messages Jun 25 – Jun 29

lore
Share & tools
lore

Awais B

author
#0 lore raw

PATCH v2 1/1 cve-update-nvd2-native: allow setting resultsPerPage

It is seen that during bulk updates on the NVD side the server struggles to keep up with the default/max of 2000 entries per page and we see a lot of incomplete read errors resulting in proper db sync failures most of the times. Lowering the per page value noticably increases the reliability of the process and hence should ideally be configurable.

---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 +++++++++++++
 1 file changed, 13 insertions(+)
meta/recipes-core/meta/cve-update-nvd2-native.bb +13 -0
index 945bd1d927..731cbb5d88 100644
@@ -34,6 +34,10 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
 # Number of attempts for each http query to nvd server before giving up
 CVE_DB_UPDATE_ATTEMPTS ?= "5"
 
+# Maximum number of CVE records per API response.
+# Lowering this value can help avoid incomplete read errors during bulk NVD updates.
+CVE_DB_RESULTS_PER_PAGE ?= ""
+
 CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}"
 CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
 CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
@@ -217,6 +221,15 @@ def update_db_file(db_tmp_file, d, database_time):
         api_key = d.getVar("NVDCVE_API_KEY") or None
         attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))
 
+        results_per_page = d.getVar("CVE_DB_RESULTS_PER_PAGE")
+        RESULTS_PER_PAGE_MAX = 2000 # imposed by NVD
+        if results_per_page:
+            results_per_page = int(results_per_page)
+            if results_per_page > RESULTS_PER_PAGE_MAX:
+                bb.warn("CVE_DB_RESULTS_PER_PAGE exceeds maximum of %d, capping" % RESULTS_PER_PAGE_MAX)
+                results_per_page = RESULTS_PER_PAGE_MAX
+            req_args['resultsPerPage'] = results_per_page
+
         # Recommended by NVD
         wait_time = 6
         if api_key:
Signature · 1 line
2.34.1
#1 lore raw

Hi, thanks for the patch!

Quoted 17 lines · Awais B

It is seen that during bulk updates on the NVD side the server struggles to keep up with the default/max of 2000 entries per page and we see a lot of incomplete read errors resulting in proper db sync failures most of the times. Lowering the per page value noticably increases the reliability of the process and hence should ideally be configurable.

---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 +++++++++++++
 1 file changed, 13 insertions(+)
meta/recipes-core/meta/cve-update-nvd2-native.bb +0 -0
index 945bd1d927..731cbb5d88 100644

meta/recipes-core/meta/cve-update-nvd2-native.bb CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" · line 39

Quoted 6 lines
meta/recipes-core/meta/cve-update-nvd2-native.bb +3 -0
@@ -34,6 +34,10 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
 # Number of attempts for each http query to nvd server before giving up
 CVE_DB_UPDATE_ATTEMPTS ?= "5"
 
+# Maximum number of CVE records per API response.
+# Lowering this value can help avoid incomplete read errors during bulk NVD updates.
+CVE_DB_RESULTS_PER_PAGE ?= ""

It'd be good to document the max value in the comment here so users can easily spot it.

Quoted 7 lines
meta/recipes-core/meta/cve-update-nvd2-native.bb +1 -0
+
 CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}"
 CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
 CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
@@ -217,6 +221,15 @@ def update_db_file(db_tmp_file, d, database_time):
         api_key = d.getVar("NVDCVE_API_KEY") or None
         attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))

meta/recipes-core/meta/cve-update-nvd2-native.bb def update_db_file(db_tmp_file, d, database_time): · line 232

Quoted 10 lines
meta/recipes-core/meta/cve-update-nvd2-native.bb +9 -0
 
+        results_per_page = d.getVar("CVE_DB_RESULTS_PER_PAGE")
+        RESULTS_PER_PAGE_MAX = 2000 # imposed by NVD
+        if results_per_page:
+            results_per_page = int(results_per_page)
+            if results_per_page > RESULTS_PER_PAGE_MAX:
+                bb.warn("CVE_DB_RESULTS_PER_PAGE exceeds maximum of %d, capping" % RESULTS_PER_PAGE_MAX)
+                results_per_page = RESULTS_PER_PAGE_MAX
+            req_args['resultsPerPage'] = results_per_page
+

This code can raise ValueError if CVE_DB_RESULTS_PER_PAGE is not convertible to an integer. That should be handled.

Best regards,

Signature · 1 line
Paul Barker