cve-update-nvd2-native: re-introduce CVE_SOCKET_TIMEOUT to bound urlopen()
PATCH 1/1 cve-update-nvd2-native: re-introduce CVE_SOCKET_TIMEOUT to bound urlopen()
From: Mehmet Fide <[email protected]>
The fetch task calls urllib.request.urlopen() with no timeout argument, so when an NVD endpoint accepts the TCP connection but stops sending data, the call blocks forever and the existing retry loop driven by CVE_DB_UPDATE_ATTEMPTS never gets a chance to run. We observed worker processes wedged for over an hour on a single recv() syscall before the build was killed manually.
Re-introduce the CVE_SOCKET_TIMEOUT variable (removed in commit d6d94eed1e "cve-update-nvd2-native: remove unused variable CVE_SOCKET_TIMEOUT" as it was a leftover from the JSON 1.1 feed) and plumb it through update_db_file() and nvd_request_next() so it is actually honoured by urlopen(). The default of 60 seconds matches the prior historical default; users behind slow proxies may raise it.
With the timeout in place, a stalled NVD endpoint produces a clean exception, the retry loop runs, and after CVE_DB_UPDATE_ATTEMPTS failures the task returns False and the build falls back to the previously cached database (bb.warn, not a hard error).
--- meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +10 -3
@@ -34,6 +34,12 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" # Number of attempts for each http query to nvd server before giving up CVE_DB_UPDATE_ATTEMPTS ?= "5" +# Per-request socket timeout (seconds) for HTTP queries to the NVD server. +# Without this, urllib uses the global default (None) and a stalled connection +# can block the do_fetch task indefinitely, preventing the retry loop driven +# by CVE_DB_UPDATE_ATTEMPTS from ever running. +CVE_SOCKET_TIMEOUT ?= "60" + CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
@@ -134,7 +140,7 @@ def cleanup_db_download(db_file, db_tmp_file): def nvd_request_wait(attempt, min_wait): return min ( ( (2 * attempt) + min_wait ) , 30) -def nvd_request_next(url, attempts, api_key, args, min_wait): +def nvd_request_next(url, attempts, api_key, args, min_wait, timeout): """ Request next part of the NVD database NVD API documentation: https://nvd.nist.gov/developers/vulnerabilities
@@ -153,7 +159,7 @@ def nvd_request_next(url, attempts, api_key, args, min_wait): for attempt in range(attempts): try: - r = urllib.request.urlopen(request) + r = urllib.request.urlopen(request, timeout=timeout) if (r.headers['content-encoding'] == 'gzip'): buf = r.read()
@@ -216,6 +222,7 @@ def update_db_file(db_tmp_file, d, database_time): url = d.getVar("NVDCVE_URL") api_key = d.getVar("NVDCVE_API_KEY") or None attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS")) + timeout = int(d.getVar("CVE_SOCKET_TIMEOUT")) # Recommended by NVD wait_time = 6
@@ -224,7 +231,7 @@ def update_db_file(db_tmp_file, d, database_time): while True: req_args['startIndex'] = index - raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time) + raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time, timeout) if raw_data is None: # We haven't managed to download data return False
Signature · 1 line
2.54.0
Quoted 16 lines · Mehmet Fide via lists.openembedded.org
From: Mehmet Fide <[email protected]>
The fetch task calls
urllib.request.urlopen()with no timeout argument, so when an NVD endpoint accepts the TCP connection but stops sending data, the call blocks forever and the existing retry loop driven byCVE_DB_UPDATE_ATTEMPTSnever gets a chance to run. We observed worker processes wedged for over an hour on a singlerecv()syscall before the build was killed manually.Re-introduce the
CVE_SOCKET_TIMEOUTvariable (removed in commitd6d94eed1e"cve-update-nvd2-native: remove unused variableCVE_SOCKET_TIMEOUT" as it was a leftover from the JSON 1.1 feed) and plumb it throughupdate_db_file()andnvd_request_next()so it is actually honoured byurlopen(). The default of 60 seconds matches the prior historical default; users behind slow proxies may raise it.
Quoted 6 lines
With the timeout in place, a stalled NVD endpoint produces a clean exception, the retry loop runs, and after
CVE_DB_UPDATE_ATTEMPTSfailures the task returns False and the build falls back to the previously cached database (bb.warn, not a hard error).Signed-off-by Mehmet Fide <[email protected]>
Hello,
Walnascar is EOL so we can't really take this patch, sorry.
That said, given the recent NVD API glitches, it does look desirable on the last supported stable using the NVD API: scarthgap. Can you please rebase & send your patch there?
Thanks!
Quoted 56 lines
--- meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)meta/recipes-core/meta/cve-update-nvd2-native.bb +10
-3@@ -34,6 +34,12 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" # Number of attempts for each http query to nvd server before giving up CVE_DB_UPDATE_ATTEMPTS ?= "5" +# Per-request socket timeout (seconds) for HTTP queries to the NVD server. +# Without this, urllib uses the global default (None) and a stalled connection +# can block the do_fetch task indefinitely, preventing the retry loop driven +# by CVE_DB_UPDATE_ATTEMPTS from ever running. +CVE_SOCKET_TIMEOUT ?= "60" + CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}" CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock" CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"@@ -134,7 +140,7 @@ def cleanup_db_download(db_file, db_tmp_file): def nvd_request_wait(attempt, min_wait): return min ( ( (2 * attempt) + min_wait ) , 30) -def nvd_request_next(url, attempts, api_key, args, min_wait): +def nvd_request_next(url, attempts, api_key, args, min_wait, timeout): """ Request next part of the NVD database NVD API documentation: https://nvd.nist.gov/developers/vulnerabilities@@ -153,7 +159,7 @@ def nvd_request_next(url, attempts, api_key, args, min_wait): for attempt in range(attempts): try: - r = urllib.request.urlopen(request) + r = urllib.request.urlopen(request, timeout=timeout) if (r.headers['content-encoding'] == 'gzip'): buf = r.read()@@ -216,6 +222,7 @@ def update_db_file(db_tmp_file, d, database_time): url = d.getVar("NVDCVE_URL") api_key = d.getVar("NVDCVE_API_KEY") or None attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS")) + timeout = int(d.getVar("CVE_SOCKET_TIMEOUT")) # Recommended by NVD wait_time = 6@@ -224,7 +231,7 @@ def update_db_file(db_tmp_file, d, database_time): while True: req_args['startIndex'] = index - raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time) + raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time, timeout) if raw_data is None: # We haven't managed to download data return False
Signature · 2 lines
Yoann Congal Smile ECS