cve-update-nvd2-native: faster requests with API keys
PATCH v2 1/2 cve-update-nvd2-native: faster requests with API keys
From: Dhairya Nagodra <[email protected]>
As per NVD, the public rate limit is 5 requests in 30s (6s delay). Using an API key increases the limit to 50 requests in 30s (0.6s delay). However, NVD still recommends sleeping for several seconds so that the other legitimate requests are serviced without denial or interruption. Keeping the default sleep at 6 seconds and 2 seconds with an API key.
For failures, the wait time is unchanged (6 seconds).
Reference: https://nvd.nist.gov/developers/start-here#RateLimits
--- meta/recipes-core/meta/cve-update-nvd2-native.bb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +6 -1
@@ -188,6 +188,11 @@ def update_db_file(db_tmp_file, d, database_time): api_key = d.getVar("NVDCVE_API_KEY") or None attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS")) + # Recommended by NVD + wait_time = 6 + if api_key: + wait_time = 2 + while True: req_args['startIndex'] = index raw_data = nvd_request_next(url, attempts, api_key, req_args)
@@ -210,7 +215,7 @@ def update_db_file(db_tmp_file, d, database_time): break # Recommended by NVD - time.sleep(6) + time.sleep(wait_time) # Update success, set the date to cve_check file. cve_f.write('CVE database update : %s\n\n' % datetime.date.today())
Signature · 1 line
2.35.6
PATCH v2 2/2 cve-update-nvd2-native: increase the delay between subsequent request failures
From: Dhairya Nagodra <[email protected]>
Sometimes NVD servers are unstable and return too many errors. There is an option to have higher fetch attempts to increase the chances of successfully fetching the CVE data.
Additionally, it also makes sense to progressively increase the delay after a failed request to an already unstable or busy server. The increase in delay is reset after every successful request and the maximum delay is limited to 30 seconds.
Also, the logs are improved to give more clarity.
--- meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +9 -4
@@ -114,7 +114,10 @@ def cleanup_db_download(db_file, db_tmp_file): if os.path.exists(db_tmp_file): os.remove(db_tmp_file) -def nvd_request_next(url, attempts, api_key, args): +def nvd_request_wait(attempt, min_wait): + return min ( ( (2 * attempt) + min_wait ) , 30) + +def nvd_request_next(url, attempts, api_key, args, min_wait): """ Request next part of the NVD dabase """
@@ -143,8 +146,10 @@ def nvd_request_next(url, attempts, api_key, args): r.close() except Exception as e: - bb.note("CVE database: received error (%s), retrying" % (e)) - time.sleep(6) + wait_time = nvd_request_wait(attempt, min_wait) + bb.note("CVE database: received error (%s)" % (e)) + bb.note("CVE database: retrying download after %d seconds. attempted (%d/%d)" % (wait_time, attempt+1, attempts)) + time.sleep(wait_time) pass else: return raw_data
@@ -195,7 +200,7 @@ def update_db_file(db_tmp_file, d, database_time): while True: req_args['startIndex'] = index - raw_data = nvd_request_next(url, attempts, api_key, req_args) + raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time) if raw_data is None: # We haven't managed to download data return False
Signature · 1 line
2.35.6