Skip to contentSkip to conversation

Patch review

PATCH In review Steve Sakoman 2 participants 7 messages Jan 5, 2024

lore
Share & tools
lore
#0 lore raw

cover Patch review

Please review this set of changes for dunfell and have comments back by end of day Tuesday, January 9

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6418

The following changes since commit 3ea36d92800b139eaaf75995cdd59912b63db9ee:

tzdata: Upgrade to 2023d (2024-01-02 03:43:26 -1000)

are available in the Git repository at:

https://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Dhairya Nagodra (2):
cve-update-nvd2-native: faster requests with API keys
cve-update-nvd2-native: increase the delay between subsequent request
failures

Dmitry Baryshkov (1):
linux-firmware: upgrade 20230804 -> 20231030

Peter Marko (2):
cve-update-nvd2-native: remove unused variable CVE_SOCKET_TIMEOUT
cve-update-nvd2-native: make number of fetch attemtps configurable

 .../meta/cve-update-nvd2-native.bb            | 27 +++++++++++++------
 ...20230804.bb => linux-firmware_20231030.bb} |  4 +--
 2 files changed, 21 insertions(+), 10 deletions(-)
 rename meta/recipes-kernel/linux-firmware/{linux-firmware_20230804.bb => linux-firmware_20231030.bb} (99%)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/5 cve-update-nvd2-native: remove unused variable CVE_SOCKET_TIMEOUT

From: Peter Marko <[email protected]>

This variable is not referenced in oe-core anymore.

(cherry picked from commit 905b45a814cb33327503b793741c19b44c8550b3)
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 3 ---
 1 file changed, 3 deletions(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +0 -3
index 67d76f75dd..64a96a46f0 100644
@@ -26,9 +26,6 @@ NVDCVE_API_KEY ?= ""
 # Use a negative value to skip the update
 CVE_DB_UPDATE_INTERVAL ?= "86400"
 
-# Timeout for blocking socket operations, such as the connection attempt.
-CVE_SOCKET_TIMEOUT ?= "60"
-
 CVE_DB_TEMP_FILE ?= "${CVE_CHECK_DB_DIR}/temp_nvdcve_2.db"
 
 CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_2.db"
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/5 cve-update-nvd2-native: make number of fetch attemtps configurable

In reply to Steve Sakoman

From: Peter Marko <[email protected]>

Sometimes NVD servers are unstable and return too many errors.

Last time we increased number of attempts from 3 to 5, but further increasing is not reasonable as in normal case too many retries is just abusive.

Keep retries low as default and allow to increase as needed.

(cherry picked from commit 6b6fd8043d83b99000054ab6ad2c745d07c6bcc1)
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +7 -3
index 64a96a46f0..dab0b69edc 100644
@@ -26,6 +26,9 @@ NVDCVE_API_KEY ?= ""
 # Use a negative value to skip the update
 CVE_DB_UPDATE_INTERVAL ?= "86400"
 
+# Number of attmepts for each http query to nvd server before giving up
+CVE_DB_UPDATE_ATTEMPTS ?= "5"
+
 CVE_DB_TEMP_FILE ?= "${CVE_CHECK_DB_DIR}/temp_nvdcve_2.db"
 
 CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_2.db"
@@ -111,7 +114,7 @@ def cleanup_db_download(db_file, db_tmp_file):
     if os.path.exists(db_tmp_file):
         os.remove(db_tmp_file)
 
-def nvd_request_next(url, api_key, args):
+def nvd_request_next(url, attempts, api_key, args):
     """
     Request next part of the NVD dabase
     """
@@ -127,7 +130,7 @@ def nvd_request_next(url, api_key, args):
         request.add_header("apiKey", api_key)
     bb.note("Requesting %s" % request.full_url)
 
-    for attempt in range(5):
+    for attempt in range(attempts):
         try:
             r = urllib.request.urlopen(request)
 
@@ -183,10 +186,11 @@ def update_db_file(db_tmp_file, d, database_time):
         index = 0
         url = d.getVar("NVDCVE_URL")
         api_key = d.getVar("NVDCVE_API_KEY") or None
+        attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))
 
         while True:
             req_args['startIndex'] = index
-            raw_data = nvd_request_next(url, api_key, req_args)
+            raw_data = nvd_request_next(url, attempts, api_key, req_args)
             if raw_data is None:
                 # We haven't managed to download data
                 return False
Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/5 cve-update-nvd2-native: faster requests with API keys

In reply to Steve Sakoman

From: Dhairya Nagodra <[email protected]>

As per NVD, the public rate limit is 5 requests in 30s (6s delay). Using an API key increases the limit to 50 requests in 30s (0.6s delay). However, NVD still recommends sleeping for several seconds so that the other legitimate requests are serviced without denial or interruption. Keeping the default sleep at 6 seconds and 2 seconds with an API key.

For failures, the wait time is unchanged (6 seconds).

Reference: https://nvd.nist.gov/developers/start-here#RateLimits

(cherry picked from commit 5c32e2941d1dc3d04a799a1b7cbd275c1ccc9e79)
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +6 -1
index dab0b69edc..0a8b6a8a0a 100644
@@ -188,6 +188,11 @@ def update_db_file(db_tmp_file, d, database_time):
         api_key = d.getVar("NVDCVE_API_KEY") or None
         attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))
 
+        # Recommended by NVD
+        wait_time = 6
+        if api_key:
+            wait_time = 2
+
         while True:
             req_args['startIndex'] = index
             raw_data = nvd_request_next(url, attempts, api_key, req_args)
@@ -210,7 +215,7 @@ def update_db_file(db_tmp_file, d, database_time):
                break
 
             # Recommended by NVD
-            time.sleep(6)
+            time.sleep(wait_time)
 
         # Update success, set the date to cve_check file.
         cve_f.write('CVE database update : %s\n\n' % datetime.date.today())
Signature · 1 line
2.34.1
#4 lore raw

PATCH 4/5 cve-update-nvd2-native: increase the delay between subsequent request failures

In reply to Steve Sakoman

From: Dhairya Nagodra <[email protected]>

Sometimes NVD servers are unstable and return too many errors. There is an option to have higher fetch attempts to increase the chances of successfully fetching the CVE data.

Additionally, it also makes sense to progressively increase the delay after a failed request to an already unstable or busy server. The increase in delay is reset after every successful request and the maximum delay is limited to 30 seconds.

Also, the logs are improved to give more clarity.

(cherry picked from commit 7101d654635b707e56b0dbae8c2146b312d211ea)
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
meta/recipes-core/meta/cve-update-nvd2-native.bb +9 -4
index 0a8b6a8a0a..69ba20a6cb 100644
@@ -114,7 +114,10 @@ def cleanup_db_download(db_file, db_tmp_file):
     if os.path.exists(db_tmp_file):
         os.remove(db_tmp_file)
 
-def nvd_request_next(url, attempts, api_key, args):
+def nvd_request_wait(attempt, min_wait):
+    return min ( ( (2 * attempt) + min_wait ) , 30)
+
+def nvd_request_next(url, attempts, api_key, args, min_wait):
     """
     Request next part of the NVD dabase
     """
@@ -143,8 +146,10 @@ def nvd_request_next(url, attempts, api_key, args):
             r.close()
 
         except Exception as e:
-            bb.note("CVE database: received error (%s), retrying" % (e))
-            time.sleep(6)
+            wait_time = nvd_request_wait(attempt, min_wait)
+            bb.note("CVE database: received error (%s)" % (e))
+            bb.note("CVE database: retrying download after %d seconds. attempted (%d/%d)" % (wait_time, attempt+1, attempts))
+            time.sleep(wait_time)
             pass
         else:
             return raw_data
@@ -195,7 +200,7 @@ def update_db_file(db_tmp_file, d, database_time):
 
         while True:
             req_args['startIndex'] = index
-            raw_data = nvd_request_next(url, attempts, api_key, req_args)
+            raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time)
             if raw_data is None:
                 # We haven't managed to download data
                 return False
Signature · 1 line
2.34.1
#5 lore raw

Patchtest results for [OE-core][dunfell 4/5] cve-update-nvd2-native: increase the delay between subsequent request failures

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/dunfell-4-5-cve-update-nvd2-native-increase-the-delay-between-subsequent-request-failures.patch

FAIL: test shortlog length: Edit shortlog so that it is 90 characters or less (currently 92 characters) (test_mbox.TestMbox.test_shortlog_length)

PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#6 lore raw

PATCH 5/5 linux-firmware: upgrade 20230804 -> 20231030

In reply to Steve Sakoman

From: Dmitry Baryshkov <[email protected]>

License-Update: additional firmwares

(cherry picked from commit 7c725d1f2ed9a271d39d899ac2534558c2d103fc)
---
 ...{linux-firmware_20230804.bb => linux-firmware_20231030.bb} | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
 rename meta/recipes-kernel/linux-firmware/{linux-firmware_20230804.bb => linux-firmware_20231030.bb} (99%)
meta/recipes-kernel/linux-firmware/linux-firmware_20231030.bb renamed from meta/recipes-kernel/linux-firmware/linux-firmware_20230804.bb +2 -2
similarity index 99%
rename from meta/recipes-kernel/linux-firmware/linux-firmware_20230804.bb
rename to meta/recipes-kernel/linux-firmware/linux-firmware_20231030.bb
index 507a003224..65cbca798e 100644
@@ -134,7 +134,7 @@ LIC_FILES_CHKSUM = "file://LICENCE.Abilis;md5=b5ee3f410780e56711ad48eadc22b8bc \
                     "
 # WHENCE checksum is defined separately to ease overriding it if
 # class-devupstream is selected.
-WHENCE_CHKSUM  = "41f9a48bf27971b126a36f9344594dcd"
+WHENCE_CHKSUM  = "ceb5248746d24d165b603e71b288cf75"
 
 # These are not common licenses, set NO_GENERIC_LICENSE for them
 # so that the license files will be copied from fetched source
@@ -212,7 +212,7 @@ SRC_URI:class-devupstream = "git://git.kernel.org/pub/scm/linux/kernel/git/firmw
 # Pin this to the 20220509 release, override this in local.conf
 SRCREV:class-devupstream ?= "b19cbdca78ab2adfd210c91be15a22568e8b8cae"
 
-SRC_URI[sha256sum] = "88d46c543847ee3b03404d4941d91c92974690ee1f6fdcbee9cef3e5f97db688"
+SRC_URI[sha256sum] = "c98d200fc4a3120de1a594713ce34e135819dff23e883a4ed387863ba25679c7"
 
 inherit allarch
 
Signature · 1 line
2.34.1