Skip to contentSkip to conversation

classes: cve-check: Get shared database lock

PATCH In review Joshua Watt 4 participants 5 messages Aug 30, 2022 – Sep 2, 2022 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 classes: cve-check: Get shared database lock

The CVE check database needs to have a shared lock acquired on it before it is accessed. This to prevent cve-update-db-native from deleting the database file out from underneath it.

[YOCTO #14899]

---
 meta/classes/cve-check.bbclass | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)
meta/classes/cve-check.bbclass +12 -11
index d95465775d..5c8b512c11 100644
@@ -145,17 +145,18 @@ python do_cve_check () {
     """
     from oe.cve_check import get_patched_cves
 
-    if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
-        try:
-            patched_cves = get_patched_cves(d)
-        except FileNotFoundError:
-            bb.fatal("Failure in searching patches")
-        ignored, patched, unpatched, status = check_cves(d, patched_cves)
-        if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status):
-            cve_data = get_cve_info(d, patched + unpatched + ignored)
-            cve_write_data(d, patched, unpatched, ignored, cve_data, status)
-    else:
-        bb.note("No CVE database found, skipping CVE check")
+    with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True):
+        if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
+            try:
+                patched_cves = get_patched_cves(d)
+            except FileNotFoundError:
+                bb.fatal("Failure in searching patches")
+            ignored, patched, unpatched, status = check_cves(d, patched_cves)
+            if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status):
+                cve_data = get_cve_info(d, patched + unpatched + ignored)
+                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
+        else:
+            bb.note("No CVE database found, skipping CVE check")
 
 }
 
Signature · 1 line
2.33.0
#1 lore raw

meta/classes/cve-check.bbclass python do_cve_check () { · line 159

Quoted 13 lines · Joshua Watt

The CVE check database needs to have a shared lock acquired on it before it is accessed. This to prevent cve-update-db-native from deleting the database file out from underneath it.

[YOCTO #14899]

meta/classes/cve-check.bbclass +4 -0
+                cve_data = get_cve_info(d, patched + unpatched + ignored)
+                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
+        else:
+            bb.note("No CVE database found, skipping CVE check")

With this commit in kirkstone-nut, we're getting an error with meta-zephyr builds:

RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
do_cve_check: Error executing a python function in exec_func_python()
autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_cve_check(d)
0003:
File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
lineno: 142, function: do_cve_check
0138: Check recipe for patched and unpatched CVEs
0139: """
0140: from oe.cve_check import get_patched_cves
0141:
*** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")],
shared=True):
0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
0144: try:
0145: patched_cves = get_patched_cves(d)
0146: except FileNotFoundError:
File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper
0236: <cleanup>
0237: """
0238: @wraps(func)
0239: def helper(*args, **kwds):
*** 0240: return _GeneratorContextManager(func, args, kwds)
0241: return helper
0242:
0243:
0244:def asynccontextmanager(func):
File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__
0079:class _GeneratorContextManagerBase:
0080: """Shared functionality for @contextmanager and @asynccontextmanager."""
0081:
0082: def __init__(self, func, args, kwds):
*** 0083: self.gen = func(*args, **kwds)
0084: self.func, self.args, self.kwds = func, args, kwds
0085: # Issue 19330: ensure context manager instances have good docstrings
0086: doc = getattr(func, "__doc__", None)
0087: if doc is None:
Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
ERROR: Logfile of failure stored in:
/tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
task do_cve_check: Failed
ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) failed with exit code '1'

Kind regards,
Marta

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 58 lines · Marta Rybczynska via lists.openembedded.org

meta/classes/cve-check.bbclass python do_cve_check () { · line 159

Quoted 13 lines · Joshua Watt

The CVE check database needs to have a shared lock acquired on it before it is accessed. This to prevent cve-update-db-native from deleting the database file out from underneath it.

[YOCTO #14899]

meta/classes/cve-check.bbclass +4 -0
+                cve_data = get_cve_info(d, patched + unpatched + ignored)
+                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
+        else:
+            bb.note("No CVE database found, skipping CVE check")

With this commit in kirkstone-nut, we're getting an error with meta-zephyr builds:

RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
do_cve_check: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001:
*** 0002:do_cve_check(d)
0003:
File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
lineno: 142, function: do_cve_check
0138: Check recipe for patched and unpatched CVEs
0139: """
0140: from oe.cve_check import get_patched_cves
0141:
*** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True):
0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
0144: try:
0145: patched_cves = get_patched_cves(d)
0146: except FileNotFoundError:
File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper 0236: <cleanup>
0237: """
0238: @wraps(func)
0239: def helper(*args, **kwds):
*** 0240: return _GeneratorContextManager(func, args, kwds)
0241: return helper
0242:
0243:
0244:def asynccontextmanager(func):
File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ 0079:class _GeneratorContextManagerBase:
0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" 0081:
0082: def __init__(self, func, args, kwds):
*** 0083: self.gen = func(*args, **kwds)
0084: self.func, self.args, self.kwds = func, args, kwds
0085: # Issue 19330: ensure context manager instances have good docstrings 0086: doc = getattr(func, "__doc__", None)

Quoted 9 lines

0087: if doc is None:
Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
ERROR: Logfile of failure stored in:
/tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
task do_cve_check: Failed
ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) failed with exit code '1'

This is a mismatch with bitbake, because one commit hasn't reached 2.0. Steve, should I be using
https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for testing?

Kind regards,
Marta

#4 lore raw
Quoted 71 lines · Marta Rybczynska
Quoted 58 lines · Marta Rybczynska via lists.openembedded.org

meta/classes/cve-check.bbclass python do_cve_check () { · line 159

Quoted 13 lines · Joshua Watt

The CVE check database needs to have a shared lock acquired on it before it is accessed. This to prevent cve-update-db-native from deleting the database file out from underneath it.

[YOCTO #14899]

meta/classes/cve-check.bbclass +4 -0
+                cve_data = get_cve_info(d, patched + unpatched + ignored)
+                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
+        else:
+            bb.note("No CVE database found, skipping CVE check")

With this commit in kirkstone-nut, we're getting an error with meta-zephyr builds:

RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
do_cve_check: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001:
*** 0002:do_cve_check(d)
0003:
File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
lineno: 142, function: do_cve_check
0138: Check recipe for patched and unpatched CVEs
0139: """
0140: from oe.cve_check import get_patched_cves
0141:
*** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True):
0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
0144: try:
0145: patched_cves = get_patched_cves(d)
0146: except FileNotFoundError:
File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper 0236: <cleanup>
0237: """
0238: @wraps(func)
0239: def helper(*args, **kwds):
*** 0240: return _GeneratorContextManager(func, args, kwds)
0241: return helper
0242:
0243:
0244:def asynccontextmanager(func):
File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ 0079:class _GeneratorContextManagerBase:
0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" 0081:
0082: def __init__(self, func, args, kwds):
*** 0083: self.gen = func(*args, **kwds)
0084: self.func, self.args, self.kwds = func, args, kwds
0085: # Issue 19330: ensure context manager instances have good docstrings 0086: doc = getattr(func, "__doc__", None)

Quoted 9 lines

0087: if doc is None:
Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
ERROR: Logfile of failure stored in:
/tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
task do_cve_check: Failed
ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) failed with exit code '1'

Quoted 4 lines

This is a mismatch with bitbake, because one commit hasn't reached 2.0. Steve, should I be using
https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for testing?

I've gone ahead and backported that bitbake change to make things easier for people.

Cheers,

Richard

#5 lore raw
In reply to Marta Rybczynska
Quoted 72 lines · Marta Rybczynska
Quoted 58 lines · Marta Rybczynska via lists.openembedded.org

meta/classes/cve-check.bbclass python do_cve_check () { · line 159

Quoted 13 lines · Joshua Watt

The CVE check database needs to have a shared lock acquired on it before it is accessed. This to prevent cve-update-db-native from deleting the database file out from underneath it.

[YOCTO #14899]

meta/classes/cve-check.bbclass +4 -0
+                cve_data = get_cve_info(d, patched + unpatched + ignored)
+                cve_write_data(d, patched, unpatched, ignored, cve_data, status)
+        else:
+            bb.note("No CVE database found, skipping CVE check")

With this commit in kirkstone-nut, we're getting an error with meta-zephyr builds:

RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0
do_cve_check: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001:
*** 0002:do_cve_check(d)
0003:
File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass',
lineno: 142, function: do_cve_check
0138: Check recipe for patched and unpatched CVEs
0139: """
0140: from oe.cve_check import get_patched_cves
0141:
*** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True):
0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")):
0144: try:
0145: patched_cves = get_patched_cves(d)
0146: except FileNotFoundError:
File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper 0236: <cleanup>
0237: """
0238: @wraps(func)
0239: def helper(*args, **kwds):
*** 0240: return _GeneratorContextManager(func, args, kwds)
0241: return helper
0242:
0243:
0244:def asynccontextmanager(func):
File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ 0079:class _GeneratorContextManagerBase:
0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" 0081:
0082: def __init__(self, func, args, kwds):
*** 0083: self.gen = func(*args, **kwds)
0084: self.func, self.args, self.kwds = func, args, kwds
0085: # Issue 19330: ensure context manager instances have good docstrings 0086: doc = getattr(func, "__doc__", None)

Quoted 9 lines

0087: if doc is None:
Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared'
ERROR: Logfile of failure stored in:
/tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603
NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0:
task do_cve_check: Failed
ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) failed with exit code '1'

Quoted 4 lines

This is a mismatch with bitbake, because one commit hasn't reached 2.0. Steve, should I be using
https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for testing?

If you are going to try to use oe-core stable/kirkstone-nut then, yes, you should also use bitbake stable/2.0-nut since occasionally changes in core must be paired with a bitbake change.

However . . .

I highly discourage anyone from using the -nut branches! They are guaranteed to be rebased (and broken!), sometimes several times a day. So be prepared for pain!

Patches move to stable/kirkstone-next (and stable2.0-next) a day after I post a patchset for review. At this point they have passed all testing and are unlikely to change unless the community finds an issue during the review period. So it would be safer to use the stable/*-next branch (but I still discourage it since it is possible it will change)

Steve

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170239): https://lists.openembedded.org/g/openembedded-core/message/170239
Mute This Topic: https://lists.openembedded.org/mt/93352038/3617601
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-