cve-check: add lockfile to task
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Konrad Weihmann
this should prevent running into the very rare error sqlite3.OperationalError: attempt to write a readonly database
It's also possible that check_same_thread (that defaults to True ) in the sqlite3.connect causes this (see https://docs.python.org/3/library/sqlite3.html#sqlite3.connect) - I have too little experience with sqlite to provide any further input.
I'd be happy to get some feedback from users that do have that experience with sqlite in a highly multithreaded/multiprocess environment. Or by people that use cvecheck class in larger setups than just core.
Quoted 38 lines
As highlighted by https://www.sqlite.org/faq.html#q5
it is likely that the adapter won't allow use multiple exec calls at the same time.So it's best to prevent multiple accesses at a time, by reusing the already in place
CVE_CHECK_DB_FILE_LOCKYOCTO #14110
Signed-off-by Konrad Weihmann <[email protected]>--- meta/classes/cve-check.bbclass | 1 + 1 file changed, 1 insertion(+)meta/classes/cve-check.bbclass +1
-1@@ -111,6 +111,7 @@ python do_cve_check () { } addtask cve_check before do_build after do_fetch +do_cve_check[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}" do_cve_check[depends] = "cve-update-db-native:do_fetch" do_cve_check[nostamp] = "1" -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
View/Reply Online (#160251): https://lists.openembedded.org/g/openembedded-core/message/160251
Mute This Topic: https://lists.openembedded.org/mt/88258164/3647476
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi Konrad,
Quoted 9 lines · Konrad Weihmann
Quoted 2 lines · Konrad Weihmann
this should prevent running into the very rare error sqlite3.OperationalError: attempt to write a readonly database
It's also possible that
check_same_thread(that defaults to True ) in the sqlite3.connect causes this (see
https://docs.python.org/3/library/sqlite3.html#sqlite3.connect) - I have too little experience with sqlite to provide any further input.
Quoted 5 lines
I'd be happy to get some feedback from users that do have that experience with sqlite in a highly multithreaded/multiprocess environment. Or by people that use cvecheck class in larger setups than just core.
Afraid I am not well versed in the internals of sqlite, so I cannot comment on that part.
However, I do run cve-check on a larger setup than core, and after this commit recently landed in dunfell branch, I noticed that do_cve_check step is taking considerably longer than it used to. I have not benchmarked it carefully, but visually from watching the console during the build, it seems to be somewhere from 2 to 5 times slower. This is running a native build on an ext4 filesystem (not using containers etc).
Are others seeing similar behaviour?
While I am writing, I should also mention that the CVE JSON feeds are apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist (existing v4 format)
https://github.com/CVEProject/cvelistV5 (preview of upcoming v5 format) This would not solve the sqlite locking issue, though perhaps we could sidestep it and just search the json directly?
I guess I should really start a separate thread about this ;-)
Regards,
Ralph
Quoted 7 lines · Ralph Siemsen
While I am writing, I should also mention that the CVE JSON feeds are apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist (existing v4 format)
https://github.com/CVEProject/cvelistV5 (preview of upcoming v5 format)
This would not solve the sqlite locking issue, though perhaps we could sidestep it and just search the json directly? I guess I should really start a separate thread about this ;-)
Urgh. Thanks for bringing this to our attention.
Can you file a bug?
Ross
Quoted 21 lines · Ralph Siemsen
Hi Konrad,
On Fri, Jan 7, 2022 at 4:59 AM Konrad Weihmann <[email protected] <mailto:[email protected]>> wrote:
On 07.01.22 10:48, Konrad Weihmann wrote:
> this should prevent running into the very rare error > sqlite3.OperationalError: attempt to write a readonly databaseIt's also possible that
check_same_thread(that defaults to True ) in the sqlite3.connect causes this (see
https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
<https://docs.python.org/3/library/sqlite3.html#sqlite3.connect>) - I have
too little experience with sqlite to provide any further input.I'd be happy to get some feedback from users that do have that experience with sqlite in a highly multithreaded/multiprocess environment. Or by people that use cvecheck class in larger setups than just core.
Quoted 3 lines
Afraid I am not well versed in the internals of sqlite, so I cannot comment on that part.
However, I do run cve-check on a larger setup than core, and after this commit recently landed in dunfell branch, I noticed that
do_cve_checkstep is taking considerably longer than it used to. I have not benchmarked it carefully, but visually from watching the console during the build, it seems to be somewhere from 2 to 5 times slower. This is running a native build on an ext4 filesystem (not using containers etc).
This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides in DL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.
Quoted 13 lines
Are others seeing similar behaviour?
While I am writing, I should also mention that the CVE JSON feeds are apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record <https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record>
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist <https://github.com/CVEProject/cvelist> (existing v4 format)
https://github.com/CVEProject/cvelistV5 <https://github.com/CVEProject/cvelistV5> (preview of upcoming v5 format)
This would not solve the sqlite locking issue, though perhaps we could sidestep it and just search the json directly?
I guess I should really start a separate thread about this ;-)Regards,
Ralph
Quoted 2 lines · Konrad Weihmann
This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
That likely explains why the loadavg is only around 1 during cve-check.
Quoted 1 line
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides in
DL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.
I actually do something like this manually - in local.conf for each of my builds, I set CVE_CHECK_DB_DIR to a unique subdirectory within $DL_DIR. And I still see the occasional build failure due to "database in use".
Ralph
Quoted 8 lines · Konrad Weihmann
This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides inDL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.
Hang on - surely the cve_check tasks are read-only, so why do we need to lock the database?
Ross
Quoted 10 lines · Ross Burton
Quoted 8 lines · Konrad Weihmann
This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides inDL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.
Quoted 4 lines
Hang on - surely the
cve_checktasks are read-only, so why do we need to lock the database?Ross
TBH I don't know - I (and Ralph confirmed that) see occasionally this error from the original patch.
As I mentioned I lack the experience with sqlite to get to the bottom of it - all I can say is that since the task lock had been merged I'm error free in all the pipelines I monitor