Skip to contentSkip to conversation

cve-check: add lockfile to task

RFC PATCH Konrad Weihmann 3 participants 7 messages Jan 7, 2022 – Feb 22, 2022 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 2 lines · Konrad Weihmann

this should prevent running into the very rare error sqlite3.OperationalError: attempt to write a readonly database

It's also possible that check_same_thread (that defaults to True ) in the sqlite3.connect causes this (see https://docs.python.org/3/library/sqlite3.html#sqlite3.connect) - I have too little experience with sqlite to provide any further input.

I'd be happy to get some feedback from users that do have that experience with sqlite in a highly multithreaded/multiprocess environment. Or by people that use cvecheck class in larger setups than just core.

Quoted 38 lines

As highlighted by https://www.sqlite.org/faq.html#q5
it is likely that the adapter won't allow use multiple exec calls at the same time.

So it's best to prevent multiple accesses at a time, by reusing the already in place CVE_CHECK_DB_FILE_LOCK

YOCTO #14110

---
  meta/classes/cve-check.bbclass | 1 +
  1 file changed, 1 insertion(+)
meta/classes/cve-check.bbclass +1 -1
index 70d1988a70..6c04ff9f09 100644
@@ -111,6 +111,7 @@ python do_cve_check () {
  }
 
  addtask cve_check before do_build after do_fetch
+do_cve_check[lockfiles] += "${CVE_CHECK_DB_FILE_LOCK}"
  do_cve_check[depends] = "cve-update-db-native:do_fetch"
  do_cve_check[nostamp] = "1"
 
 
 
 
-=-=-=-=-=-=-=-=-=-=-=-

Links: You receive all messages sent to this group.
View/Reply Online (#160251): https://lists.openembedded.org/g/openembedded-core/message/160251
Mute This Topic: https://lists.openembedded.org/mt/88258164/3647476
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Hi Konrad,

Quoted 9 lines · Konrad Weihmann
Quoted 2 lines · Konrad Weihmann

this should prevent running into the very rare error sqlite3.OperationalError: attempt to write a readonly database

It's also possible that check_same_thread (that defaults to True ) in the sqlite3.connect causes this (see
https://docs.python.org/3/library/sqlite3.html#sqlite3.connect) - I have too little experience with sqlite to provide any further input.

Quoted 5 lines

I'd be happy to get some feedback from users that do have that experience with sqlite in a highly multithreaded/multiprocess environment. Or by people that use cvecheck class in larger setups than just core.

Afraid I am not well versed in the internals of sqlite, so I cannot comment on that part.

However, I do run cve-check on a larger setup than core, and after this commit recently landed in dunfell branch, I noticed that do_cve_check step is taking considerably longer than it used to. I have not benchmarked it carefully, but visually from watching the console during the build, it seems to be somewhere from 2 to 5 times slower. This is running a native build on an ext4 filesystem (not using containers etc).

Are others seeing similar behaviour?

While I am writing, I should also mention that the CVE JSON feeds are apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist (existing v4 format)
https://github.com/CVEProject/cvelistV5 (preview of upcoming v5 format) This would not solve the sqlite locking issue, though perhaps we could sidestep it and just search the json directly?
I guess I should really start a separate thread about this ;-)

Regards,
Ralph

#3 lore raw
Quoted 7 lines · Ralph Siemsen

While I am writing, I should also mention that the CVE JSON feeds are apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist (existing v4 format)
https://github.com/CVEProject/cvelistV5 (preview of upcoming v5 format)
This would not solve the sqlite locking issue, though perhaps we could sidestep it and just search the json directly? I guess I should really start a separate thread about this ;-)

Urgh. Thanks for bringing this to our attention.

Can you file a bug?

Ross

#4 lore raw
In reply to Ralph Siemsen
Quoted 21 lines · Ralph Siemsen

Hi Konrad,

On Fri, Jan 7, 2022 at 4:59 AM Konrad Weihmann <[email protected] <mailto:[email protected]>> wrote:

On 07.01.22 10:48, Konrad Weihmann wrote:
> this should prevent running into the very rare error > sqlite3.OperationalError: attempt to write a readonly database

It's also possible that check_same_thread (that defaults to True ) in the sqlite3.connect causes this (see
https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
<https://docs.python.org/3/library/sqlite3.html#sqlite3.connect>) - I have
too little experience with sqlite to provide any further input.

I'd be happy to get some feedback from users that do have that experience with sqlite in a highly multithreaded/multiprocess environment. Or by people that use cvecheck class in larger setups than just core.

Quoted 3 lines

Afraid I am not well versed in the internals of sqlite, so I cannot comment on that part.

However, I do run cve-check on a larger setup than core, and after this commit recently landed in dunfell branch, I noticed that do_cve_check step is taking considerably longer than it used to. I have not benchmarked it carefully, but visually from watching the console during the build, it seems to be somewhere from 2 to 5 times slower. This is running a native build on an ext4 filesystem (not using containers etc).

This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides in DL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.

Quoted 13 lines

Are others seeing similar behaviour?

While I am writing, I should also mention that the CVE JSON feeds are apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record <https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record>
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist <https://github.com/CVEProject/cvelist>  (existing v4 format)
https://github.com/CVEProject/cvelistV5 <https://github.com/CVEProject/cvelistV5> (preview of upcoming v5 format)
This would not solve the sqlite locking issue, though perhaps we could sidestep it and just search the json directly?
I guess I should really start a separate thread about this ;-)

Regards,
Ralph

#5 lore raw
Quoted 2 lines · Konrad Weihmann

This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.

That likely explains why the loadavg is only around 1 during cve-check.

Quoted 1 line

One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides in DL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.

I actually do something like this manually - in local.conf for each of my builds, I set CVE_CHECK_DB_DIR to a unique subdirectory within $DL_DIR. And I still see the occasional build failure due to "database in use".

Ralph

#6 lore raw
In reply to Konrad Weihmann
Quoted 8 lines · Konrad Weihmann

This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides in DL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.

Hang on - surely the cve_check tasks are read-only, so why do we need to lock the database?

Ross

#7 lore raw
Quoted 10 lines · Ross Burton
Quoted 8 lines · Konrad Weihmann

This is somehow expected from my side tbh - as the lock file disables any kind of parallelism - so just one instance of cve-check-task can run at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite db into the recipe-sysroot before running the task (currently it resides in DL_DIR), so each task would run on its own copy of the db - that would avoid having multiple access call to the same file while enabling us to remove the arbitrary task lock.

Quoted 4 lines

Hang on - surely the cve_check tasks are read-only, so why do we need to lock the database?

Ross

TBH I don't know - I (and Ralph confirmed that) see occasionally this error from the original patch.
As I mentioned I lack the experience with sqlite to get to the bottom of it - all I can say is that since the task lock had been merged I'm error free in all the pipelines I monitor