cve-check-update-nvd2-native: Incremement DL_DIR database location
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 26 lines · Richard Purdie via lists.openembedded.org
We're seeing a lot of sqlite database corruption issues in our automated testing. It is unclear why this is happening. There were process imrpovements implemented in master and it is unclear if older releases are somehow making those changes ineffective or if the problem is elsewhere.
By changing the location in
DL_DIR, we split the two sets of accesses to be separate and can isolate whether the master changes really did improve things or not. If successful, we may consider backporting those changes to the stable releases.Signed-off-by Richard Purdie <[email protected]>---
meta/recipes-core/meta/cve-update-nvd2-native.bb | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)meta/recipes-core/meta/cve-update-nvd2-native.bb +0
-0@@ -34,7 +34,7 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000" # Number of attempts for each http query to nvd server before giving upCVE_DB_UPDATE_ATTEMPTS ?= "5"
meta/recipes-core/meta/cve-update-nvd2-native.bb +1
-1-CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}" +CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}"
Quoted 2 lines
CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
CVE_CHECK_DB_TEMP_FILE?= "${CVE_CHECK_DB_FILE}.tmp"
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of failures on the new cluster, hopefully this will help isolate the problem.
Cheers,
Richard
Quoted 10 lines · Richard Purdie via lists.openembedded.org
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of failures on the new cluster, hopefully this will help isolate the problem.
The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.
Regards,
Marta
Quoted 17 lines · Marta Rybczynska
Quoted 11 lines · Richard Purdie via lists.openembedded.org
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the problem.
Quoted 4 lines
The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.
https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
Exception: sqlite3.DatabaseError: database disk image is malformed
Cheers,
Richard
Quoted 24 lines · Richard Purdie
Quoted 17 lines · Marta Rybczynska
Quoted 11 lines · Richard Purdie via lists.openembedded.org
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the problem.Quoted 4 lines
The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.
Quoted 5 lines
https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
Exception: sqlite3.DatabaseError: database disk image is malformed
Managed to get the same error with the sqlite command line.
Marta
Quoted 26 lines · Marta Rybczynska
Quoted 23 lines · Richard Purdie
Quoted 21 lines · Marta Rybczynska
Quoted 11 lines · Richard Purdie via lists.openembedded.org
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
I'm not sure why this is happening but the older releases are all accessing the DB over NFS which makes me nervous. We're seeing a lot of
failures on the new cluster, hopefully this will help isolate the problem.The database isn't completely corrupted, it looks like it is partial, however. What was the error
that you got in that build? Any or just strange results? I'm running a verification vs a fresh database.
Quoted 8 lines
Quoted 5 lines
https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
Exception: sqlite3.DatabaseError: database disk image is malformed
Managed to get the same error with the sqlite command line.
I merged this patch as it would rule out a lot of things. The error recurred:
https://valkyrie.yoctoproject.org//#/builders/76/builds/235/steps/14/logs/stdio
This means that it happens with the access patterns in master alone. I'm also noticing there seems to be a trend for this to happen on alma9 (older centos systems?).
Running those tests manually on that worker passed the tests ok.
It has python 3.9.18 with sqlite3 not available as an external command.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 42 lines · Richard Purdie via lists.openembedded.org
Quoted 41 lines · Marta Rybczynska
Quoted 35 lines · Richard Purdie
Quoted 21 lines · Marta Rybczynska
On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via lists.openembedded.org <[email protected]>
Quoted 5 lines · wrote:
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
Quoted 9 lines
I'm not sure why this is happening but the older releases are all
accessing the DB over NFS which makes me nervous. We're seeing a
lot of
failures on the new cluster, hopefully this will help isolate the
problem.Quoted 6 lines
The database isn't completely corrupted, it looks like it is partial,
however. What was the error
that you got in that build? Any or just strange results? I'm running
a verification vs a fresh database.https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
Exception: sqlite3.DatabaseError: database disk image is malformed
Quoted 4 lines
Quoted 1 line
Managed to get the same error with the sqlite command line.
I merged this patch as it would rule out a lot of things.
It looks like I didn't do this. I will do so, then we can rule out a few things.
Cheers,
Richard
Quoted 45 lines · Richard Purdie
Quoted 27 lines · Richard Purdie via lists.openembedded.org
Quoted 26 lines · Marta Rybczynska
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Marta Rybczynska
On Fri, Oct 11, 2024 at 1:02 PM Richard Purdie via lists.openembedded.org <[email protected]>
Quoted 14 lines · wrote:
I now do have an example of a corrupted database:
https://valkyrie.yocto.io/pub/shared-failure-data/nvdcve_2-1.db
I'm not sure why this is happening but the older releases are all
accessing the DB over NFS which makes me nervous. We're seeing a
lot of
failures on the new cluster, hopefully this will help isolate the
problem.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
The database isn't completely corrupted, it looks like it is partial,
however. What was the error
that you got in that build? Any or just strange results? I'm running
a verification vs a fresh database.https://valkyrie.yoctoproject.org/#/builders/23/builds/238/steps/14/logs/stdio
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Exception: sqlite3.DatabaseError: database disk image is malformed
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Managed to get the same error with the sqlite command line.
I merged this patch as it would rule out a lot of things.
It looks like I didn't do this. I will do so, then we can rule out a few things.
I've analysed the corrupted file a bit. This is somewhat complex as I do not have a "golden" copy with the
exact same content. However, what I can see:
1. This is not a partial download, quite new CVEs from 2024 are there (as from other years)
2. Damaged records come from various years (a theory to check: if they all have been recently modified)
3. Only part of the database is broken and both NVD and PRODUCTS tables. You can read various CVEs depending on how you format your SELECT (getting all cve_ids works fine, for example)
If you suspect there are jobs accessing the file that shouldn't, what about installing inotify hooks on the file?
As I've never seen such a corruption on my side, I could imagine either an external job, or an effect of a re-download of the database during tests (but this should use locks).
Regards,
Marta
Quoted 11 lines · Marta Rybczynska
I've analysed the corrupted file a bit. This is somewhat complex as I do not have a "golden" copy with the
exact same content. However, what I can see:
1. This is not a partial download, quite new CVEs from 2024 are there (as from other years)
2. Damaged records come from various years (a theory to check: if they all have been recently modified)
3. Only part of the database is broken and both NVD and PRODUCTS tables. You can read various CVEs depending on how you format your SELECT (getting allcve_idsworks fine, for example)
Quoted 6 lines
If you suspect there are jobs accessing the file that shouldn't, what about installing inotify hooks on the file?
As I've never seen such a corruption on my side, I could imagine either an external job, or an effect of a re-download of the database during tests (but this should use locks).
That is all useful data, thanks.
I can add that on the autobuilders, the file is on NFS since DL_DIR is. Any worker can therefore in theory access it and inotify could be tricky to setup and monitor correctly in that setup.
In your tests, do you run multiple releases against the database? I'm wondering if older sqlite versions in one of the older releases may be triggering this somehow it if tries to update the database?
Cheers,
Richard